Posted by Brian McCallion ● Jun 17, 2017 11:47:21 AM

AWS Lambda Market Data Integration with Bloomberg


As customers move historical market data to AWS to power applications such as deep learning, we work to help them migrate the data, as well as integrate with trusted markete data providers such as Bloomberg. This post focuses on applying AWS Lambda to automate retrieval, load, and processing of Bloomberg data on AWS.

To backfill the historical data get the historical data we created and copied the requests via SFTP a Bloomberg server that has our public key. The Bloomberg server  polls the file system, and when it finds a new request file, checks that file against pending requests,  then processes the request and writes the data to a new file containing the historical data. Once we have the historical data, this data is loaded to a table, and the tables can be queried.

The solution we've built processes the data that is the lifeblood of a Green Field application funded by two seasoned investors with a background in trading and pension fund advisory. The first part of the platform we built is a Continuous Delivery process that begins with Machine Learning PhDs checking in the manifest of experiments, and then packages these experiments and runs tens of thousands of randomly seeded training Workloads on a Spot Instance farm fed and managed by AWS Batch.

Traditional capital markets systems favor .NET and Windows and SQL Server. In our case we have no such baggage. However the customer sought a third party offshore data review firm to check the data for anomalies. The service also provides a schema on MS SQL Server. The application includes a small .NET application that parses daily market data closing prices for equities and indexes. Such services can be useful where a portfolio of equities is part of the data set. In our case, indexes like S&P 500 are the focus. Indexes tend to be stable. For example the S&P 500 ticker probably won't change often, the index doesn't split, and even where the entity that owns the S&P 500 is acquired, as in the case of SP Global (owns S&P 500) acquired by the Intercontinental Exchange, tickers don't change.

We were not thrilled that a .NET client loads the data, as it seemed to constrain us to launching a Windows instance just to load the data. I started to look at options and to think about how to repackage the little console app for Lambda. I hope .NET Core lives up to its promise. The cross platform capability of .NET Core crept up on me and I need to follow the Microsoft developer platform more closely.

Topics: AWS Batch, AWS CodeBuild, AWS ECS, Big Data, Bloomberg DataLicense, Bloomberg Enterprise, Case Study, Deep Learning, file, SQL Server, Torch 7, AWS Lambda, Lambda, .NET, Platform