UPDATE: FINRA’s Cloud Strategy Shone amid Recent Volatility
UPDATE 10/19/2018: During the week of Oct. 8, 2018, exceptional market volatility again produced record numbers of electronic messages processed by FINRA, including the three highest-volume days in history:
- The 107 billion messages processed on Oct. 10 topped the previous record of 101 billion set on March 28, 2018.
- The very next day – Oct. 11 – saw a significantly higher record set: 135 billion messages, an increase of 25 percent from the new mark set just the previous day and an increase of 33 percent over the March record.
- Closing out the week, the 108 billion records processed on Oct. 12 now stand as the second-highest mark in history.
Early last month, when U.S. markets experienced their most turbulent week in years, a record-setting 99 billion electronic messages coursed through FINRA’s regulatory platform in a single day to cap a week of record volume. We handled the unprecedented volume seamlessly, thanks to FINRA’s pioneering cloud strategy.
For investors, it was a whiplash week. The Dow Jones Industrial Average swung at least 1,000 points in all but one day the week of Feb. 5, and changed direction 53 times.
Trading activity in the stock, bond and options markets creates a variety of electronic records that we monitor for regulatory purposes. During the week of Feb. 5, we saw the volume of this information soar to almost triple the average daily volume last year – to 99 billion on Feb. 9. That week also saw a sustained level of activity we had never experienced – the volume seen on four days that week now rank as our four most active in history.
It is thanks to our cloud strategy that we were able to handle this kind of elastic demand for processing and storage without a problem. The cloud enables FINRA to instantly and automatically provision additional capacity.
In our role as a regulator, we must often access data from weeks or months ago. When you have tens of billions of records to process every day and you are storing that data for months or years at a time to analyze historical trading patterns, that means you have a lot of data—28 petabytes of data, to be exact. That’s enough data to play 40,000 years of music – without any repeats or interruption.
With a traditional data-center approach, capacity is limited to that of the servers in a data center. Had we stayed with that traditional approach, instead of moving to the cloud, the volume from any one of those exceptional days could have taken us days to finish processing. To do our job of protecting investors and ensuring market integrity, it’s important that we are on top of each day’s activity, applying our automated surveillance patterns to help our analysts look for potentially suspicious activity – instead of running to catch up.
| Year | Average Daily Records Processed (billions) |
|---|---|
| 2018* | 53.2 |
| 2017 | 35.6 |
| 2016 | 37.4 |
| 2015 | 42.3 |
* Through Feb. 15, 2018
Scalability was one of the key factors when we made the decision in 2014 to move 90 percent of our work to the cloud by the end of 2016. There were other clear benefits including:
- Analytical capability – Surveillance staff can conduct analysis across big-data “lakes” instead of having to create a separate, smaller “pond” of data for each analysis. Inquiries can run in seconds or minutes instead of hours or days.
- Security – Having a virtual footprint in multiple locations means our data is less vulnerable than proprietary data centers. Our cloud provider’s scale of operations also gives us access to a broad range of best-of-breed security solutions that could be prohibitively expensive in a stand-alone approach. These solutions include encryption of all data – at motion and at rest – and micro segmentation of our networks.
- Reliability – The ability to move data and applications across different regions provides greater protection against power interruptions, extreme weather or natural disasters.
- Cost-effectiveness – We pay for only the computing capacity we use, as you do with a power or water utility, so we do not have to provision for peaks. Also, as is common with public cloud software development, much of the development operations (Dev Ops) functions have been automated, allowing FINRA to become much more efficient, in addition to being more reliable and secure.
As a self-regulatory organization we can – and in this case, we did – make multi-year investments that support an innovative regulatory approach. We’re committed to continue looking past the next corner to be ready for the road ahead.
