Deep Learning: The Future of the Market Manipulation Surveillance Program

FINRA’s Market Regulation and Technology teams recently wrapped up an extensive project to migrate the majority of FINRA’s market manipulation surveillance program to using deep learning in what is perhaps the largest application of artificial intelligence in the RegTech space to date.

On this episode, we hear from Susan Tibbs, senior vice president of Market Manipulation in the Market Regulation Quality of Markets group, and from C.K. Chow, principal developer with the Technology team, about how the use of deep learning is making FINRA’s market surveillance data more digestible and increasing the efficiency and flexibility of the program.

How are we doing? Take the FINRA Unscripted survey today.

Resources mentioned in this episode:

Episode 13: How the Cloud and Machine Learning Have Transformed FINRA Market Surveillance

Episode 67: FINRA’s R&D Program: Exploring the Future of Advanced Analytics

Episode 68: Augmenting the Exam and Risk Monitoring Program with Data Analytics and Technology

Listen and subscribe to our podcast on Apple Podcasts, Google Podcasts, Spotify or wherever you listen to your podcasts. Below is a transcript of the episode. Transcripts are generated using a combination of speech recognition software and human editors and may contain errors. Please check the corresponding audio before quoting in print. 

FULL TRANSCRIPT 

00:00 – 00:31

Kaitlyn Kiernan: FINRA's Market Regulation and Technology teams recently wrapped up an extensive project to migrate the majority of FINRA's market manipulation surveillance program to using deep learning in what is perhaps the largest application of artificial intelligence in the RegTech space to date. Today, we're going to hear from the two leaders responsible for implementing this project about how the use of deep learning is making FINRA's market surveillance data more digestible and increasing the efficiency and flexibility of the program.

00:31 – 00:40

Intro Music

00:40 - 01:04

Kaitlyn Kiernan: Welcome to FINRA Unscripted, I'm your host, Kaitlyn Kiernan. I'm excited to welcome two new guests to the show today. From FINRA's Market Regulation and Transparency Services team we have Susan Tibbs, senior vice president of Market Manipulation and the Quality of Markets Group, and from FINRA Technology we have principal developer C.K. Chow. Susan and C.K., thanks for joining us.

01:04 - 01:05

C.K. Chow: Thank you very much.

01:05 - 01:06

Susan Tibbs: Thank you, Kaitlyn.

01:07 - 01:36

Kaitlyn Kiernan: So late last year, we took a look back at the history of FINRA's commitment to market transparency and how that has impacted FINRA's market surveillance program.

Today, we want to look ahead to the future of that program and the exciting new technology that is taking that program to the next level. But before we get into the details of that, Susan and C.K., I was hoping you could introduce yourselves and tell us a bit about your backgrounds. Susan, maybe we can start with you.

01:36 - 01:53

Susan Tibbs: Sure, thank you. I have over 25 years of experience with FINRA and Market Regulation. I started as an analyst in the options and marking area. And since that time I've worked closely with all the other stakeholders and FINRA, and especially with technology.

01:54 - 02:19

C.K. Chow: Hi, I'm C.K. A long time ago, I actually was a theoretical physicist. I got my PhD from Caltech. I joined FINRA in the year of 2010, so it has been 11 years now. And now I'm leading the data science program in Market Regulation Technology. I lead a team of ten data scientists who work very closely with Market Reg to develop advanced analytics solutions.

02:19 - 02:26

Kaitlyn Kiernan: Susan, can you tell me more about your team within Market Regulation and some of the responsibilities that fall under your group?

02:27 - 03:05

Susan Tibbs: Sure, so in Market Reg our mission is to conduct surveillance and investigations and to ensure market integrity. We process over 200 billion market events at FINRA and as you can imagine, that creates quite a challenge in surveillance. The data that makes up that number includes equities, options, fixed income and the types of patterns that are run include data integrity, customer protection, market conduct and then market manipulation. And that's really my area. We're looking at things such as wash sales, marking the close, layering, spoofing and mini-manipulation.

03:06 - 03:12

Kaitlyn Kiernan: C.K., how about you? What all does your team of ten do within the data science team?

03:12 - 04:20

C.K. Chow: We are the data science program within Market Reg Technology, so as the name suggests, we work closely with Market Regulation in devising analytics solutions. Market Regulation has a lot of surveillance patterns, so we are constantly working with them on how to make the solutions better. Our data scientists come from all over the place, some of them are physicists or mathematicians by training. But when they join this team, we want them to really learn the business. So we work very closely and understand what is the business questions and then we devise very specific solutions.

In this way, we are in-house, we are not really outsourcing anything, but we actually try to have people who are as integrated with Market Reg to understand the problem and devise solutions as much as possible. So far, I think this works very well because the people actually have ownership of the problem. They understand what Market Reg wants to achieve. In fact, we want people who are very associated with the goal of Market Reg, which is to understand market data to maintain market integrity.

04:21 - 04:47

Kaitlyn Kiernan: That makes sense. FINRA's business is pretty complicated, so you want people in Technology to really have that deep business knowledge. So C.K. and Susan, you just wrapped up a really big project to migrate the majority of Market Regulation's market manipulation surveillance program to using deep learning. That sounds really impressive. But what does it really mean to be applying deep learning to this program?

04:48 - 05:23

Susan Tibbs: So deep learning specifically is just an advanced technique. What we were looking to do here was really to answer some of the questions that were presented in surveillance: changing market conditions, increased volatility, increased volumes and change in conduct. And so, using deep learning made a lot of sense to start to answer those challenges. And in addition, just to be able to provide better visualizations for the staff to put the activity in its full context immediately so we can be more efficient and more effective at what we're looking at.

05:24 - 05:32

Kaitlyn Kiernan: And C.K., what exactly is deep learning and how does it relate to or differ from artificial intelligence?

05:32 - 07:15

C.K. Chow: Deep learning, or it is also known as deep neural network, essentially it is a type of technology which often is said to be inspired by how the human brain works. Essentially, what happens is we have data which flows through a network of neurons. Each neuron has individual weights, which represent the relationship between the input and the output of the neurons.

And in the beginning, you have a huge, complicated representation of the market data. But in the end, after they flow through the neurons, it will give you an outcome whether this is of regulatory interest or not. The neuron adjusts its weight, it adjusts how it makes decisions by inspecting a huge amount of market data. And in this way, the neural network learns how to make prediction, not by a human writing down some rules. A human inspired rule will always be limited because humans can only handle a certain amount of complexity. But instead asking the machine to write the rules by inspecting the data, it can give you a lot more fine grained and a lot more holistic predictions. And it can also learn how to make new predictions by inspecting new data.

So, we believe that that is a much more robust way of detecting market behavior. And a lot of the commonly used artificial intelligence are actually based on deep learning. For example, we all have image recognition on our cell phones. It's actually built on deep learning. A lot of the text analytic capabilities are also based on deep learning. But we are probably the first party to use deep learning for surveillance purposes at a scale which we are doing. I think that is a really important breakthrough, so we are very proud of what we have done.

07:16 - 07:28

Kaitlyn Kiernan: C.K., you mentioned, it's a big scale. FINRA processes billions of market events, and there's dozens of patterns. Just how big of a project is this?

07:28 - 08:16

Susan Tibbs: This was an enormous undertaking and it is quite a large scale, as we talked about that over 200 billion market events a day, to be able to process that quantity and really come out with something that's usable, that's digestible and that's even more efficient than what we have now was quite an undertaking. And really just not even the surveillance itself, but the process and the procedures that we put around that, the governance, the model metrics, the entire scope of how we're doing UAT change, how we're doing QA, really putting together that entire system, our model development lifecycle, all of those aspects really have to change in order to put this type of pattern or this type of model into production.

08:18 - 08:26

Kaitlyn Kiernan: The project of the scope seems like it's definitely like an all-hands-on-deck type of a project. Who else collaborated with you on this?

08:27 - 09:44

C.K. Chow: I think the people who are very much involved is, of course, my team and Susan's team, which is the Market Regulation group. We met very often for more than a year in building up this switch of surveillance models, but we have always been in communication and collaboration with others at FINRA. A lot of these ideas first started out with the FINRA R&D program. We had a project there which actually tests out some of this idea, and we work very closely with the Technology QA team and also Internal Audit to assure that these models are done in the right way. And in the end, whatever we find here will need to go to Enforcement, so Enforcement has been in communication and they know that we're working on this journey, we are trying to build better surveillance models.

On top of this, we discussed with OGC, the Office of General Counsel, to make sure that we do everything in a way which is justifiable, and we also need to change our governance structure, so the governance staff has been working very closely with us to make sure that we know how we manage these models going forward. Last but not least, the senior management of FINRA is well aware of our experiment and obviously they support us.

09:45 - 09:57

Susan Tibbs: And really on that note, C.K., I know you'd join me in saying this, that we really need to give a lot of credit too to the late Tom Gira, who was one of the early sponsors of more advanced techniques and in particular, deep learning.

09:58 - 10:07

C.K. Chow: And I want to plug, Kaitlyn has done some podcasts with Tom before, so if you want to hear it directly from Tom, look at one of the earlier FINRA Unscripted podcasts.

10:08 - 10:23

Kaitlyn Kiernan: Yes, we can link to that episode in our show notes. I'm sure just having the enthusiastic support of senior management is so important in getting a project like this off the ground. And how did this project come about initially?

10:24 - 11:03

Susan Tibbs: Early on there was definitely strong sponsorship from senior management and the identification that we needed to continuously improve surveillance. We needed to introduce more flexibility to the patterns, and we really needed to continue to experiment and improve our capabilities on the tech side. And then, as C.K. mentioned earlier, just really tighten that relationship with Technology and how we interact and how we share examples back and forth and how we teach each other about our jobs. And so many things came together at the same time to really make it possible that we had such a great outcome here.

11:04 - 12:12

C.K. Chow: I think it's good to go back a little bit in history. Now FINRA had started a FINRA advanced analytic R&D program a couple of years ago. Kaitlyn has interviewed Greg Wolff and Ivy Ho on that program. And the very first project which the FINRA R&D program has sponsored is a program which is coached by me and sponsored by Susan on experimenting on how to use deep learning to do market surveillance. We did a proof of concept over there that actually gave us a lot of confidence, say that, OK, this is a viable path. We can use deep learning to detect interesting market behavior.

And after that, essentially, we need to make our mind that OK, this is the way to go. And after we get buy in from senior management, we assemble a team and we build the model and we have the entire timeline that we want this switch of deep learning model to support surveillance after we switch over to CAT. And that is because we want to make full advantage of CAT data when we have these deep learning model.

12:13 - 12:34

Kaitlyn Kiernan: For those who don't have the encyclopedic knowledge of all these FINRA Unscripted episodes like C.K. Does, I'll also link to episode 67 in the show notes, which was our episode with Greg Wolff and Ivy Ho about the R&D program. So it started with the R&D program, and how did it evolve from there, Susan?

12:35 - 13:18

Susan Tibbs: From the initial spark, some amazing people that worked on the R&D projects, we took it further to actually employ some custom machine learning to look at the problems that we were trying to address and then very smartly C.K. suggested that we take that further and we looked at other techniques and landed on using deep learning. But it was really focused on innovation, on how we could answer problems and really solve them in different ways and looking for a more flexible approach again to those changing market behaviors and changing market conditions. We really needed to be more flexible in how we were analyzing the data.

13:19 - 13:28

Kaitlyn Kiernan: And what were some of the biggest challenges in getting this project off the ground? Susan, maybe you can start from a business point of view?

13:28 - 13:46

Susan Tibbs: One of the challenges was just the scale and scope of what we were doing and where to get started and how to take it further. And so we did a lot of experimentation starting at those first R&Ds. But then even after that to prove out and to refine our concepts.

13:47 - 13:56

Kaitlyn Kiernan: And C.K., did it require any changes in the way that we approach our surveillance parameters and the whole way we look at the program?

13:56 - 15:43

C.K. Chow: Totally. Yeah, it is a big change in my sense. Historically, we use rule-based patterns, so we are thinking in terms of OK, if things satisfy these conditions and that condition and that these conditions become something off interest. But you can see that that way of approaching markets surveillance is rather rigid and may not be the best, given that our market is changing so quickly these days. At times of high volatility or new behavior that approach may be of limited utility.

And when we approach it with deep learning, how we approach the problem is we need to find enough examples. Understand that if we have enough examples, and we can visualize and see how why they are interesting. If a pair of human eyes can understand why these charts show me something is of regulatory interest, then a machine can see the same thing. Remember, deep learning is the technology behind image recognition. Machines are very good in recognizing images these days, we need to provide the data in the way that a human can recognize this is validation. And we know that the machine can pick up the same thing.

So, it is a very different way of thinking about the problem. It is no longer about rules, it's actually about image. And understandably, we spend a lot of time in fine tuning our visualization package. In the end, it is useful both for machines but also for the human being. In the end, because we still need to review everything which come out of the machine. But I think that is a cultural change which we have undergone, and we are interested in seeing that change to be impacting a larger part of the enterprise.

15:44 - 15:56

Kaitlyn Kiernan: And beyond that cultural shift and change in the way we're thinking about the problem. Were there any other noteworthy technological challenges to getting these deep learning patterns into production?

15:57 - 18:34

C.K. Chow: Yeah, I think there are three major challenges. One is simply scale. The volume of data is huge. As Susan mentioned, we have 200 billion events a day. It is a very large amount of data. Simply how to corral and orchestrate such a huge data processing pilot is very challenging.

And on top of this, we need to help the machine to learn how to recognize these cases of interest, which mean that we need to find enough examples. These are what we call labels in machine learning language. We need to find enough labels of, these are the interesting, manipulative behavior. But sometimes we have a lot, but sometimes we do not. So in those cases, we would need to understand the behavior better. The data scientists need to be able to find more examples of those or to be able to construct examples of those to a scale which the machine can look at enough of them and say, now I got it, this is what you mean by manipulation and then it can build a model of it. Then the scarcity of label is one of the limiting factor. To be honest, the first version of every model is not perfect. But when we gather more and more labels as we use them, more and more the models is going to get better and better.

The last challenge is the challenge of how do we even know the models are working? Now in the past, if you build models according to rules, it's pretty straightforward. You see, OK, the rules say that you should check these things and let me go to the code to see whether the code is checking exactly those. If that's the case, you know that you are correct. But in deep learning, or any machine learning approach, you really do not have rules to check. So in the end how do we know the model is working? How do we know the model is working well enough that we can put it to production? And how do we know that after you go to production, it is still working? We do not need to replace that. That is the question of testing and monitoring, and we need a lot of new thinking over that. Well fortunately, we are supported by the quality assurance team and also by that the team which develop the model development lifecycle who specified how the models should be developed, tested and monitor after it goes to production. So we have a lot of help there. A lot of these technological challenges take a lot of help from other parties and we intend to make them better as time goes on.

18:35 - 18:55

Kaitlyn Kiernan: One thing that was mentioned a little bit ago was that this project was going on at the same time that CAT was being implemented, which is also a huge undertaking. What were some of the challenges in implementing this deep learning project at the same time that we were building and implementing CAT?

18:56 - 20:22

C.K. Chow: Yeah, that actually gave us an extra degree of challenge because when we start working on the model, the CAT data was not mature enough to be used for training. Mature enough in the sense that the firms and the exchanges has just started to submit the data to CAT. But the data have a lot of variability. And I know that now the CAT data is very stable, but it was not like that a year ago, the firms are still working out how they report to CAT.

So, in the beginning, we do not have enough CAT data to support this model process. So, we need to build the model on pre-CAT data and later when the CAT data had become more mature, then we migrate the model to CAT. But as anyone who has done data migration, we know that after migration the data actually look quite different. And this is understandable if the data pre-CAT and the CAT data is exactly the same, there's no point of doing CAT, right? CAT is giving us better data. So, after we migrated model to CAT, we say, Hey, there's some difference in the outputs we need to adjust and tune the model and to make them work better on CAT. And also, there are new data policy in CAT which we realized we could have taken advantage of. So, we did some of those when we do the migration, but these add to the complexity of the project. But we have talented data scientists, and we have tons of data engineers as well. So, they help us to go through this journey

20:23 - 21:26

Susan Tibbs: With a challenge also comes opportunity. And so, CAT was really a game changer in that the added granularity and the data is amazing for surveillance. And it was an opportunity to look at the scope of the current patterns to adjust them in some cases, to combine patterns and really take a holistic look at how we can do things better, not just from the technical aspects of it, but also from the regulatory aspects. How could we be more efficient? How can we be more effective? How can we provide those better visualizations?

And so, by doing it together and by doing it with multiple things changing, we also could make a major improvement and really take it even further than just having one aspect change at a time. So, we're really proud of what came out of it and what it looks like and what we're able to find with it. So, it was definitely quite helpful to have especially the CAT data, but then also these new surveillance techniques on top of that.

21:28 - 21:41

Kaitlyn Kiernan: That definitely sounds like something to be proud of. Susan, I want to get an understanding of the impact. Can you tell us at a high level how the change will impact the market manipulation surveillance program?

21:42 - 22:23

Susan Tibbs: One of the first things that we've kind of touched on as we started this conversation is really the flexibility, the flexibility to understand the data, the market and to combine that all in the model metrics, the improved visualizations which also improves our effectiveness, more efficient process and really being able to focus our efforts on more of the problematic conduct and really focusing on specific things. And at the same time, remaining flexible and being able to provide feedback to the models and have that process really of continuous improvement. And so that all makes for a much stronger surveillance program.

22:24 - 22:39

Kaitlyn Kiernan: C.K. had mentioned the difficulty originally in getting the right labels, so as the market shifts and those looking to manipulate the market shifts their approach, how quickly can the models learn and adapt with the deep learning?

22:39 - 23:37

Susan Tibbs: So, the process that we'll be utilizing is that the alerts will be reviewed or the output from the models will be reviewed. And then we'll have discussions with our data scientists and then propose changes to the models by introducing new examples or reducing some of the examples that we're already seeing.

And so, it won't be an automatic or automated process, but it'll be one we're very hands on with the business experts and their data scientists to really improve the quality of the patterns through this retraining process and as we step through the retraining and determine the cadence that we're making these changes, we're also looking at how that works in the overall framework and process and some of our governance aspects as well. It can be done quite quickly, but we don't believe that will ever have an automated process to do that.

23:38 - 23:41

Kaitlyn Kiernan: And how many patterns have been switched over so far?

23:41 - 23:58

Susan Tibbs: So currently, we took 11 of our rule-based patterns through this process and came out with eight deep learning models that we're using, but we're certainly looking at the rest of the suite of patterns and where it might make sense to also employ deep learning.

24:01 - 24:05

Kaitlyn Kiernan: And will employing deep learning make markets safer for investors?

24:05 - 24:54

Susan Tibbs: I definitely think so. Market surveillance is really essential to maintaining market integrity. Deep learning is an incredibly powerful tool and helps create better surveillance and definitely the detection and perhaps deterring manipulation and protecting investors.

But I do want to add, though, that it's definitely not the only tool and great surveillance can be achieved through a variety of different tools. So, while we're using deep learning to solve these particular problems, it definitely does not mean that that is the only tool or the only tool for every problem, that's actually far from the case. And as we get further along in our looking and technology, expand our toolbox, we're definitely looking at a variety of different approaches. Deep learning just being one of them.

24:55 - 25:02

Kaitlyn Kiernan: And what's next? Will FINRA continue to roll out the use of deep learning to other parts of the organization.

25:04 - 25:42

Susan Tibbs: As we're having this conversation, we definitely think that advanced technologies are so important to continue to improve our ability to assess changing market conditions and market conduct. We're looking at new opportunities to use deep learning and other perhaps more risk-based techniques. There's also a major corporate initiative underway to promote advanced analytics, and there are amazing teams throughout the company that are working on this now. We think in this overall corporate initiative that there's always more to do and that innovation that is so important to surveillance and to regulation.

25:43 - 25:54

Kaitlyn Kiernan: And C.K., now that you have this deep experience working with the market manipulation group, will it be easier to roll out this new technology in other parts of the organization?

25:55 - 27:57

C.K. Chow: For this project, we have built up a core of people. We have built a core of experience in methodology. We have also worked out some of the issues around governance of these models. So at least we would be able to help if the larger part of FINRA want to implement this kind of technology or other advanced analytic technology across the board. We certainly would be able to help them.

One other point I also want to mention is we are doing this and now the exchanges and the other regulators know that we are doing it. And that also helps in the sense that Market Reg often receive examples from firms or other regulators or from the exchange say Hey, these examples look like something wrong is happening here, there's something interesting is happening here. Let's see what they tell us. Well, we'd go back and do some analysis and sometimes they turn out to be not so interesting. Sometimes they turn out to be really interesting. In the past, we need to figure out how do we fit that into our existing surveillance program? Where or how do we change the rules to account for these interesting cases? And that is not easy.

In the deep learning world, we actually can integrate these inputs from other regulators because we can actually pass these examples or other things like that to our models and the models can learn Oh, this is also of interest. And the models can learn to be smarter.

To go back to your earlier point, it would make the market safer. It would help us to react faster because other regulators know that they can send us information and it can help our model to get smarter. And I think they also have a feedback effect because once you start doing things that way, it would accelerate. It would help to change the culture from a more static rule-based way into a more dynamic, risk-based paradigm, which Susan has mentioned earlier. And I think that is the direction not only Market Reg, but the whole enterprise, is going.

27:58 - 28:04

Kaitlyn Kiernan: And Susan, just to wrap up, how do you see the market surveillance program continuing to evolve from here?

28:05 - 28:55

Susan Tibbs: Continuing to evolve, right, is the core of that. We definitely think that it's so important to continue to experiment and explore. We're presently looking at tools to enhance our clustering ability to identify like and unlike behaviors and how we can use that to enhance surveillance. We have all this amazing data from CAT, and we're looking forward to continuing to experiment and enhance our ability to really detect and review problematic market activity.

But that is definitely a process, and deep learning is one of those amazing tools that help us to look at things differently and to assess the conduct and the markets themselves. So, this is just been such an exciting project to work on, to be able to tie all these things together.

28:56 - 29:34

Kaitlyn Kiernan: Well, Susan and C.K, thank you so much for joining me to talk about these exciting developments with the market regulation program and FINRA technology. I look forward to hearing how this impacts the future of the program.

Listeners if you don't already, you can be sure to subscribe to FINRA Unscripted wherever you listen to podcasts. And if you have ideas for future episodes or thoughts to share on today's episode, you can email us at [email protected].

Today's episode was produced by me, Kaitlyn Kiernan, coordinated by Stephanie Van den Berg and engineered by John Williams. That's it for today's episode. Until next time.

29:34 – 29:40

Outro Music

29:40 – 30:07

Disclaimer: Please note FINRA podcasts are the sole property of FINRA and the information provided is for informational and educational purposes only. The content of the podcast does not constitute any rule amendment or interpretation to such rules. Compliance with any recommended conduct presented does not mean that a firm or person has complied with the full extent of their obligations under FINRA rules, the rules of any other SRO or securities laws. This podcast is provided as is. FINRA and its affiliates are not responsible for any human or mechanical errors or omissions. Parties may not reproduce these podcasts in any form without the express written consent of FINRA. 

30:07 – 30:14

Music Fades Out