Working Programmatically with FINRA APIs
People

FINRA has provided transparency to over-the-counter equities (OTCE) trading for years through its public website. Earlier this year, FINRA released companion REST APIs for this data as a prelude to an upcoming major API product. In this article, we demonstrate how to leverage the API in a programmatic way by building an HTTP client in Java.

For this article, we’ll make use of the monthly top 100 data set. All of the datasets have a unique composite identifier consisting of a group and a name. There might be multiple data sets that belong to the same group, but no two datasets in a group have the same name. For the monthly top 100 dataset, the group is "otcmarket" and the name is "monthlytop100".

Using a Web Browser (Chrome)

Before we get to work with the dataset in Java, let's take a look at the data through a web browser. Point your browser at https://api.finra.org/data/group/otcmarket/name/monthlytop100 to execute a GET call to the endpoint. By default, requests return only the first 1000 records available. Here’s what the results should look like:

    
    "monthStartDate","issueSymbolIdentifier","issueName","marketDescription","numberOfSharesTraded","dollarVolume","closingPrice"
    "2018-09-01","KGET","Kleangas Energy Technologies, Inc.  Common Stock","Other OTC","3010196595","312678","0.00010000"
    "2018-09-01","SPRV","Supurva Healthcare Group, Inc. Common Stock","Other OTC","2970846347","320103","0.00020000"
    "2018-09-01","HCMC","Healthier Choices Management Corp. Common Shares","Other OTC","2885946348","304386","0.00020000"
    "2018-09-01","MLHC","M Line Holdings, Inc. Common Stock","Other OTC","2490793487","441354","0.00030000"
    "2018-09-01","USMJ","North American Cannabis Holdings, Inc. Common Stock","Other OTC","2384546808","871383","0.00040000"

We can see the following from the response:

  • By default, the response is in .CSV format
  • The first line contains the column headings
  • Each subsequent line represents a record, where all of the field values are, by default, inside quotation marks and separated by commas

Now let's take a closer look at the dataset by executing a GET call to the endpoint https://api.finra.org/metadata/group/otcmarket/name/monthlytop100, which returns the following:

  
      {
        "datasetGroup" : "OTCMARKET",
        "datasetName" : "MONTHLYTOP100",
        "partitionFields" : [ "monthStartDate" ],
        "version" : 0,
        "fields" : [ {
            "name" : "monthStartDate",
            "type" : "Date",
            "format" : "yyyy-MM-dd",
            "description" : "First Business Day of the Month"
        }, {
            "name" : "issueSymbolIdentifier",
            "type" : "String",
            "description" : "Symbol"
        }, {
            "name" : "issueName",
            "type" : "String",
            "description" : "Symbol Name"
        }, {
            "name" : "marketDescription",
            "type" : "String",
            "description" : "Market - OTCCBB or Other OTC. All OTC includes both markets"
        }, {
            "name" : "numberOfSharesTraded",
            "type" : "Number",
            "description" : "Number of Shares for the issue"
        }, {
            "name" : "dollarVolume",
            "type" : "Number",
            "description" : "Dollar Volume for the issue"
        }, {
            "name" : "closingPrice",
            "type" : "Number",
            "description" : "Closing Price for the issue for a given month"
        } ]
      }

We can see the following from the response:

  • The group and name of the data set
  • Which fields constitute a partition field
  • Each field, including name, type, and a brief description
  • If the field type is a date, an additional format field indicates the format of the date

Using Java

Now let’s build a Gradle Java project that will be used to consume the data set in 5 easy steps.

Step 1 - Create a Gradle Java Project

Create the following directory structure within your project, including the files build.gradle and settings.gradle in the root directory with the content shown below:

  
  Directory Structure
     root
     ---src
        ---main
            ---java
            ---resources
        ---test
            ---java
            ---resources
      ---build.gradle
      ---settings.gradle
  
            
  
   build.gradle
      buildscript {
          repositories {
              mavenLocal()
              mavenCentral()
          }
      }

      ext {
          restAssuredVersion = '3.2.0'
          lombokVersion = '1.18.4'
          junitVersion = '5.3.1'
          jacksonVersion = '2.9.9'
      }

      apply plugin: 'java'

      group 'org.finra'
      version '0.0.1-SNAPSHOT'

      sourceCompatibility = 1.8

      repositories {
          mavenLocal()
          mavenCentral()
      }

      dependencies {
          implementation "io.rest-assured:rest-assured:$restAssuredVersion"
          compileOnly "org.projectlombok:lombok:$lombokVersion"
          testCompileOnly "org.projectlombok:lombok:$lombokVersion"
          annotationProcessor "org.projectlombok:lombok:$lombokVersion"
          testAnnotationProcessor "org.projectlombok:lombok:$lombokVersion"
          testImplementation "org.junit.jupiter:junit-jupiter:$junitVersion"
          implementation "com.fasterxml.jackson.core:jackson-databind:$jacksonVersion"
      }

      test {
          useJUnitPlatform()
      }

      
            
  
   settings.gradle
      rootProject.name = 'finra-api-client'

Next, let’s use Rest Assured, a REST API testing framework, as an HTTP client. Some benefits to using Rest Assured include:

  • Very easy to use and well documented
  • It's the most widely used Java library for testing REST APIs
  • You get the benefits of Apache HTTP client without having to write all of the boiler plate code

Step 2 - Create the Monthly Top 100 Model

Within the root/src/main/java directory, create a class called MonthlyTop100.java. Records returned in the response from the APIs will be automatically converted into objects of this class.

  
      package org.finra.api.client;

      import java.util.Date;
      import lombok.Data;

      @Data
      public class MonthlyTop100 {

          private Date monthStartDate;

          private String issueSymbolIdentifier;

          private String issueName;

          private String marketDescription;

          private Long numberOfSharesTraded;

          private Double dollarVolume;

          private Double closingPrice;

      }

Note that the name of the fields in this class were directly extracted from the metadata response. In other words, the metadata determines the name and type for each field inside the model. The @Data annotation comes from the Project Lombok, which accomplishes the following:

  • Generates setters and getters for all fields
  • Creates a toString method
  • Creates equals and hashCode implementations
  • Builds a constructor

All of this for free with just one line of code!

Step 3 - Create the Response Classes

Next, we create the FinraResponse and DataFinraResponse classes. The FinraResponse class will be a general wrapper that contains the response from the API. The DataFinraResponse class will extend the FinraResponse class, and it will be more specific containing responses from the data endpoint https://api.finra.org/data/group/{group}/name/{name}. Moreover, the DataFinraResponse class will provide the ability to easily convert the response into an array or list of MonthlyTop100 objects.

Within the root/src/main/java directory, create two classes called FinraResponse.java and DataFinraResponse.java as follows:

  
    package org.finra.api.client;

    import io.restassured.response.Response;
    import lombok.AllArgsConstructor;
    import lombok.Getter;

    @AllArgsConstructor
    @Getter
    public class FinraResponse {

        private Response response;
    }
  
  
  
    package org.finra.api.client;

    import io.restassured.response.Response;
    import java.util.Arrays;
    import java.util.List;

    public class DataFinraResponse extends FinraResponse {

        public DataFinraResponse(Response response) {
          super(response);
        }

        public MonthlyTop100[] asArray() {
          return getResponse().as(MonthlyTop100[].class);
        }

        public List<MonthlyTop100> asList() {
          return Arrays.asList(getResponse().as(MonthlyTop100[].class));
        }
    }

The @AllArgsConstructor and @Getter annotations also comes from Project Lombok, and they generate the constructor and all of the getters for all of the class fields, respectively.

Step 4 - Create the API Client

Next, we create the API client that ties everything together: Rest Assured, the response classes, and the model classes. Within the root/src/main/java directory, create a class called FinraApi.java as follows:

  

    package org.finra.api.client;

    import static io.restassured.RestAssured.given;

    import io.restassured.http.Header;
    import io.restassured.http.Headers;
    import io.restassured.response.Response;
    import org.apache.http.HttpHeaders;

    public class FinraApi {

        public static final String BASE_URI = "https://api.finra.org";

        private Response getResponse(Headers headers, String uri) {
            return given().headers(headers)
                .get(uri)
                .then()
                .extract()
                .response();
        }

        public DataFinraResponse getDataAsJson(String group, String name) {
            Header header = new Header(HttpHeaders.ACCEPT, "application/json");
            Headers headers = new Headers(header);
            String uri = BASE_URI + "/data/group/" + group + "/name/" + name;
            Response response = getResponse(headers, uri);
            return new DataEndpoint(response);
        }

    }

This class contains a public method called getDataAsJson responsible for:

  • Building the HTTP request
  • Sending the request to FINRA's API
  • Returning the response wrapped inside the DataFinraResponse

The request is built specifically so that the response is returned in JSON format rather than the default .CSV format through the use of the standard HTTP Accept header. The DataEndpoint class provides extra functionality so that the caller can convert the response into a list of records that will simplify the data processing.

Step 5 - Consume FINRA APIs

Finally, we’ll use JUnit 5 to create a series of tests that will consume and process the data set. Junit 5 provides a quick way of writing small units of code execution for experimenting and building demos. Within the root/src/test/java directory, create a class called MonthlyTop100Tests.java as follows:

  
    package org.finra.finra.api.client;

    import java.util.DoubleSummaryStatistics;
    import java.util.LongSummaryStatistics;
    import org.junit.jupiter.api.Test;

    public class MonthlyTop100Tests {

        private static FinraApi finraApi = new FinraApi();

        @Test
        public void numberOfSharesTraded() {
            LongSummaryStatistics statistics = finraApi.getDataAsJson("otcmarket", "monthlytop100")
                .asList()
                .stream()
                .mapToLong(record -> record.getNumberOfSharesTraded())
                .summaryStatistics();
            System.out.println(statistics);
        }

        @Test
        public void dollarVolume() {
            DoubleSummaryStatistics statistics = finraApi.getDataAsJson("otcmarket", "monthlytop100")
                .asList()
                .stream()
                .mapToDouble(record -> record.getDollarVolume())
                .summaryStatistics();
            System.out.println(statistics);
        }

        @Test
        public void closingPrice() {
            DoubleSummaryStatistics statistics = finraApi.getDataAsJson("otcmarket", "monthlytop100")
                .asList()
                .stream()
                .mapToDouble(record -> record.getClosingPrice())
                .summaryStatistics();
            System.out.println(statistics);
        }
    }

Output when the tests are executed is shown next:

  
    dollarVolume: DoubleSummaryStatistics{count=1000, sum=612056137.000000, min=1.000000, average=612056.137000, max=96875451.000000}
    numberOfSharesTraded: LongSummaryStatistics{count=1000, sum=399066225707, min=20, average=399066225.707000, max=8659694883}
    closingPriceDoubleSummaryStatistics{count=1000, sum=3609.613768, min=0.000050, average=3.609614, max=102.650000}

Let’s dissect what’s taking place in each of these tests:

  • The FINRA API client makes a request to get the data for the monthly top 100 data set
  • The response is converted into a list of MonthlyTop100 objects by the DataFinraResponse class
  • The list of MonthlyTop100 objects is converted to a stream
  • Each record is mapped to a numeric field
  • We display the results and statistics

Conclusion

When working with APIs, building an API HTTP client provides a robust and reusable way to interact with data sets or endpoints in the FINRA API ecosystem. Once the client and the supporting code is in place, you can see how easy (with just a couple of Java statements!) it is to consume and get a summary of the statistics for a data set. Once the data is gathered and saved into a list of Java objects, you can start to work with the data programmatically and deliver business value for your customers.