OPRA 96-line Expansion: The Big Boost in Latency and Infrastructure Requirements

The Options Price Reporting Authority (OPRA) consolidated feed is the largest market-data feed on the planet, with minimum dual 40Gbps cross-connects required per side and over 200 billion messages per day. And it just got bigger.

Back in August 2022, SIAC announced plans to double the number of multicast lines from 2x 48 to 2x 96, to increase capacity. After several delays to give the industry time to prepare, this change went live on February 5, 2024. Ahead of the change, SIAC published guidance on how the 1ms and 100ms microburst rates could almost double. So what’s actually happened since this change went live?

Pico has precision visibility into OPRA, as we have instrumented our 4x 40G Mahwah cross-connects (where we receive the OPRA feed) with Corvil Analytics, giving us a complete history of hardware timestamped OPRA packet-capture data that is available via our “Pico Raw Historical Data” (PRHD) product.

More than that, we monitor the entire feed in real-time, including the key metrics of Microburst, Gap Detection, and Publish-to-Receive Latency. Pico Operations use this to assure the quality and completeness of the feed we deliver to our customers, and detect latency changes intraday. Also, this gives us the perfect vantage point to see exactly how the move to 96 lines has changed the feed, and how the client experience has changed.

Before delving into detail, let’s summarize what we’ve seen:

Microburst rates have approximately doubled. Subscribers need more than 40Gbps per side (A and B) to avoid latency and gaps. The aggregate bandwidth across both copies of OPRA is already regularly exceeding 70Gbps.
Latency has decreased. Prior to the move to 96-lines, the OPRA publishers were unable to keep up with the data rates and at busy times ticks were regularly delayed by 400 microseconds. After the upgrade, these spikes have been eliminated. Clients need to be able to handle the increased microbursts to avoid introducing latency in their own systems.
Daily volumes, average bandwidth and one-second peak rates have not increased substantially, showing the importance of monitoring at microburst timescales to avoid underestimating peak demands on infrastructure.

In anticipation of these changes, in 2023 PicoNet upgraded to 100Gbps infrastructure in the US, and ensured physical separation of the A and B copies of the OPRA stream for full resiliency. We have also been encouraging our customers to move to 100Gbps service to avoid congestion, protect against gapping, and minimize variable latency. For more details on our OPRA feed availability, please get in touch or watch back this webinar: “Staying Ahead of OPRA – 100Gbps Networking and Analytics”

Now let’s get into a little detail.

Microburst – the 1ms microburst has approximately doubled from 40Gbps to over 70Gbps. These values are showing the aggregate 1ms microburst across the A and B feeds, in Gbps. We can see the dramatic increase in microburst at precisely 9.30 EST. With microburst per side already approaching 40 Gbps, clearly there is no longer any headroom for receiving even one copy of OPRA over 40Gbps, underlining the need to move to 100Gbps or dual 40G per side for OPRA service.

OPRA A+B 1ms Microburst – before and after the 96-line upgrade

Latency – We have seen a dramatic reduction in the OPRA publisher latency. Using Corvil, we compare the time at which we receive the packets from OPRA with the embedded timestamp of each published tick. This tells us how old the data is when we receive it. Prior to the 96-line upgrade, we saw regular peaks of up to 400 microseconds (average latency was 11 microseconds). This max latency reveals the buffering that was taking place in the 48-line feed assembler during market peaks, as they were unable to keep up.

After the upgrade, as can be seen below, these outliers have disappeared, the daily max latency is now reliably below 60 microseconds.

What this means is that receivers who are able to keep up with the new microburst level are getting an improvement of over 300 microseconds in OPRA latency during the busiest market events. But if your feed-handler is not able to keep up, and is instead forced to buffer data during these microbursts, then tick-to-order latency will be impacted. If you are are trading off OPRA data and are latency sensitive, you should be monitoring wire-to-wire tick-to-order latency. Corvil is able to keep up with the stringent throughput requirements for such measurements.

OPRA Publish-to-Receive latency, showing reduction after the 96-line upgrade

Daily Volumes and Average Bandwidth have not changed substantially. Daily message volumes in the first week after upgrade ranged 130-145 billion messages per day, compared to a range of 130-170 the previous week. Also, we can see that one-second peak bandwidths did not change substantially, growing from 23Gbps to 28Gbps (aggregate of A and B). This highlights the need to monitor microburst and latency: if you were relying on one-second peaks you would have missed the almost 2x increase in activity that happened at the timescales that matter for trading.

OPRA A+B one-second peak bandwidth, before and after 96-line upgrade

So, in summary:

The OPRA 96-line upgrade has had a dramatic impact.
Microburst has almost doubled, and clients need more than 40Gbps per side if they want to avoid introducing latency and potentially gaps at peak times.
Latency outliers have reduced by 75%, creating opportunities for traders able to keep up with the new increased bursts. To avoid processing latency, clients should ensure feed-handlers can keep up with microburst-timescale peak message rates of about 75 million messages per second (so 75,000 messages in a one-millisecond interval)
Corvil 100G Appliances (CNE-8800, CNE-10000) can monitor and capture to disk the entire OPRA feed in realtime, including microburst, gap-detection and publisher latency analytics.

To learn more about how Corvil Analytics can help you monitor and capture OPRA at the timescales that matter, how Redline feed handlers can keep up with market-peaks, and how Pico can deliver you the fastest gap-free market-data in the business, please get in touch, or visit https://www.pico.net.