How NOT to use Sierra Chart’s DTC Server to Pull Granular Futures Data
This one was a flop, but I wanted to share anyway.
We have covered the basics of market data and wrote some simple Python code to get data from yfinance and Alpaca’s web API. Today, we turn it up a notch. We are going to use Sierra Chart’s DTC server and Denali data feed to get historical tick data for ES (futures,baby!), aggregate it into 1-minute OHLCV bars, and then save it to CSV.
Buckle up.
UPDATE:
This project ended up being a failure. Not because I couldn’t figure it out, but because you can’t get CME data through the Sierra Chart DTC server. This is because of data usage agreements. I was going to just scrap this post, but I decided to publish it as is and leave the code I was working through as an example for anyone interested in seeing how to encode/decode and parse binary data connections. This will be useful in the future if I decide that I want to send trades through DTC, but I don’t know that I am there just yet.
Anyway, feel free to read through what is here. It is still accurate information, I just didn’t post any code because it didn’t work.
Live and learn.
Why am I using Sierra Chart, DTC, and Denali?
There are several reasons that we are using Sierra Chart (SC) for this tutorial. To put it simply, it’s a legit platform, but I will break it down a bit more below.
Sierra Chart
First and foremost, this platform is programmed in C++ and built for speed and efficiency. The documentation is dense, and the support forum is sometimes difficult to navigate. The general setup is not intuitive if you are used to other platforms, and it can be generally difficult to navigate and figure out when you first start using it. However, after you climb this steeper hill, the fruits at the top are well worth it.
It is fast. Wicked fast. It doesn’t eat your RAM and it stays out of the way. Honestly, the only other program I have seen that runs this efficiently is RealTest. Everything else is slow in comparison.
It is difficult to figure out. But, so are many other things. Most hard things are worth the work. Bet you didn’t know you were gonna get a life lesson in this post too, did you?
Teton
Teton is Sierra Chart’s futures order routing service. Here is what SC says about it:
The Teton Futures Order Routing service is a high-quality order routing service with advanced risk management from Sierra Chart to provide order routing, for outright futures and spreads, direct to the major exchanges. There is no other intermediary provider.
Remember when I talked about “intermediaries” when discussing data? Well, order routing can have them too. Except, Teton doesn’t. That makes it fast. How fast?
Orders are routed direct to the exchange with high reliability and very low latency, in under 500 microseconds.
Wow. How fast is a microsecond? One millionth of a second. That’s 0.0005 seconds. Fuck me, that’s fast.
Their servers for Teton are colocated with the CME order matching computers (we will talk about colocation more in future posts). That’s cool. It is offered to clearing firms and users at no cost. There are no transaction fees per contract traded. As far as SC is aware, this is an industry first.
That is pure quant dirty talk right there, y’all.
Denali
Ok, so their futures order routing is legit good. Surely, their data must suck.
Nope.
Let’s see what they have to say about it:
Streaming real-time data.
Market depth data.
Historical intraday day data. CME data has tick by tick data back to 2011 and minute data back to 2008.
1 year of tick data for US stocks and minute data back to 2008 (assuming the stock existed then).
Historical daily data.
100% accurate historical Bid Trade Volume and Ask Trade Volume.
High precision data delivered with millisecond time stamping.
Low latency.
Direct CME FIX data feed at the SC backend.
Wow. That’s wicked cool.
It must cost a million dollars a month.
Also, nope.
It’s one of the cheapest data streams I have found.
DTC
We discussed this briefly in the last post. This is what will allow us to get our data from Sierra Chart without having to export it to CSV from inside the platform. Of course, if you want to just export the data, have at it. That’s what makes this platform a dual threat. Don’t want to code? Cool. Want to code? Also, cool.
Remember, it turns out this is untrue. You can’t get CME data via DTC. I knew that the documents mentioned this, but it also mentions being able to request historical data, so I figured I would try it anyway. Turns out, it won’t send you data. Lame.
Turn Sierra Chart into a DTC Server
So, you decided you want to give this a go. Maybe you already used Sierra Chart, maybe you got it just to test this idea. Or, maybe you are just reading bc you think this is a cool project and you like my style.
This part of the tutorial is pretty simple. Go to the SC document page that shows you how to turn on the DTC server and follow the instructions under the “Server Usage” section.
For this tutorial, we are going to set the encoding to binary. This is because I want to test how to work with it. The whole purpose of this is to ease into building our own research engine and learning how to play with this DTC protocol is part of it. We will keep everything else at the defaults.
Additionally, make sure that your intraday data is set to store as 1 tick. I believe this is default, but I am unsure.
Setting up the project
Time to work.
First, we need to setup our project environment and activate the virtual environment just like we did in the previous article.
mkdir tutorial3
cd tutorial3
mkdir data
New-Item requirements.txt
New-Item tutorial3.py
python -m venv .venv./.venv/Scipts/ActiveNext, the required dependencies for this script. I am going to avoid using Pandas this time. I am doing this so I can get a little more hands on with the data using Numpy. I am not going to use any Pandas DataFrames in quantKit, so I want to make sure I can handle the data without it. Besides, this is good practice for you too. You don’t want to rely on a bunch of dependencies if you can help it.
numpyThat’s it. Lean and mean. Install it with pip install -r requrements.txt.
The Code
The purpose of this tutorial is two-fold. I am personally testing this out as I create this walk-through, and I am showing you how to test this kind of shit so you can get used to playing around with Python. So, instead of just pasting it all into one block, I am going to break it into sections. This just helps describe what each section is doing better without trying to rely on large comment blocks.
I suggest actually typing it in as you go through. Get reps. Or, you can paste it. And if you are real lazy, just clone the repo.
Wait…
Where is the code?
Well, it turns out that this project was a wash.
So, check out the closing thoughts to see what happened.
You can find the code that I was using at the GitHub link at the top.
Closing Thoughts
Ok, obviously this didn’t work out the way I was hoping. Sierra Chart has enabled the historical data request messaging, but because of the data usage agreements with CME, they can not allow any CME data to be sent over the DTC server.
I didn’t check to see if this applies to equities data or not. It would seem odd to enable the messages but not actually be able to send any data over the server. It is possible that it works for data that isn’t Denali, but that would defeat the purpose of this project.
Everything else I discussed in this article is still correct. The Denali data feed and the Teton routing are pretty damn good. I would imagine that it’s one of the better setups for futures traders that doesn’t require you to have buckets of money to get yourself competitive. Does it let you compete with firms? Nope. That will never happen, but it does let you compete with other retail traders and maybe even some small retail shops.
The name of the game is reliable fast data and routing.
What’s next?
Since we can’t use DTC to do this, we are going a different route. I am just going to parse the binary data files that SC uses to store historical data. This way, we still get the same tick-level data that we can then use to aggregate into minute bars and turn into a continuous chart.
Hopefully it will be published soon. Until then…
Happy hunting.
This post doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.