Oh boy, here we go again. Larry is on another kick. Why can’t he just keep his shit together? Just deliver market trading strategies and research and keep it simple. Stop trying to reinvent the wheel.

If that’s what you’re thinking, I hear you. I really do.

And I only have this to say in response:

Fuck off.

Still here?

Cool. I like you. Let’s get into it.

I started this Substack in late 2023, and my only focus was learning how to become a better day trader. Over the last two years, I learned a lot. We went from using NinjaTrader and improving ATS strategies with better code. I got asked to stop doing that by the ATS creator, and that marked the first shift.

After that, I wasn’t really sure what to do. My knowledge in this field was still limited, but I knew one thing: if you’re going to create or test automated strategies, you need to understand how the coding side actually works. That sent me down a rabbit hole. I picked up some good books by Timothy Masters and started exploring.

I used the concepts from those books to build a bunch of Python scripts for testing indicators and features. That was fun, but I was still green.

I started researching different trading platforms. I looked at QuantConnect and the Lean engine. I used TradingView for a while and tested some ideas. Then I found RealTest. This was the first program I really fell in love with. The philosophy behind its development resonated with me. Marsten Parker is a wizard—programmer and trader. Seriously, he was in Unknown Market Wizards by Jack D. Schwager. But it was the design philosophy of RealTest that really drew me in.

Why?

Because details matter.

That is what separates amateurs from pros. I’m not claiming to be a pro. Not even close. But I am neurotic, and I crave precision. I’m not naive enough to think perfection is attainable in the pure sense, but striving to get as close as possible still matters. That’s how we sharpen ourselves.

Once I found RealTest, I dove hard into strategy research. I built and dropped a bunch of strategies. I used indicator testing tools to test different features, then pulled them into RealTest to see how they performed in backtests. I was testing equities, futures, and whatever else came to mind.

That ran strong for months.

Then, somewhere around the beginning of this year, I hit a wall.

I started losing sight of what I was doing. Life was kicking my ass, and I fell off for a while. The more I learned about quantitative trading, the more I wanted better tools, and the more I became aware of the disconnect between retail traders and firms. I got jaded. I started questioning what I was doing and, eventually, started disliking it.

I was still actively day trading futures into the beginning of this year, but then I decided to step away for a while. I’d been researching systematic and quantitative trading for a good bit and still hadn’t reached the point where I felt confident enough to run an automated strategy live. I did run one of my RealTest strategies live for a while, but it wasn’t fully automated and left a lot to be desired. It did well enough, but I turned it off anyway. My heart just wasn’t in it, and I wasn’t sure what I wanted to do next.

That was when I decided I was going to build my own tool for researching trade strategies.

As much as I love RealTest, it still doesn’t do everything I want. It’s proprietary. It has quirks. Marsten is an old-school C programmer, and RealTest is opinionated. Good opinions, mostly. A lot of them I agree with. But it still lacks some of the things I want.

I spent a good bit of time writing indicators in C that could be used directly in RealTest, and that taught me something: I like low-level languages like C more than I like languages like Python, C#, Go, JavaScript, or Java.

So I started the quantKit project. Since I wasn’t proficient in C yet, I decided to build the tool in Python—but without major dependencies, and with explicit code instead of all the Pythonic shit that makes code look like Arabic to me. I get why Python is the way it is, and I understand I’m taking the road less traveled, but it made sense to me. Still does.

During all of this, I stopped writing.

I haven’t produced a new strategy since February. That cost me subscribers, and I get it. That’s why most people are here. They want strategies with good results so they can test them and maybe make some money.

But that was part of the problem.

I had become one of those guys posting content that feeds directly into people’s desire to make more money. Nothing I posted was meant to be used as advice (read the disclaimer) but still. It was my research, and anyone who used it live did so on their own. That disclaimer still stands. But the more I sat with it, the more it bothered me.

I didn’t want to be one of those people.

I wanted to give real knowledge to the community. I wanted to help people actually understand what they were looking at so they could make better decisions. I couldn’t do that by just posting strategy results and backtests. That became a moral conflict.

quantKit was supposed to be the answer. It still might be.

But it is a huge endeavor, and it has taken me (just some dude trying to do this from home while being a full-time stay-at-home dad) a lot longer than I expected. There’s probably a bias for that. Kahneman wrote about it. I can’t remember the name right now, but you know the one.

While all of that was happening, life started fighting back.

Interlude: It’s about to get personal, real, and maybe a little heavy. Skip it if that isn’t your thing. Sometimes it’s easier to pretend the people you read online aren’t actually human.

Right before March, I had a pipe burst in my basement and flood the place. I had to handle all of that repair myself, except for the pipe itself—which I now know I probably could have handled too. I had to replace all the flooring in my father’s studio and deal with all the drying and repair work from the water damage. It sucked up a lot of time and energy.

Then in March, while I was already trying to find myself, my father almost died. He had cardiac issues that sent him to the ER. During that visit, he had a femoral bleed that almost killed him. After surgery, he came home, then went straight back to the ER in septic shock from the surgery site. That second visit turned into a hospital stay and rehab that lasted for months.

For context, my father lives with me and my family. When we moved into this house in 2021, he came with us so I could help take care of him. My mother passed many years ago, and he needed to be with someone. Don’t mistake that for sainthood. I am not particularly close to or fond of my father. There is history. But he is the only parent I have left, and I care enough not to leave him to figure everything out alone.

That wasn’t the end of it.

Aside from the house repairs and my father being in the hospital, my wife was also pregnant with our second child. That’s a blessing, obviously, but it changes the environment. Our first child is two and requires more attention, more energy, and a lot more patience.

Then there was me.

If you’ve been reading for a while, you know I don’t have the normal résumé for someone writing about trading systems. Before my son was born, I was a paramedic. That was my career for over a decade. Before that, I was in the Army. When I got back from Iraq in 2009, I had more time in a combat zone than I had in garrison. For those who don’t know what that means, I deployed almost immediately after training and was fast-tracked through the pipeline.

When I got out of the Army in 2010, I went straight into EMS and didn’t leave that life until my son was born in 2023. I almost missed his birth because I was in Iraq. I would not have made it back in time if my former Spades partner hadn’t just been promoted to project manager. Thanks to him, I was able to fly home fast and be there when my son was born. The day after he was born, I called my company and resigned. I wasn’t going to stay absent from my family.

That was when I started getting into trading and eventually launched this Substack.

Why am I telling you all this?

Because from 2007 to 2023, I worked nonstop in high-adrenaline and eventually high-threat environments. I never really took enough time off to recognize that I had issues I had never addressed. Then I quit. I really quit. I went all in on being a stay-at-home dad and trying to figure out how to work from home.

Once I slowed down, all the shit from deployment and EMS became impossible to ignore.

My house breaks. Then my father breaks. And during all of this, I am breaking too.

I couldn’t sleep because of nightmares. I found out I also have sleep apnea and needed to deal with that. I started therapy to work through a lot of shit I never worked through before. It was rough.

I let the Substack slide.

I couldn’t concentrate for shit. I didn’t know who I was. I was still trying to be a parent, a husband, and the stay-at-home spouse. The Substack just didn’t make the top-priority list.

On top of all that, I was trying to get a bunch of VA stuff settled and unfuck some bad financial decisions from the past couple of years. Maybe I’ll talk about that in another post, but for now it isn’t the point. It felt like every time I handled one thing, two more popped up. All the while, I was going crazy internally.

To put it bluntly, this year was a whirlwind of fuckery that had me questioning everything. Not just HGT or my own sanity. Everything.

I started seeing things differently. I started spotting patterns in the things we take for granted. I started thinking harder about profit, capitalism, and how far we’ve drifted from anything resembling a real free market. I started wondering how much of our current condition is stupidity, and how much of it is malice that finally learned how to wear a tie and speak politely.

Then I did what I always do.

I decided to handle more of it myself.

And weirdly enough, that is what helped me find solid ground again.

Once I started doing that—really doing that—things started clicking.

And now, back to our regularly scheduled programming.

That’s just life, right?

We’re all human. We all get tested. I handled this year the best I could, and this Substack absorbed a lot of the cost. So to the readers who made it this far, and to the people who have been here for a while: I’m sorry. I did not make this a priority. I believe that was the right call, but I also know I owe you an explanation.

Still, I didn’t just spiral. I kept learning.

Because that’s what we do.

If we get hit, we learn how to avoid it next time, or we get tougher. Sometimes you can’t avoid getting punched. But you can train how you respond.

Through all of this, I kept adapting. I slowly found myself again. I may not be the same shit-hot medic I used to be, but I am still sharp, still detail-oriented, and still driven. I just had to adapt my way of thinking to a different environment and a different life.

And that is what I want to talk about now.

This year taught me a lot. The biggest takeaway is this:

We are being trained to offload our lives.

Not just in trading. In everything.

We are conditioned to hand off anything difficult, tedious, or technical to someone else. We are taught that if we want to be good at one thing, we have to sacrifice competence in everything else. We are told it is impossible to be good at multiple things.

I think that is mostly bullshit.

Yes, there are some fields where elite specialization is real. If you want to be an Olympic runner, you probably are not also training to be an elite powerlifter. Fine.

But for most of life? No.

For most of life, “specialization” becomes an excuse for learned helplessness. It gives people permission not to try.

This year reminded me that I can still learn new shit. It reminded me that there are a lot of things we can become competent at without sacrificing everything else. And when we do, we gain back freedom we didn’t even realize we’d lost.

If we apply just a little more mental bandwidth across more areas of life, we often discover it doesn’t take much to get good enough at something that we no longer need to outsource it.

Let me give you a few examples.

Example 1: The basement flood

I won’t get into why the pipe burst, because the reason doesn’t matter. For the record, it wasn’t my fault, but the amount of damage that happened from it absolutely was somebody’s fault. The pipe was going to burst either way.

The first thing I did when I stepped into a couple inches of water in my basement was run to the breaker and shut off the power. Then I shut off the water at the main line. After that, I had no clue what to do next, so I called a plumber and got someone out to the house immediately.

They identified the problem (I had already found the leak and exposed the pipe) and fixed it. I watched them the whole time. The crew was cool, down-to-earth, and answered my questions. They repaired the pipe, replaced the fixture, and had a guy call me about water mitigation.

That guy couldn’t come out until the next day.

To me, that sounded like a great way to let my walls, wood, and insulation soak up as much water as possible before anyone did a damn thing. So I handled what I could. I cut the lower part of the sheet-rock, removed insulation that might have gotten wet, vacuumed up the water, assessed the flooring in the studio, and set every fan I had to work drying things out.

The next day, the mitigation guy told me it would cost five grand for his crew to come in and do exactly what I had already done.

I told him to kick rocks.

So I did the rest myself. Walls, floors, bathroom, moisture testing, all of it. It sucked. But I learned a lot. I learned that the previous owners had installed the basement room floor over carpet, so I had to pull that too and deal with the glue.

The glue. Fuck me. That was the worst part.

Point is, it became a major project that ate my attention for about two weeks. In the end, it cost me around $1,500, and most of that was LVP. I also found two other problems in the basement bathroom that I’ll need to address later.

Example 2: The kitchen

This one is quicker.

I like cooking. My favorite job was working in a kitchen. I watch cooking shows. I like a lot of different food. So naturally, I spend a lot of time in my kitchen.

And here’s something stupidly obvious that most people still ignore: if you make your own shit, you can save a lot of money and eat better food.

A pound of fresh fettuccine costs maybe one to two dollars to make, depending on the flour and eggs. A decent dried pasta at the store costs around five dollars, and the more common brands are still two to three. Most of them suck, usually get used all at once, and don’t store half-open especially well anyway.

This pattern is everywhere.

Bread? Dirt cheap if you make it yourself, and not nearly as complicated as people pretend.

Stock or broth? It’s made out of shit you were going to throw away anyway. That makes it basically free.

And once you start noticing it, you can’t stop. The question becomes: how much can you actually save by becoming an ingredient household?

The answer: more than you think.

It takes time, sure. Maybe a little skill. Maybe some love for food and process. But the payoff is bigger than most people realize.

Example 3: Finances

Big word. Scary word.

We hear “budget this” and “finance that” and “taxes this,” and eventually somebody says, “Just hire a professional.”

No.

Stop it.

The biggest secret about managing your own finances is basically the same as cooking: it isn’t hard. It can be tedious. It can be time-consuming if you start with no framework at all. But once you understand it, that’s it. The principles don’t change that often. Math definitely doesn’t.

At some point this year, it became clear that I had made financial mistakes that needed correcting. That process really started around August. So I decided to get control of it.

Luckily, I had at least some background in finance and accounting from a degree program. I decided to go full send and built double-entry books for our family finances and my self-employment finances. That forced me to see every detail, build the account structure correctly, and actually plan.

It took time, but I backfilled all of 2025, and now I know exactly what is going on with our finances. Maintenance is easy now that the hard part is done.

And it taught me something important:

I do not need a professional to manage this for me.

Even taxes become much simpler when your books are right. The IRS rules are publicly available. They are not gripping reading, but they are educational as hell. Once the new year rolls around, I can print a handful of reports and basically have the whole picture ready to go. For anything weird (like unamortized mortgage points or deductible mortgage interest allocations) I can build supporting worksheets and cite the actual IRS or state publications.

The point is that complete control creates clarity. And clarity creates leverage.

That means real accounting. Not random tables and sloppy worksheets. Real cost analysis, supported by law, maintained honestly, and handled with the same discipline you’d expect from a professional.

Good principles and honesty go a long way here.

Time to bring it home

It was a wild year. Crisis, distraction, health issues, family obligations, and everything in between—all while still being a father, a husband, and trying to stay self-employed so I could keep being a stay-at-home dad.

And through all of it, I kept wondering what I was actually doing with HGT.

I haven’t actively traded since March, and quantKit stalled out. Something stopped feeling right.

As I got a more intimate understanding of my home, my finances, and the systems that shape everyday life, I realized how much people miss because they have been trained not to look. We’ve been conditioned to outsource the important stuff. It’s either “too hard,” “too complicated,” or “not worth the time.”

Well, I say fuck that.

A lot of it is not hard. Some of it is time-consuming, yes. But that is often the price of freedom. Freedom from complacency, from consumer dependence, and from blind trust in people whose incentives you haven’t even examined.

As a family man, I can’t think of a better use of my time than understanding the systems my household depends on and making sure they work. That means learning where the points of failure are. It means understanding the tools, the constraints, the costs, and the leverage points. It means reducing the amount of your life that sits at the mercy of somebody else’s “expert” opinion.

That is why HGT is shifting gears.

Researching day trading systems has been a great adventure, but I don’t know when I’ll actively trade again. Don’t freak out. I am not walking away from market research. I’m changing how I approach it.

quantKit was originally meant to be a research engine for quantitative and systematic traders. The project was always ambitious, highly detailed, and very complex. It will live on, but I’m pivoting it from a trading backtesting engine into a broader systems-analysis tool that better aligns with what I actually believe about household independence, resilience, and control. I’ll share more on that in the next post.

Since I am no longer focused on day trading, and since this year gave me the perspective to get honest about what I actually care about, Hunt Gather Trade is expanding.

The new focus is building resilient, well-managed households.

Systematic trading taught me something useful: systems thinking applies to a hell of a lot more than markets. Personal and small-business finance. Accounting. Kitchen management. Home infrastructure. Repair. Networks. Information control. The hidden incentives inside modern convenience. These are the systems we actually live inside whether we acknowledge them or not.

And these systems contain some of the biggest levers available to a normal family trying to gain control over its time, money, and quality of life.

So that is where HGT is going.

I am still going to write about markets, but less from the perspective of day trading and more from the perspective of longer-term investing, capital allocation, and durable wealth building.

I am going to write about home infrastructure and why understanding your own systems matters.

I am going to write about the kitchen as a serious household lever, not some side hobby.

I am going to write about finance, accounting, proprietary systems, dependence disguised as convenience, networking, repair, and whatever else helps people take more control over their own lives.

Hunt Gather Trade is not going away.

It is just becoming more honest.

If that sounds like something you’re into, stick around.

The goal is simple:

Figure out how to become as free, capable, and self-sufficient as possible.

Stay curious.

Stay open-minded.

Stay alert.

Happy hunting.

This post does not represent any type of advice, financial or otherwise. Its purpose is educational and informational only. Backtest results are based on historical data, not real-time data, and there is no guarantee hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

Well, something predictable happened. Spreadsheets suck for complex tasks. I knew that, you knew that, everyone knew that. But, I wanted to test it. I wanted to KNOW why we don’t use spreadsheet apps. Besides, you can easily formulate features and trade signals and many of the other things you need when testing a trade idea. So, why can’t you do everything inside one?

Simple.

Spreadsheets aren’t relational. They are just tables. You can mimic database techniques, and you can mimic programming formulas, but it isn’t a database or a programming language, and the limits really show when you try and do all the things you can do with Python (or whatever your favorite language is).

The very first thing I noticed is how slow it is. I loaded a year’s worth of minute data into the spreadsheet and my computer started crawling. 350k rows of data is too much to handle. I switched to daily data to make it easier on myself. I got indicators calculated, I got signals produced, and then I tried aggregating trades…

And that’s when I quit. It is just unwieldy. Even generating a simple list of trades after calculating signals turned into a nightmare. It looks great in the screenshot:

Sexy, right?

BUT, from here, it would literally be easier to export it to CSV and finish in Python. What does that mean?

The test is over.

So, another failed experiment. And I was really hoping to release this strategy for subscribers to review. Guess I will just need to get quantKit up and running instead.

There is something beautiful about this, though. All of this experimenting is learning. I am learning how to visualize data and how to think about handling data. All of this is applicable to how I intend on handling data within quantKit, and it helps me visualize how to actually backtest ideas accurately.

Before all of this, I just traded like any other trader. I would build visual charts, add a couple indicators/features I used to help target a trade, draw my lines, and trade. What I realize now is that even with a strict “strategy”, this is just gambling. The game is in managing risk, and the strategy just helps with that. Just like poker (my favorite is Texas Hold’em), you can make a living managing risk. But at the end of the day, it’s gambling.

I really learned this when I started looking at testing trade ideas with code and math. First, it was difficult to put some of the strategies I had tried into strict criteria, as I couldn’t articulate why I did something. A good example is putting horizontal lines at support levels based on a aggregated OHLC chart. I didn’t understand that what I was looking for was liquidity pools and I certainly didn’t understand how to spot them with code.

Now, I know better. I understand where to look, I am learning how data actually flows from the market to my computer, I understand how people can build an edge just by taking advantage of location (whether physical co-location or outside the kernel), and I understand that what most traders do (even the ones who are building algorithmic systems) is gambling.

Anyway, this wasn’t meant to be a philosophical piece, so I am going to stop here. You get the point.

If you didn’t, here it is:

Spreadsheets aren’t good for this kind of work. I wasted more time testing another idea when I could have been coding quantKit.

They did teach me a valuable lesson: don’t force tools to do what they weren’t built for. quantKit exists (or will exist?) because research deserves real structure, not a grid pretending to be a database.

Happy hunting.

…

Oh, you want me to excuse my tardiness.

Some of the shit I decided to do was harder than anticipated. Harder? Maybe that’s not the best term. It was more confusing than I anticipated. That doesn’t really do it justice either.

Look, futures contracts are fucking weird. It makes sense once you figure it out, but building continuous contracts stumped me for a while. I also realized that I was using some comparison logic that I wouldn’t be able to use when converting this to a low-level language (I will explain in the code breakdown), so I had to spend some time refactoring those comparisons.

So, yeah. Basically, I just suck. It is what it is. I’ve got it locked in now, and I should be able to plug and play any different type of back adjustment or rollover logic I want.

Let’s not waste to much time here. There is a lot to get into, so let’s do it.

Code can be found here:

GitHub

BLUF

There is over 800 lines of code in this tutorial. It doesn’t use anything outside of the standard library and Numpy. It does import Plotly, but that was just to test that my charts mimiced the charts Sierra Chart built. I built a continuous-futures contract builder that exactly matches Sierra Chart using only raw integers internally, then export a clean, human-readable CSV.

• Load Sierra Chart static files (SCID ticks, DLY daily)

• Resample ticks to 1-minute bars

• Compute volume-based roll dates (YYYYMMDD ints)

• Back-adjust using daily settlements

• Stitch contracts into a continuous series

• Keep time raw: intraday = UTC microseconds; daily = YYYYMMDD

• Filter by start/end using integer comparisons

• Convert to local time only for export/plots

• Export final minute series to data/SC_MES_Minute.csv

Code

I am going to go through this section by section or function by function at points. There is a lot to unpack.

Imports

These are the imports needed for this project. Most of them are standard library import. Numpy is what we use to manage the data (no Pandas) and Plotly is just there for a test.

import os
import numpy as np
from datetime import datetime, timedelta, timezone
from zoneinfo import ZoneInfo
import plotly.graph_objects as go # <- Only needed to test

Constants

There are a lot of constants to define for this project, most of them are structs (look it up) that we need to format our data correctly or read Sierra Chart’s data. Sierra Chart stores it’s intraday data in binary formats, and I like this, so we also define our data structs in a similar way. We also need some time and month-code constants defined to help us later.

# SCID binary formats
HEADER_DTYPE = np.dtype(
    [
        (”FileTypeUniqueHeaderID”, “S4”),
        (”HeaderSize”, “<u4”),
        (”RecordSize”, “<u4”),
        (”Version”, “<u2”),
        (”Unused1”, “<u2”),
        (”UTCStartIndex”, “<u4”),
        (”Reserve”, “S36”),
    ]
)

RECORD_DTYPE = np.dtype(
    [
        (”DateTime”, “<i8”),
        (”Open”, “<f4”),
        (”High”, “<f4”),
        (”Low”, “<f4”),
        (”Close”, “<f4”),
        (”NumTrades”, “<u4”),
        (”TotalVolume”, “<u4”),
        (”BidVolume”, “<u4”),
        (”AskVolume”, “<u4”),
    ]
)

# Our tick struct/format
TICK_DTYPE = np.dtype(
    [
        # raw microseconds since 1899-12-30 (Sierra Chart epoch)
        (”TimeStamp”, “<i8”),
        (”Price”, “<f8”),  # trade price
        (”Volume”, “<f8”),  # trade volume
        (”Bid”, “<f8”),  # bid at time of trade
        (”Ask”, “<f8”),  # ask at time of trade
    ]
)

OHLCV_DTYPE = np.dtype(
    [
        (”TimeStamp”, “<i8”),  # start of bar, raw micros since 1899-12-30 (SC_EPOCH)
        (”Open”, “<f8”),
        (”High”, “<f8”),
        (”Low”, “<f8”),
        (”Close”, “<f8”),
        (”Volume”, “<f8”),
    ]
)

DAILY_DTYPE = np.dtype(
    [
        (”Date”, “<i8”),  # YYYYMMDD
        (”Open”, “<f8”),
        (”High”, “<f8”),
        (”Low”, “<f8”),
        (”Close”, “<f8”),
        (”Volume”, “<i8”),
        (”OpenInterest”, “<i8”),
        (”BidVolume”, “<i8”),
        (”AskVolume”, “<i8”),
    ]
)

# === Time Constants ===

# Sierra Chart epoch (NOT Unix epoch).
# All SCID .scid files store timestamps as 64-bit integers
# representing microseconds since 1899-12-30 00:00:00.
# This matches legacy Excel/Windows serial-date conventions
# and allows handling of historical equities that predate 1970.
SC_EPOCH = datetime(1899, 12, 30)

# Microsecond unit for consistency in conversions
SC_MICROS = np.timedelta64(1, “us”)

# CME month codes
MONTH_CODES = {
    “F”: 1,
    “G”: 2,
    “H”: 3,
    “J”: 4,
    “K”: 5,
    “M”: 6,
    “N”: 7,
    “Q”: 8,
    “U”: 9,
    “V”: 10,
    “X”: 11,
    “Z”: 12,
}

START_DATE = “2024-06-01”
END_DATE = “2025-09-01”

PATH_DIR = r”C:\SierraChart\Data”

Helpers

Anything that I used more than once (or plan to use more than once in the future) became it’s own helper function. Some of these aren’t used in this complete project. They were helpers I used earlier on, but as I refactored the project, I didn’t need them anymore. I left them in there just in case we might need them again in the future.

Roll boundary note. To stitch intraday futures you can’t cut at calendar midnight. CME sessions run 17:00–16:00 Central Time, so a roll date actually flips at 17:00 CT on the prior local day. In this tutorial I approximate that by converting the roll date in UTC and shifting by a fixed 7 hours. This helper is not timezone/DST aware, so the seam is early by 5–6 hours versus CME’s exact 22:00/23:00 UTC boundary. It’s good enough here because both legs are cut at the same point and back-adjustment uses daily settlements, so prices line up and the chart matches visually. For a precise implementation later, I’ll convert “(roll_date − 1) 17:00 America/Chicago → UTC” with DST handling.

def micros_to_dt64(us):
    return SC_EPOCH.astype(”datetime64[us]”) + (us.astype(”<i8”) * SC_MICROS)


def utc_to_local(us: int) -> datetime:
    “”“
    Convert UTC microseconds since Epoch
    into the local time (system timezone) as Python datetime.
    “”“
    # UTC base time
    dt_utc = datetime(1899, 12, 30, tzinfo=ZoneInfo(”UTC”)) \
             + timedelta(microseconds=int(us))

    return dt_utc.astimezone()


def yyyymmdd_to_dt64(val: int) -> np.datetime64:
    s = str(val)
    return np.datetime64(f”{s[:4]}-{s[4:6]}-{s[6:]}”)


def yyyymmdd_to_micros_roll_boundary(date_int: int) -> int:
    “”“
    Convert YYYYMMDD integer to UTC microseconds since Sierra epoch (1899-12-30),
    adjusted to 17:00 UTC of the *previous* day (CME rollover boundary).
    “”“
    # Parse YYYYMMDD to datetime (UTC midnight of that date)
    year = date_int // 10000
    month = (date_int // 100) % 100
    day = date_int % 100
    midnight = datetime(year, month, day)

    # Subtract 7 hours → previous day 17:00 UTC
    roll_time = midnight - timedelta(hours=7)

    # Return microseconds since Sierra epoch
    return int((roll_time - SC_EPOCH).total_seconds() * 1_000_000)


def parse_contract_code(sym: str) -> int:
    “”“
    Parse a futures contract code into YYYYMM integer.
    Handles any root length, just looks at the last 3 chars.

    Example:
        >>> parse_contract_code(”MESH24”)
        202403
        >>> parse_contract_code(”ESU24”)
        202409
    “”“
    code = sym[-3:]  # last three characters: month + 2-digit year
    mon = code[0]
    yy = int(code[1:])

    if mon not in MONTH_CODES:
        raise ValueError(f”Unknown month code: {mon} in {sym}”)

    year = 2000 + yy  # always assume 2000s for now
    month = MONTH_CODES[mon]

    return year * 100 + month


def get_settlement(contract: np.ndarray, roll_date: int) -> float:
    “”“Return the settlement (close) on the last bar before roll_date.”“”
    # Convert entire column of YYYYMMDD ints into datetime64[D]
    dates = contract[”Date”]
    idx = np.searchsorted(dates, roll_date) - 1
    if idx < 0:
        raise ValueError(”No settlement found before roll date.”)
    return contract[”Close”][idx]


def shift_contract(arr: np.ndarray, adj: float):
    “”“Shift OHLC values of a contract by adj (in-place).”“”
    arr[”Open”] += adj
    arr[”High”] += adj
    arr[”Low”] += adj
    arr[”Close”] += adj

Getting contracts

These two functions are how we get the contracts we need for our continuous contracts. They go to a specified directory and return our requested date range contracts by using some of the constants we defined earlier to help. You will see that the discover_contracts function just returns all the contracts with the correct name in the directory and the contracts_in_range function actually returns the correct contracts that we want for our specific request.

def discover_contracts(data_dir: str, root: str):
    “”“
    Find unique contracts for a given root symbol in a directory.

    Scans all filenames starting with the root (e.g. “MES”) and extracts
    contract codes (month+year). Returns sorted unique records.

    Args:
        data_dir (str): Directory containing contract data files.
        root (str): Futures root symbol (e.g. “MES”, “ES”, “CL”).

    Returns:
        list[tuple[int, int, str]]: Sorted unique contracts as (year, month, symbol).

    Example:
        >>> discover_contracts(”/data”, “MES”)
        [(2024, 6, “MESM24”), (2024, 9, “MESU24”)]
    “”“
    root = root.upper()
    seen = set()
    out = []

    for fn in os.listdir(data_dir):
        name = fn.upper()
        if not name.startswith(root):
            continue

        # Extract 3-char contract code (e.g. ‘M24’)
        code = name[len(root): len(root) + 3]
        if len(code) != 3:
            continue

        mon = code[0]
        yy = code[1:]

        # Validate month code and year digits
        if mon not in MONTH_CODES:
            continue
        if not yy.isdigit():
            continue

        sym = f”{root}{mon}{yy}”
        if sym in seen:
            continue
        seen.add(sym)

        # Expand YY into YYYY (assume <70 => 2000s, else 1900s)
        year = 2000 + int(yy) if int(yy) < 70 else 1900 + int(yy)
        month = MONTH_CODES[mon]

        out.append((year, month, sym))

    # Explicit insertion sort by (year, month)
    for i in range(1, len(out)):
        key_item = out[i]
        j = i - 1
        while j >= 0 and (
            out[j][0] > key_item[0]
            or (out[j][0] == key_item[0] and out[j][1] > key_item[1])
        ):
            out[j + 1] = out[j]
            j -= 1
        out[j + 1] = key_item

    return out


def contracts_in_range(data_dir: str, root: str, start: str, end: str):
    “”“
    Slice contracts for a given root symbol within a date range.

    Uses discover_contracts() to build the full contract list, then
    returns only those covering [start, end], including the immediately
    prior contract for rollover/back-adjust purposes.

    Args:
        data_dir (str): Directory containing contract data files.
        root (str): Futures root symbol (e.g. “MES”, “ES”, “CL”).
        start (str): Start date as ‘YYYY-MM-DD’.
        end (str): End date as ‘YYYY-MM-DD’.

    Returns:
        list[str]: Contract symbols covering the date range.

    Example:
        >>> contracts_in_range(”/data”, “MES”, “2024-06-01”, “2024-12-01”)
        [’MESM24’, ‘MESU24’, ‘MESZ24’]
    “”“
    syms = discover_contracts(data_dir, root)
    if not syms:
        return []

    # Convert start to (year, month)
    dt_month = np.datetime64(start, “M”)
    dt_obj = dt_month.astype(object)
    s_y = dt_obj.year
    s_m = dt_obj.month

    # Convert end to (year, month)
    dt_month = np.datetime64(end, “M”)
    dt_obj = dt_month.astype(object)
    e_y = dt_obj.year
    e_m = dt_obj.month

    # Find first contract >= start
    start_idx = len(syms) - 1
    for i, (y, m, _) in enumerate(syms):
        if (y, m) >= (s_y, s_m):
            start_idx = i
            break
    if start_idx > 0:
        start_idx -= 1  # include prior

    # Find last contract <= end
    end_idx = start_idx
    for i, (y, m, _) in enumerate(syms):
        if (y, m) <= (e_y, e_m):
            end_idx = i

    if end_idx < len(syms) - 1:
        end_idx += 1

    # Collect results
    out = []
    for i in range(start_idx, end_idx + 1):
        _, _, sym = syms[i]
        out.append(sym)

    return out

Load daily data

Two functions to get daily data files. Sierra Chart saves these files as basic text files (.dly). These two functions use the contracts list generated by the previous functions to actually load each contract and its data into memory for us to use. We use our DAILY_DTYPE struct to structure the data.

def load_daily_data(path: str) -> np.ndarray:
    “”“
    Load entire daily data file into structured array.

    Args:
        path (str): Path to .dly file (CSV-like SierraChart daily file).

    Returns:
        np.ndarray: Daily records with dtype=DAILY_DTYPE
    “”“
    # Read raw text
    with open(path, “r”) as f:
        lines = f.readlines()

    # Drop header (first line)
    lines = lines[1:]

    out = np.empty(len(lines), dtype=DAILY_DTYPE)

    price_mult = 100.0  # for ES/MES; TODO: detect per-symbol later

    for i, line in enumerate(lines):
        parts = line.strip().split(”,”)

        # Parse YYYY/MM/DD → YYYYMMDD integer
        date_str = parts[0].strip()
        year, month, day = date_str.split(”/”)
        date_int = int(year + month + day)

        out[i][”Date”] = date_int
        out[i][”Open”] = float(parts[1]) / price_mult
        out[i][”High”] = float(parts[2]) / price_mult
        out[i][”Low”] = float(parts[3]) / price_mult
        out[i][”Close”] = float(parts[4]) / price_mult
        out[i][”Volume”] = int(float(parts[5]))
        out[i][”OpenInterest”] = int(float(parts[6]))

        # Some Sierra files may not have bid/ask columns → guard it
        if len(parts) > 7:
            out[i][”BidVolume”] = int(float(parts[7]))
            out[i][”AskVolume”] = int(float(parts[8]))
        else:
            out[i][”BidVolume”] = 0
            out[i][”AskVolume”] = 0

    return out


def load_daily_contracts(data_dir: str, contracts: list[str]) -> dict[str, np.ndarray]:
    “”“
    Load full daily data for multiple contracts.

    Args:
        data_dir (str): Directory with daily files.
        contracts (list[str]): List of contract codes (e.g. [’MESU24’, ‘MESZ24’]).

    Returns:
        dict[str, np.ndarray]: Mapping {contract -> daily data array}
    “”“
    results = {}

    for sym in contracts:
        path = os.path.join(data_dir, f”{sym}_FUT_CME.dly”)

        if not os.path.exists(path):
            print(f”Warning: no daily file for {sym}”)
            continue

        data = load_daily_data(path)

        results[sym] = data

    return results

Load intraday data

Similar to the previous functions, this set gets us our intraday data. Sierra Chart stores these as flat binary files and they don’t have a different data type for ticks versus OHLCV data. So, you have to read the docs to figure a few things out. Well, I had to read the docs. You don’t have to, you can just read my code instead. Anyway, this takes their .scid files and normalizes it into a tick array we defined.

def load_scid_ticks(path):
    “”“
    Load Sierra Chart .scid file into a normalized tick array.

    Args:
        path (str): path to .scid file

    Returns:
        np.ndarray with dtype=TICK_DTYPE
    “”“
    offset = HEADER_DTYPE.itemsize
    filesize = os.path.getsize(path)
    n_records = (filesize - offset) // RECORD_DTYPE.itemsize
    if n_records <= 0:
        return np.empty(0, dtype=TICK_DTYPE)

    # Memory-map the SCID records into a NumPy array.
    #
    # This uses np.memmap to efficiently treat the binary file as an array
    # without reading the entire file into RAM. It’s the Python equivalent of
    # using mmap() or fread() with a struct in C/Zig.
    mm = np.memmap(
        path, dtype=RECORD_DTYPE, mode=”r”, offset=offset, shape=(n_records,)
    )

    price_mult = 100.0  # ES/MES uses 100 multiplier TODO: autodetect this in the future

    # Convert to normalized tick array
    out = np.empty(n_records, dtype=TICK_DTYPE)
    out[”TimeStamp”] = mm[”DateTime”].astype(”<i8”)
    out[”Price”] = mm[”Close”].astype(np.float64) / price_mult
    out[”Volume”] = mm[”TotalVolume”].astype(np.int64)
    out[”Bid”] = mm[”Low”].astype(np.float64) / price_mult
    out[”Ask”] = mm[”High”].astype(np.float64) / price_mult

    return out


def load_scid_contracts(data_dir: str, contracts: list[str]):
    “”“
    Load multiple .scid contracts into tick arrays.

    Args:
        data_dir (str): Directory containing SCID files.
        contracts (list[str]): Contract symbols (e.g. [’MESM24’, ‘MESU24’]).

    Returns:
        dict[str, np.ndarray]: Mapping contract -> tick array.
    “”“
    results = {}

    for sym in contracts:
        # Build file path (Sierra uses suffix like ‘_FUT_CME.scid’)
        path = os.path.join(data_dir, f”{sym}_FUT_CME.scid”)

        if not os.path.exists(path):
            print(f”Warning: {path} not found, skipping”)
            continue
        ticks = load_scid_ticks(path)
        if len(ticks) > 0:
            results[sym] = ticks

    return results

Resample ticks

This function resamples ticks into minute bars. It doesn’t really care about anything other than the timestamps and aggregating them into OHLCV bars for us. It doesn’t care about timezones or anything like that, so it isn’t effected by the time issues I mentioned earlier in our helpers section.

def resample_to_bars(ticks: np.ndarray, minutes: int = 1) -> np.ndarray:
    “”“
    Convert raw tick data into N-minute OHLCV bars.

    Args:
        ticks (np.ndarray): Tick array with fields (TimeStamp, Price, Volume, ...)
        minutes (int): bar length in minutes (default = 1).
            Examples:
                1 -> 1-minute bars
                5 -> 5-minute bars
                240 -> 4-hour bars

    Returns:
        np.ndarray: OHLCV bars with dtype=OHCLV_DTYPE
    “”“
    if ticks.size == 0:
        return np.empty(0, dtype=OHLCV_DTYPE)

    # Floor tick timestampes to start of their minute
    micros_per_bar = minutes * 60 * 1_000_000
    bar_bins = (ticks[”TimeStamp”] // micros_per_bar) * micros_per_bar

    bars = []
    current_bar = bar_bins[0]

    # Init OHLCV from first tick
    o = h = l = c = ticks[”Price”][0]
    v = ticks[”Volume”][0]

    for i in range(1, ticks.size):
        ts = ticks[”TimeStamp”][i]
        price = ticks[”Price”][i]
        vol = ticks[”Volume”][i]
        bar = bar_bins[i]

        if bar == current_bar:
            # Still inside this bar
            if price > h:
                h = price
            if price < l:
                l = price
            c = price
            v += vol
        else:
            # close previous bar
            bars.append((current_bar, o, h, l, c, v))
            # start new bar
            current_bar = bar
            o = h = l = c = price
            v = vol

    # Store last bar
    bars.append((current_bar, o, h, l, c, v))

    return np.array(bars, dtype=OHLCV_DTYPE)

Volume-based rollover

This is where shit got fun. Sierra Chart uses daily data files to determine rollovers. It makes sense. It is a much smaller file and relatively trivial to use. I will do something similar in quantKit. This particular rollover calculation just compares daily volume in contracts and returns a list of rollover dates from contract to contract. It rolls over when volume of the oncoming contract exceeds the daily volume of the current contract. Simple.

def volume_based_rollover(
    contracts: list[str], daily_data: dict[str, np.ndarray]
) -> list[dict]:
    “”“
    Determine rollover dates using Sierra-style volume-based rollover.

    Args:
        contracts (list[str]): Ordered list of contract codes (e.g. [’MESH24’, ‘MESM24’, ...]).
        daily_data (dict): Mapping {contract -> daily bars}, dtype must have
                           fields: (”Date”, “Open”, “High”,
                                    “Low”, “Close”, “Volume”)

    Returns:
        list[dict]: rollover schedule, e.g.
            [{”date”: int, “from”: “MESH24”, “to”: “MESM24”}, ...]
    “”“
    roll = []

    for i in range(len(contracts) - 1):
        front = contracts[i]
        back = contracts[i + 1]

        df = daily_data.get(front)
        dn = daily_data.get(back)
        if df is None or dn is None:
            continue

        # Get expiry year, month from the contract code
        ym = parse_contract_code(front)
        y = ym // 100
        m = ym % 100

        # Restrict to that month/year only
        win_dates = []
        for d in np.intersect1d(df[”Date”], dn[”Date”]):
            s = str(int(d))
            dy, dm = int(s[:4]), int(s[4:6])
            if dy == y and dm == m:
                win_dates.append(int(d))

        rollover_date = None
        for d in sorted(win_dates):
            vol_front = df[df[”Date”] == d][”Volume”][0]
            vol_back = dn[dn[”Date”] == d][”Volume”][0]
            if vol_back > vol_front:
                rollover_date = d
                break

        if rollover_date:
            roll.append(
                {
                    “from”: front,
                    “to”: back,
                    “date”: rollover_date,
                }
            )

    return roll

Back-adjust contracts

The back_adjust_contracts function uses the rollover dates from the previous function (or any other similar rollover function) and calculates back adjustments and returns the adjusted contracts. The daily file already uses the settlement price as its close price. Meaning, it is not the last tick of the day. There is a difference, and this will be something to address in quantKit when I get to it. Not all daily data is built the same.

def back_adjust_contracts(daily_contracts: dict[str, np.ndarray],
                          intraday_contracts: dict[str, np.ndarray] | None,
                          rolls: list[dict]) -> tuple[dict[str, np.ndarray], dict[str, np.ndarray] | None]:
    “”“
    Perform back-adjustment using daily settlements.

    Args:
        daily_contracts (dict): {symbol -> OHLCV array with “Date”}
        intraday_contracts (dict or None): {symbol -> OHLCV array with “TimeStamp”} (optional)
        rolls (list): [{”from”: “H24”, “to”: “M24”, “date”: int}, ...]

    Returns:
        (dict, dict|None): adjusted daily contracts, adjusted intraday contracts
    “”“
    if intraday_contracts is not None:
        adjusted = {sym: arr.copy() for sym, arr in intraday_contracts.items()}
    else:
        adjusted = {sym: arr.copy() for sym, arr in daily_contracts.items()}

    expired = []

    for roll in rolls:  # rolls must be sorted by date
        from_sym = roll[”from”]
        to_sym   = roll[”to”]
        roll_date = roll[”date”]

        # settlement always comes from daily
        old_contract = daily_contracts[from_sym]
        new_contract = daily_contracts[to_sym]

        old_settle = get_settlement(old_contract, roll_date)
        new_settle = get_settlement(new_contract, roll_date)

        diff = new_settle - old_settle

        # All expired contracts get shifted by this new diff
        for sym in expired + [from_sym]:
            if sym in adjusted:
                shift_contract(adjusted[sym], diff)

        expired.append(from_sym)

    return adjusted
 

Stitch contracts

Almost done, y’all.

This function takes all of our contracts and stitches them together at the rollover dates. It doesn’t care whether or not the contracts have been adjusted.

def stitch_contracts(
    contracts: list[str], data: dict[str, np.ndarray], rolls: list[dict]
) -> np.ndarray:
    “”“
    Stitch together multiple contracts into a continuous series using rollover dates.

    Args:
        contracts (list[str]): Ordered list of contract codes.
        data (dict): {contract -> bar array}, bar dtype must include “TimeStamp”.
        rolls (list[dict]): Rollover events [{”from”:, “to”:, “date”:}, ...]

    Returns:
        np.ndarray: Stitched bars, no back adjustment.
    “”“
    stitched = []

    # Build quick lookup for roll dates
    roll_dates = {(r[”from”], r[”to”]): r[”date”] for r in rolls}

    # Detect date/timestamp field
    use_date = “Date” in data[contracts[0]].dtype.names

    for i in range(len(contracts)):
        sym = contracts[i]
        bars = data.get(sym)
        if bars is None or bars.size == 0:
            continue

        # Determine slice window for this contract
        start_date = None
        end_date = None

        # If this contract has an incoming rollover
        if i > 0:
            prev = contracts[i - 1]
            if (prev, sym) in roll_dates:
                start_date = roll_dates[(prev, sym)]

        # If this contract has an outgoing rollover
        if i < len(contracts) - 1:
            nxt = contracts[i + 1]
            if (sym, nxt) in roll_dates:
                end_date = roll_dates[(sym, nxt)]

        # Apply filters
        out = []
        for j in range(bars.shape[0]):
            if use_date:
                # daily: already YYYYMMDD int
                dts = bars[”Date”][j]

                if start_date is not None and dts < start_date:
                    continue
                if end_date is not None and dts > end_date:
                    continue
            else:
                # intraday: convert TimeStamp micros -> date
                ts = bars[”TimeStamp”][j]

                if start_date is not None:
                    start_date_ts = yyyymmdd_to_micros_roll_boundary(start_date)
                else:
                    start_date_ts = None
                
                if end_date is not None:
                    end_date_ts = yyyymmdd_to_micros_roll_boundary(end_date)
                else:
                    end_date_ts = None

                if start_date_ts is not None and ts < start_date_ts:
                    continue
                if end_date_ts is not None and ts > end_date_ts:
                    continue


            out.append(bars[j])

        if out:
            stitched.extend(out)

    if len(stitched) == 0:
        return np.empty(0, dtype=data[contracts[0]].dtype)

    return np.array(stitched, dtype=data[contracts[0]].dtype)

Build contracts wrapper

We made it!

The final piece. This is the wrapper that brings it all together. It returns our finished chart.

def build_continuous_chart(
    root: str,
    data_dir: str,
    start: str,
    end: str,
    chart_type: str = “intraday”,   # “daily” or “intraday”
    interval: int = 1,              # minutes for intraday; ignored if daily
    rollover: str = “volume”,       # only “volume” supported right now
    back_adjust: bool = True
) -> np.ndarray:

    contracts = contracts_in_range(data_dir, root, start, end)

    daily_data = load_daily_contracts(data_dir, contracts)

    if rollover == “volume”:
        rolls = volume_based_rollover(contracts, daily_data)
    else:
        rolls = []

    if chart_type == “daily”:
        bars_by_contract = daily_data
    elif chart_type == “intraday”:
        scid_ticks = load_scid_contracts(data_dir, contracts)
        bars_by_contract = {}
        for sym, ticks in scid_ticks.items():
            bars_by_contract[sym] = resample_to_bars(ticks, minutes=interval)
    else:
        raise ValueError(f”Unknown chart_type: {chart_type}”)
    
    if back_adjust:
        bars_by_contract = back_adjust_contracts(daily_data, bars_by_contract if chart_type==”intraday” else None, rolls)

    stitched = stitch_contracts(contracts, bars_by_contract, rolls)

    final = []
    for i in range(stitched.shape[0]):
        if “Date” in stitched.dtype.names:
            dts = stitched[”Date”][i]
            start_int = int(start.replace(”-”, “”))
            end_int = int(end.replace(”-”, “”))
            if dts < start_int or dts > end_int:
                continue
        else:
            ts = stitched[”TimeStamp”][i]
            start_micros = yyyymmdd_to_micros_roll_boundary(int(start.replace(”-”, “”)))
            end_micros = yyyymmdd_to_micros_roll_boundary(int(end.replace(”-”, “”)))

            if ts < start_micros or ts > end_micros:
                continue
            
        final.append(stitched[i])


    if len(final) == 0:
        return np.empty(0, dtype=stitched.dtype)

    return np.array(final, dtype=stitched.dtype)

Make a damn CSV

The final touch. Let’s make a human-readable CSV.

chart = build_continuous_chart(
    root=”MES”,
    data_dir=PATH_DIR,
    start=”2025-06-01”,
    end=END_DATE,
    chart_type=”intraday”,
    interval=1,
    rollover=”volume”,
    back_adjust=True
)

# ----------------------
# Save to CSV
# ----------------------
if chart.size > 0:
    output_path = os.path.join(”data”, “SC_MES_Minute.csv”)

    # make sure directory exists
    os.makedirs(os.path.dirname(output_path), exist_ok=True)

    if “Date” in chart.dtype.names:
        x_vals = [yyyymmdd_to_dt64(int(d)).item() for d in chart[”Date”]]
        cols = [name for name in chart.dtype.names if name != “Date”]
    else:
        x_vals = [utc_to_local(ts) for ts in chart[”TimeStamp”]]
        cols = [name for name in chart.dtype.names if name != “TimeStamp”]

    export_data = np.column_stack([x_vals] + [chart[name] for name in cols])
    header = “DateTime,” + “,”.join(cols)
    fmt = [”%s”] + [”%.6f”] * (export_data.shape[1] - 1)

    np.savetxt(
        output_path,
        export_data,
        delimiter=”,”,
        fmt=fmt,
        header=header,
        comments=”“
    )

    print(f”✅ Saved: {output_path} ({chart.size} bars)”)
else:
    print(”⚠️ No data to export.”)

And the outcome:

Closing

As you can see, there is a lot of scaffolding here for working with local data files in the future. I like the way Sierra Chart handles local data files without using a database, and I will probably implement something similar for quantKit when it’s time. There is a lot to build from here, and I look forward to adding it into quantKit soon. For now, it’s just a solid tutorial to help you see how you can find, manipulate, and export data for your needs without having to get funky with APIs.

What’s next?

The next post will be a paid post. I am going to create a testing suite for strategies in a spreadsheet. The goal is to have 2-3 strategy ideas in this suite to showcase how it works. I will also build a daily oriented template that will generate orders for daily strategies. This shouldn’t be too difficult to setup, but as we saw from this one, sometimes I misjudge how hard something can be.

Until next time.

Happy hunting.

This post doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

Buckle up.

UPDATE:

This project ended up being a failure. Not because I couldn’t figure it out, but because you can’t get CME data through the Sierra Chart DTC server. This is because of data usage agreements. I was going to just scrap this post, but I decided to publish it as is and leave the code I was working through as an example for anyone interested in seeing how to encode/decode and parse binary data connections. This will be useful in the future if I decide that I want to send trades through DTC, but I don’t know that I am there just yet.

Anyway, feel free to read through what is here. It is still accurate information, I just didn’t post any code because it didn’t work.

Live and learn.

GitHub Repo

Why am I using Sierra Chart, DTC, and Denali?

There are several reasons that we are using Sierra Chart (SC) for this tutorial. To put it simply, it’s a legit platform, but I will break it down a bit more below.

Sierra Chart

First and foremost, this platform is programmed in C++ and built for speed and efficiency. The documentation is dense, and the support forum is sometimes difficult to navigate. The general setup is not intuitive if you are used to other platforms, and it can be generally difficult to navigate and figure out when you first start using it. However, after you climb this steeper hill, the fruits at the top are well worth it.

It is fast. Wicked fast. It doesn’t eat your RAM and it stays out of the way. Honestly, the only other program I have seen that runs this efficiently is RealTest. Everything else is slow in comparison.

It is difficult to figure out. But, so are many other things. Most hard things are worth the work. Bet you didn’t know you were gonna get a life lesson in this post too, did you?

Teton

Teton is Sierra Chart’s futures order routing service. Here is what SC says about it:

The Teton Futures Order Routing service is a high-quality order routing service with advanced risk management from Sierra Chart to provide order routing, for outright futures and spreads, direct to the major exchanges. There is no other intermediary provider.

Remember when I talked about “intermediaries” when discussing data? Well, order routing can have them too. Except, Teton doesn’t. That makes it fast. How fast?

Orders are routed direct to the exchange with high reliability and very low latency, in under 500 microseconds.

Wow. How fast is a microsecond? One millionth of a second. That’s 0.0005 seconds. Fuck me, that’s fast.

Their servers for Teton are colocated with the CME order matching computers (we will talk about colocation more in future posts). That’s cool. It is offered to clearing firms and users at no cost. There are no transaction fees per contract traded. As far as SC is aware, this is an industry first.

That is pure quant dirty talk right there, y’all.

Denali

Ok, so their futures order routing is legit good. Surely, their data must suck.

Nope.

Let’s see what they have to say about it:

• Streaming real-time data.

• Market depth data.

• Historical intraday day data. CME data has tick by tick data back to 2011 and minute data back to 2008.

• 1 year of tick data for US stocks and minute data back to 2008 (assuming the stock existed then).

• Historical daily data.

• 100% accurate historical Bid Trade Volume and Ask Trade Volume.

• High precision data delivered with millisecond time stamping.

• Low latency.

• Direct CME FIX data feed at the SC backend.

Wow. That’s wicked cool.

It must cost a million dollars a month.

Also, nope.

It’s one of the cheapest data streams I have found.

DTC

We discussed this briefly in the last post. This is what will allow us to get our data from Sierra Chart without having to export it to CSV from inside the platform. Of course, if you want to just export the data, have at it. That’s what makes this platform a dual threat. Don’t want to code? Cool. Want to code? Also, cool.

Remember, it turns out this is untrue. You can’t get CME data via DTC. I knew that the documents mentioned this, but it also mentions being able to request historical data, so I figured I would try it anyway. Turns out, it won’t send you data. Lame.

Turn Sierra Chart into a DTC Server

So, you decided you want to give this a go. Maybe you already used Sierra Chart, maybe you got it just to test this idea. Or, maybe you are just reading bc you think this is a cool project and you like my style.

This part of the tutorial is pretty simple. Go to the SC document page that shows you how to turn on the DTC server and follow the instructions under the “Server Usage” section.

For this tutorial, we are going to set the encoding to binary. This is because I want to test how to work with it. The whole purpose of this is to ease into building our own research engine and learning how to play with this DTC protocol is part of it. We will keep everything else at the defaults.

Additionally, make sure that your intraday data is set to store as 1 tick. I believe this is default, but I am unsure.

Setting up the project

Time to work.

First, we need to setup our project environment and activate the virtual environment just like we did in the previous article.

mkdir tutorial3
cd tutorial3
mkdir data
New-Item requirements.txt 
New-Item tutorial3.py
python -m venv .venv./.venv/Scipts/Active

Next, the required dependencies for this script. I am going to avoid using Pandas this time. I am doing this so I can get a little more hands on with the data using Numpy. I am not going to use any Pandas DataFrames in quantKit, so I want to make sure I can handle the data without it. Besides, this is good practice for you too. You don’t want to rely on a bunch of dependencies if you can help it.

numpy

That’s it. Lean and mean. Install it with pip install -r requrements.txt.

The Code

The purpose of this tutorial is two-fold. I am personally testing this out as I create this walk-through, and I am showing you how to test this kind of shit so you can get used to playing around with Python. So, instead of just pasting it all into one block, I am going to break it into sections. This just helps describe what each section is doing better without trying to rely on large comment blocks.

I suggest actually typing it in as you go through. Get reps. Or, you can paste it. And if you are real lazy, just clone the repo.

Wait…

Where is the code?

Well, it turns out that this project was a wash.

So, check out the closing thoughts to see what happened.

You can find the code that I was using at the GitHub link at the top.

Closing Thoughts

Ok, obviously this didn’t work out the way I was hoping. Sierra Chart has enabled the historical data request messaging, but because of the data usage agreements with CME, they can not allow any CME data to be sent over the DTC server.

I didn’t check to see if this applies to equities data or not. It would seem odd to enable the messages but not actually be able to send any data over the server. It is possible that it works for data that isn’t Denali, but that would defeat the purpose of this project.

Everything else I discussed in this article is still correct. The Denali data feed and the Teton routing are pretty damn good. I would imagine that it’s one of the better setups for futures traders that doesn’t require you to have buckets of money to get yourself competitive. Does it let you compete with firms? Nope. That will never happen, but it does let you compete with other retail traders and maybe even some small retail shops.

The name of the game is reliable fast data and routing.

What’s next?

Since we can’t use DTC to do this, we are going a different route. I am just going to parse the binary data files that SC uses to store historical data. This way, we still get the same tick-level data that we can then use to aggregate into minute bars and turn into a continuous chart.

Hopefully it will be published soon. Until then…

Happy hunting.

This post doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

In this post I want to zoom in on the practical side: how does this data actually get to your terminal, platform, or research environment? The answer lies in the delivery models. Whether you are downloading static data files (CSV), making requests to REST APIs, or connecting to a streaming protocol, the delivery method shapes what you can do with the data and what types of strategies you can deploy.

File Exports

This is the OG method for getting historical data for testing. It is still widely used, and as we know from previous posts, one I am not a big fan of. Brokers, platforms, and data vendors will sometimes provide data in a flat file format (CSV being the most popular). This is a simple, code free way to get data to test. It is simple, portable, human-readable and works with spreadsheet applications (Excel) and coding languages (Python –> pandas). It is not real-time and often limited in granularity.

There are plenty of data vendors that provide CSV downloads or exports. Personally, I used Norgate data for daily data during my RealTest experiments. The data was reliable and great for daily strategies, especially for equities. But, it required using their software to access the data either via CSV export or connection to your testing/trading platform. Not the worst setup, but not the best either.

A quick google search and you can find plenty of providers that allow you to download aggregated OHLCV data in this type of format. This is the best way to get data if you are trying to no-code test an idea. In a soon-to-be-featured article, I will show how to use simple CSV data to test a strategy in a spreadsheet platform. Spoiler, I don’t fuck with Excel or Google Sheets, but you can do whatever you want.

Web APIs

Now, we are getting closer to the kind of data I like. I admit, I am a nerd. Being a nerd requires that newer technology gets you off a bit, but I have to admit, web APIs aren’t new. The finance world just hasn’t completely caught up with the times. This is because an API is really just a set of code that allows remote, sometimes authorized, access to some form of data. The real nerdy part is how the data is stored, but I am getting ahead of myself.

REST APIs

REST stands for representational state transfer. Yeah, it doesn’t really mean dick if you aren’t a web developer. It’s just an architecture that defines how you design an API. Oh yeah, API stands for “application programming interface”. In case you didn’t know. Probably should have mentioned that more often. If you aren’t a developer, it doesn’t matter. If you are a developer, you already know. I hope.

In the Last Post, I gave a quick example of how to get data from the Alpaca API. It’s pretty simple, especially if you are just backfilling data for a research pipeline. There are other protocols (maybe?), but REST is the standard on the inter-webs.

Web Sockets

Web sockets are responsible for our live data streams. It is a persistent connection that pushes updates to you in real-time. Alpaca (and many others) provides a WebSocket API. You can request bars, trades, or quotes for a given symbol and timeframe. It is also dependent on the type of data you ask for. Remember, many services apply their own magic to the data before it gets to you, so you have to know how they handle their data. It matters.

This is how you get real-time data to feed live order books or trades to an algo or dashboard. It’s pretty flexible, easy to integrate, and supported by most modern platforms. However, it isn’t standardized. Each platform has its own endpoints, formats, and quirks.

Protocols

This is where it gets… fun. For professional software, APIs aren’t always enough. That’s where protocols come in. They are standardized (or semi-standardized) messaging systems designed specifically for trading and market data. Cool, right?

FIX

FIX stands for “financial information exchange”. I am learning from my hastiness, see? This is the granddaddy of trading protocols. It is used worldwide for order routing and sometimes for market data. It is essentially the industry standard and widely supported. Unfortunately, it sucks. Just kidding. Kind of. It leaves a lot to be desired. It is verbose, complex, and often overkill if all you want is some historical data.

It really is the industry standard, though. It dates back to the 90s (before some of your birthdays, I bet) and is still the backbone of institutional routing. This makes it hard to ignore.

DTC

DTC stands for Data and Trading Communications. This is Sierra Chart’s love letter to modern traders. It is an open protocol that attempts to fix (heh) some of the issues with FIX. Unfortunately, I don’t know of any other platform/data service that uses it besides Sierra Chart and their Denali Data feed.

Does that matter?

Not really. It’s open and other people can use it, and I am sure users have. But, it requires some developer knowledge. Like FIX, it is a standardized messaging protocol. It has options for binary or compact JSON and is optimized to be fast and friendly. It’s pretty damn niche though. I kind of like it.

To be clear, this protocol is open and free to adapt. Free as in spe… er, beer. Beer is the safer option here these days, Larry. Adoption is limited, but it will be one I am exploring for my needs.

Proprietary Protocols

Unlike Sierra Chart, some vendors decided to fuck FIX and open protocols. Instead, they created their own shit that is tightly integrated within their own ecosystems. Very Apple of them. Instead of listing them, I just wanted to make you aware that there was the possibility of proprietary options out there. Since I am too cool the fuck with them (read broke), there isn’t much else to say here.

Putting it All Together

As a SQR (serious quant researcher, thanks SLA), you will often mix these models. You can use CSV for backtests, make requests to REST APIs, and keep a WebSocket or protocol running to stream live ticks. Understanding the difference is key to designing a reliable research and trading pipeline.

What’s next?

Now that I have covered the basics and showed you a couple of ways to get data, and convert it to CSV, I am going to get a little more detailed. I am going to show you how to use Sierra Chart (hint, this is the platform I am playing with for trading) to get data via DTC and aggregate tick data into OHCLV bar data for testing. After that, I am going to build out a simple testing suite in a spreadsheet to show you how you can go “code free” to test out simple ideas, get stats, and maybe even do some simple statistical robustness tests on the data.

Happy hunting, folks!

This post doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

All of the code for these tutorials can be found on my Github.

Install Python and setup your environment

Before you do anything, you have to make sure that you have Python installed. The examples I am going to give are for Windows, but if you are on a different operating system, a quick search online should help you figure out what to do for your particular system.

# Install (or upgrade) via winget
winget install --id Python.Python.3.13 -e

# or upgrade if already installed
winget upgrade --id Python.Python.3.13 -e

# Verify
python --version

After this, you want to create a directory for your Python projects or scripts. Then, you want to create a virtual environment for that project. This allows us to compartmentalize the dependencies that we need for each project and not clutter up our main python install on our machine. Each tutorial in this guide will have a different environment with different dependencies.

mkdir python_tutorials\tutorial1
cd python_tutorials\tutorial1
python -m venv .venv
.\.venv\Scripts\Activate

After we get the environment activated, we need to create a requirements.txt file in our directory. This is where we will list each of the dependencies that we need for our project. In Powershell, that looks like:

New-Item requirements.txt

Then, using your favorite text editor (I use Neovim, btw), open that file and add the following two lines:

pandas
yfinance

Then, we run the command to install them:

python -m pip install -r requirements.txt

Once it installs, we are ready to start playing with data.

Using yfinance to get free data

The first thing we are going to cover is the most popular and easy way to get free, mostly unreliable (but good for learning), data. Before we get started, make sure you create a data directory inside this project to save our CSVs to when we are done. Then, create a new file tutorial1.py and then open that file with our text editor.

There, we are going to add the following code:

import yfinance as yf
import pandas as pd
from pathlib import Path

# Parameters
ticker = "SPY"
period = "15mo"  # last 15 months
interval = "1d"

# Fetch data with auto-adjust so prices reflect dividends/splits
data = yf.download(ticker, period=period, interval=interval, auto_adjust=True)

# Show raw data straight from Yahoo
print("\nRaw data (first 5 rows):")
print(data.head())

# Handle multi-index columns if present
if isinstance(data.columns, pd.MultiIndex):
    data.columns = data.columns.get_level_values(0)  # keep only column names, drop ticker level

# Clean data with pandas
out = (
        data.rename(
            columns={
                "Open": "open",
                "High": "high",
                "Low": "low",
                "Close": "close",
                "Volume": "volume",
            }
        )
        .reset_index()
        .rename(columns={"Date": "date"})
        .loc[:, ["date", "open", "high", "low", "close", "volume"]]
    )

# Save to file
Path("data").mkdir(exist_ok=True)

all_path = Path("data") / f"{ticker}_15mo.csv"
ins_path = Path("data") / f"{ticker}_in_sample.csv"
oos_path = Path("data") / f"{ticker}_out_sample.csv"

out.to_csv(all_path, index=False)

# Split ~12mo IS / ~3mo OOS using pandas slicing
insample = out.iloc[:-63]
outsample = out.iloc[-63:]
insample.to_csv(ins_path, index=False)
outsample.to_csv(oos_path, index=False)

print(f"\nSaved {len(out):,} total rows")
print(f"In-sample rows: {len(insample)} → {ins_path}")
print(f"Out-of-sample rows: {len(outsample)} → {oos_path}")

# Verify by reading back with pandas
print("\nPreview of saved CSVs:")
print(f"\nAll data:\n{pd.read_csv(all_path).head()}")
print(f"\nIn-sample:\n{pd.read_csv(ins_path).head()}")
print(f"\nOut-of-sample:\n{pd.read_csv(oos_path).head()}")

Then, run this code with:

python tutorial1.py

And behold the data.

A good bit of the code in the above code block is print statements that just confirm that we got data and that it was formatted correctly. Aside from that code, what we are doing is:

• Defining the data we want to get (SPY), timeframe (daily), and how much (15months).

• Downloading that data and formatting it. This isn't always necessary, but the example shows you how to change column names and drop columns that we don’t need.

• Lastly, we save it to CSV in our data file path

Notice that we separated our data out into three different files. One of them holds all the data, the next two are our in-sample (IS) and out-of-sample (OS) data. It is always a good practice to withhold some data from your testing. This helps us insure that we are overfitting our strategy to all of the available data. When we get to testing our data, we will create, test, and tweak our strategy with the IS data, then test it against the OS data to see how it performs.

Getting data from Alpaca (or any similar web API)

Moving along to our next example. Instead of using yfinance, this time we are going to use Alpaca. I am using Alpaca here because it is a developer first broker/provider. It has solid APIs and documentation. It requires having an account with them (which you can do for free) and that you use an API key to make calls to their API.

Make sure you deactivate your previous venv session if you are going through this in order.

deactivate

Create a new directory (tutorial2), add you data folder, create a new venv for this project, and the add the requirements file:

pandas
alpaca-py
python-dotenv

Then, just like before, we need to activate our venv and install the dependencies.

We will need to create a .env file to add our API key too.

ALPACA_API_KEY=your_api_key_here
ALPACA_API_SECRET=your_api_secret_here

If you are planning on using git for version control, be sure to add the .env file to your .gitignore so you don’t advertise your keys to the internet.

In your Alpaca account, you are going to want to go to the profile settings and manage accounts. Here you can generate the API keys you need for the .env file above. Make sure you use the paper account for this tutorial. We won’t be making any calls to the account itself, but there is no need to take chances.

Once that is done, open up the new tutorial2.py file and add:

from datetime import datetime, timedelta
import os
from pathlib import Path

import pandas as pd
from dotenv import load_dotenv
from alpaca.data.historical import StockHistoricalDataClient
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame

# Load keys from .env
load_dotenv()
API_KEY = os.getenv("ALPACA_API_KEY")
API_SECRET = os.getenv("ALPACA_API_SECRET")

# Initialize the market data client (no need for base_url here)
data_client = StockHistoricalDataClient(API_KEY, API_SECRET)

# Date range
end = pd.Timestamp.today(tz="UTC").normalize()
start = end - pd.DateOffset(months=15)

# Define the request (last 5 days of SPY daily bars)
request_params = StockBarsRequest(
    symbol_or_symbols=["SPY"],
    timeframe=TimeFrame.Day,
    start=start.to_pydatetime(),
    end=end.to_pydatetime(),
    adjustment='all',
)

# Fetch the bars
bars = data_client.get_stock_bars(request_params)

# JSON --> DataFrame
df = bars.df.reset_index().rename(columns={"timestamp": "date"})
df["date"] = pd.to_datetime(df["date"]).dt.tz_localize(None)
df = df.loc[:, ["date", "open", "high", "low", "close", "volume"]]

# Print the raw response so you can inspect its format
bar_list = bars.data["SPY"]
print("\nFirst Bar:", bar_list[0])
print("Last Bar:", bar_list[-1])
print(df.head())

# Save into data/
Path("data").mkdir(exist_ok=True)
all_path = Path("data") / f"SPY_15mo.csv"
is_path = Path("data") / f"SPY_in_sample.csv"
os_path = Path("data") / f"SPY_out_sample.csv"

df.to_csv(all_path, index=False)

# Split Data
cutoff = df["date"].min() + pd.DateOffset(months=12)
df[df["date"] < cutoff].to_csv(is_path, index=False)
df[df["date"] >= cutoff].to_csv(os_path, index=False)

# Print to confirm csv data
print("\nPreview of saved CSVs:")
print("\nAll data:\n", pd.read_csv(all_path).head())
print("\nIn-sample:\n", pd.read_csv(is_path).head())
print("\nOut-of-sample:\n", pd.read_csv(os_path).head())

Similar to the method we used before, this retrieves data from an API and then formats the fields we want and saves them into a CSV file for us. The biggest change is that we get the data in JSON format and not as a DataFrame. We use Pandas to convert the data to a DataFrame and then save it to CSV. This is pretty standard for web APIs, but there is no guarantee that the data will be in the same format. It is always good to check the data format in the documentation or just print it out like we did here to see what it looks like before saving or changing anything.

Closing Notes

The purpose of this post was to just show you some easy ways to use Python to get data. Both methods use freely available data, with the Alpaca data being more reliable than the yfinance data. Free data is still free, and you get what you pay for.

Next time

Part two will cover how to get data from your trading platform. For my example, I will be using Sierra Chart. Most platforms offer at least one to export data, but Sierra Chart actually has two methods for getting data. One, a simple export to CSV feature (the easiest if your goal is to just get data), and two, a server option that allows you to retrieve data via their own protocol (DTC) that makes it possible to use your programming language of choice to retrieve data, stream data, and even make trades.

Before part two, I will also cover the different protocols and formats of data that researchers will likely run into on their quant journeys.

Until next time, happy hunting!

This post doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

Let me answer myself...

Data matters. I have mentioned this many times before, but it wasn't until recently, while avoiding work and contemplating all the different ways to try and build quantKit, that I realized, I can't test anything until I have data. Profound, right?

Well, then I started thinking about how I wanted to handle this data. Of course, I can do what everyone else does: just use CSV files and call it a day. But that shit gets messy quick. And it's slow. Sure, you can work some magic—compress files, stream the data in chunks, ensure you aren't storing anything to memory that doesn't need to be stored—but it's still slow. Especially when you want Python to handle this.

Also, it isn't exactly easy for the user, is it?

I mean, no one wants to have to constantly manually import CSV files of data. Imagine having to do this with hundreds of equities in an index. It gets old fast, and if I am going to take the time to abstract that away for the user, I might as well go ahead and manage the data more appropriately.

This got me thinking. I started reading about market data and asking AI to take up its role as my research assistant once again. After a few weeks of trying to consume as much information as I could, I came to a conclusion.

Data matters more than you think.

And because it matters more than you think, it should get a decent amount of consideration when you are attempting to build a robust research engine for traders to use.

So, I am going to go over some basics in this post. Just some food for thought. This will be a part of a mini-series that will explain market data, why it matters, and how I intend to handle it for quantKit.

Spoiler: It ain't with fucking CSV files.

Types of Market Data

There are really only two main types of market data as far as an exchange is concerned: trade data and market depth.

Trade data is "tape". It is the source of truth: every executed transaction on the exchange, including price, size, and timestamp.

Market depth is the real-time record of resting buy/sell orders at multiple price levels, updated as orders are added, canceled, or executed.

From there, data gets aggregated into other formats such as snapshots and OHLCV data. The latter, which we are all familiar with, builds most of our charts and indicators.

Technically, exchanges also publish reference data. This is essentially metadata that the exchange broadcasts about symbols. It's not true market data, at least not as far as a quant is concerned. You can't math this data.

How the data gets from the exchange to you

This is where it gets fun.

The data has to get to your computer from the exchange somehow, and many of us don't have direct access to exchange data, so the data takes a path through several layers:

Vendors (Rithmic, CQG, DTN IQFeed, etc) aggregate and redistribute exchange data.

Brokers also received market data and redistribute it to you, usually throttled or normalized (IBKR's 250ms snapshots are a good example of this).

Platforms (Sierra Chart, NinjaTrader, TradingView) sit on top of these feeds and present data visually, while also adding their own calculations and aggregations.

Example:

Exchange —> Vendor —> Broker —> Platform—> You

This isn't always the chain of events. Some brokers and vendors simply pass through exchange data without modification. For example, Alpaca's paid IEX stream is a direct redistribution of IEX exchange data. Sierra Chart's Denali feed works the same way.

The key point: every time the data changes hands, or hops, there's potential for increased latency and distortion, especially if the intermediary is throttling, batching, or applying its own adjustments.

Why this matters for research

As quants, we aren't just consuming charts. We are building systems and testing ideas. That means we need to understand exactly what kind of data we're working with, and how it was handled before it reached us.

Historical data is what we use to populate charts and calculate our features/indicators for testing ideas. Not all historical data is the same. It could be raw data that we aggregate ourselves or pre-aggregated OHLCV bars from a vendor or a broker.

Streaming data drives real-time execution and decisions. This is true tick-by-tick data or aggregated snapshots (like IBKR) that skip over many small events.

These distinctions matter because the fidelity of your backtests and research depends on the fidelity of the underlying data. Aggregated bar data is fine for forming ideas or running simple tests. Tick-level data gives you more realistic insights into slippage and execution. Order book data adds the missing dimension of liquidity, showing you how your order would have interacted with the market at the time of the trade.

If we want our research to reflect reality, we need to match the granularity of our data to the questions we are asking.

Quant's Dilemma

There is a common saying in trading that I've even repeated myself: "Test with the data you trade on." It sounds reasonable, but for quants and traders it's misleading.

When you place an order, it's executed against the raw exchange order book. Your broker's data feed, whether throttled, aggregated, or delayed, is not what determines your fill. Execution always happens on the exchange.

This means that your research data and execution broker don't have to match. In fact, separating them often gives you an edge. I remember when I was first learning about day trading and watched Ross Cameron trade. He used Lightspeed for trade execution and eSignal for his charting and market data. It was a simple but powerful example of why it doesn't really matter if you test and analyze on different data than the broker you execute through. What matters is pairing the best data for decision-making with the fastest broker for fills.

Data feeds and execution feeds are not the same, and they do not need to come from the same source. The goal is simple: use the highest-quality data you can for research and decision-making, and choose a broker that gives you the fastest, most reliable execution.

Closing thoughts

At the exchange level, all market data boils down to two streams: trade and market depth. Everything else is derived from them. Once it passes through vendors and brokers, it's often batched, throttled, or normalized before it ever reaches you. That's why understanding how your data is handled is critical for researchers.

Data quality drives research.

Execution depends on your broker.

Data feeds and execution feeds are separate.

For quants, this isn't a liability—it's an advantage. By doing a little work, we can get the best of both worlds.

Coming up next

The next post will explore the ways that data gets delivered. I will look at different protocols, see how the plumbing of these systems shapes the speed, granularity, and reliability of the feeds we depend on.

This post doesn't represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

The first month led into summer and I made about two hundred dollars total the next 3 months with plenty of asshole puckering drawdowns in between. I decided this wasn’t the style of trading for me and did what any amateur (I am still one of those, don’t get it twisted) would do. I started looking for different ways to trade.

I switched platforms and started trading slower, with my mouse. This worked for a time, and then it didn’t. I switched again.

This went on for about 9 months, constantly changing up what I was doing until I landed on a slow and steady way to trade ES/MES that I felt comfortable with. I also discovered algorithmic trading.

Damn, I thought. That sounds like the sweet spot. You mean I can use my wasting developer skills to research strategies and then program them to trade automatically for me? Yes, please.

So, while I traded slowly, manually, I started researching this algorithmic trading thing. I launched this Substack to document it along the way. My manual trading did okay, not bad enough to quit, not good enough to own a G-wagon. I dove in head first.

The first strategy that had great backtest results and drawdowns I could handle (I thought) went live about a week before a planned trip to Disney World with the family. That first week, it went great. It had an automatic stop at $2500, which backtests showed it rarely ever hit, typically only losing a few hundred dollars with bad trades. It took profits pretty fast, and would trade several times a day when criteria was met.

So, I felt comfortable setting it up to run while out at the parks, enjoying the theme parks.

The one thing the backtest never showed was that if the market gets volatile enough (not enough testing data), and criteria is still technically being met, there is nothing saying that it couldn’t take max losses multiple times a day. Like the rookie I was, I didn’t program in a hard stop. Of course, I came back into the room to find that it had failed twice (that’s 5k for those who can’t math) and was currently in a third losing trade.

Fuck me.

I pulled the plug on the trade, taking my licks, and shut down the strategy.

Back to the drawing board… back to manual trading for a while.

I think this was when I actually found the entrance to the rabbit hole. I started looking for anything and everything that was going to help me avoid that mistake again. I wanted a better trading platform, I wanted better testing techniques, I wanted in depth risk analysis with more data, I wanted my fucking money back!

I eventually dug myself out of that drawdown, but it was slow, and I was scared to take risks again. I even quit trading for a while, telling myself that I needed to know more, to learn more, so that I wouldn’t have any more black swan events like that one. Knowing what I know now, it is unfair to call it a black swan event. It was very possible for me to have known about that risk. I was just naive.

Anyway, I kept trading manually and I kept studying. HGT (that’s this Substack, keep up) started changing. I started focusing on mathematical concepts, and researching how to more accurately test strategies and research ideas. I looked into people like Jim Simons and picked up some obscure books, like those written by Timothy Masters.

There was no "aha!” moment. No mentor came to my rescue. There were times when I was in contact with other traders, but I eventually fell away from them all as I started learning more. I felt like I didn’t want any other influence while I was searching. It all felt hollow, like everyone had their own ideas and was confident they were correct. I wasn’t convinced.

During this time, I came to a conclusion: all day trading is gambling. Some people do it by feel; The cowboys, the discretionary traders hyped up on cocaine caffeine, staring at charts all day long, year after year, fingers on the pulse (in the form of ticker tape, or the new age equivalent). Others study the data, monitor the news, and trade in a way that resembles valuation estimations, buying when an asset is undervalued, selling when it returns to value or holding it until it makes them millionaires (like anyone who bought Apple in the 90s).

People make a living gambling, right? The same principles apply: you need a system, you need to know the odds, you need to be able to read the room, and you need to know when to exit, before that big fucker in a suit who started moving closer to your table gets within reaching distance.

I asked myself, what is it that the pros are doing? What are the firms and people with too much money doing? Turns out, they are doing math. Lots of it. Hard math too. The kind that makes men weep, preferring instead to enter the breach once more, anything to avoid having to learn about something that can’t be fixed with a hammer or a bullet, let alone figure out what an epsilon means, or those tiny little numbers and letters below that weird fucking ‘E’.

After being displeased with every platform I was using, I decided to build something on my own. Enter quantKit. My way to give back to the community (assuming it is worth a damn).

But, there is a catch. It’s a pretty big undertaking, especially for someone who is self-taught. I didn’t go to school for this, I have never had a mentor (unless you count The Primeagen on YouTube), and I don’t have any idea if what I am doing is going to be any better than what others before me have done. I can guarantee this though: it will be free, it will be well documented, and it will be better than something proprietary, which requires you to conform to their way of thinking.

This undertaking has actually required me to stop trading actively, or day trading rather. It requires my full attention, which, as always, means that I ignore it until I feel overwhelming pressure and disappointment before binge studying/coding until my brain is leaking out of my ears and I am tired before noon every day.

That is the path though. This shit isn’t a flat stroll through the woods. It’s fucking treacherous. I don’t have any idea if there is anything at the end of this path or not, but I am sure as shit going to find out.

Because here's what keeps me up at night: maybe this is a game that can't be won. Maybe those retail traders making a living are just riding good luck until it runs out. Maybe when I see quant firms starting Substacks, dispensing wisdom like digital prophets, they're just the house telling the marks how to play better — keep losing, but with more style.

Am I building another tool to help people lose money more elegantly? At least it'll be free (as in speech). Maybe that's something. Maybe it's not.

The path to quant trading might not have a destination. But I'm walking it anyway.

Happy hunting everyone! Until the next one.

This post doesn't represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

In case you were wondering where my usual AI art at the beginning of my articles went, after trying to defend the use of AI generated art on a Note thread, I convinced myself that I was indeed a piece of shit for using it. In place of it, you just get this explanation of why it isn’t there.

Disclaimer: the following post is an organized representation of my research and project notes. It doesn't represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

You know me - I hate vendor lock-in. I hate proprietary platforms that force you to think their way. And I really hate paying for tools that don't do exactly what I need them to do. So when I started working through "Stochastic Finance with Python" by Avishek Nag, it wasn't just academic curiosity. It was strategic.

I'm building quantKit because I want complete control over how I research and test trading strategies. But to build something truly useful, you need to understand the mathematical foundations that make quantitative finance actually work. That's where stochastic methods come in.

Of course, none of this excuses my radio silence recently. I have been a terrible content creator and a worse developer. I had so many things happen back to back that crushed my will and determination, planting me firmly in the "burn out" zone. Each time I thought everything was done, something else would happen. Then, when everything was done, I was too shot out to jump back into my work. My mind didn't want to focus, I couldn't force the work ethic, and I rapidly spiraled downward until I hit a ledge.

But it wasn't just exhaustion keeping me away. There's been this nagging feeling I can't shake - that most of the financial trading game is a setup. Like gambling, where the house is designed to win. Every platform wants their cut. Every data provider has restrictive agreements. Every backtesting engine forces you to think their way. And most retail traders? We're just playing at tables where we don't even know the real rules.

That's when the poker analogy really clicked for me...

The Fundamental Truth About Markets (And Poker)

Here's the thing everyone seems to forget: no one can predict the future. Full stop.

Think about Texas Hold'em. Any decent player can calculate basic probabilities like the odds of hitting your flush draw or the probability of pocket aces. But that's just the starting point. The real game is about updating those probabilities in real-time based on how your opponents bet, when they get aggressive, when they get conservative, how often they bluff.

You're not just playing the cards. You're playing probability distributions that constantly shift based on new information and opponent behavior patterns.

Markets work the same way. There is no deterministic behavior in markets with unknown futures. Yet most trading platforms, backtesting engines, and even machine learning approaches treat uncertainty as noise to be filtered out rather than the fundamental nature of what we're trying to model.

This is backwards. Market uncertainty isn't a bug — it's a feature. It's the entire reason opportunities exist in the first place. And maybe, just maybe, if we stop pretending we can eliminate uncertainty and start modeling it properly, we can start figuring out the real rules of the game. Not enough to get kicked out of the casino, but enough to get what we need and walk away.

What Makes Stochastic Finance Different

Most people think quantitative finance is just about crunching numbers. It's not. It's about modeling uncertainty in a way that gives you an edge, like poker players who understand that the math is just the foundation for reading the game.

Here's the thing that clicked for me while working through this material:

Traditional approaches assume you can predict exact outcomes. Stochastic methods assume you can predict probability distributions of outcomes that update based on new information.

That's a massive difference. Instead of asking "what will this stock be worth tomorrow?", you're asking "what's the range of possible values, how likely is each one, and how should I update these probabilities as new information comes in?"

In poker terms: you're not just calculating your hand's strength - you're constantly updating your read on the entire table based on betting patterns, position, and player tendencies. For anyone building trading systems, this is gold.

Markets are fundamentally nondeterministic systems. The same market conditions can produce different outcomes because there are always unknown factors influencing price discovery. Stochastic methods embrace this reality instead of pretending it doesn't exist.

Why This Matters for Strategy Development

When I look at my previous strategies (the momentum systems, the mean reversion plays, the volatility-based approaches), they all work because they capture some aspect of market uncertainty. But most backtesting platforms treat this uncertainty as noise to be filtered out rather than signal to be modeled.

It's like playing poker but only looking at your own cards and ignoring everything else happening at the table.

Stochastic finance gives you the tools to:

• Model the randomness directly instead of pretending it doesn't exist

• Quantify uncertainty in your forecasts (crucial for position sizing - your "bet sizing" in market terms)

• Build systems that adapt to changing market conditions (reading the table)

• Generate realistic scenarios for stress testing (playing against different opponent types)

• Create synthetic data that captures multiple market regimes your historical data might miss

Take something as basic as calculating returns. Most platforms give you simple percentage returns and call it a day. But there's a reason quants prefer log returns: they have mathematical properties that make them much better for modeling and optimization. When you understand why these choices matter, you build better systems.

The Real Problem with Proprietary Platforms

Here's what really gets me: every backtesting platform makes assumptions about how you should model markets. RealTest assumes certain things about how you want to handle data. WealthLab has its own framework. Sierra Chart has its own approach. They're all good at what they do, but they're thinking for you.

It's like playing poker with someone else's strategy card that they won't let you modify.

Stochastic methods require you to think differently about fundamental concepts:

Deterministic vs. Nondeterministic Systems

• Your simple interest calculation? Deterministic. Same inputs always give same outputs.

• Stock prices? Nondeterministic. Same market conditions can produce different outcomes.

Most platforms are built for deterministic thinking. They want you to backtest as if the past will repeat exactly. Stochastic approaches assume the past gives you probability distributions, not exact forecasts.

What I'm Building: The Foundation That Actually Makes Sense

quantKit isn't trying to replace every trading platform. It's designed to give you the mathematical tools that most platforms ignore:

1. Proper uncertainty quantification - Instead of point estimates, get probability distributions

2. Multiple return calculation methods - Simple, log, multiperiod - each with different mathematical properties

3. Stochastic process modeling - Build systems that explicitly model randomness

4. Flexible data handling - Work with any data source without vendor restrictions

5. Realistic synthetic data generation - Create test scenarios based on actual stochastic processes

6. Probabilistic backtesting - Test against multiple probable futures, not just one historical path

7. Adaptive probability updating - Systems that adjust their "read" on market conditions in real-time

The goal isn't to create another black box. It's to give you transparent, mathematically sound tools that you can modify, extend, and understand completely.

Stochastic Methods + Machine Learning = The Future

Here's where this gets really interesting. Stochastic finance isn't competing with machine learning - it's providing the mathematical foundation that makes ML applications to finance more robust and realistic.

Think of it this way: basic poker probabilities are like traditional ML models. But the real edge comes from combining those probabilities with dynamic reads on opponent behavior, position, betting patterns, and meta-game considerations. That's what stochastic methods + ML gives you in trading.

For Simple Strategies:

Even an EMA crossover becomes more powerful when you can:

• Generate probability distributions for future crossover events

• Size positions based on confidence levels (your "bet sizing")

• Test against thousands of statistically valid market scenarios

• Adapt to changing market "player types" (trending vs. choppy markets)

For Advanced ML Applications:

• Use stochastic processes as feature generators that capture market uncertainty

• Train ML models on synthetically generated data that covers multiple market regimes

• Build ensemble models where stochastic processes inform the base predictions

• Get proper uncertainty quantification around your ML predictions

• Create systems that update their "read" on market conditions just like a poker player adjusts to table dynamics

The Key Advantage:

Stochastic methods are less data hungry than pure ML approaches (crucial when clean financial data is expensive), more maintainable (you understand the mathematical relationships), and provide superior uncertainty quantification as a built-in feature rather than an afterthought.

Real-World Applications

Let me give you a concrete example. When I was developing Strategy 12 (the MES futures strategy with a profit factor of 2.03), I was essentially building a system that captured VIX/SPX relationship patterns using linear regression bands. But I was doing it intuitively, not mathematically.

With stochastic methods, you can:

• Model the underlying stochastic processes directly using stochastic differential equations

• Quantify the uncertainty around your entry and exit signals

• Generate multiple scenarios to test your position sizing

• Build adaptive systems that adjust to changing volatility regimes (different "table dynamics")

• Train ML models on synthetic data that captures regimes your historical data missed

This isn't theoretical bullshit. These are practical tools that make your systems more robust.

The Learning Process

Working through this material has been eye-opening. Chapter 1 alone covers concepts that most trading platforms completely ignore:

• Why randomness in markets isn't noise -- it's the fundamental nature of price discovery

• How to think about financial instruments in terms of their stochastic properties

• Mathematical frameworks for modeling uncertainty that actually work

The beautiful thing about building your own tools is that you understand every assumption being made. No black boxes. No "trust us, this works" algorithms. Just mathematical relationships you can verify and modify.

Where This Is Heading

I'm not abandoning everything else to become a pure quant. But understanding these foundations is making me a better system developer. When you know why log returns behave differently from simple returns, you make better choices about how to handle your data. When you understand the mathematical properties of different stochastic processes, you can design systems that are more aligned with how markets actually behave.

The crypto pivot I have been mentioning? That's partly about escaping restrictive data agreements, but it's also about having clean datasets to test these methods on. Crypto markets are perfect laboratories for stochastic methods because they're highly volatile, relatively new, and not constrained by traditional market structures.

The Bottom Line

Building quantKit isn't just about creating another backtesting tool. It's about building the first platform that properly integrates stochastic methods with modern machine learning techniques, giving users everything from simple probability-enhanced strategies to sophisticated ML systems built on sound mathematical foundations.

Most retail traders use tools built on these principles without understanding them. That's fine for some people, but I want to know exactly what's happening under the hood. More importantly, I want tools that acknowledge the fundamental truth about markets: they're uncertain by nature, and that uncertainty is where the opportunities live.

Just like in poker - the best players don't try to eliminate uncertainty. They embrace it, model it, and use it to their advantage.

If you're tired of being limited by what your platform thinks you should be able to do, this approach is worth considering. The math isn't insurmountable, and the tools you build will be exactly what you need them to be.

Plus, there's something deeply satisfying about solving problems with code that you actually understand, built on mathematical foundations that acknowledge reality instead of pretending it away.

Next week, I'll dive into the probability foundations and start showing some actual code. This is going to be a journey, but I think it's going to be worth it.

Happy hunting.

Feel free to comment below or email me if you need help with anything, wish to criticize, or have thoughts on improvements. You can also DM me or start a chat thread. Red Team members can access this code and more at the private HGT GitHub repo. As always, this newsletter represents refined versions of my research notes. That means these notes are plastic. There could be mistakes or better ways to accomplish what I am trying to do. Nothing is perfect, and I always look for ways to improve my techniques.

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

Okay, this is a new thing for me. Usually, I just stick to developing and finance specific research. However, the projects I want to explore for the rest of the year are going to require I do some heavy lifting first in Python. It seems very boring to write about the process and redundant to document it in more than one place. So, I decided to write about a topic I have been thinking about for some time and that seems to be on everyone's mind: US tariffs and Chinese (PRC) manufacturing.

First and foremost, I am not an expert. In anything. So, this is just an editorial or opinion piece. Whatever you want to call it. Where I can, I will reference sources I use to make my points.

Secondly, some of my information is second hand information that I received from US Federal Agents who worked overseas in China around the 2015 - 2019 time frame. So, that information is old, but still applicable. Other portions of my information come from literally living in China. In case you misunderstood that last sentence, I will state it again. I actually lived in China for months or so at a time. My wife lived outside of Shanghai and worked at private Chinese international school for 2 years. So, when I was in Iraq and got leave, I went to China (or other south Asian countries). I was in Hong Kong during the protests against the PRC taking over the SAR of Hong Kong. I will speak later on what happened with that...

Now that we have that out of the way, let's get into it.

A collection of partial truths

I'm going to post a few links here and briefly summarize the points of the speakers.

First up, Tim Cook (Apple CEO). I want to point out that I have been mostly impressed with Tim Cook taking over Apple. I wrote several papers on him in college that revolved around contrasting leadership styles. Anyway, the video is short, and he just gives his opinion on Chinese manufacturing and why they use it.

• Tim Cook says that we don't have the manufacturing infrastructure in the USA that the PRC has.

• He also says they have more skilled laborers and engineers.

Next, we've got one from Substack. I have seen a shit ton of these videos from various accounts, but they all pretty much say the same thing.

• There is a knowledge gap in how this manufacturing works.

• A lot of this technology was developed in China by Chinese engineers and scientists.

• A lot of this technology has never been made in the USA. Full stop.

• White folk don’t tend to research brushless motor technology…

Ok, that last bullet point is a joke, but he did point out that you don’t see many American (or European) names on the research papers, or they aren’t the majority. Brushless motors are pretty specific, so we can chock that up to an anecdotal joke. However, I have a theory as to why you might see this type of disparity and why there are more Chinese engineers and scientists.

These are the only two videos I am going to use because ever since Tim Cook’s comments, everyone seems to be jumping onto this same concept to make points against US tariff’s on China and to try and change the perspective of the American majority that seems to not quite understand everything that is going on here.

I think they missed the point too. As a matter of fact, I believe the information they are giving is purposefully misleading. They aren’t wrong. Not even close. They are stating facts, but they aren’t stating the ones that matter…

Cost

Tim Cook makes the claim that Apple (and presumably other technology companies) doesn’t go to China because it’s cheaper. He says that they stopped being cheap some time ago and that they use them solely for their expertise and infrastructure. I want to break this down, because what he is saying isn’t exactly true. Actually, it's nowhere near true. It omits some pretty crucial points to consider when discussing “costs”.

Let's be real here - China's impressive manufacturing setup didn't just magically appear. It was deliberately built using our money, our technology, and massive government manipulation. The U.S.-China Economic and Security Review Commission (USCC, 2023) documented how American companies handed over billions in tech and manufacturing know-how just to access the Chinese market. We literally funded the creation of our future competition. And according to the Congressional Research Service, the Chinese government then piled on with massive subsidies, practically free land, and artificially cheap energy for manufacturing zones. The type of stuff that would get you slapped with WTO violations in any Western country (Morrison, 2023).

And the human cost? Absolutely staggering. The Lancet found that air pollution from this rapid industrialization killed about 1.24 million Chinese people annually (Burnett et al., 2020). That's not a typo. The International Labour Organization has documented systematic abuse of workers, with hours and safety conditions that would get you shut down instantly in the U.S. (ILO, 2022). Then there's China's "hukou" system, which Harvard's Ash Center studied extensively, that basically creates a second-class workforce of migrants with zero local rights or protections (Liang & Lu, 2021). So yeah, of course Chinese manufacturing is "cheaper" than U.S. production—they're playing a completely different game with different rules, cutting corners that American companies legally can't.

In case you forgot about what working conditions in even Apple’s Chinese factories, here is a post from The Guardian to help remind you:

Life and death in Apple's forbidden city

Expertise

I am going to take the low hanging fruit first. The USA has a population of 334 million and the PRC has 1.4 billion. Do I need to explain how this math works?

When Tim Cook and others brag about China having more engineers and skilled laborers, they're conveniently ignoring basic population math. Of course China produces more engineers - they have more than four times our population! But the story goes much deeper than raw numbers.

What they're not telling you is how China's education system actually works. Unlike America, where students generally choose their own career paths, China's government actively directs education priorities based on national economic goals. The Ministry of Education literally sets quotas for engineering graduates to meet industrial needs (Li, 2019). This isn't about natural interest or market demand - it's central planning in action.

And those Chinese engineers? They earn dramatically less than their American counterparts. According to salary data from Glassdoor, the average engineer in China makes about $35,000 annually compared to $105,000 in the US (Glassdoor, 2023). So when tech execs praise China's "engineering talent," they're really saying "we can hire three Chinese engineers for the price of one American." That's not about superior skill - it's about cheaper labor dressed up as expertise.

Then there's the uncomfortable truth about innovation in China. A 2022 report from the Office of the U.S. Trade Representative estimated that Chinese intellectual property theft costs American businesses between $225 billion and $600 billion annually (USTR, 2022). Much of China's technological "development" comes from stealing and reproducing Western innovations. So when you see those Chinese names on research papers, remember that foundation was often built on technology that was either forcibly transferred or outright stolen.

Let's be honest about what's happening: China's engineering workforce isn't some magical resource that America lacks. It's the product of government-directed education, dramatically lower wages, and a system that has historically rewarded copying rather than creating. That's the reality behind the "expertise" argument.

I'm not suggesting China is incapable of innovation - far from it. In recent years, China has made advances in areas like mobile payment systems, high-speed rail, and renewable energy technology. However, there's an important context to this innovation timeline. The IP Commission, a bipartisan initiative led by Admiral Dennis Blair and Governor Jon Huntsman Jr., specifically recommended in 2021 that the U.S. "set a national goal to delegitimize Chinese indigenous innovation efforts that are dependent on the theft of foreign IP" (IP Commission, 2021). The Commission also noted that IP theft costs the U.S. economy "hundreds of billions of dollars annually and reduces U.S. companies' R&D investment and innovation." China's technological leap forward came only after decades of acquiring foreign technology through joint ventures, technology transfer requirements, and forced technology transfer. It's much easier to innovate when you're building on a foundation of existing technology rather than developing it from scratch. The key point isn't that China can't innovate - it's that their current innovation capacity was jump-started through years of policies that appropriated technologies developed elsewhere at significant cost.

Personal Experience

Like I mentioned in the beginning of this post, my wife actually lived in and worked full-time in China. I spent several months there myself when I was on leave between the years of 2017 and 2020. Something tells me that a lot of people who speak about China, or defend their actions, have never actually been to China.

First, getting a visa isn't easy. Even with my spouse there, I had to hang out in Hong Kong for several weeks to work the system, eventually getting a 10-year business visa. This process wasn't easy, and it required business practices that are probably illegal in the USA.

While I was in Hong Kong, there were a lot of protests and riots. These were in an attempt to keep the PRC from having complete control over the SAR area of Hong Kong. This was a huge deal for the local nationals of the SAR (mostly Cantonese if I remember correctly). Then, in 2020, while the whole world was too busy paying attention to COVID-19, the PRC implemented the 2020 National Security Law and cleaned house. According to Human Rights Watch, this law criminalized virtually any form of dissent with vague charges like "subversion" and "collusion with foreign forces" (HRW, 2020). Within days, pro-democracy activists were arrested, books were removed from libraries, and freedom of speech effectively died. The PRC now has complete control of the area. This is just an example of how their government operates.

Then there is China actual and let me tell you, it reads just like your favorite books about society's potential future. Shanghai was a cool city. Public transportation was really good too. The air quality is terrible (read cancerous) and even if the streets don't have cars on them for miles, no one will cross the street until the crosswalk light turns and they wouldn't cross the street anywhere except where there is a designated cross walk.

At first, I thought that this was just a side effect of a culture built around respect, like Japan. But after actually talking to Chinese people (the few that spoke English well enough), I discovered that isn't the case. What keeps them from jay-walking? Fear. This fear of their government is so real that I had people return change to me at the wet market because they were afraid someone would report them for stealing it. This was change that didn't even add up to fifty cents in USD.

Research from the University of California confirms this observation - China's social credit system has created a pervasive surveillance state where citizens monitor their own behavior out of fear of consequences (Creemers, 2022). Every aspect of daily life - from crossing streets to online comments - carries the potential for punishment. This is the social control system that our manufacturing dollars help support.

I told you that my wife was a teacher, right? One day, while at work, she went to lunch, did some planning with folk, and then went back to her classroom. In the time that she was gone, the government had people inspect rooms (foreigner rooms). She had a bunch of random posters and such on the wall in English. All American-oriented posters (as all of her students were attempting to get into Western colleges) were ripped off the wall and replaced with PRC flags, creeds, and propaganda. Why do I say propaganda? Well, what would you call government advertising that says Tiananmen Square never happened and that Americans made it up?

This isn't isolated. According to a report from Freedom House, the Chinese government employs over two million "internet opinion analysts" to monitor and censor online content and extends this control into every aspect of education and daily life (Freedom House, 2021). What Tim Cook and others conveniently ignore is that when we manufacture in China, we're not just getting "infrastructure" and "expertise" - we're financially supporting one of the most sophisticated authoritarian control systems ever created.

Closer to enemies than friends

While American companies pour billions into Chinese manufacturing, we're overlooking a disturbing reality: China actively works against critical U.S. interests in ways that resemble hostile action more than friendly competition. The devastating fentanyl crisis killing over 70,000 Americans annually has direct links to China. According to a 2022 DEA report, Chinese chemical companies remain the primary source of fentanyl precursors and manufacturing equipment for Mexican cartels like Jalisco New Generation (DEA, 2022). Even after China's supposed crackdown on direct fentanyl production, they simply shifted to supplying the ingredients and equipment to Mexican partners, maintaining their role in a supply chain that's killing Americans daily.

This pattern extends beyond narcotics. The FBI has identified China as "the greatest long-term threat to our nation's information and intellectual property, and to our economic vitality" (FBI, 2023). Their analysis documented over 2,000 active investigations tied to the Chinese government, with new cases opening every 12 hours on average. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) has attributed numerous sophisticated attacks on American infrastructure, government agencies, and businesses to state-sponsored Chinese hackers, including the massive Microsoft Exchange breach affecting tens of thousands of organizations (CISA, 2022). These aren't random criminals but coordinated operations with government backing.

Perhaps most concerning is China's strategic relationship with America's clearest adversaries. A 2023 Pentagon report detailed China's provision of dual-use technology and components to both Russia and Iran, materially supporting countries that openly threaten U.S. interests (DoD, 2023). The report noted that Chinese companies have supplied drone components used in Ukraine and missile technology utilized by Iran's proxies across the Middle East. This creates the disturbing scenario where American consumer dollars spent on Chinese-manufactured goods indirectly fund technologies being used against our allies and interests. When Tim Cook and others defend Chinese manufacturing as simply about "infrastructure" and "expertise," they conveniently omit this darker reality: we're economically strengthening a nation that systematically undermines American security across multiple fronts.

Now, let’s talk tariffs

Finally, we arrive at the topic at hand. When critics suggest that tariffs on Chinese goods are misguided or harmful, they're deliberately obscuring the complex reality we've just explored. Tariffs aren't simply about protecting American jobs or bringing manufacturing back - though these are worthy goals. They're about addressing a fundamentally uneven playing field created by decades of strategic manipulation.

The economic case for tariffs is straightforward: they help offset the artificial cost advantages China has created through labor exploitation, environmental degradation, intellectual property theft, and massive state subsidies. According to a 2023 Economic Policy Institute analysis, the U.S. lost approximately 3.7 million jobs to China between 2001-2021, with the greatest impact felt in manufacturing communities that haven't recovered (Scott & Mokhiber, 2023). Tariffs create space for American manufacturing to rebuild without competing against a system designed to undercut fair competition.

Critics argue that tariffs ultimately hurt American consumers through higher prices. This narrow view ignores the broader economic and social costs of our China dependence. A RAND Corporation study estimated that IP theft alone costs the U.S. economy between $225-600 billion annually - costs that are distributed across all Americans whether they realize it or not (RAND, 2022). The Peterson Institute calculated that increased production diversity and reduced supply chain vulnerability resulting from reshoring critical industries would provide long-term economic benefits that outweigh short-term price increases (Lovely & Xu, 2023).

Beyond economics, tariffs represent one of the few meaningful policy tools available to address China's systematic human rights abuses. The U.S. Holocaust Memorial Museum and the State Department have officially recognized China's treatment of Uyghurs as genocide, with forced labor being a central component (USHMM, 2021). When American companies manufacture in China without consequence, they become complicit in a system that violates our most basic values. Tariffs create economic consequences for this complicity and provide leverage for demanding change.

The fundamental question isn't whether tariffs will cause some short-term economic disruption - they will. The question is whether Americans are willing to pay a modest premium on consumer goods to support a manufacturing system that doesn't rely on authoritarian control, environmental destruction, worker exploitation, and technology theft. When viewed through this lens, tariffs aren't just economically justified; they're a moral imperative for a nation that claims to stand for freedom, fairness, and human dignity.

My Conclusions

I want to make sure that you understand that the conclusions I draw from this are personal. I know that people will see things differently than me. That's cool. It's good for conversation. The points that I bring up here are all backed by sound references and are rooted (to the best of my ability) in some core truths about China and how they operate.

So, what is my take?

I look at this through a very simple lens. I am not just American, I actually believe in what we are supposed to stand for and represent: freedom and equality for all. I don't believe that any company that sells to Americans should be able to profit more by abusing countries that do not have the standards and beliefs we do. I believe when you look at the facts, and the terrible/inhumane history of modern China, it becomes very difficult to support ANYONE who wishes to take advantage of disadvantaged people.

The fundamental disconnect in how we talk about Chinese manufacturing comes down to values. When Tim Cook and others frame their manufacturing choices as being about "infrastructure" and "expertise," they're deliberately avoiding the moral dimension of their decisions. According to business ethics research from the University of Pennsylvania's Wharton School, companies engage in this type of "ethical fading" when they reframe moral choices as purely business decisions (Tenbrunsel & Messick, 2021). This allows executives to sleep at night while their companies profit from systems they would never tolerate in their home countries.

What's particularly frustrating is the dishonesty. If Cook simply said, "Look, we manufacture in China because it's dramatically more profitable despite the ethical concerns," I'd at least respect the transparency. Instead, we get this carefully constructed narrative about Chinese manufacturing superiority that conveniently omits the human rights abuses, environmental devastation, and authoritarian control that make their "efficiency" possible. As documented by the Business & Human Rights Resource Centre, American companies have repeatedly failed to meaningfully address forced labor in their Chinese supply chains despite public commitments to ethical standards (BHRRC, 2022).

The tariff debate ultimately isn't about economics or even politics – it's about what kind of world we want to live in. When we purchase products manufactured in China without any economic consequences, we're tacitly endorsing a system that contradicts our most fundamental values. The Princeton philosopher Peter Singer has argued convincingly that our purchasing decisions have moral weight – they reflect what we're willing to accept and what we're willing to ignore (Singer, 2022). By this measure, our current trade relationship with China represents a massive moral compromise that we've normalized through clever marketing and willful ignorance.

I'm not suggesting we cut off all trade with China overnight or pretend there aren't complex economic realities at play. What I am suggesting is honesty about what we're actually doing and the true costs involved. Tariffs represent one imperfect but necessary step toward a trading system that aligns with our values rather than undermining them. If that means paying a bit more for our iPhones or accepting that some things can't be as cheap as they once were, that seems like a small price to pay for a world where prosperity doesn't depend on exploitation.

I don't have a solution for our problems, but I don't think we can go wrong if we keep our values and look at this through a lens that considers the humans that make it happen. When we strip away the marketing language, the economic theories, and the geopolitical complexities, we're left with a simple truth: our manufacturing choices have human consequences. Every purchase decision connects us to the hands that assembled our products and the systems that govern their lives. If we truly believe in human dignity, environmental stewardship, and fair competition, then we must demand that our economic relationships reflect these values. Perhaps that's the true measure of "expertise" and "infrastructure" that should matter most. The question isn't just how efficiently we can produce goods, but whether we can do so in ways that uplift rather than exploit the people involved. The most advanced manufacturing system in the world isn't worth much if it's built on human suffering. That's a truth no tariff debate can ever change.

As always, comment if you have something to say. If I got something wrong, let me know. If you disagree with my assessment, tell me why. Just remember, I strongly believe that ALL humans are worthy of freedom, equality, and entitled to live a life of love and peace. If you believe that I am not supposed to apply that to people who aren't American, we can't have a constructive conversation. I don't bend those beliefs just because another country or government doesn't believe in them. The way I see it, if we're willing to compromise on our fundamental values for economic convenience, then those values weren't really fundamental to begin with. And that's a far greater cost than any tariff could ever impose.

References

1. USCC. (2023). Annual Report to Congress. U.S.-China Economic and Security Review Commission. Washington, DC: U.S. Government Publishing Office.

2. Morrison, W. M. (2023). China's Economic Rise: History, Trends, Challenges, and Implications for the United States. Congressional Research Service Report RL33534.

3. Burnett, R., Chen, H., Szyszkowicz, M., et al. (2020). Global estimates of mortality associated with long-term exposure to outdoor fine particulate matter. The Lancet, 391(10130), 1639-1649.

4. ILO. (2022). Labour standards in global supply chains: A programme of action for Asia and the garment sector. International Labour Organization. Geneva: ILO Publishing.

5. Liang, Z., & Lu, J. (2021). Hukou Reform and Migration in China: Causes and Consequences. Harvard Kennedy School Ash Center for Democratic Governance and Innovation.

6. Li, Y. (2019). China's educational policy making: A policy analysis perspective. Journal of Education Policy, 34(5), 763-786.

7. Glassdoor. (2023). Salary Comparison: Engineers in United States vs. China. Glassdoor Economic Research.

8. USTR. (2022). Annual Report on Intellectual Property Protection and Enforcement. Office of the United States Trade Representative.

9. WIPO. (2020). World Intellectual Property Indicators 2020. World Intellectual Property Organization.

10. IP Commission. (2021). The Report of the Commission on the Theft of American Intellectual Property. National Bureau of Asian Research.

11. HRW. (2020). China: New Hong Kong Law a Roadmap for Repression. Human Rights Watch.

12. Creemers, R. (2022). China's Social Credit System: An Evolving Practice of Control. University of California Press.

13. Freedom House. (2021). Freedom on the Net 2021: China. Freedom House

14. DEA. (2022). 2022 National Drug Threat Assessment. Drug Enforcement Administration.

15. FBI. (2023). China's Economic Espionage and Theft of American IP. Federal Bureau of Investigation.

16. CISA. (2022). Chinese Government-Sponsored Cyber Operations: Observed TTPs. Cybersecurity and Infrastructure Security Agency.

17. DoD. (2023). Military and Security Developments Involving the People's Republic of China. Department of Defense Annual Report to Congress.

18. Scott, R. E., & Mokhiber, Z. (2023). The China Trade Toll Revisited: U.S. Jobs Lost, Trade Deficit Increase, Twenty Years After China's Entry Into the WTO. Economic Policy Institute.

19. RAND. (2022). Economic Costs of Intellectual Property Theft: An Analysis of the Impact on the U.S. Economy. RAND Corporation.

20. Lovely, M., & Xu, B. (2023). Resilient Supply Chains and Economic Security in a Post-Pandemic World. Peterson Institute for International Economics.

21. USHMM. (2021). "To Make Us Slowly Disappear": The Chinese Government's Assault on the Uyghurs. United States Holocaust Memorial Museum.

22. Tenbrunsel, A. E., & Messick, D. M. (2021). Ethical Fading: The Role of Self-Deception in Unethical Behavior. Wharton School, University of Pennsylvania.

23. BHRRC. (2022). Beyond Compliance: Effective Reporting Under Modern Slavery Laws. Business & Human Rights Resource Centre.

24. Singer, P. (2022). The Most Good You Can Do: How Effective Altruism Is Changing Ideas About Living Ethically. Yale University Press.

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

Ok, my break was too long. After the basement flooding issue, my father had a medical emergency and has been in and out of ICU for the last month. With everything going on, I fell off for a bit, but now I am back and ready to work.

As we know, I am shifting focus for the time being to research and creating tools to help us research trade systems better, and without vendor lock-in. That's the goal, anyway. Ultimately, it is going to be a learning process, so we can learn why our tools matter and how they improve our testing and research process. For me (HGT), this will consist of 3-4 distinct projects.

I am going to focus on creating a set of tools for testing in Python, building on the statistical testing tools I created last year for feature/indicator testing. I am also going to focus on creating strategy and indicator libraries for RealTest and Wealth 8 for the HGT users to play with and use. While learning how to create tools in Python, I will also work on building a platform for testing and running algorithmic strategies in Zig.

Today, we focus on what has been updated so far in the Python tools and where to access the project ( it's open source).

First, the name of the project is quantKit, and the goal is to make an efficient backtesting and researching tool set for algorithmic trade system development. I have no intentions of using this tool for live trading, so I won't focus on that part.

If you recall from any of the statistical testing posts, we had several tests that we created in this project. Some tests were simple, others more complex. While they worked, and matched the calculations and logic from the source material, they weren't the most efficiently written functions. So, my goal is to go back through these functions and refactor them to be more efficient.

I also intend to begin orienting the tool to be used from the command line. This means that everything will be printed to the command line first, and saved to file second (or at user request). The idea is to streamline development and remove any abstractions that could create more overhead or limit developer capability.

As an example, I went through the first test in the library and updated it to use Numpy's built in vector functions to increase the speed of the test by 85%. This is the simplest test of the bunch, and was already pretty fast, so I don't know what kind of performance increase we will see at the end, but I am hopeful that we can get the overall testing speed down to something more reasonable when testing a large amount of features. The photo below shows an example of how long it took to generate the simple stats report ( with relative entropy).

Not too shabby, right? This section is so simple that adding in multiprocessing for this test actually slowed down the overall speed of the report. I imagine I will see multiprocessing become more useful as we get to the more computational intensive tasks and permutation testing.

Next up is the mutual information function, then the mean break test, and lastly the optimal threshold test. Once these are complete, I will shift over to the backtesting engine and data management.

Check out the Python project repo below:

quantKit Repository

Happy hunting!

Feel free to comment below or e-mail me if you need help with anything, wish to criticize, or have thoughts on improvements. You can also DM me or start a chat thread. Red Team members can access this code and more at the private HGT GitHub repo. As always, this newsletter represents refined versions of my research notes. That means these notes are plastic. There could be mistakes or better ways to accomplish what I am trying to do. Nothing is perfect, and I always look for ways to improve my techniques.

What about markets? What is general lean here.

How do you like to learn/consume content?

I don’t do video, but I could get into it.

Let’s talk about programming languages.

What do you think about the recent shift in focus? Lay it on me.

I am just summing up the changes and sketching a tentative timeline.

Changes

• No more proprietary platforms. This means I am shifting my entire focus to modelling and testing trade systems in Python.

• No more Windows only and restrictive data providers. The previous article talked about why this change is happening.

• Because of the two aforementioned points, I will be shifting my focus to using Cryptocurrency data for this development and testing process.

• My Windows machine will become a network machine. This will give me access to any Windows specific needs, such as…

• WealthLab 8 (WL8) will become the Windows platform of choice (at least for the next year). This platform appears to offer the most bang for your buck.

• Paid subscribers will get access to the WL8 libraries and the paid content on Substack (obviously). All articles will still breakdown strategies in a way that they can be used in your platform of choice and will be accompanied by Python code for them as well.

• Python tools will be made open source, so anyone can access them and play with them.

Why?

I want to enjoy what I do, and this change will facilitate that. I love coding and I love having an intuitive coding environment that gives me control over everything I need. I also like using open-source tools and non-proprietary solutions for my needs. Embracing this will allow me to produce more content for the subscribers as a higher rate. I can document the entire process and provide value to other people who endeavor to participate in this field.

Tasks and Timeline

• Install Linux on desktop machine and setup current laptop as a network machine.

• Move quantKit (previously just called pyquanttools) to a public repo and add open-source licensing. This will be the base of the tools I develop for research in Python.

• Create extension libraries for WL8 to house indicators and strategies.

  • This will take some a little time. Subscription based licensing will be first, and then I will start translating indicators and strategies into the libraries as we progress forward.

• Begin developing a trade simulation engine for quantKit turning it into a proper Python library/command line tool for easier use.

• Archive HGT repos and setup new structure for GitHub repo to house Python indicators and strategies.

I estimate that by the end of this week I will have my Windows machine setup as a remote network device. After that, the first thing we have to develop is the trade simulation (or backtesting) engine in Python. This is going to start out pretty basic, and we will fine tune it as we go along.

So, by next week I should be working the Python tools and getting indicators and strategies translated into C# for use in WL8. From there, it becomes much harder to create a proper timeline. I don’t know how long it will take developing the simulation engine before we get to test a strategy idea. I plan on trying to make this engine reliable, with a focus on statistical analysis and math.

During the engine development phase, I will still be able to create new indicators and translate indicators into C#, so I won’t just be posting about the development process.

Ideally, I would like to have up to date WL8 extensions and a working simulation engine inside 30 days. Unfortunately, I am keenly aware of how bad we (humans) are at guessing at how long it will take us to accomplish something. With that in mind, I will double it to 60 days, just to be safe.

Summary

This is a big shift. I know. But I don’t enjoy the games I am currently having to play, and I don’t want to stop researching, writing, and trading. So, I need to do it my way. A way that aligns with my principles and that I enjoy doing.

I plan to dabble in producing video content as well. We shall see how this goes. My time without toddler noise is limited, so I will have to play around with how I do this.

If anyone has any questions or concerns, reach out to me via message or email.

I am going to put out a poll or chat (or both, idk how they work yet) to see what strategies and indicators y’all want to see translated into C# first and what subjects you want to focus on.

As always, remember that I am not a professional anything. This is just something I love doing and researching.

Happy hunting.

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

My wife was chasing our 2-year-old from around the camper towards to the car the get packed up to go to dinner. It was then that I noticed my father at the side of the house, red in the face and out of breadth.

"I hate to be a bother, but we have a problem."

My father is a bit extra from time to time, so I blinked. Real slow like. After that millennia passed. I just replied, "show me."

I made my way down the hill and around the house to the basement entrance. He was talking to me while we were heading down there, but I wasn't paying attention. I heard something about a leak and water on the floor. I open the door and step inside, splashing straight into an inch or so of water.

A million things went through my head while I bolted to the pressure tank and to hit the emergency shut off valve for the water supply to the house. Where are the main water lines located downstairs? How far are they from the fucking door? How long was this happening to fill all the way to the door an inch deep? How many live electric outlets and power strips are about to start blowing circuits?

All of these questions got answered in pretty rapid succession. Of the square footage in my mostly unfinished basement, about a quarter of it got covered in water. The only finished portion of my basement (the in-law studio my father occupies) got hit pretty hard, of course. There were multiple power strips running real live electricity through them floating in the water like the door in the Atlantic from that one movie we've all seen more than once. You know the one, don't pretend. It was just one teeter totter away from blowing out a circuit and probably giving me a good kick.

In the end, I got the water off and the slammed all my basement circuits to off to assess the potentially deadly situation. I did pause, though. My dad has a cat that I wouldn't mind...

Anyway, that's what happened to me a couple of weeks ago. I got the pipe problem solved and a good lesson in basic water plumbing for houses. I had a guy come out to give me a quote on mitigation and repair the very next morning. I'm not sure if prices for this type of work are targeted at insurance prices plus average deductibles or contractor types just think people are stupid, but when I saw the quote, I realized I was going to need to do it myself.

This type of work isn't complicated. It's actually pretty damn simple. Pull water soaked/damaged materials, dry and dehumidify everything, continuously assess moisture levels, test for mold, and be proud of this proprietary knowledge you've just come up with. Insert eye roll. Then, of course, you need to be able to replace the damaged pieces you just cut out of your walls and floors.

Too easy. I'm no stranger to hard work and I'm not stranger to this kind of work. I've got all the tools, I've done it before, and anything I don't already know is just a few finger taps away on ye ole YouTube machine.

Then it took two weeks. Why? Because I was doing it alone, while trying to work from home (HGT and client work), watch and care for a toddler, and every couple of steps I took forward I had to backtrack because I discovered a new gift from the home's previous owners. Such as the fact that the plank flooring (now ruined, cause Linoleum ain't waterproof) that was previously installed directly on top of carpet (hydrophilic) which had been glued to the concrete with no less than 3 gallons of glue.

This went on the entire project. One new discovery after another.

But that's past me now. I mean, mostly. I have a trailer full of construction trash I need to get rid of, a little furniture to move around, and I have to have a conversation with an old Vietnam vet about hoarding in a studio apartment. Other than that, this nightmare is past me.

Now I am sitting at my keyboard, I have no idea what day it is unless I look at the calendar and I have to do it multiple times a day because I forget. My toddler has become an official and credentialled 2-year-old. I'm operating off of caffeine fumes and the occasional nicotine pouch to help me focus.

And I just stare. I just stare at the fucking screen and can't figure out what I'm supposed to do. Before this event I had plans, and they are still written out, but I look at them and I don't want to fucking do them.

If you ever need clarity in your life, or you want to start thinking straight (about the shit that matters), go do something hard. Physically hard. Not just once. Do it for a while. Wake up every day and do hard physical work for as many hours as you can manage. Think only about how much it fucking sucks (this will be easy, because if it's hard enough, it is all you can think about), and resign yourself to doing it until it's done at whatever pace you have to do the thing in. Then, when it's done, go back to the other thing in your life that needed some perspective.

That's what happened to me. Not by design. It just sort of happened, but here I am. I've been trying to dig back into my projects this week, but the drive isn't there. There is so much that I am unhappy with when it comes to researching trade systems. I hate trying to find the right program or having to make a program sort of do thing I want/need. I hate that there is a concept of "public" companies yet everything you need to play (platforms, data, etc) the game is private, proprietary and costs money.

Well, I don't want to play that game anymore. I don't want to keep getting lost in the different platforms and methods for researching strategies and trading. I don't want to have one program for chart trading, one for hot key trading, another for backtesting strategies, one for equites, another for futures, and STILL have to use 5 different coding languages to glue shit together or make the things I want to make. It's fucking exhausting and, probably by definition, insane.

So, I have to make some changes. I am solidifying the tools I use and switching my focus using my own code for researching and testing. I am going to focus on Python (as that is commonly used and the lowest barrier to entry) and a low-level language such as C or Zig. The latter of which will just be a side project to work on along the way.

As of right now, for the platforms that I will use that run on Windows, I am looking at keeping Sierra Chart (but not why you think) and... well, that's it, really. I will continue to use Norgate Data for end-of-day data for futures and equities, and Sierra Chart has a built-in DTC messaging protocol that allows users to stream or get historical data, send and receive trades, etc. This is really cool, because Sierra Chart uses very minimal resources when running and can be used as a platform for testing and running algorithmic systems with, without having to use their built-in framework. We can let Python do all the work and generate trades directly to the platform.

Important Note

The quote above is something I wrote the other day, when I first started this post. At the time, I was thinking that I would use Sierra Chart as my catch all for futures and equity data. Since they have a DTC protocol, this seemed like a good way to get good historical and streaming data through Sierra Chart for both testing and running live strategies on. However (this is the important part), it turns out that even sending data over your own home/personal network is a violation of the data usage agreements.

Wait, what?

That's right. Technically, sending the data from one machine to another, over any type of network, could be considered distribution of data and thus violate the data usage agreement. This got me thinking. Is this type of restriction/verbiage common amongst data providers?

Well, let's check out the ones I use and see...

First, let's look at how Sierra Chart words it.

And this is how they interpret it.

That's pretty explicit.

Next, let's check Norgate's license agreement.

Hmm, "redistribute the Data in any way or form" seems to jump out, doesn't it? I guess the argument would focus on the definition of "redistribution". Regardless, it is ambiguous enough to make me uneasy. You see, my goal was to use Norgate data and the Python library to just copy the data bases and the data I need and make it accessible to my desktop machine for testing. It seems that this would probably qualify as "redistribution" to some, should they wish to get upset by it. Since this verbiage is similar to Sierra Chart's, a conservative person would believe that sharing across a network is probably prohibited.

If my intention was to have Windows be a secondary machine attached to my home network, I would probably be in violation of the data usage agreements with the two providers that I was looking towards for my data needs.

Fascinating development.

Back to our regular programming...

You may notice I didn't mention RealTest. That is because I am at a weird point where I don't really want to keep or use ANY proprietary backtesting engine. I still really enjoy RealTest, and I think it is one of the best backtesting engines there is. I also love that it is written in C (just C, not C++), and is very efficient. However, it lacks some of the features that I really want to start testing with my research. I don't have as much control as I would like when creating custom features for testing. Of course, I can always find work arounds, but I don't want to spend time on workarounds when I could spend that time developing a good system that does exactly what I need. I would also like the ability to test both end-of-day and intraday strategies.

I am currently trialing WealthLab too (for the next year). What I am thinking about doing with this is using it as a way to store libraries of strategies, indicators, and tools that I develop for the platform. It is written in C# (sigh), and the framework is in C#, but that's ok. It has extensive documentation for its framework and extension API. It also makes it easy to offer libraries and tools as subscriber-based extensions that are easy to install.

This would mean I could restructure how and what is offered to paid subscribers. GitHub access will remain the same (with some restructuring interiorly) for Red Team members, but paid subscribers will gain access to the HGT Indicators and Strategies extensions for WealthLab. This way, I can try and address all levels of reader. Research and ideas can remain focused on the development using Python (or whatever else seems fun), but users who want to test the strategies and indicators without having to code or compile something themselves, can access to the versions for WealthLab.

In case you are wondering, I chose WealthLab because it seems to have the most features that would suite many different levels of trader’s needs. The system is robust enough to handle strategies like the ones I have been testing in RealTest, and offers a strong extension eco system of tools, data and broker connections, and many others. It seems to have a pretty active user base and responsive development team.

While this endeavor may take several weeks of constant work, I think it will be worth it in the end. This way, I can start to focus on the work that I truly enjoy, solving problems with code and trade research.

Conclusions

Now, you might be wondering, "what does this mean, Larry?"

Let's recap:

• Hard work is good for the soul.

• I hate Windows and feel very restricted using proprietary platforms for my research and trading needs.

• I am going to stop using Windows regularly, turning my laptop into a network machine for remote access when I have a specific need for something on Windows.

• I am no longer going to be testing my ideas on proprietary platforms. Instead, I am switching focus to using Python and (insert low-level language here) for my research needs.

• I don't like restrictive data usage agreements that technically don't even let me share the data across my own network to another personal machine. So, I'm not going to use those data providers.

Okay, so you don't like Windows, and you don't like the commonly used restrictive language in the data usage agreements. What are you going to do about it?

Well, I'm going to stop using them. I already didn't like the fact that data can't be retrieved for free. There is something about having a public domain for retail traders/investors that gate keeps data that rubs me the wrong way. Top this with the excessive restrictions on your own machine or network and it's just something that I can't get behind.

There are data providers out there that are API first and have less restrictive verbiage on their data usage agreements. I plan on looking at these types of providers in the future, but for now...

I'm switching to crypto.

Ha. Wait, are you serious?

Yup.

What better way is there to escape the platform lock in, restrictive data usage agreements, and still be able to provide useful and informational content to your subscribers?

If you can think of one, I'll be willing to listen. Put it in the comments.

Until next time, happy hunting.

Feel free to comment below or e-mail me if you need help with anything, wish to criticize, or have thoughts on improvements. You can also DM me or start a chat thread. Red Team members can access this code and more at the private HGT GitHub repo. As always, this newsletter represents refined versions of my research notes. That means these notes are plastic. There could be mistakes or better ways to accomplish what I am trying to do. Nothing is perfect, and I always look for ways to improve my techniques.

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

Part two of this series covers creating the intraday volatility bands indicator that is being used in Maróy's^[1] paper on intraday momentum strategies for SPY and QQQ. I created this indicator in using Sierra Chart's (SC) Advanced Custom Study Interface Language (ACSIL), which is just C++. The logic for this indicator can be translated into whatever flavor of language you want to use. I have done my best to mimic the logic presented in the research paper. I think I have done a good job, but I'm not a mathematician or computer engineer, so it's just my best attempt. We will really be able to tell how I did when I start the backtesting process.

In the previous post (the daily test of this concept), I believe I mentioned that my use of standard deviation for a quick calculation and that it differed from Maróy's^[1-1] calculation, but now that I am going back through it to create this indicator it looks like they do use the standard deviation of mean returns. What they don't use are log returns, which I did use in the previous post. I will probably update this indicator in during the testing phase to have an option to use log returns instead. That way we can test the difference and see if we can use that to help make this indicator/strategy more robust.

I am leaving most of this post open for all readers. The pay wall will block the completed code, but there will be portions of the code used to create this indicator in the breakdown that can be used to help you recreate this indicator for yourself.

If you are a Sierra Chart user and you want to use this indicator, you will need to add all three files to the ACS_Source folder and then build/compile the indicator using all three files in the built in build tool in SC. I am using my own headers and helper function files to create indicators, so it isn't all contained in a single file. This way, I can eventually create and entire library of custom indicators without having to type code more than once.

Intraday Volatility Bands

The Intraday Volatility Bands identify significant price movements by creating dynamic boundaries that adapt to the time of day. The core idea is that a price move should be compared to what's typically expected at that specific time in the trading session. Below is the core formula for this indicator1:

and

1. The Main Volatility Formula (σ):

   • σ (sigma) represents the historical volatility

   • It's taking a sum of squared differences between returns and their mean

   • The formula uses 14 days of data (hence the 14 in denominator)

   • The square root (√) is used to get back to the original units of measurement

2. The Mean Return Formula (μ):

   • μ (mu) represents the average return over the period

   • It's a simple arithmetic mean of returns over 14 days

   • Each ret₍ₜ₋ᵢ₎ represents the return from a previous day

We calculate the volatility by first finding the average return over the past 14 days. Then for each day, we subtract this average from the actual return, square the difference, sum up all these squared differences, divide by 14, and take the square root. This gives us a measure of how much returns typically deviate from their average.

This volatility measure then determines how wide our bands should be around the price. Higher volatility = wider bands, lower volatility = narrower bands.

Basic Calculation Logic

For any given time during the trading day (let's say 9:45 AM), we:

1. Look back at the same time (9:45 AM) on previous days for our lookback period

2. Calculate how much price typically moves from the open to that time

3. Use this average movement (σ) to create upper and lower boundaries

4. Adjust these boundaries if there was a gap at today's open

For each of those historical bars, it calculates the percentage move from that day's gap-adjusted open (max of that day's open or previous close) to the price at 9:45. Then it averages all those moves to get σ (sigma), which represents the typical percentage move you might expect at that specific time of day.

Gap Handling

The bands handle gaps using a simple max/min approach:

• Upper Boundary Base = max(today's open, yesterday's close)

• Lower Boundary Base = min(today's open, yesterday's close)

Then the volatility multiplier is applied to these bases:

• Upper = upperBase × (1 + multiplier × σ)

• Lower = lowerBase × (1 - multiplier × σ)

This approach automatically adjusts the noise area for gaps by anchoring one band to the previous close when appropriate, without needing explicit gap percentage calculations.

Implementation Details

Let's break down how this is implemented in the code:

1. Session Management

First, we determine if we're in an active trading session. If we are not in session, we will carry forward the previous values and skip calculations. The code below also handles overnight sessions, so we can turn off the chart session times being used and define our own time window if we want to.

bool isInSession;
if (MarketOpenSeconds > MarketCloseSeconds) {
    // Overnight session handling
    isInSession = (currentTimeSeconds >= MarketOpenSeconds || 
                  currentTimeSeconds <= MarketCloseSeconds);
} else {
    // Regular session handling
    isInSession = (currentTimeSeconds >= MarketOpenSeconds && 
                  currentTimeSeconds <= MarketCloseSeconds);
}

2. Historical Volatility (σ) Calculation

For each historical day, we:

• Find the same time of day

• Calculate the return from that day's effective open

• Calculate the mean of these returns

• Calculate the standard deviation using squared deviations from the mean

float sumReturns = 0.0f;
float returns[60];  // Assuming max lookback is 60 days
int validDays = 0;

// First pass - collect returns and calculate mean
for (int day = 1; day <= LookbackDays; day++) {
    // Get historical date
    SCDateTime HistoricalDate = sc.BaseDateTimeIn[sc.Index];
    HistoricalDate.SubtractDays(day);
    
    // Get session start and effective open
    SCDateTime SessionStart = sc.GetTradingDayStartDateTimeOfBar(HistoricalDate);
    float histOpen = sc.Open[SessionStartIndex];
    float histPrevClose = (SessionStartIndex > 0 ? 
                          sc.Close[SessionStartIndex - 1] : histOpen);
    float histEffectiveOpen = (histOpen > histPrevClose ? 
                              histOpen : histPrevClose);
    
    // Calculate historical return
    float historicalReturn = (sc.Close[targetBarIndex] / histEffectiveOpen) - 1;
    returns[validDays] = historicalReturn;
    sumReturns += historicalReturn;
    validDays++;
}

float meanReturn = sumReturns / validDays;
float sumSquaredDeviations = 0.0f;

// Calculate squared deviations from mean
for (int i = 0; i < validDays; i++) {
    float deviation = returns[i] - meanReturn;
    sumSquaredDeviations += deviation * deviation;
}

float sigma = sqrt(sumSquaredDeviations / validDays);

3. Band Calculation With Gap Adjustments

Finally, we calculate today's bands with appropriate gap adjustments:

float todayOpen = sc.Open[TodayStartIndex]; 
float prevClose = (TodayStartIndex > 0 ? sc.Close[TodayStartIndex - 1] : todayOpen); 

float effectiveUpperBasis = max(todayOpen, prevClose); 
float effectiveLowerBasis = min(todayOpen, prevClose);

float upperMultiplier = (1 + VolatilityMultiplier * sigma); 
float lowerMultiplier = (1 - VolatilityMultiplier * sigma); 

UpperBoundary[sc.Index] = effectiveUpperBasis * upperMultiplier; 
LowerBoundary[sc.Index] = effectiveLowerBasis * lowerMultiplier;

Key Implementation Features

1. Update Intervals: Bands only update at specified intervals to reduce noise

2. Gap Adjustment: One band gets "anchored" to the previous close during gaps

3. Effective Open: Uses the higher of today's open or previous close as the base

4. Data Validation: Includes checks for valid data and proper session times

Usage

The resulting bands create a "noise area" that:

• Adapts to normal volatility patterns for each time of day

• Accounts for overnight gaps

• Updates at controlled intervals

• Provides clear boundaries for significant moves

When price moves outside these bands, it signals a potentially significant trend that has broken out of the normal volatility pattern for that time of day. At least, that is the idea.

Complete Code

The rest of this post is for paid subscribers. Paid subscribers will have access to all paid publications, strategies, code associated with posts, and access to the HGT chat room. Red Team members will have access to the community GitHub and more.

Read more

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

This strategy is based on the research found in references Maróy's1 & Zarattini's2. Both of them discuss the same strategy, but reference 1 is the newer paper that attempts to expand on and improve the original strategy. The purpose of this test is to see if there is an efficacy to the strategy idea. Maróy's paper uses several different methods to improve on the original paper, but it is highly optimized and only has results on 1 instrument in a short time period for back testing results. With that in mind, my goal is to recreate the logic from the papers and then see if we can make it more robust and applicable to more instruments.

Before testing this logic on the 1-minute or 1-second chart (used in Maróy's paper), I want to do a fast and dirty check of the logic using a daily historical chart. The purpose is to create a simple daily strategy that uses the same concepts from the paper to see if there is any reason to continue down to intraday time periods for further investigation.

So, let's summarize the main points of the paper and create a strategy in RealTest (RT) that has similar logic. We will use QQQ for this test, since that is the test the most recent paper decided to focus on. I will use historical returns to create the volatility bands and then set the strategy to take positions only if a bar crosses the upper or lower band. I will also set the strategy use the actual fill price of the trade to place our limit exit orders. This will ensure that orders look more like what you would see with a true intraday strategy.

Setting QtyPrice: FillPrice technically introduces a "lookahead bias" in RT. Since RT operates on EOD data, and doesn't handle true streaming data throughout the day, it has to “look forward” to test this type of system. That means that this version has some inherent problems, if you were to try and use this strategy to actually auto trade a strategy. It can't know what price the instrument is going to open at so it can't generate accurate child/bracket orders if you try and use this strategy as is. That is perfectly fine for what I am doing with this strategy (and the reason it's freely available to all readers), but it should not be traded exactly how it is. At least, it shouldn't be traded with RT as it is.

Here's a quick breakdown of the logic I am applying for this test:

• Instrument being tested is QQQ

• Dates match the paper: 10/01/2014 to 10/01/2025

• Allow optimizable parameters for volatility multipliers and lookback periods separately for long and short strategies.

• Calculate historical volatility bands.

  • This is where I do something a little different. Since the paper uses an intraday chart to calculated historical volatility by time of day, I can't match that logic. Instead, I just measure volatility with historical daily returns.

  • I also use logarithmic returns here, instead of simple returns. When I recreate the intraday version, I will use both methods to see if we can make this strategy more robust. In our case here, since this is just a rapid idea test, using logarithmic returns should be fine. Feel free to experiment with simple returns if that is more your vibe.

  • I use a regular standard deviation calculation along with the volatility multipliers to create the upper and lower bands.

• Use target volatility and historical volatility to calculate dynamic entry sizes.

• Set a simple drawdown-based stop for the strategy.

• Set exit targets based on percent gain and take a partial position exit at target. Else, hold the position until close.

RealTest, EOD Test Code:

Notes: 	
	Based on a research paper.
	
Import:	// requires Norgate Platinum subscription
	DataSource:	Norgate
	IncludeList:	QQQ
	IncludeList:	SPY
	StartDate:	1/1/2014
	EndDate:	Latest
	SaveAs:	27_QQQ.rtd
	
Settings:	
	DataFile: 	27_QQQ.rtd
	StartDate: 	10/01/2014
	EndDate: 	10/01/2024
	BarSize: 	Daily
	AccountSize:	1e5
	HolidayList:	us_auto
	
Parameters:
	lookback_long:	from 2 to 14 step 1 def 9	// 2 - 14 days are most effective according to paper
	lookback_short:	from 2 to 14 step 1 def 6
	volatility_mult_long:	from 0.5 to 1.5 step 0.1 def 1
	volatility_mult_short:	from 0.5 to 1.5 step 0.1 def 1.3
	target_volatility:	0.01  	// Target daily volatility (1%)
	risk:	0.2
		
Data:
	log_returns:	Log(Close / Close[1])
	
	// Long indicators
	historical_volatility_long:	StdDev(log_returns, lookback_long)
	upper_noise_band_long:	C * Exp(volatility_mult_long * historical_volatility_long)
	lower_noise_band_long:	C * Exp(-volatility_mult_long * historical_volatility_long)
	
	// Short indicators
	historical_volatility_short:	StdDev(log_returns, lookback_short)
	upper_noise_band_short:	C * Exp(volatility_mult_short * historical_volatility_short)
	lower_noise_band_short:	C * Exp(-volatility_mult_short * historical_volatility_short)
	
Library:
	dynamic_position_size_long:	(S.Equity * Min(1, target_volatility / historical_volatility_long)) / Close
	dynamic_position_size_short:	(S.Equity * Min(1, target_volatility / historical_volatility_short)) / Close
	
	drawdown_pct:	(S.MaxEquity - S.Equity) / S.MaxEquity	
	
Charts:
	upperNB:	{}	upper_noise_band_long[1]
	lowerNB:	{}	lower_noise_band_long[1]

Benchmark: buy_and_hold
	Side: 	Long
	EntrySetup: 	Symbol = $SPY
	ExitRule: 	Dividend  // reinvest

Strategy: strategy_x_long
	Allocation:	S.Equity
	QtyType:	Shares
	// This changes the exit limits and stops to "look ahead" and simulates the orders based on entry price instead of the predetermined entry order price.
	// This is for when price opens above the entry price. RT assumes entry on open if entry stop/limit is above/below the order price.
	// This behavior will cause some orders to exit immediately, if price gapped up/down far enough to have gone past the exit order price.
	QtyPrice:	FillPrice
	Commission:	Min(0.01 * FillValue, Max(0.005 * Shares, 1))	
	Side: 	Long
	Quantity:	dynamic_position_size_long
	EntrySetup:	InList(2) and drawdown_pct <= risk
	EntryStop:	upper_noise_band_long
	ExitLimit:	FillPrice * 1.01
	ExitLimitQty: 	Shares * 0.75
	ExitTime:	ThisClose
	ExitRule:	Select(BarsHeld == 0, "EOD Exit")
	
Strategy: strategy_x_short
	Allocation:	S.Equity
	QtyType:	Shares
	// QtyPrice changes the exit limits and stops to "look ahead" and simulates the orders based on entry price instead of the predetermined entry order price.
	// This is for when price opens above the entry price. RT assumes entry on open if entry stop/limit is above/below the order price.
	// This behavior will cause some orders to exit immediately, if price gapped up/down far enough to have gone past the exit order price.
	QtyPrice:	FillPrice
	Commission:	Min(0.01 * FillValue, Max(0.005 * Shares, 1))	
	Side: 	Short
	Quantity:	dynamic_position_size_short
	EntrySetup:	InList(2) and drawdown_pct <= risk
	EntryStop: 	lower_noise_band_short
	ExitLimit:	FillPrice * 0.98
	ExitLimitQty: 	Shares * 0.75
	ExitTime:	ThisClose
	ExitRule:	Select(BarsHeld == 0, "EOD Exit")

Results:

The results aren't terrible, right? Test 2 shows the results for the strategy if we remove the exit logic and limits. There isn't much of a difference, but it does help the drawdown to simulate taking partial profits intraday.

What happens next?

I am going to move over to Sierra Chart and turn this into a true intraday strategy. First, I will create the indicator (gotta do a little C++ judo to set it up right). Then, I will create the actual strategy from the research papers we are looking at and see if I can get close to the same results.

And before I talk about what comes after that, I want to point out a few things about these papers. As far as I can tell, neither of them has been published in a traditional peer-reviewed journal. Maróy's paper doesn't have any academic affiliation (just the author's personal e-mail) and Zarattini's is published in a journal, but not a traditional peer-reviewed journal. The earlier paper also has affiliations with private trading and research firms. These facts already give me pause, but it doesn't mean there isn't some sort of edge here.

I think it's also important to point the very obvious problems that we know are present in the logic being presented to us in these papers. First, the strategies only focus on 1 instrument. This means they can easily be overfit. Second, the most recent paper uses a large amount of optimization to get the results it is presenting. It even points out that the results for the strategy are better on QQQ, so they are only focusing on QQQ and not SPY, like the original paper. That is basically saying (in the paper, none the less) that this strategy is overfit to a specific market/instrument.

We don't need to reproduce the strategy to know that it has some shortcomings. The papers practically admit their own faults while presenting their findings. Oddly enough, I don't believe that any of this means that you couldn't use this strategy to make money. I think it needs some real work, to go into the portfolio, but I am hopeful that it's possible.

Conclusion

Thanks for reading and hopefully you are excited about seeing what this strategy turns into in the end. The next post will cover creating the intraday volatility bands indicator. After that, we will dive into creating trying to recreate the strategy results from the paper. Finally, we will end with my attempt at updating the strategy so that it is more reliable. Ideally, I would like to be able to use something like this with ES. The idea is sound, I think.

Anyway, until the next one...

Happy hunting!

Feel free to comment below or e-mail me if you need help with anything, wish to criticize, or have thoughts on improvements. You can also DM me or start a chat thread. Red Team members can access this code and more at the private HGT GitHub repo. As always, this newsletter represents refined versions of my research notes. That means these notes are plastic. There could be mistakes or better ways to accomplish what I am trying to do. Nothing is perfect, and I always look for ways to improve my techniques.

References:

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

A reader reached out to me and asked me a couple of questions about Strategy 6. When I went back and looked at it, I realized that they were right to ask questions. This strategy was one of the earlier ones, and thus didn't get updated as I learned better ways to script strategies. So, I decided to fix the issues and see if that strategy was still usable.

During this process, I discovered that the long variation of the strategy was lackluster. I removed it and focused on the short side of the strategy since it still showed promise. I corrected some of the main issues (ranking and checking index status) and adjusted some of the calculations to use more reliable measures (such as log returns). The strategy looks like it could be a promising addition to a portfolio looking for some short-term short exposure.

Check out the results. The strategy is capped at using 25k (the starting account balance) each time it makes a trade. Meaning, there is no compounding here. This is to help make it more realistic, because incredibly large short positions on these stocks are unlikely.

Important Notes about this Strategy:

• This strategy is a short-term strategy that shorts instruments in the Russel 2000. I have had several people advise me that it is difficult to trade strategies like this (especially with EOD data and simple entries) and get the same results in the real world. I trust their advice, since they have history trading these types of strategies and I do not.

• Russel 2000 stocks can be illiquid and difficult to short (not as many stocks available).

• Lump sum limit entries will be difficult to fill during volatility spikes.

• This version uses EOD data to get an idea about the efficacy of the idea. It makes the assumption that the entire position will get filled at the predetermined limit price should price hit that order. Because of this, and the aforementioned issues with shorting stocks in this index, this strategy is meant to be a framework for further investigation on an intraday chart. It would likely benefit from scaled, dynamic entries that take advantage of volatility spikes and liquidity.

• I make my best effort to limit issues within the strategy criteria, but it should be understood that taken exactly as is, this strategy has some holes that I plan to try and address with an intraday platform (rapidly scaling into positions).

Strategy 6_RUT

This strategy is still the same concept as the original strategy 6, but with upgrades and dropping of the long variation. It uses volatility and overbought conditions to time short-term mean reversion opportunities in small cap, low price stocks. The concept is that these stocks will see heightened volatility short-term and then revert back to the mean. The goal is to capitalize on those days and short the stock after the move.

You will notice that this strategy does use a custom indicator in its calculations. I will attach that custom code here as well. This is just a custom ATR (average true range) that calculates either a regular ATR or log ATR with simple smoothing. Code is attached below. You can change this in the strategy if you wish to use a different version of ATR.

The rest of this post is for paid subscribers. Paid subscribers will have access to all paid publications, strategies, code associated with posts, and access to the HGT chat room. Red Team members will have access to the community GitHub and more.

Read more

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

I use continuous contracts when backtesting futures in RealTest. The data comes from Norgate. They also have individual contracts. In RealTest, you can import the individual contracts and manage the rollover logic yourself. I believe you can also use CSI futures data and manage the rollover logic in their software, but I don't use CSI data.

I tend to use continuous contracts in my backtesting because it is the fastest way to model and test out ideas. Since I day trade futures (ES, specifically), I actually use the strategies I built in RealTest as a scanner. I'll generate orders and see what the orders are generated to help me get a "lean" on the day. Since the tests are ran assuming buy at the open and exit at close, I can use the accuracy percentages of each strategy to gauge the strength of the edge.

However, if you want to actually generate accurate orders for trading these strategies live (or automating the process), you will want to use the individual contracts. This also gives you more control of the rollover logic. If that is what you want to do, I am going to show you how I have done this in the past.

First, the results of both variations on the most recent strategy (26_NQ).

As you can see, it has very similar results.

I will show how to do this witch strategy 26 below the paywall, but for free subsribers, I am going to provide snippets in case you want to implement this. Before you get started, make sure you have a watchlist of the idnividual contracts that you want to trade and import them into RealTest. I use Norgate for this so it is very simple.

Import:

DataSource:	Norgate
IncludeList:	.@NQ_Individual_Contracts	{"NQ"}

We need to define our rollover logic. Below is an example of a simple volume based rollover and a check to make sure we aren't trading the outgoing contract during triple witching.

Data:

// Contract Rollover
turnover_min:	5e4
lowest_turnover:	Lowest(Extra ,5)
turnover_ma:	MA(Extra, 5)
DaysToExp:	Days(BarDate, InfoExpiry, True)
TripleWitching:	DaysToExp < 5
liquid:	lowest_turnover > turnover_min and not TripleWitching
	
MktRank:	#Rank #ByMkt If(liquid and inlist("MicroES"), turnover_ma, -1)

We can use the Library section to store a constant variable that checks the market rank to determine if we can hold the contract per the logic defined about

Library:

CanHold: 	MktRank = 1

From there, you can use the CanHold boolean to check for the correct contract in the Strategy section.

Strategy:

EntrySetup: 	long and CanHold

Too easy. If you are using indicators with long lookbacks, it would be wise to import a second list with the continuous contracts that you are looking to trade as well. That way you can use them to calculate whatever indicators you need.

Code

The rest of this post is for paid subscribers. Paid subscribers will have access to all paid publications, strategies, code associated with posts, and access to the HGT chat room. Red Team members will have access to the community GitHub and more.

Read more

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

Important!

This strategy has been updated. When I was working on adding this strategy to a portfolio project, I ran across an error I made in the short logic. I accidentally had it set to take longs instead of shorts. When I fixed this, it changed the results. The short portion of the strategy only accounted for about 1/5 of the trades. The new results for just the long side are posted directly below this paragraph. The strategy code has been updated for the paid subscribers. Sorry for the mistake!

I have been doing a lot of experimenting with creating custom indicators for RealTest (RT) using C. Just a few weeks ago, I didn't even know that this was something that could be done in RT. Now, I am working on converting all of my favorite indicators to C so I can program them in a way that works better for my brain. Of course, this requires that I learn some C code along the way. You don't have to though. If you are interested (and a paid subscriber), the code will be provided for you to experiment with or use.

Below are the results from this strategy. You can see longs and shorts tested separately and in-sample/out-of-sample results.

Stats for the entire strategy are below. I am updating my subdomain for strategy results, so bare with me for a couple of weeks until I get that updated. Once updated, all strategy results can be found there again.

Chaikin's Money Flow (CMF), C-style

I have come full circle with this indicator. I learned about it and it's logic from "Statistically Sound Indicators" by Timothy Masters. In that book, the code examples provided are in C. So, I took that logic and translated it into Python (infinitely easier to learn and work with), and tested it from there. Then, I took the calculation logic and figured out how to create it using RT script. I got annoyed with creating indicators in RT Script and at some point in my questioning on the forum, I was instructed to look at the RTDLL directory inside the RT install directory. There, I discovered that I can actually code an indicator in C and call it from inside RealTest. The following is the first indicator I completed and tested.

It's important to mention that RT exposes a header file with pointers you can use to design your indicators. It works on a single pass principle, meaning that the indicators designed this way will work with 1 array of data at a time unless you actively pass in different arrays of data with the function call (more on this in future posts). In this indicator, we actually need to calculate and store a rolling window of data to make further calculations. I attempted to comment this as best I could.

The rest of this post is for paid subscribers. Paid subscribers will have access to all paid publications, strategies, code associated with posts, and access to the HGT chat room. Red Team members will have access to the community GitHub and more.

Read more

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

This is a practice in controlled word vomit, folks. I am just talking about things that I either did or thought about this week while trading and researching.

Best trade of the week

I've been working on my day trading game recently and thought it would be cool to review any good trades (or good, bad trades) I made during the week. I only got to trade 3 days this week, so I don't have too many candidates for good trades, but I did take a nice ten point move in ES. As you can see from the screenshot below, I don't use too much in the way of indicators and my technical analysis is very simple.

Most of the day (Friday), all I did was watch. This moved here came near lunchtime and was an easy setup. The day had been trending up since early morning, so I was looking for a pullback into an area of interest. In this case, just a recent low. I only use the RSI indicator you see below as extra confirmation and the ATR is just there so I know how to place my first target if I am trading multiple contracts. The blue represents the high of day.

That's it. Just two lines.

It's been a decent week for me, despite only trading 3 days. Or maybe it's been a good week because I only traded 3 days? Either way, it doesn't matter. Just gotta keep refining the process.

Strategy 9

While working on setting up my trading environments and getting ready to start forward testing some of the HGT strategies, I went ahead and got Strategy 9 running in a small account I have. If you don't recall what Strategy 9 is, check out the link below:

Strategy 9 is a rotational Nasdaq 100 strategy that we created back in September. We tested the strategy from 2007 up to 07/12/2024. It had a CAGR of 22%, Max Drawdown of -20%, a 62% win rate, and a profit factor of 2 and Sharpe of 1.21.

If we run the backtest up to December (giving up almost six months of true OOS data), we get these results:

As you can see, it had a drawdown after the summer and has since then been slowly grinding it's way out. It looks like it was probably just a hair away from triggering the drawdown filter. Regardless, I decided to start trading this strategy live to see how it does. It rebalances every Friday, so I figured I could keep a rolling update of it's performance here.

This strategy took it's first positions on Monday (01/13/2025). We close this week out at X%...

I took the positions on Monday well after open, which is not how it is supposed to be done. The orders are meant to be placed on open. I will make sure that happens going forward.

Here are the trades that I took Monday:

And here is the performance from this week:

Not bad, for the first week. Hopefully I can create a better way to track this strategy, so I don't have to take screenshots of IBKR's desktop application...

Important Note: I am not going to be providing any “signals”. The update for 9 is going to come on a weekly basis AFTER the trade week. What you see are orders generated for this week. New orders are generated after close on Friday. If you want to see what it generates each week, you will need to get the strategy and do it at your own risk.

Platform overload

Remember when I said, "all trading platforms suck"? I was only kidding a little bit. Like, a tiny little bit. A negligible amount, really. It seems like a big sweeping generalization, but I think it's one of the more accurate ones. Want to know why I feel this way? Sure you do.

Time to vent a little (cracks knuckles, positions keyboard).

Why doesn't IBKR offer direct API access (to non-institutional customers)? Why does IBGateway and TWS take so much fucking memory (between 750MB to 1.5G in my experience) to run in the background doing nothing? It's almost like they really just want you running their software on your desktop as often as possible. Why?

Why do good data providers (Norgate, CSI) require you use their software to download the data? This goes back to APIs. I just want to access the data when I need to and then I can store and backup local copies on my own without having to worry about yet another program running in the background all day long.

Why do other traders seem to focus on the shit that doesn't matter? I know that IBKR has super cheap data. But you get what you pay for. Also, streaming data is the one part of a platform that requires the most work to make sure it works well. When I was trading equities, I used a program called DAS Trader Pro to connect to my IBKR account. This was the smoothest shit ever. I literally only used it for direct access order routing (look it up) and high-speed, efficient streaming data. Both of these where offerings from the platform. The only thing it did with IBKR was connect to my account to place orders. And it did it WITHOUT the TWS or IBGateway software. I wonder how they managed to do that...

Why are people happy with Python performance? Do you know how long it would take to backtest a strategy across all SP500 constituents for the last 20 years? It might only take an hour to create the backtesting engine, but it will also take the better part of the hour to pre-calculate all your extra values and loop through the history to perform the test. No thanks. I would rather take the 10 years to learn C++... <-- Jokes.

Ok, I'm done. Moving on.

My Current Platform list (absolutely subject to change, and hopefully soon)

I have been doing a lot of messing around with different platforms this week. I can't begin to explain how much it annoys me that there isn't a single platform that does everything I need it to do. Right now, the platforms I am having to deal with are:

• Interactive Brokers (IBKR) TWS or IBGateway

  • There is no good option here. Both of these products kind of suck a lot of ass. They use excessive resources just to run in the background and IBKR doesn't offer direct API access unless you are institutional customer.

• Rithmic Trader Pro

  • This is used for trading futures prop firm accounts. There is no getting around it. Oddly, I don't have any issues with this software. It's simple and I only use it to monitor trades and orders when I'm day trading.

• Sierra Chart

  • This is the software I chart and day trade from. Really, I don't have anything bad to say about it. It's a bit harder to get the hang of in the beginning, but it does everything you need it to do for an all-in-one platform. If I had to keep only one platform, this would be it.

• RealTest

  • This is for researching and backtesting EOD strategies. When it finally gets intraday capability, I will use it for that too. I don't think there is anything better that handles multiple strategies and multiple instrument universes better. This one stays.

• OrderClerk

  • This software comes with RealTest and creates a bridge between RealTest and IBKR. It places orders to IBKR via CSV and tracks trades from IBKR to feed accurate trade lists back to RealTest so it generates accurate order sizes for any strategy that you are trading live. It's pretty dope, but it still requires the localhost gateway API's through IBKR's TWS or IBGateway, the latter of which is mother fucker to work with for this purpose due to trade history request limitations.

• Norgate Data

  • This is for good EOD data. For now, this is what I use. I don't like having to use their software to get their data, but the software is pretty simple and does everything you need it to do without consuming too many system resources. Still, I would like direct API access.

• TradeStation

  • Shocking, I know. I just recently opened up a tiny account with TradeStation so I could learn how to make strategies and indicators in EasyLanguage (EL). I don't imagine that I will be doing any trading or running any automations through this platform for myself. I have it mostly for any prospective clients or subscribers who want something in EL.

That's a lot of different platforms just to research and day trade, right? And not a single one does everything I would like to see in a single platform. Sierra Chart comes hella close, but it lacks good connections for trading stocks. It connects to IBKR, but like every third-party connection to IBKR, this is only a half-assed solution that requires use of their software as a middleman to connect to their API.

As you can tell, I am quite frustrated with IBKR's lack of a modern API. I am stuck with them for now, since I am using them to test out Strategy 9. I plan on rectifying this situation by creating my own tool to submit CSV orders to IBKR. They have a small client portal option that can be used with docker to get access to the API without all the resource hogging (and God only knows what else). This solution will probably work until IBKR opens the API up to regular clients. Or maybe I will just switch to something like Alpaca, who knows.

Since I am currently day trading in a prop firm account (evaluation, I am testing if my day trading methodology lines up with their consistency rules), I can't avoid using Rithmic. Sierra Chart lets you run multiple instances of itself (so does RealTest), so it's fairly simple to just dedicate one install to the Rithmic data feed and leave it alone. For futures trading, this setup is just about perfect.

Actions to take

First, keep trading well.

Second, slim up this platform overload. I'm thinking I'll drop the IBKR software and OrderClerk and replace this with my own command line tool for submitting orders to IBKR (looking at Go for this). I will still have to use a gateway, but they have a tiny little java program that I can use in a docker image for this. It's better, but still not the perfect setup. For now, everything else will have to stay.

I want to drop Norgate by the time my renewal comes up in about 5 months, but I will need to really figure out how to handle certain things in RealTest, like historical constituency checks... ok, that might actually be all I have to solve. Then, I need to test backtest performance for the different data sources. I know data matters, but my hypothesis is that I don't see that much of a difference. We shall see.

TradeStation stays because it's just there for prospective clients (hit me up if you want something developed privately) and to learn EasyLanguage.

Sierra Chart is my favorite trading platform for active day trading right now. So, it ain't going anywhere. Rithmic is required for prop accounts, so I am stuck with it for the time being.

These actions would really just leave me with Sierra Chart, Rithmic, and RealTest (since TradeStation is just kind of there). That's way better than all the shit I currently have plaguing my machine. I know that these are first-world problems, but they are problem that irk me to no end. So, I just need to create my own solutions.

Learning C/C++

If you don't know, RealTest and Sierra Chart are both programs that are written in highly efficient C code. If you want to make any custom functions for either of them (bet you didn't know that you could do that with RealTest), it is a good idea to understand C. Which I don't, really. So, I am going to learn.

In order to learn this, I am hoping to create some research tools in C. They will probably be open source, but they will be oriented towards the developer/researcher. I do not intend to abstract anything away for the "user". I think this is the issue I am having with all of the proprietary platforms. It's about making it "easy" for the user. In order to make things easier for the user, certain things get sacrificed. Simple scripting languages are cool, but I would rather be able to use the full power of a programming language. I'm sure I fall into a very small group of users, but that's fine. The goal is to build something that can be built upon to suite whatever needs the researcher/trader has.

Conclusion

That's it for this week’s updates. Next week I am going to start breaking down a stupid complicated indicator and how I am using it in RealTest. Then, we will explore whether or not it is actually useful for trading by backtesting it's signals with some new strategies. Get ready for some silliness, because this indicator is a little complicated and it will require that I make some parts in C and leave some parts as scripts inside RealTest. I will not create this indicator in Python, because it would just take too long for it make its calculations across large universes.

Happy hunting.

Feel free to comment below or e-mail me if you need help with anything, wish to criticize, or have thoughts on improvements. You can also DM me or start a chat thread. Red Team members can access this code and more at the private HGT GitHub repo. As always, this newsletter represents refined versions of my research notes. That means these notes are plastic. There could be mistakes or better ways to accomplish what I am trying to do. Nothing is perfect, and I always look for ways to improve my techniques.