Measuring Linear Trend -- The statistically sound way

There is big math in this one. Don't let it scare you. You've got this.

Aug 21, 2024

∙ Paid

Disclaimer: the following post is an organized representation of my research and project notes. It doesn’t represent any type of advice, financial or otherwise. Its purpose is to be informative and educational. Backtest results are based on historical data, not real-time data. There is no guarantee that these hypothetical results will continue in the future. Day trading is extremely risky, and I do not suggest running any of these strategies live.

In the last post I discussed an impromptu indicator that came across my radar. When someone asked me about looking at a discussion about how to implement the indicator, I thought it might be a good time for a dry run at indicator testing. I had an indicator that didn't have any previous code and there appeared to be an interest in it. What better way is there to test my current ability than to test myself against an unknown problem. This worked out well enough that I published the fruits of that test. In the end, I ended up adding two new functions to the pqt library and I learned about vectorization, which is going to help (re)shape the way the I code my functions when working with arrays (data). Not a bad two days’ work.

In this post, I am going to discuss the linear trend indicator. This indicator was not as simple as the last one. I mean, I guess it’s simple if you are a statistician/mathematician. The general concepts are relatively simple in that world. There is nothing bleeding edge here. But I'm not either of those things and this one was chock full of shit I had to try and wrap my nugget around in a short period of time.

We're talking about learning terminology like "curvilinear" and "polynomials of the second order". I don't need to be able to whiteboard these concepts from memory, but I do need to know what is going on if I am going to try and hack together some Python code to create this indicator. This indicator measures trend by calculating a best fit line over a lookback period using a least squares regression technique and several normalization measures for volatility and scaling.

I will do my best to briefly (and simply) explain the concepts necessary to understand how we are calculating this indicator and why we do it this way. Like the indicators before it, I will leave the plain English calculation open for all subscribers. The idea is that you can still create this indicator from the concepts and formulas, as long as you have a little bit of skill or know how to navigate an LLM. Whichever route you prefer.

The concepts for this indicator come from the book "Statistically Sound Indicators for Financial Market Prediction" by Timothy Masters. I attempt to take the concepts he discusses in his book and turn it into an indicator/function in Python that can be used for trade system research. I also attempt to explain the concepts in a more knuckle-dragger friendly manner, just for you.

And me.

Mostly for me.

I suggest grabbing a coffee or an adult beverage (author's choice: a gin martini, extra olive) before getting into this one. It is going to a be a little bit lengthy. We will have several general concepts to touch on our way to the test results and coding section. I do not have a variation of this strategy in RealTest yet, but once I do, it will be added to the list on GitHub.

Cheers!

Addendum:

I wrote the intro to this post last Friday. This post was originally intended to be published on Friday. I lost two days working on the QP indicator from the previous post, but I thought that this one would be easy enough to research and code in a day or two, maybe push the article publication date back to Saturday or Sunday...

It's now Wednesday, and we are finally publishing.

This indicator was way more complex than I thought it was going to be. It probably isn't that complex for someone well-versed in statistical math and linear regression. I am not, so there was a lot of reading and researching to make sure I grasped the basic concepts of what was going on both mathematically and in the code. This indicator happens to use a basic linear regression technique to create a trend line, but it executes that equation in a novel way, making it harder to understand with almost zero knowledge about the subject.

On top of that subtle complexity, I also had a couple of changes to make in the pqt library. The mutual information functionality seemed to produce odd results in the last test. This time it was producing zeros. Debugging that issue led to refactoring the entire report suite and half of the indicators in the library. All parallel processes in the code were treated and converted to use Python's built-in multiprocessing capability and not an external dependency. Report functions were also cleaned up and parallelization was applied to multiple chunks of code to help speed up the process.

Many of the indicators and all of the report functions had unnecessary looping functionality removed and replaced with vectorization techniques that should speed up the calculations. The next step will be to apply parallelization to the loops that are still required, assuming there are any. Most of the indicator functionality works without explicit loops, which is nice.

I think that's it. Back to the post.

Oh. I almost forgot. I won’t judge you for going “adult beverage” as your drink choice for this article, but I did write that part in the afternoon on Friday. I just wanted to make sure you didn’t think that I was enjoying a dry gin martini at nine o’clock in the morning. It’s a bloody Mary. Just kidding, its coffee and water for me in the mornings.

Measuring Linear Trend

Trend is a term thrown around often in the trading community. Everyone knows what it means, but we all have a different way of measuring it when it comes to trading. For systematic or quantitative trading, we want to use measures of trend that are going to be statistically sound and machine friendly. Linear trend is a great (and mostly simple) method for doing just that.

Linear trend is, in the most basic sense, the slope of a line that is optimally centered.[1] By itself, the slope has some flaws that need to be addressed.

The line (slope) should be scaled by the price activity within the lookback window in which it is fit.
The line should also be scaled by a broader, more long-term measure of market behavior.

To illustrate the first problem, look at the following two arrays of close prices:

array_A = [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110]
array_B = [100, 102, 95, 110, 120, 100, 103, 99, 109, 112, 110]
Copy

In this example, you could trust the slope of array_a to signify a real trend since it goes up by the same increment each iteration. array_b is much more erratic and should have a negligible positive trend. However, just calculating a simple slope would not account for this erratic behavior.

The second problem is illustrated with the following two arrays:

array_A = [100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110]
array_B = [1000, 1001, 1002, 1003, 1004, 1005, 1006, 1007, 1008, 1009, 1010]
Copy

Here, array_a represents the price of the same instrument at an earlier time frame. It can also represent the price a different market/instrument all together. Either way, the linear trend line would need to be scaled to account for this. It is common to see people use fractional returns (percent returns) as a way to normalize this across time or across markets. However, there is an inherent problem with using regular percent returns. The return from 100 to 105 (a move of 5 points) is not the same as a move from 1000 to 1005. The former is 5% and the latter 0.5%.

We can solve the second problem using Logarithmic Returns. Log returns handle the problem with scaling and the natural relationship between scale and numerical volatility.[1] It is a simple and robust solution for problem two. The returns are even across the board and make measuring them consistent across time and markets.

To solve the first problem, we need to consider two things. We need to consider the volatility within the lookback window and we need to look at a much broader market volatility.

For the latter, we will use the instruments ATR across a long time frame, much longer than the lookback window of the linear trend. Using a long period for calculating the ATR helps us capture a more general market volatility, as opposed to just the activity within the lookback window. In order to achieve this, the lookback window should be several hundred bars. I tend to default to 252, which is about a year in trading days.

To solve the problem with volatility within the lookback window (array example 1), we will need to calculate the (r-squared) of the slope line to determine the fit. If the fit is high, like the very first array example, then the line will be adjusted very little. If the data is not fit, like the second array example, then the trend line will be adjusted accordingly.

The method we are going to use to calculate this slope is ordinary least squares regression.

Least Squares Regression [2]

Least squares regression (or least-squares estimation of the parameters), is one of the most basic linear regression models. It is a model with a single regressor that has a relationship with a response . The simple linear regression model looks like this:

where the intercept and the slope are unknown constants and is a random error component.

Least-squares is used to estimate and , meaning that they are estimated so the sum of the squares of the differences between the observations and the straight line is minimal. The least-squares criterion is:

This formula could be used in the way it is written, but instead we will precompute a vector containing the first-order Legendre polynomial (the linear one) over the span of the trend lookback and then get the dot product of this vector with a sequence of log prices.[1] The dot product is a mathematical operation that multiplies corresponding elements of two vectors and sums the results, providing a scalar value that represents the degree of alignment between the vectors.[3] In the context of trend calculation, the dot product is used to combine the log-transformed prices with the coefficients from Legendre polynomials, producing an initial estimate of the best-fit line or curve over a specified lookback period.

Here is a breakdown of how we will accomplish this:

Log Returns: First, we calculate the log of the closing prices.
Polynomials: Then, we use the lookback length to calculated the first, second, and third-order orthogonal polynomial coefficients.
Dot Product (dot_prod): This calculation represents the initial estimate of the trend (best-fit line or curve) over the specified lookback period. It combines the log-transformed prices with the coefficients from the Legendre polynomials. This is the raw trend measurement.
Normalize w/ ATR: We normalize the dot product with a long period ATR that has been adjusted for the total change over the lookback period.
Residuals: After calculating the best-fit line/curve using the dot product, the function computes the residuals, which are the differences between the actual log prices and the predicted log prices from the trend.
R-squared (rsq): The R-squared value is then calculated to assess how well the trend fits the data. It represents the proportion of the variance in the dependent variable that is predictable from the independent variable(s). This step normalizes the output, ensuring that poor fits (higher residuals) reduce the impact of the trend on the final output.
Normalization and Final Adjustment: The output value is then adjusted by multiplying it by the R-squared value. This step degrades the impact of the trend if the fit is poor (i.e., if R-squared is low).
Compression: The output is further compressed with a Normal CDF function.

Legendre Polynomials[5]

Before we can move on to the tests and code, we need to cover what the polynomials are and how they are being used. From what I can tell, the methods that Masters' uses in his book are slightly different than the traditional equations used to calculate the first, second, and third-order polynomials. The purpose of these changes is to make sure that polynomials are orthogonal to one another and that they are scaled so the vector has unit length. [1]. I am still not 100% confident in my understanding of this, but I am going to give it a shot.

Polynomials are used in situations where an analyst knows that there are curvilinear properties present in the response function.[1] In financial market analysis, the relationship between a predictor (independent variable, indicator) and the target (dependent variable, price data) is not a consistent relationship (making it "curvilinear"). When a predictor moves, the target price does not change with it in a consistent manner. Polynomials are able to model complex nonlinear relationships (such as those we see in financial market analysis) over small ranges and make them ideal when trying to measure trend in a financial instrument.

Legendre polynomials are particularly useful because they are orthogonal over the interval [−1,1], meaning the dot product of any two different polynomials in this set is zero. This property makes them ideal for regression, as they prevent multicollinearity (correlated predictors) when modeling complex trends. In the legendre_3 function, the coefficients generated for these polynomials are used to calculate how much a given set of prices aligns with these specific trends, allowing for the analysis of linear, quadratic, or cubic patterns in financial data.

Visualizations and Test Results

The generated report provides a comprehensive analysis of trading indicators, offering insights into their statistical properties, predictive power, mean stability over time, and optimal thresholds for profitability. It combines detailed statistical summaries, mutual information scores, mean break tests, and profit factor evaluations.

Simple Statistics and Relative Entropy Report

The Simple Statistics Table summarizes key metrics for each trading indicator, including the number of cases, mean, minimum, maximum, interquartile range (IQR), range/IQR ratio, and relative entropy. In the table, a lower range/IQR ratio suggests a tighter, more predictable dataset, while an optimal relative entropy indicates a balance of diversity and uniqueness without excessive noise.

Ncases: Number of cases (bars) in feature (indicator).

Mean: Average value of the feature across all cases.

Min/Max: The minimum and maximum value of the feature across all cases.

IQR: Interquartile Range, measures range minus the top and bottom 25% of the raw range.

Range/IQR: A unitless measure of data dispersion relative to its middle 50%.

Relative Entropy: Measures the difference between two probability distributions; a value of zero indicates identical distributions.

Mutual Information Report

High MI scores indicate a strong relationship between the indicator and the target variable, suggesting potential predictive power. Low p-values further validate the indicator's statistical significance.

MI Score: Measures the mutual dependence between the feature and the target.

Solo p-value: Initial significance estimate, proportion of permuted MI scores equal to or higher than the original MI scores.

Unbiased p-value: Adjusted solo p-value considering the number of permutations plus one, reducing bias.

Serial Correlated Mean Break Test Report

The Serial Correlated Mean Break Test Report identifies potential breaks in the mean of each trading indicator, taking into account serial correlation. This test helps detect significant shifts in the mean over time, indicating nonstationary behavior in the data.

nrecent: The number of recent observations considered in the test.

z(U): The greatest break encountered in the mean across the user-specified range.

Solo p-value: Measures the significance of the greatest break while accounting for the entire range of boundaries searched. If this value is not small, it suggests that the indicator does not have a significant mean break.

Unbiased p-value: Adjusted solo p-value considering the number of permutations plus one, reducing bias.

Optimal Thresholds w/ Profit Factor Report

The Optimal Thresholds w/ Profit Factor Report evaluates various threshold levels for trading indicators to identify the most profitable long and short positions. The report includes the fraction of data points greater than or equal to the threshold, the corresponding profit factor for long and short positions, and the fraction of data points less than the threshold with their respective profit factors. The optimal thresholds at the bottom indicate the threshold levels with the highest profit factors for long and short positions, while the p-values provide statistical significance for these thresholds.

The rest of this post is for paid subscribers. Paid subscribers will have access to the private GitHub where all the Hunt Gather Trade code and strategies live. It is continuously being worked on and will expand as we explore the world of automated trade systems.

References:

Masters, T. (2020). Statistically sound indicators for financial market prediction.
Montgomery, D. C., Peck, E. A., & Vining, G. G. (2021). Introduction to linear regression analysis (6th ed.). Wiley.
Dot product - Wikipedia
numpy.dot — NumPy v2.0 Manual
Legendre polynomials - Wikipedia

Hunt Gather Trade