The answer is sometimes and for some equities.
“Some equities” because individual stocks are too volatile and affected by too many semi-random events (eg, the CEO says something controversial one day and stock declines or some random post on reddit causes a meme stock to “go to the moon”).
Exchange Traded Funds (ETF) consist of a combination of many equities and, as a result, are typically less volatile compared to individual stocks. They can be traded just like stocks while the market is open (unlike mutual finds where the trade gets executed the next day). Therefore, ETFs seem to be a natural candidate for our prediction challenge.
I mentioned that prediction is possible sometimes because price prediction is a stochastic problem and, as such, cannot be predicted with 100% certainty. However, to make a profit in the stock market one does not need to be right all the time; you just need to be right more than wrong.
Let’s try to illustrate this with an example. Suppose you are invited to play a game of flipping coins. In this game you get to bet $X and a “head” outcome earns you $X while a “tail” event costs you $X. The good news is that the coin is biased by 60% in favor of “head” events (ie, 60% of times the outcome is “head”). This seems like a money making machine if you play it right.
One greedy approach would be to spend a significant portion of your money at each round. Doing so can cause you to go bankrupt quickly as there is 40% chance that you lose a significant portion of your wealth at each trial and eventually everything is lost and the game stops. To take advantage of the biased coin, you must play in such a way that allows you to play the game as much as possible so that the average win probability approaches 60%. Playing the game only a small number of times does not guarantee that you actually win 60% of times.
In statistics, this is explained by Central Limit Theorem which asserts that the sample mean approaches to mean of the underlaying random variable with increasing certainty as sample size grows.
So, to be able to profit, one needs an algorithm that can predict future values of an ETF with reasonable accuracy and an opportunity to trade using that algorithm enough times to make the algorithm that is biased towards winning work.
How to Predict
Machine learning has been used extensively for predicting future values of a sequence given its historical data. This article explains such techniques in reasonable detail. The predictor is referred to as Recurrent Neural Network (RNN) which is a particular type of Neural Network with properties that are suitable for sequences (eg, historical stock values or historical temperature values in a specific region).
Essentially an RNN predictor is trained on many samples of historical values for a sequence such that the error between the output of the predictor and the actual future value (which is known during training) is minimized.
Fortunately, for the purpose of predicting future prices of ETF, there is additional information available to us which may lead to more accurate predictions. Specifically, instead of using just the historical values of the ETF, we can also provide historical values of various indices (eg, Dow Jones, S&P 500, Russel 1000, etc). As long as there is some correlation (positive or negative) between the extra information and the ETF of interest, the RNN predictor can make use of them to improve prediction accuracy. The following image shows an abstract representation of information that can be used as input to the predictor. Each string represents historical values associated with one of the inputs.
Relationship to Technical Analysis
It is likely that you have come across a series of techniques referred to as Technical Analysis (TA) for predicting the direction of stocks and other equities based on past values. In that sense, TA techniques strive to achieve the same goal as we are attempting to do using RNN predictors. One salient difference is that TA techniques tend to be simple and intuitive
For example, consider one popular TA method called Bollinger Bands. This method computes moving average of the price certain standard deviations above and below the actual price. When the price breaches an upper or lower band, it could indicate an overbought or oversold condition according to this TA method. An example is shown in the following image.
This method is certainly simple and intuitive. But intuition may not be enough for a complex problem such as price prediction. TA methods tend to analyze single-dimensional information which we as humans can understand and visualize. We cannot visualize more than three dimensions and cannot reason about and find patters in higher dimensions intuitively.
Neural networks are very efficient at processing and finding patterns in higher dimensions. That is one of the reasons why RNN predictors are suitable for sequence prediction.
Neural networks minimize the error between the true value (eg, future price) and the predicted value by iterating over the training data and improving the prediction after each iteration. An example of such process is shown in the following image. The vertical axis is the average prediction error and the horizontal axis is the iteration number.
Does it work: Back-Testing
To train a neural network, historical data is divided into training set and test set (there is also another important validation set that for sake of brevity we will not describe here). Separating the data into disjoint training and test set is important to make sure the model does not overfit (ie, work really well for the data that is used to train it but not for new data that the model has not seen before). Also, the test set allows us to perform back-testing: the process of determining how the model performs on data that it has not seen. For example you can train the model on data from years 2000 through 2015 and then ask it to perform prediction on data from year 2024.
In the following graph we show price of QQQ (a popular ETF) from mid 2022 to early 2025 together with the prediction values. The prediction and actual values are quite close to each other.
To see the deviations, we can examine the zoomed-in versin of the same graph.
Quantitatively, the prediction falls within 1% of the actual values 96% of the time.
Another way to quantify the prediction error of a model is to use the RMSPE metric (Root Mean Squared Percentage Error). This metric measures the distance between a model’s predicted values and the actual values in the dataset (a.k.a. the “residual” or prediction error) . The lower the score, the better the predictive performance of the model. The following is the definition of this metric.
The RMSPE in our QQQ example (shown in the figure above) is 0.15% which is quite small (smaller is better).
Sample Prediction Results
The following graph shows an example of prediction results for QQQ. The horizontal axis is date and each dor represents a day. It is important to point out that the model attempts to predict the moving average value of the price. This removes the noise associated with high frequency fluctuations. The orange dots represent the future value predictions of the moving average in the next few days. The blue graph is the actual moving average. The gray band shows the 1% band above and below the prediction (for visualization purposes).
Value Metric
It would be useful to use the model to assign a normalized metric to each ETF that represents how undervalued/overvalued that ETF is. This is similar to the RSI metric (Relative Strength Index) in Technical Analysis only in terms of representation but, obviously, obtained based on out RNN model and not based on TA.
One way to achieve this is to pass the difference between the prediction and the current price to a sigmoid function and use the output as a Value Metric (VM). The following is the formula for the sigmoid function. In practice, we use 1 minus the sigmoid to compute the VM.
Buy/Sell Opportunities
We can then use the value metric to come up with but or sell opportunities notifications. For example, an ETF is undervalued/overvalued if the VM is below/above a certain threshold.
How to use the Value Metric
As we pointed out at the beginning of this article, no prediction algorithm is perfect (in particular, the ones that attempt to predict future price of equities). But we strive to make the prediction as accurate as possible. It is similar to our biased coin example where we hope that by using our algorithm we have a better handle on “buying low and selling high”.
The algorithm computes a normalized metric that indicates how overvalued or undervalued an ETF is. So, we can use this knowledge to buy an ETF when the VM for the ETF is below a certain threshold (eg, 20%).
Once we have purchased an ETF, when do we sell? Depending on your style of investment you may adopt one of the following strategies:
- You can sell the ETF when the associated VM approaches ~50%. This means that the algorithm predict that in the near future (eg, a week), the average price will not change much compared to the current price. Alternatively, you can wait until the associated VM approaches a higher threshold (eg, 80%) which indicates that the algorithm predicts that average short-term future price is going to drop. Technically, VM above 50% indicates a future drop.
- If you are a type of investor that prefers to “buy and hold” and the ETF is in question is one that you typically invest in periodically, you may use the buy indications (eg, VM below 20%) as opportunity to add to your position.
- Finally, you can also sell if the price drops below your personal risk threshold or goes above a personal percentage for profit taking.
FundOracle App
I have packaged all this in a simple and easy-to-use app called FundOracle.
The app includes models for more than 60 popular ETFs and updates every 30 minutes (when the market is open). The user receives notifications for buy and sell opportunities based on the criteria described in the previous sections.
The app even provides a Home Widget so that you can see the top undervalued/overvalued ETFs at a glance without launching the app.
