# Quant Forecast Lab: scoring an LSTM against a naive baseline

> The repository started as a 2020 LSTM price forecasting experiment and is now a research lab that treats one question as primary, which is whether a model has out-of-sample skill beyond the assumption that tomorrow's price equals today's. That is a stricter and more useful goal than a chart that tracks a price series, and it comes with a cost in complexity and in what the project is willing to claim.

**JordiCorbilla/stock-prediction-deep-neural-learning** — Predicting stock prices using a TensorFlow LSTM (long short-term memory) neural network for times series forecasting

- Repository: https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning
- Website: https://jordicorbilla.github.io/stock-prediction-deep-neural-learning/
- Stars: 688 · Forks: 127
- Language: Python
- License: Apache-2.0
- Published: 2026-10-21 · Updated: 2026-10-21 · Language: en
- Canonical page: https://hysenlabs.com/projects/jordicorbilla-stock-prediction-deep-neural-learning

## The stated goal is to beat assuming no change

The README states the objective directly, and the wording is unusual enough to quote: the objective is not to make a chart that visually tracks a price series. The objective is to measure whether a model has out-of-sample forecasting skill beyond simple baselines, whether uncertainty is calibrated, and what remains after realistic trading frictions.

That reframing is the project. Financial price levels are persistent, so a forecast can look excellent on a chart while failing to beat the assumption that tomorrow's price equals today's price. The evaluation layer therefore reports RMSE skill as one minus the model RMSE divided by the naive RMSE, and a positive value means the model beats the last-observation baseline on RMSE for the evaluated sample.

The repository history explains why the emphasis is there. It began as an LSTM stock-price forecasting experiment in 2020, and the current architecture keeps those historical entry points compatible while evolving the project into a more rigorous quantitative forecasting laboratory. The old scripts still run; what was added is the measurement apparatus.

For a reader, this is the difference between a demo and an evaluation. Most forecasting repositories show a fitted line over a price chart, and that picture is close to worthless as evidence, because a model with too much capacity can fit the noise in a price series and still have no skill. Scoring against a naive forecast on data the model never saw is a much less flattering test, and it is the test this project is built around.

## Walk-forward splits and directional accuracy, not a single train/test cut

The evaluation primitives are listed in the highlights and they are worth reading as a set, because they describe the whole discipline rather than one metric.

There is training, validation and test separation for model fitting and evaluation, and expanding-window walk-forward split primitives, which means the model is refit as the window moves rather than being trained once on a prefix and tested on a suffix. For a series where the data generating process changes, that difference decides whether your test result means anything.

Directional accuracy is reported alongside RMSE, which answers a different question from the error in the price level. A model can have poor RMSE and still call the sign of the next move correctly, or have a small RMSE because it mostly predicts level while never calling direction. Reporting both stops either number from being quoted alone.

The naive benchmark is a first-class component rather than an afterthought, since the RMSE skill score is meaningless without a denominator. The README also insists that published benchmark results belong under the benchmarks directory and should never be generated from training observations or a hand-picked single run. That is an unusual instruction in a research repository, and it is the one that keeps the headline numbers honest, because the failure mode in this field is a good run being reported as the result.

The uncertainty side is where the project is careful about its own claims. One-step held-out residual bands exist, and they are explicitly labelled as unvalidated for recursive horizons, which is exactly the distinction most often glossed over when a model is asked to forecast many steps ahead.

## Three model versions, one of them opt-in because it failed its own test

The model history is documented and the naming is versioned rather than hidden.

The compatibility default is v7, which does direction and magnitude forecasting with recursive future inference. The cleanup keeps v7 as the default so existing users do not silently receive different model semantics, and the compatibility CLI still exposes it with a -model_version flag.

v8 is the shared multi-task architecture, in which one temporal representation feeds two heads, one predicting the direction of the move and one predicting its magnitude, rather than training two separate models. The README shows that shape as an ASCII diagram: the input flows into stacked LSTM layers and then branches to the direction output and the magnitude output.

v9 is the most recent and the most interesting, because of how it is presented. The v9 notebook trains a shared LSTM with return, direction and ordered quantile heads, and it saves the model, the scalers, the input-data hash, the test metrics, the future forecast and the inference settings, with the future forecast including independent raw and anchored recursive paths. The quantiles and the two forecast paths are the machinery for calibrated uncertainty rather than a single point estimate.

But the README says the model remains opt-in because the current held-out FTSE run did not beat the zero-return baseline on RMSE, and points to the model semantics document. A project that keeps a model available while stating that it failed its own benchmark, and that makes it opt-in rather than the default, is behaving very differently from a repository whose headline result is the best run of a profitable configuration.

## The arena scores saved predictions and charges for transaction costs

The two command line entry points are worth understanding separately, because they answer different questions.

The benchmark command takes an aligned CSV of actuals and predictions and reports RMSE, MAE, directional accuracy and RMSE skill against the previous-price naive baseline. The input format is a header row naming the actual column and the prediction columns, then one row per date, and the point is that it scores predictions you already have rather than training anything. Two prediction columns can be scored in the same run, so a baseline and a candidate can be compared on identical rows.

The arena command does the same for multiple assets or horizons, grouping by columns such as Ticker and Horizon, and it takes an explicit cost assumption in basis points and an output path for a self-contained HTML report. The report table includes RMSE and MAE, directional accuracy, RMSE skill against the naive forecast, annualised directional-strategy return and volatility, a Sharpe-like diagnostic, maximum drawdown, average turnover, and the explicit transaction cost assumption.

Two design decisions in that command are the interesting part. The arena consumes predictions that are already out of sample rather than training models inside the evaluator, which keeps model fitting and evaluation cleanly separated. And it reports turnover and cost, so a strategy that wins on directional accuracy but trades constantly can be seen to lose after costs.

The cost number is an assumption you supply, not something inferred from your fills, so the report is only as honest as that single input. And the diagnostic is described as a Sharpe-like measure, which is a reminder that the arithmetic resembles a Sharpe ratio without being one.

## Two installs, and a package that deliberately omits the old scripts

There are two documented setup paths, one for conda and one for a virtual environment.

```bash
conda env create -f environment.yml
conda activate stock-prediction
```

```bash
python -m venv .venv
python -m pip install -r requirements.txt
python -m pip install -e .
```

The dependency set is pinned tightly where it matters. The pyproject requires Python 3.11 or newer and below 3.13, and pins tensorflow at 2.18.1, yfinance at 1.0 and exchange-calendars at 4.13.2, with numpy, pandas, scikit-learn, matplotlib, h5py and absl-py alongside. The exchange calendar pin is a detail worth noticing, since it is what makes a trading day grid consistent across assets.

The split between what you install and what you run from a checkout is stated explicitly. The published package installs the quant-forecast command line tool and the library, and it does not bundle the root scripts or the historical notebooks. So the stock_prediction_*.py compatibility scripts and the notebooks are run from a source checkout, while the packaged artefact is the research utility layer.

Development installs add the optional extras.

```bash
python -m pip install -e ".[dev]" --no-deps
python -m pip install -r requirements.txt
pytest
ruff check quant_forecast_lab tests tests_runtime
```

The unit tests live in tests, the runtime tests in tests_runtime, and ruff is configured with a line length of 120 and a rule set of E, F, I, B and UP.

## A compatibility-first migration that changes your numbers

The repository layout shows how the refactor was done. The maintained, testable research utilities sit in a quant_forecast_lab package with modules named arena, backtest, benchmark, config, evaluation, experiment, reproducibility, uncertainty and validation. Alongside them are the stock_prediction_*.py compatibility entry points, the tests, a docs directory with ARCHITECTURE.md and METHODOLOGY.md, an examples directory with notebooks, and the packaging files pyproject.toml and environment.yml.

The README calls the migration deliberately compatibility-first, with stable research components introduced around the historical scripts rather than breaking old commands. That is the right instinct for a research codebase, and it comes with a documented cost: new training runs fit scalers only on the chronological fit window, so their numerical outputs can differ from earlier runs. In other words, the same command line and the same seed can produce different numbers from the version you ran last month, because the fitting window changed rather than the seed.

Reproducibility is treated as a spectrum rather than a promise. Experiment seeds and runtime metadata are recorded, and the README says exact numeric reproducibility can vary by hardware and backend. That is an honest statement about floating point and backend differences, but it also means the reproducibility module is a provenance record, not a guarantee of bit-identical results.

A first training run uses the compatibility model with the flags the CLI documents.

```bash
python stock_prediction_deep_learning.py \
  -ticker=GOOG \
  -start_date=2017-11-01 \
  -validation_date=2022-09-01 \
  -epochs=150 \
  -batch_size=32 \
  -time-steps=30
  -use_returns=false \
  -model_version=v7 \
  -forecast_horizon=1 \
  -seed=42
```

The v8 run uses the same flags with -model_version=v8, and the shared multi-task model is opt-in rather than the default.

## Apache 2.0, recent releases, and a research tool rather than a strategy

The licence is Apache-2.0, with the LICENSE file at the top of the repository, the same value in the pyproject metadata and the badge in the README. The copyright in the pyproject lists Jordi Corbilla as the author.

Releases have come quickly. The list shows v2.0 on 2026-09-28, and v1.0 and v0.5 both on 2025-12-28, with the last push on 2026-10-10 and the repository not archived. A CHANGELOG.md and a CITATION.cff sit at the top level, so the project supports citation of the work, which is the convention for anything published as a result.

The honest scope statement is that this is a laboratory for measuring forecasting skill. It trains models, evaluates them out of sample, prices the frictions, and refuses to promote a model that did not beat its baseline. What it does not provide is a trading system, a broker connection, a position sizing policy or any claim that a strategy built on it will make money. The most likely misreading of this repository is to treat the presence of a directional strategy return and a drawdown figure in the arena report as a result rather than as a diagnostic of a supplied prediction file.

The alternative approach, and the one most people start from, is a single notebook that downloads prices with yfinance, fits an LSTM, and plots the fit against the series. The difference is not the model, it is the measurement. A notebook has no out-of-sample split to report skill against, no naive denominator, no transaction cost assumption, and no record that a published number came from a representative run rather than the best one. The laboratory supplies the parts that make the claim checkable, at the cost of a package, a CLI, a CSV format and a slower learning curve.

## Conclusion

Quant Forecast Lab fits a quant who wants to know whether a forecasting model is actually better than assuming no change, and it does not fit a developer looking for a strategy to trade or a chart that looks convincing. Before trusting any result from it, run the benchmark command on your own saved predictions and read the skill figure, then check docs/MODELS.md for the models that are explicitly marked as not beating their baseline.

## FAQ

### How do I install the stock prediction deep learning lab?

Either create the conda environment with conda env create -f environment.yml and conda activate stock-prediction, or make a virtual environment and run python -m pip install -r requirements.txt followed by python -m pip install -e . The project requires Python 3.11 or newer and below 3.13.

### What is RMSE skill in this project and why does it matter?

The evaluation layer reports RMSE skill as one minus the model RMSE divided by the naive RMSE, where the naive forecast is that tomorrow's price equals today's price. A positive value means the model beats the last-observation baseline on RMSE for the evaluated sample, which is why a forecast that merely tracks the price series is not treated as a success.

### Which model versions are available and which is the default?

v7 is the compatibility default with direction and magnitude forecasting and recursive future inference, v8 is a shared multi-task LSTM for direction and magnitude, and v9 is a notebook with return, direction and ordered quantile heads. v9 stays opt-in because the held-out FTSE run did not beat the zero-return baseline on RMSE.

### How do I benchmark saved predictions with quant-forecast?

Run quant-forecast benchmark with a CSV that has an actual column and one or more prediction columns, for example quant-forecast benchmark --csv results.csv --actual-col Actual --prediction-col LSTM. The output includes RMSE, MAE, directional accuracy and RMSE skill against the previous-price naive baseline.

### Does the published package include the old notebooks and scripts?

No. The README states that the published package installs the quant-forecast CLI and library and does not bundle the root stock_prediction_*.py scripts or the historical notebooks, which are run from a source checkout.

## Sources

- [JordiCorbilla/stock-prediction-deep-neural-learning on GitHub](https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning)
- [License: Apache-2.0](https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/blob/master/LICENSE)
- [Project website](https://jordicorbilla.github.io/stock-prediction-deep-neural-learning/)
- [README](https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/blob/master/README.md)
- [Releases](https://github.com/JordiCorbilla/stock-prediction-deep-neural-learning/releases)

---

Hysen Labs editorial analysis, written from the project's own repository and release notes. Cite the canonical page: https://hysenlabs.com/projects/jordicorbilla-stock-prediction-deep-neural-learning