118 lines
4.7 KiB
Markdown
118 lines
4.7 KiB
Markdown
# Training Dataset Design
|
|
|
|
## Initial Goal
|
|
|
|
Build a supervised learning dataset from stored IBKR daily candle data. The first dataset predicts whether `SPY` closes higher five trading days after the observation date.
|
|
|
|
The first implementation is [../notebooks/spy_direction_dataset.ipynb](../notebooks/spy_direction_dataset.ipynb), a Python notebook using pandas. This keeps the feature calculations inspectable while the dataset design is still changing. Once the feature contract settles, reusable loading and feature-building code can move into `src/trading_bot/models` or `src/trading_bot/data`.
|
|
|
|
## Raw Data
|
|
|
|
Raw daily candles are stored under:
|
|
|
|
```text
|
|
data/ibkr/daily
|
|
```
|
|
|
|
Storage expectations:
|
|
|
|
- one Parquet file per symbol;
|
|
- file name is the symbol, such as `SPY.parquet`;
|
|
- each row contains one daily candle for that symbol;
|
|
- expected columns are `date`, `symbol`, `open`, `high`, `low`, `close`, and `volume`.
|
|
|
|
The initial dataset requires at least these symbols:
|
|
|
|
- `SPY`;
|
|
- `VIX`;
|
|
- `TLT`;
|
|
- `USO`.
|
|
|
|
## Observation Rows
|
|
|
|
Each dataset row represents one trading date. Feature values are derived from data available on or before that date. The target is derived from `SPY` closing price five trading days after that date.
|
|
|
|
Rows should be dropped when any required feature or target value cannot be computed.
|
|
|
|
## Input Features
|
|
|
|
| Feature | Description |
|
|
| --- | --- |
|
|
| `SPY_ret_5` | `SPY` fractional return over the previous 5 trading days. |
|
|
| `SPY_ret_20` | `SPY` fractional return over the previous 20 trading days. |
|
|
| `SPY_dist_sma50` | `SPY` distance from its 50 trading day simple moving average, divided by that average. |
|
|
| `VIX_change_5` | `VIX` fractional change over the previous 5 trading days. |
|
|
| `VIX_rank_20` | `VIX` percentile rank within a rolling 20 trading day window. |
|
|
| `TLT_ret_10` | `TLT` fractional return over the previous 10 trading days. |
|
|
| `USO_ret_5` | `USO` fractional return over the previous 5 trading days. |
|
|
| `SPY_TLT_ratio_ret` | Fractional 5 trading day change in the `SPY` to `TLT` close-price ratio. |
|
|
|
|
Fractional percentage changes should follow pandas `pct_change` convention. For example, a five percent return is represented as `0.05`.
|
|
|
|
## Output Target
|
|
|
|
The target column is a binary indicator of `SPY` forward return over the next five trading days.
|
|
|
|
| Target | Description |
|
|
| --- | --- |
|
|
| `spy_up_5d` | `1.0` when `SPY` closes above today's close five trading days later; `0.0` when `SPY` closes below today's close five trading days later; `0.5` when the future close equals today's close. |
|
|
|
|
Using `0.5` for unchanged prices preserves the row while making the target explicitly neutral.
|
|
|
|
## Data Alignment
|
|
|
|
The dataset should use trading-day alignment rather than calendar-day alignment. A five day lookback or forecast horizon means five available trading sessions for the relevant symbol, not five calendar days.
|
|
|
|
`SPY` should define the primary observation calendar because the first target is an `SPY` forward-return label. Other symbols should be joined to the `SPY` observation dates after their own features are computed.
|
|
|
|
## Notebook Feature Sketch
|
|
|
|
The first notebook should use pandas transformations close to this shape:
|
|
|
|
```python
|
|
import numpy as np
|
|
import pandas as pd
|
|
|
|
df["SPY_ret_5"] = df["SPY_close"].pct_change(5)
|
|
df["SPY_ret_20"] = df["SPY_close"].pct_change(20)
|
|
|
|
sma_50 = df["SPY_close"].rolling(50).mean()
|
|
df["SPY_dist_sma50"] = (df["SPY_close"] - sma_50) / sma_50
|
|
|
|
df["VIX_change_5"] = df["VIX_close"].pct_change(5)
|
|
df["VIX_rank_20"] = df["VIX_close"].rolling(20).rank(pct=True)
|
|
|
|
df["TLT_ret_10"] = df["TLT_close"].pct_change(10)
|
|
df["USO_ret_5"] = df["USO_close"].pct_change(5)
|
|
df["SPY_TLT_ratio_ret"] = (df["SPY_close"] / df["TLT_close"]).pct_change(5)
|
|
|
|
spy_forward_close = df["SPY_close"].shift(-5)
|
|
df["spy_up_5d"] = np.select(
|
|
[
|
|
spy_forward_close > df["SPY_close"],
|
|
spy_forward_close < df["SPY_close"],
|
|
],
|
|
[1.0, 0.0],
|
|
default=0.5,
|
|
)
|
|
|
|
df_model = df.dropna().copy()
|
|
```
|
|
|
|
## Dataset Splits
|
|
|
|
Training, validation, and test splits should be chronological:
|
|
|
|
- oldest rows for training;
|
|
- newer rows for validation;
|
|
- newest rows for the final test set.
|
|
|
|
A split indicator column such as `split` is useful in the dataset artifact for auditability and reproducibility. It should be treated as metadata, not as a model input feature. The model training code should build `X` from the explicit feature column list and exclude metadata columns such as `date`, `split`, raw close prices, and the target.
|
|
|
|
## Open Decisions
|
|
|
|
- exact adjusted versus unadjusted close handling;
|
|
- whether same-day `VIX`, `TLT`, and `USO` values are acceptable for the intended trading decision timing;
|
|
- exact train, validation, and test date boundaries or split percentages;
|
|
- output dataset file location, schema metadata, and versioning.
|