Documentation update

docs/training-dataset.md

@@ -2,16 +2,16 @@
 
 ## Initial Goal
 
-Build a supervised learning dataset from stored IBKR daily candle data. The first dataset predicts whether `SPY` closes higher five trading days after the observation date.
+Build a supervised learning dataset from stored Alpaca daily candle data. The first dataset predicts whether `SPY` closes higher five trading days after the observation date.
 
-The first implementation is [../notebooks/spy_direction_dataset.ipynb](../notebooks/spy_direction_dataset.ipynb), a Python notebook using pandas. This keeps the feature calculations inspectable while the dataset design is still changing. Once the feature contract settles, reusable loading and feature-building code can move into `src/trading_bot/models` or `src/trading_bot/data`.
+The current implementation is `src/trading_bot/data/train_pipeline.py`, which combines the original dataset notebook and XGBoost training notebook into one runnable script. The earlier notebook, [../notebooks/spy_direction_dataset.ipynb](../notebooks/spy_direction_dataset.ipynb), remains useful background for inspecting the initial feature design.
 
 ## Raw Data
 
 Raw daily candles are stored under:
 
 ```text
-data/ibkr/daily
+data/alpaca/daily
 ```
 
 Storage expectations:
@@ -21,10 +21,10 @@ Storage expectations:
 - each row contains one daily candle for that symbol;
 - expected columns are `date`, `symbol`, `open`, `high`, `low`, `close`, and `volume`.
 
-The initial dataset requires at least these symbols:
+The initial dataset requires at least these logical inputs:
 
 - `SPY`;
-- `VIX`;
+- volatility proxy, currently `VIXY` in `config/train_config.json`;
 - `TLT`;
 - `USO`.
 
@@ -57,7 +57,7 @@ The target column is a binary indicator of `SPY` forward return over the next fi
 | --- | --- |
 | `spy_up_5d` | `1.0` when `SPY` closes above today's close five trading days later; `0.0` when `SPY` closes below today's close five trading days later; `0.5` when the future close equals today's close. |
 
-Using `0.5` for unchanged prices preserves the row while making the target explicitly neutral.
+The original dataset design used `0.5` for unchanged prices as an explicit neutral target. The current training pipeline drops unchanged `0.5` rows before training so the XGBoost model remains a binary classifier.
 
 ## Data Alignment
 
@@ -109,9 +109,24 @@ Training, validation, and test splits should be chronological:
 
 A split indicator column such as `split` is useful in the dataset artifact for auditability and reproducibility. It should be treated as metadata, not as a model input feature. The model training code should build `X` from the explicit feature column list and exclude metadata columns such as `date`, `split`, raw close prices, and the target.
 
+## Training Pipeline
+
+Run the current pipeline from the repository root:
+
+```sh
+mise exec -- uv run python src/trading_bot/data/train_pipeline.py
+```
+
+The script reads configuration from `config/train_config.json`, writes the dataset to `data/training/spy_direction_5d.parquet`, and saves model artifacts to:
+
+- `models/spy_xgb_v1.json`;
+- `models/spy_xgb_v1_meta.json`.
+
+The metadata file stores the training base probability, feature column list, last trained date, and model configuration.
+
 ## Open Decisions
 
 - exact adjusted versus unadjusted close handling;
-- whether same-day `VIX`, `TLT`, and `USO` values are acceptable for the intended trading decision timing;
+- whether same-day volatility proxy, `TLT`, and `USO` values are acceptable for the intended trading decision timing;
 - exact train, validation, and test date boundaries or split percentages;
-- output dataset file location, schema metadata, and versioning.
+- output dataset schema metadata and versioning.