jarno/trading-bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Documentation update

Browse Source
This commit is contained in:
jarno
2026-08-11 20:23:59 +03:00
parent 900b70d6df
commit c1c02e2b0d
7 changed files with 110 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+1 -1
View File
@@ -2,7 +2,7 @@
## Project Purpose
This repository is for a Python trading bot that uses machine learning to decide which assets should be held. The bot will use the IBKR API to facilitate trades, starting with paper trading and allowing a carefully controlled path to real trading later.
This repository is for a Python trading bot that uses machine learning to decide which assets should be held. The bot uses the Alpaca Markets API for market data and paper trading, with any future live trading behind a carefully controlled path.
## Current Direction
README.md
+38 -7
View File
@@ -1,10 +1,10 @@
# Trading Bot
Python trading bot project using machine learning to decide which assets should be held. The bot is intended to trade through the IBKR API, beginning with paper trading and potentially supporting real trades later behind explicit safeguards.
Python trading bot project using machine learning to decide which assets should be held. The bot currently uses the Alpaca Markets API for market data and paper trading, with any future live trading kept behind explicit safeguards.
## Project Status
This repository is at the planning and scaffolding stage. The machine learning model, trading parameters, asset universe, and risk rules will be designed later.
This repository now has a working prototype flow for Alpaca market data, dataset generation, XGBoost training, prediction, and paper-trading rebalancing. The strategy is still intentionally narrow: it currently focuses on `SPY` exposure using a small market-regime feature set.
## Main Parts
@@ -29,22 +29,53 @@ The current local Python version is pinned in `.mise.toml`.
Runtime dependencies are declared in `pyproject.toml`.
- `ib-insync` for the IBKR API connection.
- `alpaca-py` for Alpaca market data and trading clients.
- `pandas` for tabular candle data handling.
- `pyarrow` for Parquet file support.
- `xgboost` and `scikit-learn` for model training and evaluation.
- `python-dotenv` for loading local Alpaca credentials from `.env`.
## Trading Safety
The default target is paper trading. Real trading should only be added later with explicit configuration, clear documentation, and tests around order generation and broker integration.
The default target is Alpaca paper trading. Real trading should only be added later with explicit configuration, clear documentation, and tests around order generation and broker integration.
Do not commit secrets such as IBKR credentials, account identifiers, API tokens, or private configuration.
Do not commit secrets such as Alpaca API keys, account identifiers, API tokens, or private configuration.
## Current Commands
Fetch Alpaca daily candles:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_alpaca_daily.py SPY
```
Build the training dataset and train the current model:
```sh
mise exec -- uv run python src/trading_bot/data/train_pipeline.py
```
Run the current Alpaca paper-trading rebalance flow:
```sh
mise exec -- uv run python src/trading_bot/models/trade.py \
--paper \
--fetch-recent-data \
--model-path models/spy_xgb_v1.json \
--metadata-path models/spy_xgb_v1_meta.json
```
The main Python entry points are:
- `src/trading_bot/data/train_pipeline.py` for dataset generation and model training.
- `src/trading_bot/models/trade.py` for Alpaca account inspection and SPY rebalancing.
## Documentation
See [docs/README.md](docs/README.md) for the initial architecture notes and decision log.
The first data collection design note is [docs/data-fetcher.md](docs/data-fetcher.md), and the first training dataset contract is [docs/training-dataset.md](docs/training-dataset.md).
The current data collection note is [docs/data-fetcher.md](docs/data-fetcher.md), and the training dataset contract is [docs/training-dataset.md](docs/training-dataset.md).
The first dataset generation notebook is [notebooks/spy_direction_dataset.ipynb](notebooks/spy_direction_dataset.ipynb).
Manual test instructions for the IBKR fetcher skeleton are in [docs/manual-test/README.md](docs/manual-test/README.md).
Manual test instructions for the Alpaca fetcher are in [docs/manual-test/README.md](docs/manual-test/README.md).
docs/README.md
+13 -9
View File
@@ -2,7 +2,7 @@
## Objective
Build a Python-based trading bot that uses machine learning to determine which assets should be held, then uses the IBKR API to facilitate trades. The first supported trading mode should be paper trading.
Build a Python-based trading bot that uses machine learning to determine which assets should be held, then uses Alpaca Markets for market data and broker execution. The first supported trading mode is paper trading.
## Planned Modules
@@ -10,9 +10,9 @@ Build a Python-based trading bot that uses machine learning to determine which a
Responsible for acquiring and storing market, asset, and any future feature data needed for model training and evaluation.
The first planned tool is an IBKR daily candle fetcher. It should fetch open, high, low, close, and volume data for a ticker and date range, then eventually persist that data to a ticker-named Parquet file. See [data-fetcher.md](data-fetcher.md).
The current data tool is an Alpaca daily candle fetcher. It fetches open, high, low, close, and volume data for a ticker and date range, then persists that data to a ticker-named Parquet file. See [data-fetcher.md](data-fetcher.md).
The initial supervised training dataset is documented in [training-dataset.md](training-dataset.md). It derives market-regime features from `SPY`, `VIX`, `TLT`, and `USO`, then labels whether `SPY` closes higher five trading days later.
The initial supervised training dataset is documented in [training-dataset.md](training-dataset.md). It derives market-regime features from `SPY`, a volatility proxy, `TLT`, and `USO`, then labels whether `SPY` closes higher five trading days later. The checked-in configuration currently maps the volatility input to `VIXY` for Alpaca data availability.
Parquet files are partitioned by ticker, not by date.
@@ -29,6 +29,8 @@ Open decisions:
Responsible for building datasets, training models, evaluating candidates, and writing versioned model artifacts.
The main entry point is `src/trading_bot/data/train_pipeline.py`. It reads raw Alpaca Parquet files from `data/alpaca/daily`, builds `data/training/spy_direction_5d.parquet`, trains an XGBoost classifier, and writes model artifacts to `models/`.
Open decisions:
- prediction target;
@@ -41,11 +43,13 @@ Open decisions:
### Trading Bot
Responsible for loading a model, generating portfolio signals, deciding target holdings, and using the IBKR API to place or simulate orders.
Responsible for loading a model, generating portfolio signals, deciding target holdings, and using the Alpaca trading API to place paper-trading orders.
The main entry point is `src/trading_bot/models/trade.py`. It loads a model and metadata, optionally refreshes recent Alpaca market data, estimates a target `SPY` exposure from the model probability, cancels open Alpaca orders, and submits a day market order when the desired rebalance exceeds the configured minimum notional amount. Pass `--model-path models/spy_xgb_v1.json --metadata-path models/spy_xgb_v1_meta.json` to trade with artifacts produced by the current training pipeline.
Initial expectations:
- paper trading first;
- Alpaca paper trading first;
- real trading later only behind explicit configuration;
- clear logging of model version, signals, target holdings, generated orders, and broker responses;
- separation between signal generation, portfolio construction, and broker execution.
@@ -56,7 +60,7 @@ Open decisions:
- position sizing;
- risk limits;
- cash handling;
- order types;
- order types and time-in-force choices;
- failed order handling;
- market hours behavior;
- manual override behavior.
@@ -79,9 +83,9 @@ Possible scope:
1. Decide the initial project package structure.
2. Keep Python packaging and dependency management current with `uv`.
3. Add a minimal configuration system.
4. Define interfaces for data collection, model artifacts, and broker execution.
5. Add tests for the core trading decision boundaries before connecting real broker behavior.
4. Harden interfaces for data collection, model artifacts, and Alpaca broker execution.
5. Expand tests around trading decision boundaries, portfolio sizing, stale data handling, and broker API boundaries.
## Decisions Deferred
The model, features, labels, asset selection rules, risk management rules, and trading cadence are intentionally deferred for later discussion.
Broader model design, asset selection rules, risk management rules, live-trading gates, and trading cadence are intentionally deferred for later discussion.
docs/data-fetcher.md
+11 -12
View File
@@ -2,7 +2,7 @@
## Initial Goal
Create a Python module/tool that fetches daily candlestick data from the IBKR API for a specific ticker and date range.
Create a Python module/tool that fetches daily candlestick data from the Alpaca Market Data API for a specific ticker and date range.
The intended example workflow is:
@@ -14,41 +14,40 @@ The intended example workflow is:
## Current Implementation
The first implementation is intentionally small:
The current implementation is intentionally small:
- ticker passed as a required command line argument;
- end date passed with `--end-date YYYYMMDD`, defaulting to yesterday;
- end date can also be derived with `--end-date-from-parquet`;
- duration passed with `--duration`, defaulting to `1 W`;
- hard-coded IBKR Gateway target: `127.0.0.1:4002`;
- uses the IBKR API through `ib_insync`;
- prints fetched candles as CSV-like rows;
- uses the Alpaca Market Data API through `alpaca-py`;
- reads credentials from `--api-key` / `--secret-key`, `ALPACA_API_KEY` / `ALPACA_SECRET_KEY`, or `.env`;
- writes candles to a symbol-named Parquet file;
- appends to an existing symbol file and keeps one row per date.
Run it with:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY
mise exec -- uv run python src/trading_bot/data/fetch_alpaca_daily.py SPY
```
To override the requested range:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date 20250605 --duration "1 M"
mise exec -- uv run python src/trading_bot/data/fetch_alpaca_daily.py SPY --end-date 20250605 --duration "1 M"
```
To fetch backward from the oldest date already stored in the symbol file:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date-from-parquet
mise exec -- uv run python src/trading_bot/data/fetch_alpaca_daily.py SPY --end-date-from-parquet
```
`--end-date` and `--end-date-from-parquet` cannot be used together. If the symbol Parquet file does not exist or has no rows, `--end-date-from-parquet` uses today's US/Eastern date.
This expects a local IBKR Gateway session to be running and accepting API connections on `127.0.0.1:4002`.
This expects Alpaca API credentials to be available via CLI arguments, environment variables, or `.env`.
By default, output is written to `data/ibkr/daily/SPY.parquet`. Use `--output-dir` to choose another directory.
By default, output is written to `data/alpaca/daily/SPY.parquet`. Use `--output-dir` to choose another directory.
Manual test instructions are in [manual-test/README.md](manual-test/README.md).
@@ -78,5 +77,5 @@ Candidate output schema:
Open decisions:
- how to handle adjusted versus unadjusted prices;
- how to handle missing sessions and IBKR pacing limits;
- whether to use `ib_insync` long term or a lower-level IBKR client wrapper.
- how to handle missing sessions, market holidays, and Alpaca API rate limits;
- whether to add explicit feed selection, adjustment settings, or data entitlement checks.
docs/manual-test/README.md
+23 -30
View File
@@ -1,30 +1,24 @@
# Manual Test Instructions
## IBKR Daily Fetcher
## Alpaca Daily Fetcher
This test checks the IBKR data fetcher and confirms it writes a symbol-named Parquet file.
This test checks the Alpaca data fetcher and confirms it writes a symbol-named Parquet file.
The fetcher currently requests:
- gateway: `127.0.0.1:4002`;
- client id: `101`;
- symbol: provided as a command line argument;
- end date: provided with `--end-date YYYYMMDD`, defaulting to yesterday;
- end date can also be derived with `--end-date-from-parquet`;
- duration: provided with `--duration`, defaulting to `1 W`;
- bar size: `1 day`;
- data type: `TRADES`;
- regular trading hours only;
- output: printed CSV-like rows and a Parquet file in `data/ibkr/daily/`.
- data source: Alpaca stock bars through `alpaca-py`;
- output: a Parquet file in `data/alpaca/daily/`.
## Prerequisites
1. Start IBKR Gateway.
2. Log in to the paper trading account.
3. Confirm API access is enabled in IBKR Gateway.
4. Confirm the API socket port is `4002`.
5. Confirm no other API client is already using client id `101`.
6. Sync Python dependencies:
1. Create or confirm Alpaca API credentials.
2. Provide credentials through `ALPACA_API_KEY` and `ALPACA_SECRET_KEY`, a local `.env`, or CLI arguments.
3. Sync Python dependencies:
```sh
mise exec -- uv sync
@@ -35,52 +29,51 @@ mise exec -- uv sync
From the repository root, run:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY
mise exec -- uv run python src/trading_bot/data/fetch_alpaca_daily.py SPY
```
To fetch a specific IBKR range, pass an end date and duration:
To fetch a specific Alpaca range, pass an end date and duration:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date 20250605 --duration "1 M"
mise exec -- uv run python src/trading_bot/data/fetch_alpaca_daily.py SPY --end-date 20250605 --duration "1 M"
```
To fetch backward from the oldest date already stored in `data/ibkr/daily/SPY.parquet`:
To fetch backward from the oldest date already stored in `data/alpaca/daily/SPY.parquet`:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date-from-parquet
mise exec -- uv run python src/trading_bot/data/fetch_alpaca_daily.py SPY --end-date-from-parquet
```
`--end-date` and `--end-date-from-parquet` cannot be used together. If the symbol Parquet file does not exist or has no rows, `--end-date-from-parquet` uses today's US/Eastern date.
## Expected Output
The tool should first print the request range and connection target:
The tool should first print the request range:
```text
Fetching SPY daily candles ending 2025-06-05 for duration 1 M
Connecting to IBKR Gateway at 127.0.0.1:4002 with client id 101
Fetching SPY daily candles ending 2025-06-05 for duration 1 M via Alpaca API
```
If the request succeeds, it should then print a header and one row per returned trading day:
If the request succeeds, it should print how many candles were fetched and where the merged Parquet file was written:
```text
date,symbol,open,high,low,close,volume
2025-06-05,SPY,...
Fetched 21 daily candles for SPY.
Wrote 21 total daily rows to data/alpaca/daily/SPY.parquet
```
Exact prices and volume depend on what IBKR returns.
Exact row counts depend on the requested date range and market calendar.
The tool should then write or update:
```text
data/ibkr/daily/SPY.parquet
data/alpaca/daily/SPY.parquet
```
If the Parquet file already exists, rows from the latest fetch are merged into it. The trading date is used as the row key, so a symbol file keeps only one row for each date.
## Common Issues
- Connection refused: IBKR Gateway is not running, the port is not `4002`, or API access is disabled.
- Client id already in use: change `IBKR_CLIENT_ID` in the fetcher or disconnect the other client.
- No historical bars: confirm the account has market data permissions and that IBKR accepts the requested historical data range.
- Pacing or permission errors: note the IBKR error message before changing the request.
- Missing credentials: set `ALPACA_API_KEY` and `ALPACA_SECRET_KEY` or pass them with CLI flags.
- No historical bars: confirm the symbol, requested range, and Alpaca market data permissions.
- Authentication or entitlement errors: confirm the keys belong to the intended Alpaca account and data plan.
- Rate-limit errors: wait before retrying or reduce repeated requests.
docs/training-dataset.md
+23 -8
View File
@@ -2,16 +2,16 @@
## Initial Goal
Build a supervised learning dataset from stored IBKR daily candle data. The first dataset predicts whether `SPY` closes higher five trading days after the observation date.
Build a supervised learning dataset from stored Alpaca daily candle data. The first dataset predicts whether `SPY` closes higher five trading days after the observation date.
The first implementation is [../notebooks/spy_direction_dataset.ipynb](../notebooks/spy_direction_dataset.ipynb), a Python notebook using pandas. This keeps the feature calculations inspectable while the dataset design is still changing. Once the feature contract settles, reusable loading and feature-building code can move into `src/trading_bot/models` or `src/trading_bot/data`.
The current implementation is `src/trading_bot/data/train_pipeline.py`, which combines the original dataset notebook and XGBoost training notebook into one runnable script. The earlier notebook, [../notebooks/spy_direction_dataset.ipynb](../notebooks/spy_direction_dataset.ipynb), remains useful background for inspecting the initial feature design.
## Raw Data
Raw daily candles are stored under:
```text
data/ibkr/daily
data/alpaca/daily
```
Storage expectations:
@@ -21,10 +21,10 @@ Storage expectations:
- each row contains one daily candle for that symbol;
- expected columns are `date`, `symbol`, `open`, `high`, `low`, `close`, and `volume`.
The initial dataset requires at least these symbols:
The initial dataset requires at least these logical inputs:
- `SPY`;
- `VIX`;
- volatility proxy, currently `VIXY` in `config/train_config.json`;
- `TLT`;
- `USO`.
@@ -57,7 +57,7 @@ The target column is a binary indicator of `SPY` forward return over the next fi
| --- | --- |
| `spy_up_5d` | `1.0` when `SPY` closes above today's close five trading days later; `0.0` when `SPY` closes below today's close five trading days later; `0.5` when the future close equals today's close. |
Using `0.5` for unchanged prices preserves the row while making the target explicitly neutral.
The original dataset design used `0.5` for unchanged prices as an explicit neutral target. The current training pipeline drops unchanged `0.5` rows before training so the XGBoost model remains a binary classifier.
## Data Alignment
@@ -109,9 +109,24 @@ Training, validation, and test splits should be chronological:
A split indicator column such as `split` is useful in the dataset artifact for auditability and reproducibility. It should be treated as metadata, not as a model input feature. The model training code should build `X` from the explicit feature column list and exclude metadata columns such as `date`, `split`, raw close prices, and the target.
## Training Pipeline
Run the current pipeline from the repository root:
```sh
mise exec -- uv run python src/trading_bot/data/train_pipeline.py
```
The script reads configuration from `config/train_config.json`, writes the dataset to `data/training/spy_direction_5d.parquet`, and saves model artifacts to:
- `models/spy_xgb_v1.json`;
- `models/spy_xgb_v1_meta.json`.
The metadata file stores the training base probability, feature column list, last trained date, and model configuration.
## Open Decisions
- exact adjusted versus unadjusted close handling;
- whether same-day `VIX`, `TLT`, and `USO` values are acceptable for the intended trading decision timing;
- whether same-day volatility proxy, `TLT`, and `USO` values are acceptable for the intended trading decision timing;
- exact train, validation, and test date boundaries or split percentages;
- output dataset file location, schema metadata, and versioning.
- output dataset schema metadata and versioning.
pyproject.toml
+1 -1
View File
@@ -1,7 +1,7 @@
[project]
name = "trading-bot"
version = "0.1.0"
description = "Python trading bot using machine learning and the IBKR API."
description = "Python trading bot using machine learning and the Alpaca Markets API."
readme = "README.md"
requires-python = ">=3.11,<3.12"
dependencies = [