83 lines
2.8 KiB
Markdown
83 lines
2.8 KiB
Markdown
# Data Fetcher Design
|
|
|
|
## Initial Goal
|
|
|
|
Create a Python module/tool that fetches daily candlestick data from the IBKR API for a specific ticker and date range.
|
|
|
|
The intended example workflow is:
|
|
|
|
- ticker: `SPY`;
|
|
- date range: `2026-06-01` to `2026-06-30`;
|
|
- bar size: one trading day;
|
|
- fields: open, high, low, close, volume;
|
|
- output: one Parquet file named for the ticker, such as `SPY.parquet`.
|
|
|
|
## Current Implementation
|
|
|
|
The first implementation is intentionally small:
|
|
|
|
- ticker passed as a required command line argument;
|
|
- end date passed with `--end-date YYYYMMDD`, defaulting to yesterday;
|
|
- end date can also be derived with `--end-date-from-parquet`;
|
|
- duration passed with `--duration`, defaulting to `1 W`;
|
|
- hard-coded IBKR Gateway target: `127.0.0.1:4002`;
|
|
- uses the IBKR API through `ib_insync`;
|
|
- prints fetched candles as CSV-like rows;
|
|
- writes candles to a symbol-named Parquet file;
|
|
- appends to an existing symbol file and keeps one row per date.
|
|
|
|
Run it with:
|
|
|
|
```sh
|
|
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY
|
|
```
|
|
|
|
To override the requested range:
|
|
|
|
```sh
|
|
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date 20250605 --duration "1 M"
|
|
```
|
|
|
|
To fetch backward from the oldest date already stored in the symbol file:
|
|
|
|
```sh
|
|
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date-from-parquet
|
|
```
|
|
|
|
`--end-date` and `--end-date-from-parquet` cannot be used together. If the symbol Parquet file does not exist or has no rows, `--end-date-from-parquet` uses today's US/Eastern date.
|
|
|
|
This expects a local IBKR Gateway session to be running and accepting API connections on `127.0.0.1:4002`.
|
|
|
|
By default, output is written to `data/ibkr/daily/SPY.parquet`. Use `--output-dir` to choose another directory.
|
|
|
|
Manual test instructions are in [manual-test/README.md](manual-test/README.md).
|
|
|
|
## Intended Future Behavior
|
|
|
|
Later, this tool should broaden configuration around data source and normalization choices.
|
|
|
|
Decided storage behavior:
|
|
|
|
- Parquet files are partitioned by ticker, not by date.
|
|
- Each ticker should have its own Parquet file, such as `SPY.parquet`.
|
|
- The trading date is used as the row key for merges.
|
|
- Re-fetching a date replaces the existing row for that date in that symbol's file.
|
|
|
|
Candidate output schema:
|
|
|
|
| Column | Type | Description |
|
|
| --- | --- | --- |
|
|
| `date` | date index | Trading session date |
|
|
| `symbol` | string | Asset ticker |
|
|
| `open` | float | Daily open price |
|
|
| `high` | float | Daily high price |
|
|
| `low` | float | Daily low price |
|
|
| `close` | float | Daily close price |
|
|
| `volume` | integer | Daily traded volume |
|
|
|
|
Open decisions:
|
|
|
|
- how to handle adjusted versus unadjusted prices;
|
|
- how to handle missing sessions and IBKR pacing limits;
|
|
- whether to use `ib_insync` long term or a lower-level IBKR client wrapper.
|