jarno/trading-bot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Data fetcher implementation

Browse Source
This commit is contained in:
jarno
2026-07-26 14:18:07 +03:00
parent 8d3e663e71
commit e2eb5dd574
4 changed files with 273 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/data-fetcher.md
+29 -11
View File
@@ -12,44 +12,63 @@ The intended example workflow is:
- fields: open, high, low, close, volume;
- output: one Parquet file named for the ticker, such as `SPY.parquet`.
## Current Skeleton
## Current Implementation
The first implementation is intentionally small:
- hard-coded ticker: `SPY`;
- hard-coded range: `2026-06-01` to `2026-06-05`;
- ticker passed as a required command line argument;
- end date passed with `--end-date YYYYMMDD`, defaulting to yesterday;
- end date can also be derived with `--end-date-from-parquet`;
- duration passed with `--duration`, defaulting to `1 W`;
- hard-coded IBKR Gateway target: `127.0.0.1:4002`;
- uses the IBKR API through `ib_insync`;
- prints fetched candles as CSV-like rows;
- does not write Parquet yet;
- does not expose CLI arguments yet;
- does not define storage paths yet.
- writes candles to a symbol-named Parquet file;
- appends to an existing symbol file and keeps one row per date.
Run it with:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY
```
To override the requested range:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date 20250605 --duration "1 M"
```
To fetch backward from the oldest date already stored in the symbol file:
```sh
mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date-from-parquet
```
`--end-date` and `--end-date-from-parquet` cannot be used together. If the symbol Parquet file does not exist or has no rows, `--end-date-from-parquet` uses today's US/Eastern date.
This expects a local IBKR Gateway session to be running and accepting API connections on `127.0.0.1:4002`.
By default, output is written to `data/ibkr/daily/SPY.parquet`. Use `--output-dir` to choose another directory.
Manual test instructions are in [manual-test/README.md](manual-test/README.md).
## Intended Future Behavior
Later, this tool should accept a ticker and date range, fetch daily candles from IBKR, normalize the schema, and write the result to a ticker-named Parquet file.
Later, this tool should broaden configuration around data source and normalization choices.
Decided storage behavior:
- Parquet files are partitioned by ticker, not by date.
- Each ticker should have its own Parquet file, such as `SPY.parquet`.
- The trading date is used as the row key for merges.
- Re-fetching a date replaces the existing row for that date in that symbol's file.
Candidate output schema:
| Column | Type | Description |
| --- | --- | --- |
| `date` | date | Trading session date |
| `ticker` | string | Asset ticker |
| `date` | date index | Trading session date |
| `symbol` | string | Asset ticker |
| `open` | float | Daily open price |
| `high` | float | Daily high price |
| `low` | float | Daily low price |
@@ -58,7 +77,6 @@ Candidate output schema:
Open decisions:
- where raw and normalized data files should live;
- how to handle adjusted versus unadjusted prices;
- how to handle missing sessions and IBKR pacing limits;
- whether to use `ib_insync` long term or a lower-level IBKR client wrapper.