Data fetcher implementation

docs/data-fetcher.md

@@ -12,44 +12,63 @@ The intended example workflow is:
 - fields: open, high, low, close, volume;
 - output: one Parquet file named for the ticker, such as `SPY.parquet`.
 
-## Current Skeleton
+## Current Implementation
 
 The first implementation is intentionally small:
 
-- hard-coded ticker: `SPY`;
-- hard-coded range: `2026-06-01` to `2026-06-05`;
+- ticker passed as a required command line argument;
+- end date passed with `--end-date YYYYMMDD`, defaulting to yesterday;
+- end date can also be derived with `--end-date-from-parquet`;
+- duration passed with `--duration`, defaulting to `1 W`;
 - hard-coded IBKR Gateway target: `127.0.0.1:4002`;
 - uses the IBKR API through `ib_insync`;
 - prints fetched candles as CSV-like rows;
-- does not write Parquet yet;
-- does not expose CLI arguments yet;
-- does not define storage paths yet.
+- writes candles to a symbol-named Parquet file;
+- appends to an existing symbol file and keeps one row per date.
 
 Run it with:
 
 ```sh
-mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py
+mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY
 ```
 
+To override the requested range:
+
+```sh
+mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date 20250605 --duration "1 M"
+```
+
+To fetch backward from the oldest date already stored in the symbol file:
+
+```sh
+mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date-from-parquet
+```
+
+`--end-date` and `--end-date-from-parquet` cannot be used together. If the symbol Parquet file does not exist or has no rows, `--end-date-from-parquet` uses today's US/Eastern date.
+
 This expects a local IBKR Gateway session to be running and accepting API connections on `127.0.0.1:4002`.
 
+By default, output is written to `data/ibkr/daily/SPY.parquet`. Use `--output-dir` to choose another directory.
+
 Manual test instructions are in [manual-test/README.md](manual-test/README.md).
 
 ## Intended Future Behavior
 
-Later, this tool should accept a ticker and date range, fetch daily candles from IBKR, normalize the schema, and write the result to a ticker-named Parquet file.
+Later, this tool should broaden configuration around data source and normalization choices.
 
 Decided storage behavior:
 
 - Parquet files are partitioned by ticker, not by date.
 - Each ticker should have its own Parquet file, such as `SPY.parquet`.
+- The trading date is used as the row key for merges.
+- Re-fetching a date replaces the existing row for that date in that symbol's file.
 
 Candidate output schema:
 
 | Column | Type | Description |
 | --- | --- | --- |
-| `date` | date | Trading session date |
-| `ticker` | string | Asset ticker |
+| `date` | date index | Trading session date |
+| `symbol` | string | Asset ticker |
 | `open` | float | Daily open price |
 | `high` | float | Daily high price |
 | `low` | float | Daily low price |
@@ -58,7 +77,6 @@ Candidate output schema:
 
 Open decisions:
 
-- where raw and normalized data files should live;
 - how to handle adjusted versus unadjusted prices;
 - how to handle missing sessions and IBKR pacing limits;
 - whether to use `ib_insync` long term or a lower-level IBKR client wrapper.

docs/manual-test/README.md

@@ -1,19 +1,21 @@
 # Manual Test Instructions
 
-## IBKR Daily Fetcher Skeleton
+## IBKR Daily Fetcher
 
-This test checks the first hard-coded IBKR data fetcher without writing any data files.
+This test checks the IBKR data fetcher and confirms it writes a symbol-named Parquet file.
 
 The fetcher currently requests:
 
 - gateway: `127.0.0.1:4002`;
 - client id: `101`;
-- ticker: `SPY`;
-- date range: `2026-06-01` to `2026-06-05`;
+- symbol: provided as a command line argument;
+- end date: provided with `--end-date YYYYMMDD`, defaulting to yesterday;
+- end date can also be derived with `--end-date-from-parquet`;
+- duration: provided with `--duration`, defaulting to `1 W`;
 - bar size: `1 day`;
 - data type: `TRADES`;
 - regular trading hours only;
-- output: printed CSV-like rows.
+- output: printed CSV-like rows and a Parquet file in `data/ibkr/daily/`.
 
 ## Prerequisites
 
@@ -33,32 +35,52 @@ mise exec -- uv sync
 From the repository root, run:
 
 ```sh
-mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py
+mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY
 ```
 
+To fetch a specific IBKR range, pass an end date and duration:
+
+```sh
+mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date 20250605 --duration "1 M"
+```
+
+To fetch backward from the oldest date already stored in `data/ibkr/daily/SPY.parquet`:
+
+```sh
+mise exec -- uv run python src/trading_bot/data/fetch_ibkr_daily.py SPY --end-date-from-parquet
+```
+
+`--end-date` and `--end-date-from-parquet` cannot be used together. If the symbol Parquet file does not exist or has no rows, `--end-date-from-parquet` uses today's US/Eastern date.
+
 ## Expected Output
 
-The tool should first print the hard-coded request range and connection target:
+The tool should first print the request range and connection target:
 
 ```text
-Fetching SPY daily candles from 2026-06-01 to 2026-06-05
+Fetching SPY daily candles ending 2025-06-05 for duration 1 M
 Connecting to IBKR Gateway at 127.0.0.1:4002 with client id 101
 ```
 
 If the request succeeds, it should then print a header and one row per returned trading day:
 
 ```text
-date,ticker,open,high,low,close,volume
-2026-06-01,SPY,...
+date,symbol,open,high,low,close,volume
+2025-06-05,SPY,...
 ```
 
 Exact prices and volume depend on what IBKR returns.
 
+The tool should then write or update:
+
+```text
+data/ibkr/daily/SPY.parquet
+```
+
+If the Parquet file already exists, rows from the latest fetch are merged into it. The trading date is used as the row key, so a symbol file keeps only one row for each date.
+
 ## Common Issues
 
 - Connection refused: IBKR Gateway is not running, the port is not `4002`, or API access is disabled.
 - Client id already in use: change `IBKR_CLIENT_ID` in the fetcher or disconnect the other client.
 - No historical bars: confirm the account has market data permissions and that IBKR accepts the requested historical data range.
 - Pacing or permission errors: note the IBKR error message before changing the request.
-
-This skeleton does not write Parquet files yet.