Add comprehensive project documentation and fix data pipeline

- Add detailed README with architecture diagram and usage instructions - Add API, configuration, and development documentation - Fix price data column handling for yfinance auto_adjust=True - Fix model feature dimension indexing and temporal attention batching - Add missing imports and position tracking in paper broker - Add python-dotenv support for environment variables - Update .gitignore with Python artifacts and environment files
2026-05-26 14:10:48 +02:00
parent 4bf7394a0a
commit 0cf37e786a
17 changed files with 1536 additions and 277 deletions
@@ -0,0 +1,406 @@
+# API Reference
+
+Base URL: `http://localhost:8000`
+
+---
+
+## Dashboard
+
+### `GET /api/dashboard/metrics`
+
+Returns current system metrics.
+
+**Response:**
+```json
+{
+  "memory": {
+    "allocated_gb": 4.2,
+    "max_allocated_gb": 5.1,
+    "total_gb": 32.0,
+    "limit_gb": 28.8,
+    "usage_percent": 13.1,
+    "free_gb": 24.6
+  },
+  "account": {
+    "cash": 100000.00,
+    "total_value": 100000.00,
+    "positions_count": 0,
+    "positions": {}
+  },
+  "model": {
+    "status": "loaded",
+    "device": "cuda",
+    "amd_gpu": true,
+    "mixed_precision": true,
+    "precision": "bf16",
+    "hidden_channels": 128,
+    "num_heads": 16,
+    "batch_size": 128,
+    "learning_rate": 0.0005
+  },
+  "system": {
+    "project_name": "StockGNN_R9700",
+    "version": "1.0.0",
+    "training_active": false,
+    "trading_active": false
+  }
+}
+```
+
+### `GET /api/dashboard/logs?limit=100`
+
+Returns recent log entries.
+
+**Response:**
+```json
+{
+  "logs": [
+    "[2024-01-15 09:30:00] Trading started",
+    "[2024-01-15 09:30:02] Model loaded from models/stock_gnn_r9700.pt"
+  ],
+  "total": 2
+}
+```
+
+### `POST /api/dashboard/logs/clear`
+
+Clears stored logs.
+
+**Response:**
+```json
+{"status": "cleared"}
+```
+
+---
+
+## Trading
+
+### `GET /api/trading/status`
+
+**Response:**
+```json
+{
+  "active": false,
+  "cash": 100000.00,
+  "total_value": 100000.00,
+  "positions": {},
+  "orders_count": 0
+}
+```
+
+### `POST /api/trading/start`
+
+Starts live trading.
+
+**Response:**
+```json
+{"status": "started"}
+```
+
+### `POST /api/trading/stop`
+
+Stops live trading.
+
+**Response:**
+```json
+{"status": "stopped"}
+```
+
+### `GET /api/trading/orders`
+
+Lists all orders.
+
+**Response:**
+```json
+[
+  {
+    "order_id": "abc-123",
+    "ticker": "AAPL",
+    "action": "buy",
+    "quantity": 100,
+    "price": 150.0,
+    "timestamp": "2024-01-15 09:30:00",
+    "type": "market",
+    "status": "filled"
+  }
+]
+```
+
+### `POST /api/trading/order`
+
+Submits a manual order.
+
+**Request Body:**
+```json
+{
+  "ticker": "AAPL",
+  "action": "buy",
+  "quantity": 100,
+  "price": 150.0,
+  "timestamp": "2024-01-15 09:30:00",
+  "type": "market"
+}
+```
+
+**Response:**
+```json
+{"status": "submitted", "order_id": "abc-123"}
+```
+
+### `POST /api/trading/cancel/{order_id}`
+
+Cancels an order.
+
+**Response:**
+```json
+{"status": "cancelled"}
+```
+
+### `POST /api/trading/positions/close/{ticker}`
+
+Closes a position.
+
+**Response:**
+```json
+{"status": "submitted", "order_id": "def-456"}
+```
+
+### `POST /api/trading/positions/close-all`
+
+Closes all positions.
+
+**Response:**
+```json
+{
+  "status": "submitted",
+  "results": [
+    {"ticker": "AAPL", "order_id": "ghi-789"}
+  ]
+}
+```
+
+---
+
+## Models
+
+### `GET /api/models/status`
+
+**Response:**
+```json
+{
+  "status": "loaded",
+  "training_active": false,
+  "device": "cuda",
+  "amd_gpu": true,
+  "mixed_precision": true,
+  "precision": "bf16",
+  "model_name": "stock_gnn_r9700",
+  "hidden_channels": 128,
+  "num_heads": 16,
+  "dropout": 0.3,
+  "learning_rate": 0.0005,
+  "batch_size": 128,
+  "epochs": 200,
+  "sequence_length": 60
+}
+```
+
+### `POST /api/models/train`
+
+Starts training.
+
+**Response:**
+```json
+{"status": "started"}
+```
+
+### `POST /api/models/train/stop`
+
+Stops training.
+
+**Response:**
+```json
+{"status": "stopped"}
+```
+
+### `POST /api/models/save`
+
+Saves model weights.
+
+**Response:**
+```json
+{"status": "saved"}
+```
+
+### `POST /api/models/load`
+
+Loads model weights.
+
+**Response:**
+```json
+{"status": "loaded"}
+```
+
+### `POST /api/models/benchmark`
+
+Runs performance benchmark.
+
+**Response:**
+```json
+{
+  "status": "complete",
+  "inference_time_ms": 2.341,
+  "training_time_ms": 8.567,
+  "inference_throughput": 427.0,
+  "training_throughput": 116.7,
+  "memory_allocated_gb": 4.5,
+  "device": "cuda"
+}
+```
+
+---
+
+## Data
+
+### `GET /api/data/tickers`
+
+**Response:**
+```json
+{
+  "initial": ["AAPL", "MSFT", "GOOGL", ...],
+  "index": "^GSPC",
+  "count": 30
+}
+```
+
+### `GET /api/data/pipeline/status`
+
+**Response:**
+```json
+{
+  "status": "ready",
+  "tickers_loaded": 30,
+  "db_path": "data/processed/stock_data.db"
+}
+```
+
+### `POST /api/data/update`
+
+Triggers a full data update.
+
+**Response:**
+```json
+{"status": "started"}
+```
+
+### `GET /api/data/features/{ticker}`
+
+**Response:**
+```json
+{
+  "ticker": "AAPL",
+  "timestamp": "2024-01-15 09:30:00",
+  "features": {
+    "AAPL": {
+      "ticker": "AAPL",
+      "timestamp": "2024-01-15 09:30:00",
+      "return": 0.0,
+      "volatility": 0.2,
+      ...
+    }
+  }
+}
+```
+
+### `GET /api/data/price/{ticker}`
+
+**Response:**
+```json
+{
+  "ticker": "AAPL",
+  "date": "2024-01-15 00:00:00",
+  "open": 150.0,
+  "high": 152.0,
+  "low": 149.0,
+  "close": 151.0,
+  "adj_close": 151.0,
+  "volume": 50000000
+}
+```
+
+### `GET /api/data/prices/{ticker}?limit=30`
+
+**Response:**
+```json
+[
+  {
+    "date": "2024-01-15",
+    "open": 150.0,
+    "high": 152.0,
+    "low": 149.0,
+    "close": 151.0,
+    "volume": 50000000
+  }
+]
+```
+
+### `GET /api/data/corporate-actions/{ticker}`
+
+**Response:**
+```json
+{
+  "ticker": "AAPL",
+  "actions": {
+    "splits": {"2020-08-31": 4.0},
+    "dividends": {"2024-01-10": 0.24}
+  }
+}
+```
+
+---
+
+## WebSocket
+
+### Connection
+
+```javascript
+const ws = new WebSocket('ws://localhost:8000/ws');
+```
+
+### Server → Client Messages
+
+**Type: `metrics`**
+```json
+{
+  "type": "metrics",
+  "timestamp": "2024-01-15T09:30:00.000000",
+  "memory": {
+    "allocated_gb": 4.2,
+    "total_gb": 32.0,
+    "usage_percent": 13.1
+  },
+  "account": {
+    "cash": 100000.00,
+    "total_value": 100000.00,
+    "positions": 0
+  },
+  "model": {
+    "status": "loaded",
+    "device": "cuda",
+    "mixed_precision": true,
+    "precision": "bf16"
+  }
+}
+```
+
+### Client → Server Messages
+
+**Ping:**
+```json
+{"action": "ping"}
+```
+
+**Subscribe:**
+```json
+{"action": "subscribe", "channel": "all"}
+```
@@ -0,0 +1,69 @@
+# Configuration Guide
+
+All configuration is centralized in `config.py` via the `Config` class.
+
+## GPU Settings
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `DEVICE` | `'cuda'` | PyTorch device (auto-detected) |
+| `AMD_GPU` | `True` | Enable AMD-specific optimizations |
+| `GPU_MEMORY_LIMIT` | `0.9` | Fraction of GPU memory to use (0.0–1.0) |
+| `ROCM_OPT_LEVEL` | `'O2'` | ROCm JIT optimization level |
+| `MIXED_PRECISION` | `True` | Enable AMP |
+| `PRECISION` | `'bf16'` | `'bf16'` or `'fp16'` |
+
+## Model Hyperparameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `HIDDEN_CHANNELS` | `128` | GNN hidden dimension |
+| `NUM_HEADS` | `16` | Attention heads |
+| `DROPOUT` | `0.3` | Dropout rate |
+| `LEARNING_RATE` | `0.0005` | AdamW learning rate |
+| `BATCH_SIZE` | `128` | Training batch size |
+| `SEQUENCE_LENGTH` | `60` | Temporal window length |
+| `EPOCHS` | `200` | Max training epochs |
+
+## Trading Settings
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `TRADING_FREQUENCY` | `'5min'` | Signal generation interval |
+| `INITIAL_CAPITAL` | `100000` | Starting portfolio value |
+| `MAX_POSITION_SIZE` | `0.03` | Max position as fraction of portfolio |
+| `MAX_DAILY_LOSS` | `0.01` | Daily loss circuit breaker |
+| `MAX_DRAWDOWN` | `0.05` | Max portfolio drawdown |
+
+## Data Providers
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `DATA_PROVIDER` | `'polygon'` | Live data source |
+| `POLYGON_API_KEY` | — | Polygon.io API key |
+| `ALPHA_VANTAGE_API_KEY` | — | Alpha Vantage API key |
+| `IB_HOST` | `'127.0.0.1'` | Interactive Brokers TWS host |
+| `IB_PORT` | `7497` | TWS API port |
+
+## Feature Definitions
+
+Three feature vectors are defined and referenced by the models:
+
+```python
+NEWS_FEATURES = ['sentiment', 'volume', 'recency', 'source_reliability', 'topic_relevance']
+SOCIAL_FEATURES = ['twitter_sentiment', 'twitter_volume', 'reddit_sentiment', 'reddit_volume', 'social_momentum']
+INTRADAY_FEATURES = ['return', 'volatility', 'momentum', 'volume_momentum', 'bid_ask_spread', 'bid_ask_spread_pct', 'volume_imbalance', 'order_flow', 'vwap_deviation']
+```
+
+## Environment Variables
+
+Sensitive keys can be overridden via environment variables:
+
+```bash
+export POLYGON_API_KEY="your_key"
+export ALPHA_VANTAGE_API_KEY="your_key"
+export NEWS_API_KEY="your_key"
+export TWITTER_BEARER_TOKEN="your_token"
+```
+
+These are read in `config.py` and fall back to hardcoded placeholders if not set.
@@ -0,0 +1,194 @@
+# Development Guide
+
+## Setting Up Development Environment
+
+```bash
+# Clone repo
+git clone <repository-url>
+cd stock_gnn_r9700
+
+# Create virtual environment
+python -m venv venv
+source venv/bin/activate
+
+# Install in editable mode + dev dependencies
+pip install -r requirements.txt
+pip install black isort mypy pytest pytest-asyncio
+```
+
+## Code Style
+
+Format with `black` and `isort`:
+
+```bash
+black src/ config.py main.py live_trading.py benchmark.py
+isort src/ config.py main.py live_trading.py benchmark.py
+```
+
+Type-check with `mypy`:
+
+```bash
+mypy src/
+```
+
+## Running Tests
+
+```bash
+pytest tests/ -v
+```
+
+## Adding a New Data Processor
+
+1. Create file in `src/data/`:
+
+```python
+# src/data/earnings_processor.py
+import logging
+from typing import Dict, List
+
+logger = logging.getLogger(__name__)
+
+class EarningsProcessor:
+    def __init__(self):
+        self.earnings_data = {}
+
+    def fetch_earnings(self, tickers: List[str], start_date: str, end_date: str):
+        """Fetch earnings data."""
+        pass
+
+    def get_earnings_features(self, ticker: str, date: str) -> Dict:
+        """Return earnings-based features."""
+        return {"earnings_surprise": 0.0, "eps_growth": 0.0}
+```
+
+2. Import and initialize in `StockDataPipeline.__init__`:
+
+```python
+from src.data.earnings_processor import EarningsProcessor
+
+class StockDataPipeline:
+    def __init__(self):
+        ...
+        self.earnings_processor = EarningsProcessor()
+```
+
+3. Call `fetch_earnings` in `update_alternative_data` and include features in `create_training_dataset`.
+
+## Adding a New Model
+
+1. Create file in `src/models/`:
+
+```python
+# src/models/my_model.py
+import torch
+import torch.nn as nn
+from config import config
+
+class MyModel(nn.Module):
+    def __init__(self, num_features: int):
+        super().__init__()
+        self.fc = nn.Linear(num_features, 1)
+
+    def forward(self, data):
+        return self.fc(data.x)
+```
+
+2. Wrap with `AMDOptimizer` before training:
+
+```python
+from src.amd.optimizations import AMDOptimizer
+
+model = MyModel(num_features)
+model = AMDOptimizer().optimize_model(model)
+```
+
+## Adding a New Trading Strategy
+
+1. Subclass `Broker` in `src/trading/`:
+
+```python
+from src.trading.broker import Broker
+
+class AlpacaBroker(Broker):
+    def submit_order(self, order):
+        # Implementation
+        pass
+```
+
+2. Instantiate in `live_trading.py` and pass to `RealTimeTrader`.
+
+## Adding API Endpoints
+
+1. Create router in `src/web/api/`:
+
+```python
+from fastapi import APIRouter
+
+router = APIRouter()
+
+@router.get("/health")
+async def health():
+    return {"status": "ok"}
+```
+
+2. Include in `src/web/app.py`:
+
+```python
+from src.web.api import my_endpoints
+app.include_router(my_endpoints.router, prefix="/api/my")
+```
+
+## Environment Variables
+
+Set in `.env` file (not committed):
+
+```bash
+POLYGON_API_KEY=your_key
+ALPHA_VANTAGE_API_KEY=your_key
+IB_CLIENT_ID=1
+```
+
+Load with `python-dotenv` if needed.
+
+## Git Workflow
+
+```bash
+# Feature branch
+git checkout -b feature/my-feature
+
+# Commit
+git add .
+git commit -m "feat: add my feature"
+
+# Push
+git push origin feature/my-feature
+```
+
+## Debugging
+
+### GPU Memory Issues
+
+```python
+from src.utils.memory_manager import MemoryManager
+
+mm = MemoryManager()
+print(mm.get_memory_stats())
+```
+
+### Model Inspection
+
+```python
+from src.models.gnn_model import CorporateActionAwareGNN
+
+model = CorporateActionAwareGNN(num_features=10)
+print(sum(p.numel() for p in model.parameters()))  # parameter count
+```
+
+### Data Inspection
+
+```python
+from src.data.pipeline import StockDataPipeline
+
+pipe = StockDataPipeline()
+print(pipe.price_data['AAPL'].tail())
+```