\n\n日本語要約: Binance WebSocketから取得した1億5500万件超のオーダーブックレコードを効率的に処理するデータパイプラインの構築記録です。バイナリサーチローダー、Numba JITによる高速集計、LOBシミュレーターの設計に加え、AR-MLPやQuant-GANなどのML手法も試した結果と考察を共有します。
\n
I’ve been working on crypto microstructure analysis for a while now, and the hardest part isn’t the modeling — it’s wrangling the data. When you’re looking at tick-level BTC/USDT orderbook dynamics, you’re dealing with a firehose of information that will happily eat all your RAM and leave you staring at a frozen terminal.
\nThis post covers how I built a data pipeline that can efficiently load and process 155+ million orderbook records, run aggregation at sub-second intervals, and feed the results into ML models and LOB simulators.
\nCrypto microstructure research requires tick-level data. Not 1-minute candles, not even 1-second bars — actual individual bid/ask updates as they stream from the exchange. For BTC/USDT on Binance, that means:
\nAll of this collected over several months from the Binance WebSocket feed. Each record has a timestamp, and we need to slice arbitrary time windows, compute derived features (mid price, VWAP, volume profiles), and feed them into models.
\nThe goal was simple: given any arbitrary time range, produce aggregated microstructure features at 0.5-second intervals in under 2 seconds. Sounds reasonable until you realize a naive approach takes 45+ minutes just to load the data.
\nMy first attempt was embarrassingly straightforward:
\nimport pandas as pd\n\n# Don't do this with 48GB of CSVs\ndf = pd.read_csv(\"binance_btcusdt_depth_updates.csv\")On my 64GB workstation, this OOM’d spectacularly. Even with chunksize, iterating through 155M rows to find a 30-minute window was painfully slow (~3 minutes per query). Parquet helped with compression but the full scan was still the bottleneck.
The key insight: these files are already sorted by timestamp. We don’t need to scan — we need to seek.
\nThe idea is simple. Since our CSV files are sorted by timestamp (they come from a WebSocket feed, after all), we can binary search for the starting position and only read what we need.
\nimport os\nimport bisect\nimport numpy as np\nimport pandas as pd\nfrom pathlib import Path\nfrom typing import Tuple, Optional\n\n\nclass BinarySearchCSVLoader:\n \"\"\"\n Loads time-sliced data from large sorted CSV files using binary search.\n Builds a sparse index on first access, then seeks directly to the\n relevant byte offset for subsequent queries.\n \"\"\"\n\n def __init__(self, csv_path: str, timestamp_col: str = \"timestamp\",\n index_granularity: int = 100_000):\n self.csv_path = Path(csv_path)\n self.timestamp_col = timestamp_col\n self.granularity = index_granularity\n self._index: Optional[np.ndarray] = None\n self._offsets: Optional[np.ndarray] = None\n self._header: Optional[str] = None\n self._build_index()\n\n def _build_index(self):\n \"\"\"Build sparse index: sample every N-th row's timestamp and byte offset.\"\"\"\n timestamps = []\n offsets = []\n\n with open(self.csv_path, 'r') as f:\n self._header = f.readline()\n row_count = 0\n\n while True:\n offset = f.tell()\n line = f.readline()\n if not line:\n break\n\n if row_count % self.granularity == 0:\n ts = int(line.split(',')[0]) # timestamp is first column\n timestamps.append(ts)\n offsets.append(offset)\n\n row_count += 1\n\n self._index = np.array(timestamps, dtype=np.int64)\n self._offsets = np.array(offsets, dtype=np.int64)\n print(f\"Index built: {len(self._index)} entries covering {row_count:,} rows\")\n\n def load_range(self, start_ts: int, end_ts: int) -> pd.DataFrame:\n \"\"\"Load only rows within [start_ts, end_ts] using binary search.\"\"\"\n # Find the index entry just before our start timestamp\n idx_start = max(0, bisect.bisect_left(self._index, start_ts) - 1)\n idx_end = min(len(self._index) - 1,\n bisect.bisect_right(self._index, end_ts))\n\n byte_start = self._offsets[idx_start]\n # Read until well past the end to account for granularity\n if idx_end + 1 < len(self._offsets):\n byte_end = self._offsets[idx_end + 1]\n else:\n byte_end = os.path.getsize(self.csv_path)\n\n bytes_to_read = byte_end - byte_start\n\n # Read only the relevant chunk\n from io import StringIO\n with open(self.csv_path, 'r') as f:\n f.seek(byte_start)\n chunk = f.read(bytes_to_read)\n\n df = pd.read_csv(StringIO(self._header + chunk))\n # Filter to exact range\n mask = (df[self.timestamp_col] >= start_ts) & \\\n (df[self.timestamp_col] <= end_ts)\n return df[mask].reset_index(drop=True)The index build takes about 90 seconds for the 155M-row file (one-time cost), and after that, loading any 30-minute window takes 0.8-1.2 seconds instead of 3+ minutes. Memory usage drops from “all of it” to just the slice you need — typically 50-200MB for a 30-minute window depending on market activity.
\n| Operation | \nNaive (pandas) | \nBinary Search Loader | \n
|---|---|---|
| Load 30-min window | \n185s | \n1.1s | \n
| Load 2-hour window | \n185s (same full scan) | \n3.8s | \n
| Memory (30-min) | \n48GB+ (OOM) | \n~150MB | \n
| Index build (one-time) | \nN/A | \n92s | \n
Once we have the raw tick data loaded, we need to aggregate it into features at regular intervals. Computing mid price, VWAP, and volume at 0.5-second intervals across millions of ticks is the hot path that runs on every query.
\nPure pandas/numpy was taking 800ms+ for a 30-minute window. With Numba JIT, this drops to about 45ms.
\nimport numba\nimport numpy as np\nfrom numba import njit, prange\n\n\n@njit(cache=True)\ndef compute_aggregated_features(\n timestamps: np.ndarray, # int64, microseconds\n bid_prices: np.ndarray, # float64\n ask_prices: np.ndarray, # float64\n bid_volumes: np.ndarray, # float64\n ask_volumes: np.ndarray, # float64\n trade_prices: np.ndarray, # float64\n trade_volumes: np.ndarray, # float64\n trade_timestamps: np.ndarray, # int64\n interval_us: int = 500_000 # 0.5 seconds in microseconds\n) -> tuple:\n \"\"\"\n Compute mid price, VWAP, and volume at fixed intervals.\n All arrays must be sorted by timestamp.\n \"\"\"\n t_start = timestamps[0]\n t_end = timestamps[-1]\n n_intervals = int((t_end - t_start) / interval_us) + 1\n\n mid_prices = np.empty(n_intervals, dtype=np.float64)\n vwaps = np.empty(n_intervals, dtype=np.float64)\n volumes = np.empty(n_intervals, dtype=np.float64)\n interval_times = np.empty(n_intervals, dtype=np.int64)\n\n tick_idx = 0\n trade_idx = 0\n\n for i in range(n_intervals):\n t_interval_start = t_start + i * interval_us\n t_interval_end = t_interval_start + interval_us\n interval_times[i] = t_interval_start\n\n # Advance tick pointer to end of this interval\n last_bid = 0.0\n last_ask = 0.0\n while tick_idx < len(timestamps) and timestamps[tick_idx] < t_interval_end:\n last_bid = bid_prices[tick_idx]\n last_ask = ask_prices[tick_idx]\n tick_idx += 1\n\n mid_prices[i] = (last_bid + last_ask) / 2.0\n\n # Compute VWAP and volume from trades in this interval\n vol_sum = 0.0\n pv_sum = 0.0\n while trade_idx < len(trade_timestamps) and \\\n trade_timestamps[trade_idx] < t_interval_end:\n v = trade_volumes[trade_idx]\n p = trade_prices[trade_idx]\n vol_sum += v\n pv_sum += p * v\n trade_idx += 1\n\n volumes[i] = vol_sum\n vwaps[i] = pv_sum / vol_sum if vol_sum > 0 else mid_prices[i]\n\n return interval_times, mid_prices, vwaps, volumesThe first call takes ~2s due to JIT compilation, but subsequent calls with the same type signature hit the cache and run in 40-50ms. For interactive exploration this is a huge win — you get near-real-time iteration on feature engineering.
\nWith the pipeline working, I built a Limit Order Book simulator to generate synthetic orderbook dynamics. The idea was to test whether ML models could learn microstructure patterns from simulated data before deploying on real data.
\nThe simulator supports multiple price process models:
\nEach model generates price paths, and the LOB simulator wraps these with realistic order flow dynamics — queue sizes, arrival rates, and cancellation patterns calibrated from the real Binance data.
\nHere’s where things got interesting (and humbling).
\nThe AR-MLP takes a window of past mid-price returns and orderbook imbalance features, and predicts the next 0.5s return. Architecture is straightforward: 3 hidden layers (256, 128, 64), batch norm, dropout 0.3, trained on 2 months of data.
\nThis produced more visually interesting results. The distribution of log returns from the GAN actually matches the empirical distribution reasonably well — you can see the heavy tails are captured. But in practice:
\nAfter weeks of ML experiments, I ended up using a self-exciting jump diffusion model for the LOB simulator. The reasons:
\nThe ML models are interesting academically, and I think with more data and better architectures (transformers, perhaps) they could eventually win. But for a practical LOB simulator where you need reliable synthetic data for strategy testing, a well-calibrated stochastic model is hard to beat.
\nThe full pipeline looks like:
\nBinance WebSocket --> Raw CSVs (sorted by timestamp)\n |\n Binary Search Index (sparse, in-memory)\n |\n Time-Slice Loader (seek + read)\n |\n Numba Aggregator (0.5s intervals)\n |\n +--------------------+\n | |\n Feature Store LOB Simulator\n | |\n ML Models Strategy BacktesterTotal latency from “give me features for this time range” to having aggregated data ready: ~1.5 seconds for a typical 30-minute window. This includes loading from disk, so if you’re iterating on features interactively, it feels snappy enough.
\nBinary search on sorted files is underrated. Everyone reaches for databases or Parquet partitioning, but if your data is already sorted and you need arbitrary range queries, a simple sparse index + seek gets you 90% of the way there with zero dependencies.
\nNumba is magic for numerical hot paths. The constraint is that you need to write “numpy-style” code inside the JIT function (no pandas, no Python objects), but for aggregation loops it’s a perfect fit. The cache=True flag is essential — without it you pay compilation cost on every restart.
ML models for price simulation are harder than they look. The drift problem in AR models and mode collapse in GANs are well-known, but experiencing them firsthand on real financial data is educational. The gap between “interesting paper results” and “useful tool for my workflow” is wider than I expected.
\nStart with the data pipeline, not the model. I spent the first two weeks just on the loader and aggregator, and it paid off massively. Every model experiment after that was fast to iterate on because the data was always ready in seconds.
\nThe full pipeline code is roughly 2,500 lines of Python. Not a massive codebase, but it handles the core problem well: turning a firehose of raw WebSocket data into something you can actually work with for research.
","tableOfContents":"","frontmatter":{"title":"Processing 155 Million Orderbook Records: Building a Fast Data Pipeline for Crypto Microstructure Analysis","date":"February 20, 2026","description":"How I built a data pipeline to process 155M+ Binance orderbook records for crypto microstructure research, using binary search, Numba JIT, and LOB simulation."}}},"pageContext":{"slug":"/crypto-data-pipeline/","previous":{"fields":{"slug":"/self-exciting-jump-diffusion/"},"frontmatter":{"title":"Self-Exciting Jump-Diffusion for Crypto: Why Vanilla Models Miss Momentum and Volatility Clustering"}},"next":{"fields":{"slug":"/polymarket-orderbook-analysis/"},"frontmatter":{"title":"Analyzing Bid-Ask Spreads and Liquidity Shifts in Polymarket's 5-Minute Binary Option Orderbooks"}}}}}