Engine benchmarks
One strategy spec, implemented identically on each engine. Same data, same execution assumptions, same costs. Parity is checked at two levels: whether the engines took the same decisions (sides and sizes, in order), and whether their fill prices are bit-identical. The second is stricter than any published comparison we know of, and it is where the interesting differences show up. The code is public and every run is reproducible from a fixed seed.
Strategy: SMA crossover 10/50, long-only. SMA(10/50) on closes; buy 100 shares when fast crosses above slow, sell all when it crosses below; fills at next bar open; zero costs; $100,000 starting cash; dataset synthetic-250k-seed42.
Run date: 2026-08-04 · Machine: Apple M1 · Darwin 23.6.0 · Python 3.12.9
Trade parity: identical trade sequences across all engines
Price parity: 2 distinct price sets, differing by at most 1 unit in the last place of a float64 (relative 1.7e-16). Representation noise, not behaviour.
| Engine | Runtime | Bars | Trades | Return | Max DD | Parity |
|---|---|---|---|---|---|---|
| NumPy/pandas (vectorized baseline)pandas 3.0.5 / numpy 2.4.6 Reference implementation of the spec; other engines are checked against its parity hash. | 0.021s | 250,000 | 3439 | 3.28% | -18.07% | dba8ef4c |
| VectorBT (vectorized)1.1.0 Includes one-time numba JIT compilation in the first run; median of runs reported. | 0.234s | 250,000 | 3439 | 3.28% | -18.07% | dba8ef4c |
| Backtesting.py (hybrid)0.6.6 Fills reconstructed from the trade log; the engine's end-of-series force-close is excluded to match the spec. | 0.808s | 250,000 | 3439 | 3.27% | -18.07% | dba8ef4c |
| Backtrader (event-driven)1.9.78.123 | 16.168s | 250,000 | 3439 | 3.28% | -18.07% | dba8ef4c |
How to read this
- Runtime. Wall-clock time for the full backtest, median of 3 runs, same machine.
- Trade parity. A hash of sides and sizes in order. Identical hashes mean the engines made the same decisions, so any remaining difference belongs to the engine rather than to the strategy.
- Price parity. A second hash over the fill prices as raw IEEE-754 doubles, with no rounding. Rounding before hashing would let genuinely different fills collide, which would quietly invalidate the whole exercise. Here it exposed that Backtrader's prices differ from the other three on 911 of 6,879 fills, by at most 5.7e-14 absolute. That is one unit in the last place of a float64: the number travelling through different internal machinery, not a different number.
- The dataset is synthetic (seeded random walk) on purpose: it is redistributable, identical for every engine, and free of survivorship bias. We benchmark engines, not strategies, so absolute returns are meaningless here and that is fine.
Full protocol on the methodology page. The benchmark code ships in the open repo (bench/ directory). Run python bench/run_all.py to reproduce.
Educational content only, not investment advice. Benchmarks measure software performance, not strategy quality.