We had a problem. Our automated trading system had a positive expected value: the math checked out, the backtests looked great, and initially, it made money. But over time, it was bleeding. Small losses that accumulated faster than the occasional wins could compensate. This wasn't a bug in the code. It was a fundamental misunderstanding of what matters in production. The Expected Value Trap Most trading tutorials, academic papers, and online courses teach you to maximize expected value. The logic seems bulletproof: E[profit] = Σ(probability_i × outcome_i) E[profit] = Σ(probability_i × outcome_i) If this number is positive, you should take the trade. If you can make this number bigger, you should optimize for it. Simple, right? Except in production, this optimization strategy has a fatal flaw: it doesn't account for the path you take to reach that expected value. it doesn't account for the path you take to reach that expected value Let me show you what I mean with a real scenario from our system. The Bleeding System Our strategy was designed to capture price spikes in volatile markets. The model would: Analyze possible price directions for each trading window Optimize position sizing using quadratic programming Execute trades to capture spread opportunities Analyze possible price directions for each trading window Optimize position sizing using quadratic programming Execute trades to capture spread opportunities On paper, the expected value was solidly positive. In practice: Day 1-3: Caught a major spike, made $15,000 Day 4-12: Small losses every day, total -$8,000 Day 13-14: Another spike, made $12,000 Day 15-28: Gradual bleed, total -$11,000 Day 1-3: Caught a major spike, made $15,000 Day 1-3 Day 4-12: Small losses every day, total -$8,000 Day 4-12 Day 13-14: Another spike, made $12,000 Day 13-14 Day 15-28: Gradual bleed, total -$11,000 Day 15-28 The problem? Our optimizer had developed a structural bias. It was systematically taking positions that won big occasionally but lost small amounts frequently. The expected value calculation said this was fine: the big wins would eventually compensate. But "eventually" requires infinite capital and infinite time horizon. We had neither. Seeing The Difference: A Simulation To illustrate why these risk controls matter, let's compare two strategies trading the same market over one year: Strategy A (EV Maximization): Aggressive position sizing based purely on expected value, using 150% leverage when opportunities look good. Strategy A (EV Maximization) Strategy B (Risk-Controlled): Same market signals, but with fractional Kelly sizing (40% of aggressive) and CVaR-based position reduction during high tail risk periods. Strategy B (Risk-Controlled) The results tell a crucial story. Look at the left chart closely - most EV-maximization paths aren't catastrophically failing. They're just... not compounding. You can see the sawtooth pattern: occasional spikes up, followed by slow erosion. This is the insidious bleeding that positive expected value misses. Notice how a few paths reached $500k? Those outliers pull the mean up to $146k. But the median is only $136k, and 29 out of 100 paths end below starting capital. In a backtest, you might have gotten lucky and seen one of those winner paths. In production, you get one random draw. median In a backtest, you might have gotten lucky and seen one of those winner paths. In production, you get one random draw. The right chart is "boring", and that's exactly the point. No moonshots to $500k, but also no catastrophic drawdowns. The risk-controlled strategy clusters tightly around modest growth. It survives to compound returns over multiple years. This is the production reality: the strategy that survives gets to compound. The strategy that bleeds out makes nothing, regardless of what the expected value calculation promised. What Expected Value Doesn't Capture 1. Risk of Ruin This is the classic gambler's problem, formalized by the Kelly Criterion. Even with positive expected value, if your position sizing is wrong, you will go broke. will Consider: You have $100,000 capital and a trade with 60% win probability that either doubles your bet or loses it. Expected value is positive (+20%). But if you bet everything, you have a 40% chance of losing it all on the first trade. Kelly tells you the optimal bet size is: kelly_fraction = (p * b - q) / b # where p = win probability, q = loss probability, b = odds kelly_fraction = (p * b - q) / b # where p = win probability, q = loss probability, b = odds But here's what we learned in production: even Kelly is too aggressive. even Kelly is too aggressive Why? Because: Your probability estimates are wrong (always) Markets change (your 60% edge becomes 52%) Correlations break down during stress (when you need them most) You can't rebalance instantly (slippage, latency, market impact) Your probability estimates are wrong (always) Markets change (your 60% edge becomes 52%) Correlations break down during stress (when you need them most) You can't rebalance instantly (slippage, latency, market impact) We ended up using fractional Kelly (25-50% of the theoretical Kelly bet) because the real-world costs of overestimating your edge are catastrophic. 2. Numerical Instability in Extreme Events One morning, our system crashed during an extreme weather event. Not a software crash, but a mathematical one. Our covariance matrix became singular. The optimizer couldn't find a solution. We were frozen, unable to trade, during the exact conditions where our strategy should have made the most money. The problem: we had optimized for expected scenarios. But extreme events have different correlation structures. Assets that normally move independently suddenly become perfectly correlated. Your carefully estimated covariance matrix, built from thousands of normal days, becomes useless. The fix wasn't better expected value calculations. It was regularization: regularization from sklearn.covariance import LedoitWolf # Instead of sample covariance cov_matrix = np.cov(returns.T) # Use shrinkage towards structured estimator lw = LedoitWolf() cov_matrix_robust = lw.fit(returns).covariance_ from sklearn.covariance import LedoitWolf # Instead of sample covariance cov_matrix = np.cov(returns.T) # Use shrinkage towards structured estimator lw = LedoitWolf() cov_matrix_robust = lw.fit(returns).covariance_ This trades off some accuracy in normal times for stability in extremes. Your expected value calculations will be slightly worse. Your system will survive black swans. Time Horizon Mismatch Here's a problem that doesn't show up in backtests: your expected value calculation assumes you can wait long enough for the law of large numbers to work. In production, you can't. We discovered this when our system showed strong positive expected value over 90-day windows but consistently lost money over 30-day windows. The problem wasn't the math. It was the business reality. Our capital providers reviewed performance monthly. Our risk limits were adjusted quarterly based on recent results. If we had three bad months, our position limits got cut, regardless of what the long-term expected value said. The theoretical strategy needed 6-12 months to reliably show profit. The operational reality gave us 3 months before consequences kicked in. We had to add explicit time-horizon constraints to our optimization: def optimize_with_horizon_constraint(scenarios, max_horizon_days=30): """ Optimize not just for long-term EV, but for probability of positive returns within operational time horizon """ # Standard expected value ev = np.mean(scenarios) # But also: what'sthe probability we're profitable # within our actual time horizon? rolling_returns = pd.Series(scenarios).rolling(max_horizon_days).sum() prob_profitable_in_horizon = (rolling_returns > 0).mean() # Penalize strategies with low short-term win probability # even if long-term EV is great if prob_profitable_in_horizon < 0.6: return ev * 0.5 # Heavily discount return ev def optimize_with_horizon_constraint(scenarios, max_horizon_days=30): """ Optimize not just for long-term EV, but for probability of positive returns within operational time horizon """ # Standard expected value ev = np.mean(scenarios) # But also: what'sthe probability we're profitable # within our actual time horizon? rolling_returns = pd.Series(scenarios).rolling(max_horizon_days).sum() prob_profitable_in_horizon = (rolling_returns > 0).mean() # Penalize strategies with low short-term win probability # even if long-term EV is great if prob_profitable_in_horizon < 0.6: return ev * 0.5 # Heavily discount return ev This meant accepting strategies with slightly lower theoretical expected value but higher probability of showing profit within our operational constraints. It's not mathematically optimal, but it's practically necessary. What to Optimize Instead After painful lessons, here's what we learned to optimize for: 1. Risk-Adjusted Returns with CVaR Instead of maximizing E[profit], we minimize CVaR (Conditional Value at Risk): the expected loss in the worst 5% of scenarios import cvxpy as cp # Decision variable: position sizes positions = cp.Variable(n_assets) # Scenarios returns scenario_returns = get_price_scenarios() # shape: (n_scenarios, n_assets) portfolio_returns = scenario_returns @ positions # CVaR constraints alpha = 0.05 # 5% tail var = cp.Variable() u = cp.Variable(n_scenarios) constraints = [ u >= 0, u >= -(portfolio_returns - var), ] cvar = var + cp.sum(u) / (n_scenarios * alpha) # Optimize for return while constraining tail risk objective = cp.Maximize(cp.sum(portfolio_returns) / n_scenarios - lambda_risk * cvar) import cvxpy as cp # Decision variable: position sizes positions = cp.Variable(n_assets) # Scenarios returns scenario_returns = get_price_scenarios() # shape: (n_scenarios, n_assets) portfolio_returns = scenario_returns @ positions # CVaR constraints alpha = 0.05 # 5% tail var = cp.Variable() u = cp.Variable(n_scenarios) constraints = [ u >= 0, u >= -(portfolio_returns - var), ] cvar = var + cp.sum(u) / (n_scenarios * alpha) # Optimize for return while constraining tail risk objective = cp.Maximize(cp.sum(portfolio_returns) / n_scenarios - lambda_risk * cvar) This explicitly penalizes strategies that have good average returns but catastrophic tail risk. 2. Robustness to Model Error We assume our model is wrong and optimize for the worst-case within a reasonable uncertainty bound: worst-case # Instead of single expected return estimate mu_estimated = historical_returns.mean() # Assume uncertainty mu_lower_bound = mu_estimated - 2 * historical_returns.std() / np.sqrt(len(historical_returns)) # Optimize for worst-case in uncertainty range # (Robust optimization / minmax approach) # Instead of single expected return estimate mu_estimated = historical_returns.mean() # Assume uncertainty mu_lower_bound = mu_estimated - 2 * historical_returns.std() / np.sqrt(len(historical_returns)) # Optimize for worst-case in uncertainty range # (Robust optimization / minmax approach) This protects against overconfident parameter estimates. 3. Kelly-Constrainted Position Sizing We explicitly limit position sizes based on Kelly criterion, even when the optimizer wants more: def kelly_position_limit(edge, volatility, capital, max_kelly_fraction=0.25): """ edge: expected return per unit risk volatility: standard deviation of returns max_kelly_fraction: fraction of theoretical Kelly to actually use """ kelly_full = edge / (volatility ** 2) kelly_fraction = capital * kelly_full * max_kelly_fraction return kelly_position def kelly_position_limit(edge, volatility, capital, max_kelly_fraction=0.25): """ edge: expected return per unit risk volatility: standard deviation of returns max_kelly_fraction: fraction of theoretical Kelly to actually use """ kelly_full = edge / (volatility ** 2) kelly_fraction = capital * kelly_full * max_kelly_fraction return kelly_position We use 25% Kelly as a hard constraint. Yes, this reduces expected value. It also ensures we'll still be trading next month. The Production Mindset The shift from expected value thinking to production thinking is philosophical: Research mindset: "What strategy has the highest expected return?" Research mindset Production mindset: "What strategy will survive being wrong about my assumptions?" Production mindset Here are the practical shifts we made: Backtests: Added worst-month analysis, not just average returns Position sizing: Conservative by default, with kill-switches for anomalies Risk metrics: Track CVaR daily, not just P&L Model validation: Assume 30% parameter uncertainty on all estimates Disaster planning: Explicit code paths for "model is completely wrong" scenarios Backtests: Added worst-month analysis, not just average returns Backtests Position sizing: Conservative by default, with kill-switches for anomalies Position sizing Risk metrics: Track CVaR daily, not just P&L Risk metrics Model validation: Assume 30% parameter uncertainty on all estimates Model validation Disaster planning: Explicit code paths for "model is completely wrong" scenarios Disaster planning The Lesson Expected value is a beautiful mathematical concept. It's clean, intuitive, and theoretically optimal. It's also not enough. In production, you're not trading against a probability distribution. You're trading against: Your own imperfect risk models Markets that change Operational constraints that aren't in your backtest The psychological reality of watching your capital decline day after day even though the "expected value is positive" Your own imperfect risk models Markets that change Operational constraints that aren't in your backtest The psychological reality of watching your capital decline day after day even though the "expected value is positive" The systems that survive aren't the ones with the highest expected value. They're the ones that remain robust when the model is wrong, markets shift, and black swans appear. Optimize for survival first. Profitability second. Expected value is a component of that calculation, but it's not the objective function.
