We had a problem. Our automated trading system had a positive expected value: the math checked out, the backtests looked great, and initially, it made money. But over time, it was bleeding. Small losses that accumulated faster than the occasional wins could compensate.
This wasn't a bug in the code. It was a fundamental misunderstanding of what matters in production.
The Expected Value Trap
Most trading tutorials, academic papers, and online courses teach you to maximize expected value. The logic seems bulletproof:
E[profit] = Σ(probability_i × outcome_i)
If this number is positive, you should take the trade. If you can make this number bigger, you should optimize for it. Simple, right?
Except in production, this optimization strategy has a fatal flaw: it doesn't account for the path you take to reach that expected value.
Let me show you what I mean with a real scenario from our system.
The Bleeding System
Our strategy was designed to capture price spikes in volatile markets. The model would:
- Analyze possible price directions for each trading window
- Optimize position sizing using quadratic programming
- Execute trades to capture spread opportunities
On paper, the expected value was solidly positive. In practice:
- Day 1-3: Caught a major spike, made $15,000
- Day 4-12: Small losses every day, total -$8,000
- Day 13-14: Another spike, made $12,000
- Day 15-28: Gradual bleed, total -$11,000
The problem? Our optimizer had developed a structural bias. It was systematically taking positions that won big occasionally but lost small amounts frequently. The expected value calculation said this was fine: the big wins would eventually compensate. But "eventually" requires infinite capital and infinite time horizon.
We had neither.
Seeing The Difference: A Simulation
To illustrate why these risk controls matter, let's compare two strategies trading the same market over one year:
Strategy A (EV Maximization): Aggressive position sizing based purely on expected value, using 150% leverage when opportunities look good.
Strategy B (Risk-Controlled): Same market signals, but with fractional Kelly sizing (40% of aggressive) and CVaR-based position reduction during high tail risk periods.
The results tell a crucial story. Look at the left chart closely - most EV-maximization paths aren't catastrophically failing. They're just... not compounding. You can see the sawtooth pattern: occasional spikes up, followed by slow erosion. This is the insidious bleeding that positive expected value misses.
Notice how a few paths reached $500k? Those outliers pull the mean up to $146k. But the median is only $136k, and 29 out of 100 paths end below starting capital. In a backtest, you might have gotten lucky and seen one of those winner paths. In production, you get one random draw.
The right chart is "boring", and that's exactly the point. No moonshots to $500k, but also no catastrophic drawdowns. The risk-controlled strategy clusters tightly around modest growth. It survives to compound returns over multiple years.
This is the production reality: the strategy that survives gets to compound. The strategy that bleeds out makes nothing, regardless of what the expected value calculation promised.
What Expected Value Doesn't Capture
1. Risk of Ruin
This is the classic gambler's problem, formalized by the Kelly Criterion. Even with positive expected value, if your position sizing is wrong, you will go broke.
Consider: You have $100,000 capital and a trade with 60% win probability that either doubles your bet or loses it. Expected value is positive (+20%). But if you bet everything, you have a 40% chance of losing it all on the first trade.
Kelly tells you the optimal bet size is:
kelly_fraction = (p * b - q) / b
# where p = win probability, q = loss probability, b = odds
But here's what we learned in production: even Kelly is too aggressive.
Why? Because:
- Your probability estimates are wrong (always)
- Markets change (your 60% edge becomes 52%)
- Correlations break down during stress (when you need them most)
- You can't rebalance instantly (slippage, latency, market impact)
We ended up using fractional Kelly (25-50% of the theoretical Kelly bet) because the real-world costs of overestimating your edge are catastrophic.
2. Numerical Instability in Extreme Events
One morning, our system crashed during an extreme weather event. Not a software crash, but a mathematical one.
Our covariance matrix became singular. The optimizer couldn't find a solution. We were frozen, unable to trade, during the exact conditions where our strategy should have made the most money.
The problem: we had optimized for expected scenarios. But extreme events have different correlation structures. Assets that normally move independently suddenly become perfectly correlated. Your carefully estimated covariance matrix, built from thousands of normal days, becomes useless.
The fix wasn't better expected value calculations. It was regularization:
from sklearn.covariance import LedoitWolf
# Instead of sample covariance
cov_matrix = np.cov(returns.T)
# Use shrinkage towards structured estimator
lw = LedoitWolf()
cov_matrix_robust = lw.fit(returns).covariance_
This trades off some accuracy in normal times for stability in extremes. Your expected value calculations will be slightly worse. Your system will survive black swans.
Time Horizon Mismatch
Here's a problem that doesn't show up in backtests: your expected value calculation assumes you can wait long enough for the law of large numbers to work.
In production, you can't.
We discovered this when our system showed strong positive expected value over 90-day windows but consistently lost money over 30-day windows. The problem wasn't the math. It was the business reality.
Our capital providers reviewed performance monthly. Our risk limits were adjusted quarterly based on recent results. If we had three bad months, our position limits got cut, regardless of what the long-term expected value said.
The theoretical strategy needed 6-12 months to reliably show profit. The operational reality gave us 3 months before consequences kicked in.
We had to add explicit time-horizon constraints to our optimization:
def optimize_with_horizon_constraint(scenarios, max_horizon_days=30):
"""
Optimize not just for long-term EV, but for probability of
positive returns within operational time horizon
"""
# Standard expected value
ev = np.mean(scenarios)
# But also: what'sthe probability we're profitable
# within our actual time horizon?
rolling_returns = pd.Series(scenarios).rolling(max_horizon_days).sum()
prob_profitable_in_horizon = (rolling_returns > 0).mean()
# Penalize strategies with low short-term win probability
# even if long-term EV is great
if prob_profitable_in_horizon < 0.6:
return ev * 0.5 # Heavily discount
return ev
This meant accepting strategies with slightly lower theoretical expected value but higher probability of showing profit within our operational constraints. It's not mathematically optimal, but it's practically necessary.
What to Optimize Instead
After painful lessons, here's what we learned to optimize for:
1. Risk-Adjusted Returns with CVaR
Instead of maximizing E[profit], we minimize CVaR (Conditional Value at Risk): the expected loss in the worst 5% of scenarios
import cvxpy as cp
# Decision variable: position sizes
positions = cp.Variable(n_assets)
# Scenarios returns
scenario_returns = get_price_scenarios() # shape: (n_scenarios, n_assets)
portfolio_returns = scenario_returns @ positions
# CVaR constraints
alpha = 0.05 # 5% tail
var = cp.Variable()
u = cp.Variable(n_scenarios)
constraints = [
u >= 0,
u >= -(portfolio_returns - var),
]
cvar = var + cp.sum(u) / (n_scenarios * alpha)
# Optimize for return while constraining tail risk
objective = cp.Maximize(cp.sum(portfolio_returns) / n_scenarios - lambda_risk * cvar)
This explicitly penalizes strategies that have good average returns but catastrophic tail risk.
2. Robustness to Model Error
We assume our model is wrong and optimize for the worst-case within a reasonable uncertainty bound:
# Instead of single expected return estimate
mu_estimated = historical_returns.mean()
# Assume uncertainty
mu_lower_bound = mu_estimated - 2 * historical_returns.std() / np.sqrt(len(historical_returns))
# Optimize for worst-case in uncertainty range
# (Robust optimization / minmax approach)
This protects against overconfident parameter estimates.
3. Kelly-Constrainted Position Sizing
We explicitly limit position sizes based on Kelly criterion, even when the optimizer wants more:
def kelly_position_limit(edge, volatility, capital, max_kelly_fraction=0.25):
"""
edge: expected return per unit risk
volatility: standard deviation of returns
max_kelly_fraction: fraction of theoretical Kelly to actually use
"""
kelly_full = edge / (volatility ** 2)
kelly_fraction = capital * kelly_full * max_kelly_fraction
return kelly_position
We use 25% Kelly as a hard constraint. Yes, this reduces expected value. It also ensures we'll still be trading next month.
The Production Mindset
The shift from expected value thinking to production thinking is philosophical:
Research mindset: "What strategy has the highest expected return?"
Production mindset: "What strategy will survive being wrong about my assumptions?"
Here are the practical shifts we made:
- Backtests: Added worst-month analysis, not just average returns
- Position sizing: Conservative by default, with kill-switches for anomalies
- Risk metrics: Track CVaR daily, not just P&L
- Model validation: Assume 30% parameter uncertainty on all estimates
- Disaster planning: Explicit code paths for "model is completely wrong" scenarios
The Lesson
Expected value is a beautiful mathematical concept. It's clean, intuitive, and theoretically optimal.
It's also not enough.
In production, you're not trading against a probability distribution. You're trading against:
- Your own imperfect risk models
- Markets that change
- Operational constraints that aren't in your backtest
- The psychological reality of watching your capital decline day after day even though the "expected value is positive"
The systems that survive aren't the ones with the highest expected value. They're the ones that remain robust when the model is wrong, markets shift, and black swans appear.
Optimize for survival first. Profitability second. Expected value is a component of that calculation, but it's not the objective function.