Joint weight optimization via coordinate descent

Question: After the sentiment-only bump found by the marginal sweep + refinement (documented in the entry above), can a proper joint optimization find further weight changes that survive joint testing?

Methodology: scripts/coord_descent_weights.py runs proper coordinate descent on the current DEFAULT_HORIZON_WEIGHTS_BACKTEST: each cell swept on a 9-point grid {0, 0.05, 0.10, 0.15, 0.20, 0.25, 0.35, 0.50, 0.70} holding all other cells at their current best (not the baseline). Up to 3 passes; stops when a full pass produces no improvement above 0.005 Sharpe. news skipped (always 0 in backtest mode — no information). Total: 21 non-news cells × 9 weights × max 3 passes ≤ 567 backtests.

This contrasts with the naive "marginal sweep argmax stack" tested in the previous entry, which gave Sharpe -0.746 — every cell optimized against the SAME baseline, ignoring the fact that good cells overlap. Coordinate descent fixes this by optimizing each cell given the running state.

Setup: sp500 2014-2026, weekly rebalance, n_top=10. Starting point = the post-sentiment-bump default (Sharpe 1.582).

Result: CD converged in 2 passes with 3 changes locked in:

Total CD lift on sp500 12yr: Sharpe 1.582 → 1.606 (+0.025), CAGR +1.19 pp, MDD −0.42 pp (slight worsening).

Pass 2 found no further improvements above the 0.005 threshold — the algorithm explicitly REJECTED every other proposed change, including all sentiment further tweaks, sector cuts, fundamental adds, and turnaround upweights.

Anti-overfit validation across universes/windows

4/6 positive, 2/6 modestly negative. sp500 in-sample / OOS Sharpe lift agree to 1 basis point (+0.025 vs +0.024) — again exceptionally consistent on the main universe.

Negative cells (nasdaq100/sp600 OOS) are small but real. The insider/long jump from 0 → 0.70 is the biggest individual change and the most suspect — insider Form 4 data coverage is much thinner on small-caps than on sp500, which is where the regression lands hardest.

Decision

Adopt all 3 CD-found changes. New DEFAULT_HORIZON_WEIGHTS_BACKTEST (deltas from prior version marked with **):

Total cumulative lift vs original repo defaults (sp500 12yr): Sharpe 1.461 → 1.606 (+0.145), CAGR 51.12 % → 56.01 % (+4.9 pp), MDD −44.54 % → −40.31 % (~4 pp shallower).

Methodology notes

1. Proper coordinate descent finds smaller but cleaner gains than marginal stacking. CD locked in only 3 additional changes after the sentiment bump, vs the 24 cells the marginal sweep proposed argmax weights for. The marginal sweep over-promised by 5-10× per cell because it ignored joint state.
2. The "best" cells from marginal sweep mostly did NOT survive coordinate descent. Sector cuts, sentiment-to-0.35-or-higher, technical-long-to-0.60 — none of these appeared in the CD optimum. The marginal-sweep findings are useful as candidate pointers but should not be treated as optimization targets.
3. Coverage-thin layers may overfit. Insider/long going from 0 → 0.70 gave only +0.008 Sharpe lift but is the change most associated with the sp600/nasdaq100 OOS regression. Future calibration should consider per-name data coverage when interpreting large jumps on sparse-data layers.

Reports: - research/cd_weight_optimum_sp500_12yr.json (per-eval trajectory, final weight matrix, 345 evals)