STRATEGY SPECIFICATION - Walk-Forward Factor Selection with Meta-Selection

This is the complete, self-contained specification of the strategy. An AI
or a student can implement it from this document alone, without access to
any other course file. Follow it exactly; the constraints are the point.

You are implementing a quantitative equity strategy in Python
(numpy/pandas/scipy only; deterministic; no external data).

DATA YOU WILL RECEIVE

Three parquet files:

1. ctff_chars.parquet - one row per stock-month.
   Columns: id (stock identifier), eom (month-end date), sic, size_grp,
   ret_exc_lead1m, a boolean ctff_test, and ~400 float columns of stock
   characteristics (e.g. market_equity, ret_12_1, be_me, ...). About 1.4M
   rows, monthly 1951-2023, US stocks.
   CRITICAL: ret_exc_lead1m on the row for month t is the stock's excess
   return over month t+1. It is the prediction target, not a feature. It
   becomes known only at the end of month t+1.
2. ctff_features.parquet - a single column `features` listing the ~400
   characteristic column names.
3. ctff_daily_ret.parquet - id, date, ret_exc daily returns (optional;
   you may ignore it).

To build these files yourself from WRDS, see 02-DownloadJKPdata.py.

REQUIRED INTERFACE

def main(chars: pd.DataFrame, features: pd.DataFrame,
         daily_ret: pd.DataFrame) -> pd.DataFrame:
    """Returns a DataFrame with exactly the columns id, eom, w
    (stock-level portfolio weights, no NaNs). Weights at eom t are held
    over month t+1 and earn ret_exc_lead1m of row t."""

THE STRATEGY: FOUR STAGES

Stage 1 - Factor portfolio returns. For every characteristic and every
month: percentile-rank stocks cross-sectionally, weight each stock by
(rank - 0.5), and compute the factor's long-short return
LS = SUM(w_i * ret_i) / SUM(|w_i|) using ret_exc_lead1m. Require >= 20
stocks with valid ranks, else NaN. Drop factors with fewer than 60 valid
months. Result: a (months x factors) matrix. The value stored at month t
is realized over month t+1.

Stage 2 - Walk-forward simulation of a configuration menu. Define a fixed
symmetric menu of 36 configurations:
window in {10y, 20y, expanding} x n_factors in {5, 10} x selection in
{Sharpe, ClusterSharpe} x weighting in {EW, RankSharpe, MeanVar}.

- Sharpe selection: top-N factors by annualized Sharpe over the window.
- ClusterSharpe: hierarchically cluster factors on (1 - |corr|) of their
  LS returns (average linkage, max(3*N, 20) clusters), take the
  best-Sharpe factor from each of the top N clusters.
- EW: equal weights. RankSharpe: weights proportional to Sharpe rank
  among the selected (best gets largest). MeanVar: w proportional to
  inv(Sigma) * mu with 30% shrinkage of the covariance toward its scaled
  identity, clipped to >= 0, normalized to sum to 1; fall back to EW if
  degenerate.

Every 6 months (rebalance date rp), each configuration selects factors and
weights using ONLY LS rows <= rp - 1 month, then earns the realized LS
returns of its chosen factors until the next rebalance. Record each
configuration's resulting out-of-sample return series. Skip rebalances
with under 24 months of accumulated history.

Stage 3 - Meta-selection. At each rebalance, compute each configuration's
annualized Sharpe over its own out-of-sample returns up to rp - 1 month
(expanding window). Among configurations with >= 120 months of history,
blend the top 3 equally. If none qualifies yet, blend ALL active
configurations equally. The methodology live at any date thus depends only
on data before that date - this is the design's entire purpose: it removes
configuration selection bias.

Stage 4 - Stock weights. For each month, take the governing rebalance's
chosen configurations; for each, map factor weights to stocks: stock i
gets fw_k * (rank_i - 0.5) / SUM_j(|rank_j - 0.5|) summed over its
factors k; average equally across the chosen configurations. Output all
non-zero weights as (id, eom, w).

HARD CONSTRAINTS

- No look-ahead, three layers. Factor selection, factor weighting, and
  meta-selection at rebalance rp must each use only information realized
  by the end of month rp, i.e. LS-matrix rows <= rp - 1 month (because row
  rp - 1 is realized at end of month rp). Stock ranks in Stage 4 use only
  month-t characteristics.
- Deterministic: no randomness anywhere.
- Handle small datasets (< 200 months): scale down minimum-observation
  thresholds and the 120-month meta requirement (e.g. to n_months/3) so
  the strategy still produces output; the EW-prior fallback must work.
- Performance: full dataset must run in well under one hour on a laptop.
  Vectorize the monthly loops over the ~400-column matrices; cache factor
  selections shared between configurations that differ only in weighting.

DELIVERABLES

1. strategy.py with main() as specified, plus a __main__ runner that
   loads the parquets from a --data directory, runs the backtest, prints
   annualized gross Sharpe for 1960-1989, 1990-2003, 2004-2013, 2014-2023,
   1990-2023, and average monthly one-way turnover, and writes the
   portfolio weights to weights.csv in the SUBMISSION format below.
2. A rebalance-by-rebalance log of which configurations the meta layer
   chose (date, mode, labels, their trailing OOS Sharpes), saved as TSV.
3. A short README: how each stage avoids look-ahead, and which design
   constants were fixed a priori.

SUBMISSION (JKP CTF) - rules: https://jkpfactors.com/ctf/rules

The strategy is built to be submittable to the JKP Characteristics Trading
Factor challenge. A complete submission is:
- The model script (.py/.r), self-contained, exposing main() with the
  REQUIRED INTERFACE above; <= 1 MB, UTF-8. No manual steps to run.
- A pinned dependency file: requirements.txt or pyproject.toml with exact
  versions (R: renv.lock).
- The portfolio weights as CSV: columns id, eom (YYYY-MM-DD), w; no missing
  values; <= 50 MB. Generated from main()'s return value. Only rows with
  ctff_test = True are scored, but all rows may be exported.
- A methodology document (PDF) describing the approach - highly encouraged.
Monthly rebalancing is mandatory. The challenge auto-detects look-ahead by
re-running the script on truncated data and requiring identical predictions
(this is exactly verification check V2); determinism (V5) is also required.

VERIFICATION CHECKLIST (run these before declaring success)

- V1 (shift test): shift the entire LS matrix one month LATER
  (ls.shift(1)) before Stages 2-3 and rerun. Performance should degrade
  only mildly (information is one month staler). If instead it collapses
  or jumps dramatically, the original code was reading contemporaneous or
  future rows somewhere.
- V2 (truncation test): run main() on data truncated at 1990-01, and on
  the full sample. The factor selections and meta-choices at every
  rebalance before 1989 must be IDENTICAL in both runs. Any difference
  proves future data leaks into past decisions.
- V3 (replication identity): for a sample of months, confirm
  SUM_i(w_i * ret_exc_lead1m_i) equals the equally-blended factor-level
  return of the chosen configurations to numerical precision.
- V4 (sanity ranges): monthly gross portfolio returns should have
  annualized volatility roughly 2-6% at unit gross leverage; an annualized
  Sharpe above ~4 in any long period almost certainly means a bug (most
  often ret_exc_lead1m used as a characteristic, or selection on rows
  that include the formation month).
- V5 (determinism): two consecutive runs produce byte-identical output.

BACKGROUND READING

- Jensen, Kelly & Pedersen (2023), "Is There a Replication Crisis in
  Finance?", Journal of Finance - the characteristics dataset.
  Documentation: https://jkpfactors.com and the bkelly-lab/ReplicationCrisis
  GitHub repository.
- Brandt, Santa-Clara & Valkanov (2009) - parametric portfolio policies
  (the linear-rank weighting used here is a close cousin).
- Kelly, Kuznetsov, Malamud & Xu (2025), NBER WP 33351 - transformer-based
  asset pricing (a deep-learning extension of the same managed-portfolio
  idea).