# MEME pre-run badge — results summary & what's possible

**Audience:** DM3 product + reviewers deciding whether/how to ship the pre-run "MEME outcome" badge.
**One-line:** the gate ranks well and is now honestly calibrated to the displayed event — three bands
are supportable, but the *word* "unlikely" can only truthfully mean "~1 in 4," not "almost certainly
nothing." That's a copy/product decision, not a model limit.

---

## Where we started

DM3's badge showed **"Unlikely to report a site"** below gate score 0.35. Validated against real MEME
jobs, alignments carrying that badge **actually had MEME report a site 72% of the time.** The badge's
headline claim was false roughly three times out of four — a release blocker.

Two root causes (both confirmed):
1. **Wrong training target.** The model predicted `max LRT ≥ 5`; the badge claims "MEME reports a site,"
   which MEME reports at p ≤ 0.1. Different events.
2. **Wrong population + no calibration.** Trained on a distribution unlike production traffic; scores
   weren't probabilities of anything users see.

Plus two open defects from veg/datamonkey3#151: the score was **non-monotone** (adding sequences could
*lower* it) and carried a **dead feature** (`frac_p_defined`, 0 of 2,205 tree splits).

---

## What we did

Retrained on the **user-visible label** (MEME reports ≥1 site at p ≤ 0.1), on **production traffic**,
with monotonicity enforced and the dead feature dropped. Then validated two ways — and critically,
**not** on cross-validation of the training set (that inflation is exactly what let the original slip
through), but on:
- a **held-out production test split** (900 jobs), and
- the **entire `/data` MEME corpus** (8,981 jobs — every result-with-tree on the server).

---

## Results

| | shipped model | v2 |
|---|---|---|
| Predicts | max LRT ≥ 5 | **MEME reports a site at p ≤ 0.1** (what users see) |
| Monotone | no (18–31% violations) | **yes** (native XGBoost monotone_constraints, 0 violations) |
| Features | 4 (one dead) | **3** |
| Calibrated | no (scores meaningless) | **yes** (scores are P(site)) |
| Production AUC | **0.745** | **0.87** (full corpus) / 0.91 (held-out) |

The ranking was always real; we fixed the *meaning* of the score.

---

## What's possible — the band structure (full /data corpus, 8,981 jobs, base rate 85%)

**Three honest bands, at score cuts 0.30 / 0.70:**

| Band | Score | **Actual chance MEME reports a site** | Share of jobs |
|---|---|---|---|
| Unlikely | < 0.30 | **25%** | 5.2% |
| Uncertain | 0.30 – 0.70 | **52%** | 11.2% |
| Likely | ≥ 0.70 | **93%** | 83.6% |

These three rates are cleanly separated — 25 / 52 / 93 — and the middle is a genuine coin-flip. **Three
bands work.**

### The one hard constraint (this is the decision)

At an **85% base rate, most alignments genuinely do have a site.** So:
- "Unlikely" can honestly mean **"~1 in 4 chance"** — but **not** "almost certainly nothing."
- There is **no** score cut that makes an "unlikely" band mean <5% *and* fire on a meaningful share of
  jobs. A "<5%" band would fire on **under 0.3%** of datasets — a badge that essentially never speaks.

So the product choice is between honest wordings, not between accurate and inaccurate models:

| Option | What it looks like | Trade |
|---|---|---|
| **A — 3 bands, honest copy** (recommended) | "MEME often finds little here (~1 in 4)" / "Could go either way" / "Likely to report a site" | Keeps the "unlikely" affordance; requires wording that admits 21%, not 0% |
| **B — 2 bands** | single 0.70 cut: "May find little here (~45%)" / "Likely to report a site (92%)" | Simplest, least copy nuance; loses the low-end signal |
| **C — drop the badge** | — | Only correct if the team won't ship any claim that "unlikely" ≈ 25% |

**Not viable:** keeping "Unlikely to report a site" as-is. At best it's 21%; users read it as ~5%.

---

## Recommendation

**Ship Option A** — 3 bands at 0.30/0.70 with truthful copy. The model is now honest, monotone,
exportable, and validated on the full production corpus. The only thing standing between here and
release is agreeing on wording that tells the truth at the observed rate — which is a one-meeting
product decision, not more modeling.

Fall back to **Option B** if the team judges that a 25%-"unlikely" badge is more confusing than helpful.

---

## Deliverables (in `stevenweaver/meme-gate-v2/`)

- **`meme_gate.json`** — XGBoost native format (single artifact, no converter). Dev team's runtime reads it directly.
- **`calibration_full_universe.json`** / **`calibration_report.json`** — the exact observed hit-rates per
  threshold (full corpus and held-out sample), so UI copy can be written against real numbers.
- **`GATE-V2-REPORT.md`** — technical write-up.
- Reproducible: **`train_gate_xgb.py`** (trainer -> meme_gate.json), `validate_full.py`, `validate_full_xgb.py`.

**One integration note:** the badge input contract changes from 4 features to 3 (`frac_p_defined`
dropped). Either migrate the app to `[1,3]` or slice the 4th slot before scoring.

*(Raw per-job validation data is real user submissions and was deliberately kept off the webshare and
out of the repo; it stays on silverback for reproduction.)*