The afternoon breakout

Day 3 — simulate your evaluation before you model

Evaluation & Study Design · Matthew McDermott

AHLI HEALTH AI SUMMER CAMP 2026

Breakouts change gear today

  • Days 1–2: discussion groups around a sub-modality of health data.
  • Days 3–4: you build — a Colab / Jupyter notebook that simulates your evaluation (today) or your methods (tomorrow) setup.

If you'd like, drop your notebook into your folder in the workbook GitHub.

AHLI HEALTH AI SUMMER CAMP 2026

Why set up your evaluation first?

Design how you'll evaluate before you build the model. Doing it now helps you:

  • optimize for the right thing as you develop,
  • see where you're vulnerable to noise or to being misled by a metric,
  • know up front how to present and interpret your results.
AHLI HEALTH AI SUMMER CAMP 2026

Methods are the easy case — evaluation is the interesting one

For methods, a synthetic experiment is usually straightforward: build fake data so the factor of your model that matters is present (or absent) by construction. (More on this tomorrow — with examples.)

For evaluation, "synthetic experiments" are trickier and far less common. Pick whichever of these prototypes fits your project:

  1. You care about X, but work on Y
  2. You can measure X — but only noisily / through proxies
  3. The interpretation of X depends on hidden population properties
  4. Many metrics, many comparisons
AHLI HEALTH AI SUMMER CAMP 2026

Prototype 1 · You care about X, but work on Y

You ultimately care about impact X; the problem in front of you is Y.

  • First, can you reframe? Either assume Y is valuable because of X and just optimize Y — or work on X directly.
  • If not: write a simple mathematical model of how Y turns into X.

Even a rough model links what you can measure to what you care about — it makes your assumptions explicit and checkable, and shows how much a gain in Y is actually worth.

AHLI HEALTH AI SUMMER CAMP 2026

Prototype 1, worked · which region of the score matters?

McDermott et al., "A Closer Look at AUROC and AUPRC under Class Imbalance." NeurIPS 2024.

AHLI HEALTH AI SUMMER CAMP 2026

Prototype 2 · You measure X, but only noisily / via proxies

You can measure what you care about — but imperfectly, through proxies that are vulnerable to noise.

Ask: how sensitive is my evaluation to which kinds of noise?

  • Simulate different noise sources and see what flips your conclusion.
  • Write a sensitivity-analysis protocol — and test it on fake model results.
  • Track additional proxy metrics that help triangulate the true signal.
AHLI HEALTH AI SUMMER CAMP 2026

Prototype 3 · Interpretation depends on hidden population properties

You can measure X — but its meaning may hinge on properties of your population that aren't obvious.

Simulate different patient / sample properties, then ask:

  • How would my evaluation mislead me in each case?
  • How would I detect that it's happening?
AHLI HEALTH AI SUMMER CAMP 2026

Prototype 4 · Many metrics, many comparisons

You're comparing many options across many metrics, and need to communicate and prioritize them.

Less about simulating noise — more about how you aggregate and report:

  • average metric across tasks? average rank order? win-rate?
  • a Pareto frontier over competing metrics?

Decide early so you can tell when a model is actually winning.

Your task this afternoon

Build a notebook that simulates or sets up your evaluation: pick the prototype(s) that fit, generate fake data or fake model results, and pressure-test how you'd measure success — before you start modeling.

Add it to your workbook GitHub folder if you'd like. We'll regroup to share what we found.

SPEAKER NOTES — Slide 1 (Title) Quick framing before the breakout. Today's session is a build session, not a discussion group. Keep this short — the goal is to send them off with a concrete thing to try and a way to think about it.

SPEAKER NOTES — Slide 2 (The shift) Name the change explicitly: the first two days were about exploring data types together; the back half is hands-on. It's fine if the notebook is rough — the point is the thinking, not a polished artifact. The GitHub folder is optional.

SPEAKER NOTES — Slide 3 (Why first) The core motivation. If you wait until the model exists to think about evaluation, you've already been optimizing — possibly for the wrong thing. Setting it up first is cheap insurance against a whole class of mistakes.

SPEAKER NOTES — Slide 4 (Methods easy, eval tricky + menu) Contrast the two. Methods: isolate the factor by construction — they'll see real examples tomorrow. Evaluation: there's no single recipe, so offer prototypes. Tell them to pick one (or two) that match their project; they don't all apply.

SPEAKER NOTES — Slide 5 (Prototype 1) The most common situation. Push the reframing first — often the cleanest fix is to honestly assume Y→X and just optimize Y, or to go work on X. If you can't, a back-of-envelope model of Y→X is worth a lot, even if it's crude.

SPEAKER NOTES — Slide 6 (Metric ↔ use-case) A real worked example of Prototype 1: tie the metric ("classification accuracy", AUROC, AUPRC) to the downstream use — cancer screening, public-health targeting, top-k molecule screening — and it tells you which region of the score actually matters, and therefore which metric is the right one to optimize and report.

SPEAKER NOTES — Slide 7 (Prototype 2) Here you can measure the right thing, just not cleanly. The notebook move: inject controlled noise and watch whether your ranking/decision survives. Testing the protocol on fake model outputs means it's ready before the real model exists.

SPEAKER NOTES — Slide 8 (Prototype 3) Subtler than noise: the metric is "right" but conditional on population structure (case mix, subgroup balance, label timing). Simulate those properties and look for where the same number means different things — then design a detector.

SPEAKER NOTES — Slide 9 (Prototype 4) The foundation-model situation: dozens of tasks and metrics. The choice of aggregation (mean AUC vs. mean delta vs. win-rate vs. Pareto) changes the story. Pin it down before you have results so you're not tempted to pick the flattering one.

SPEAKER NOTES — Slide 10 (The task) Send them off. Concrete deliverable: a notebook that sets up the evaluation, not a model. Emphasize fake data / fake results — the whole point is to reason about the measurement before committing to a model. Offer to circulate during the session.