Evaluation & Study Design

Day 3 — How would you know if it worked?

Shalmali Joshi · Columbia · June 24, 2026

Terminology

• ML model:
• AI system: Model + ecosystem in which it sits, including how is represented to the end-user

We will stick to "AI system" for the rest of this talk, but conceptually focus on some abstraction of (predictive risk score, regressed lab value, radiology report, synthetic image, etc.)

• Objective: To characterize what would count as success.

• Scope: We will not focus on metrics post deployment, continual monitoring, feedback and improvement

This talk will only focus on aspects of human-AI collaboration to the extent it relates to validation and evaluation choices

Precursor

• Assumptions:
  • Task, data, and how the model output is integrated into the workflow are known
  • Construct validity^* (what you are measuring is what you are intending to measure¹)
  • Design choices have been carefully considered: no leakage (e.g., hypertension medication as features for predicting risk of hypertension², antibiotics order to predict sepsis risk³)

Predicting 1-year risk of hypertension

Important features

Background/Primer: Types of distribution shifts

Models can deteriorate due to many factors, but we conceptually focus on distribution shifts. The two fundamental types of distribution shifts are:

• Covariate shift/X-shift: Commonly assumed factor contributing to lack of generalization
• Concept shift/Y|X-shift: The relationship between feature and outcome changed. Causal view: Concept shift points to presence of unobserved confounding

Background/Primer: Types of distribution shifts

• Accuracy on the line: Models improving on one dataset typically improve on other related datasets¹
• Accuracy on the line only holds under covariate shift²
• Which shifts are anticipated is usually an assumption, but we now have diagnostic tools to identify which shifts may be causing model deterioration^2,3

Case study: Epic Sepsis Model

• Definition of sepsis: differed in the initial UMich study (CDC + Sepsis-1 criterion)
• Sepsis-3 criterion needed to be used
• External validity fails despite these fixes
• Performance worse where patients are sicker (higher comorbidity rates)
• Some sites used antibiotics as feature to predict sepsis risk (leakage!)

Background/Primer: Shortcut learning

• Model learns from non-generalizable patterns¹
• Phenomenon is fairly general: happens even in multiple domains, including other forms of learning (learning on one modality because it is easier to learn)

The canonical and famous case: COVID-CXR deep learning

A deep learning diagnostic model was trained to predict COVID risk from chest X-rays¹

• The model flagged COVID from chest X-rays with high reported accuracy
• But was actually predicting on scanner make, patient position, and dataset source (DeGrave et al., 2021).
• The held-out split came from the same distribution, make it challenging to detect the shortcut.

Stages of evaluation: in silico to prospective trials

Target population vs. deployment population

• The population you evaluate on (retrospective/in silico evaluation) is rarely the population you deploy on.
• The gap should be identified apriori
• The gap must inform design and evaluation choices, including in fact whether the model is worth building!
• Example: SHD detection from ECGs using deep learning¹
  • Everyone with a cardiac complain gets an ECG, but SHD is only diagnosed using an ultrasound (selection bias)
  • Your X-Y pairs are complete only for those who were referred to an ultrasound by the cardiologist they saw

Target population vs. deployment population (cont.)

Example: SHD detection from ECGs (cont.)

• Doctors only refer patients to ultrasounds if they suspect SHD or want to rule out arrhythmias; ultrasounds are expensive, need specialized expertise to read and interpret
• Your model is trained only on this distribution, which is also your eval population because you need labels, but your test population is everyone who gets an ECG
• Will the model generalize? Is it worth building this model? Why or why not?

Metrics for In silico/Retrospective evaluation

Accurate prediction and harmful self-fulfilling prophecies¹

Treatment as confounding or source of distribution shift

• If we don't model how treatment impacts outcomes, to make treatment decisions using a predictive model, we will have very good AUROC but a worthless model
• This is a case of differing "treatment effects being heterogeneous conditioned on patient covariates (patients with fast growing tumors)"
• In RL language, you're generating your predictions based on a "historic/behavior policy", but your model itself changes the "treatment policy", model cannot disambiguate because model only approximates not

Your estimand might need to be a causal¹

• If your goal is to predict an outcome to make decisions about some intervention, then one must model the relationship between the intervention and the outcome!
• Prediction under intervention is a causal estimand:
• Observational data (your typical training data collected at point-of-care) does not give you this, basic deep learning gives you:
  • Causal identifiability assumptions are necessary to ensure that can be estimated from
  • Assumptions: No hidden confounding
  • Positivity: , consistency:
• Causal estimands can be empirically estimated using observational data if above conditions are satisfied or by running RCTs

In silico discrimination (AUROC, AUPRC)

• AUROC is probability a random positive outranks a random negative; prevalence-independent.
• AUPRC: area under the precision–recall curve; prevalence-dependent.
• Both aggregate behavior across thresholds
• Both metrics help rank models but say nothing about whether the probabilities are usable, or whether any single operating point helps downstream decision-making.

Discrimination is necessary, not sufficient. Calibration and decision utility come next.

Healthcare data is imbalanced and diseases can be very low prevalnce

A sepsis risk prediction model, disease prevalence is 1% prevalence, AUROC = 0.85-0.90, consider the setting where threshold is set for 80% sensitivity (TPR):

• False positive rates can wildly differ
• PPV or precision (TP/TP+FP) can be single-digits
• Alarm fatigue renders model clinically useless

AUPRC may be better for imbalanced datasets for in silico validation, though not always¹

• Given a threshold , AUPRC weighs false positives in inverse proportion to the likelihood of the score being greater than , whereas AUROC weighs all false positives equally
• Emphasizing AUPRC can result in choosing models that overly favor improvements in higher prevalence subgroups

Measuring discrimination at deployment

• Treatment decisions: If your model is enabling treatment decisions, and AUROC looks great at deployment, recall the self-fulfilling prophecy
• AUROC/AUPRC: Will necessarily change if the model is impacting decisions (not necessarily related to a treatment, but any intervention)
  • AI is an intervention in the system, that will introduce a shift in the patient population, data-collection, outcomes, etc.
  • If the AI intervention works, the AUROC could worsen (because metrics will emphasize subpopulations where the intervention did not work!)

In silico calibration is necessary but not sufficient

• Population-level calibration¹ usually plotting as a reliability diagram measures how well confidence scores generated by a model match the empirical/frequentist likelihood in the data: ; where
• Discrimination and calibration are independent.
• Subgroup calibration can differ from population calibration (called multicalibration²)
• Calibration is population-level statistic (see how it doesn't depend on ) and therefore is limited in what it tells us about using (the model score) at an individual level

In silico calibration and confidence scores

• A model can be over confident, under confident or a mix of the two¹
• Multiclass extensions is known as "confidence calibration": ; where
• Because its a population-level metric, it doesn't help identify what individual level decisions are the best
• We need both, good calibration AND good discrimination, only then can we reliably use the scores as actual confidence scores that should drive decision-making

In silico calibration versus calibration at deployment

• Deploying an AI model introduces a distribution shift induced by downstream decisions.
• That is AI itself is an intervention in the system (patient, healthcare institution, clinical workflow)
• Calibration is only guaranteed so long as the distribution does not shift!
• If your model approximates a causal estimand, or it predicts outcomes under interventions to make decisions, it will remain calibrated under shifts introduced as a consequence of the model!¹

Decision-curve analysis

• Define:
  , where is the threshold probability
• If is large, we're more worried about harm, if is small, we're more worried about the event (risk of cancer)

• Decision-curve analysis: does using the model beat "treat all" / "treat none"?
• Different stakeholders may have different threshold operating points
• Heterogeneous utilities may call for stratified or individualized threshold policies, not a single population-level threshold.

Decision-curve analysis versus discrimination

• Credit: David M. Kent (Tufts University) — example of a biopsy result (Prostate-Specific Antigen)
• A model with a worse AUROC and slightly worse Brier score may provide higher benefit at least for certain disease risk ranges
• Also means that probability of disease (threshold probability) needs to correlate with true risk uncertainty

Decision-curves can help identify harms of miscalibration

parameter simply introduces various levels of miscalibration by varying. Overestimation: overestimating risk, Underestimation: underestimating risk

Subpopulation analysis and fairness

• Population-level performance can hide subgroup failure modes.
• Minimally, all metrics should be evaluated at subpopulation levels
• Critical design choices: identify important subgroups (need not be demographic, could be healthcare utilization¹)
• When should we be demographic/subgroup aware versus not?

Subpopulation analysis and fairness (cont.)

When should we be demographic/subgroup aware versus not?

• Case studies¹: Race-unaware prediction were substantially miscalibrated compared to race-aware predictions
• However, the net clinical benefit of race awareness vs unawareness was very small (since majority of the people would receive the same downstream decision)
• Among patients who receive different decisions, the benefit was marginal, if focused on screening since the disease risks were closer to decision thresholds
• However, maybe when resources are very costly, race awareness may be a bit more beneficial
• We will not cover algorithms to enforce various fairness criterion here

Silent evaluation or validation

• Importance: First stage of prospective evaluation, model is integrated in the workflow but remains "non-interventional". That is, model does not inform care
• Goal: identify potential discrepancies and shifts in data collection and what the module consumes
  • Case study¹: Age distribution differences, proportion of left and right kidneys, and format differences caused an AI predicting obstructive hydronephrosis to deteriorate but was detected at the silent trial stage

Silent evaluation or validation (cont.)

• Preliminary validation against an in situ clinical ground-truth (this may be challenging depending on the end-point)
  • For example, outcome may not be available for a long time
  • If model is supposed to inform treatment decisions, then the best we're doing is comparing to clinician decisions
• Identifying issues early will increase the likelihood that your actual prospective evaluation (a clinical trial succeeds)¹
• Discrepancies persist in how silent trials are currently reported for predictive models²

Prospective validation: A causal view of Randomized Controlled Trials (RCTs)¹

• The goal of prospective evaluation is to estimate the causal effect of the AI intervention on a clinically relevant endpoint.

Prospective validation: A causal view of RCTs (cont.)

• If you simply integrate the model and observe patient outcomes, you will be able to estimate a prospective value of . To understand whether the AI model is any useful, what you need instead is
• An RCT randomizes the treatment (here an AI intervention), which severs the arrow from observed confounders (criteria you've selected subjects on) into :

• This makes — the average treatment effect (ATE) can now be read out by just averaging outcomes between groups for whom decisions were made with the AI and those without. There will be no need for further assumptions

AI as the intervention

• The AI system is the treatment : clinician makes decisions with vs. without AI support
• The outcome is a patient-level clinical endpoint, not a model performance metric, not whether you can imitate a clinician decision
  • Remember that AUROC improvement patient outcome improvement
• The estimand should be specified before the trial: for whom, what outcome, over what horizon
• You CANNOT peak into trial data while the trial is ongoing. That is why RCTs are "registered" before study begins.
• Deployment context matters: the same model may have heterogeneous effects across sites, clinician experience levels, and patient populations, this means we need to know different mechanisms of randomization and what they help with

Internal vs. external validity of an RCT

• Internal validity: the trial correctly estimates the causal effect for the enrolled population under the trial conditions
  • Your study design should account for: selection bias, potential non-compliance or automation, contamination, Hawthorne effects, clinician learning
  • Hawthorne effects: performance temporarily improves because subject knows they're being observed
• External validity (generalizability): the effect holds in the target deployment population
  • Threatened by: site-specific variation, different patient distributions, different workflow integration and protocols
• For AI interventions, clinician learning and system-level adoption are additional threats not present in drug trials
• A silent trial phase prior to the RCT reduces both classes of threats¹

Taxonomy of RCT designs for AI evaluation

Three organizing dimensions:

No single design is universally appropriate. The right design follows from the estimand, the unit at which the AI is deployed, and operational constraints.

Group assignment designs: parallel and crossover

Parallel RCT

• Each subject is randomized once to intervention or control; arms run concurrently
• Most basic design, no carryover; requires sufficient sample for statistical power
• Standard when the AI intervention cannot be reversed or washed out

Crossover RCT

• Each subject receives both conditions in sequence, separated by a washout period (try to come up with examples where you might need this)
• Increases efficiency (each subject is their own control); reduces required samples or power
• Requires: washout period long enough to eliminate carryover; outcome must be reversible
• AI-specific concern: clinician learning during the first period constitutes carryover and may not wash out, might warrant longer washouts

Group assignment designs: cluster RCTs

• Unit of randomization: groups of subjects (hospitals, wards, clinician panels) rather than individuals
• Required when the AI is deployed at the system level and individual randomization is not feasible or would cause contamination. Preferred for many AI interventions because AI interventions tend to be system/workflow level interventions
• Intracluster correlation (ICC): subjects within a cluster are more similar than subjects across clusters; reduces effective sample size
• Design effect: , where = cluster size, = ICC
• Sample size must be inflated by the design effect; ICC should be estimated from pilot or historical data

Examples: CONCERN early warning system at Columbia

Used real-time nursing documentation patterns to identify risk of deterioration upto 42 hours earlier than other early warning systems

One-year multisite cluster randomization of acute and intensive care units

Tool displays a categorical risk score (low, increased, high)

Primary endpoints: in-hospital mortality, LOS; Secondary outcomes: cardiopulmonary arrest, sepsis, unanticipated ICU transfers, 30-day readmission

Results: 35.6% decreased risk of death, 11.2% decreased LOS, 7.5% decreased risk of sepsis

Intervention and factor configurations: factorial and multi-arm RCTs

Factorial RCT

• Tests two or more interventions (two non-competing AI models) simultaneously in a single trial
• design: four arms — , , ,
• Efficient when interaction between factors is of scientific interest or assumed absent
• AI use case: test AI alert () and a clinical decision support checklist () simultaneously
• If interaction is present, main effects cannot be interpreted in isolation

Multi-arm RCT

• Three or more arms with a shared control; tests multiple doses, variants, or competing interventions
• More efficient than separate two-arm trials sharing no participants
• Multiplicity must be controlled; pre-specify primary comparison and adjustment method

Patient characteristics and adjustments: stratified RCTs

Stratified RCT

• Randomization is performed within pre-specified strata (e.g., site, severity, age group) to ensure balance on prognostic covariates
• Reduces chance imbalance; increases precision; enables pre-specified subgroup analyses
• Strata should be chosen based on variables known to moderate the treatment effect

Patient characteristics and adjustments: adaptive RCTs

Adaptive RCT: So far this is most commonly done with mobile health interventions/nudges

• Allocation probabilities or trial parameters update as data accumulate
• Response-adaptive randomization: shifts allocation toward better-performing arms; raises ethical and statistical concerns (non-stationarity, inflation of type-I error)
• Group sequential designs: pre-specified interim analyses with stopping rules for efficacy or futility; widely accepted; controls type-I error
• Bayesian adaptive designs: update priors continuously; require careful pre-specification to avoid gaming

For AI interventions with high upfront uncertainty about effect size, group sequential designs with pre-specified interim analyses are a pragmatic choice (only time you can peak).

Pilots or early live clinical evaluations

Before running a full prospective study, small-scale live evaluations are helpful to identify issues that silent trials cannot:

1. Silent trials cannot establish safety, utility, human factors (since model is not integrated into care)
2. Might be necessary for certain trial designs, e.g., cluster randomizations to estimate intra-cluster correlations or baseline variances and covariances in crossover designs
3. It may not be worth investing in full scale prospective randomized studies if the model does not impact workflow as intended
4. See DECIDE-AI for guidelines on what needs to be reported.

Clinical trials for dynamic AI systems: the gap¹

• Current standards (SPIRIT-AI, CONSORT-AI) require the exact algorithm version to be prespecified — treating AI as a static intervention
• In practice, AI is a dynamic socio-technical system: it is embedded in health-IT infrastructure, clinical workflows, and user interfaces, and requires ongoing performance monitoring and model refinement
• Continuous updating is not a deviation from the protocol — it is intrinsic to how the intervention works and persists after the trial ends
• Precedent exists: implementation trials allow protocolized local tailoring; titrated drug trials allow dose adjustment per prespecified algorithm — the same principle applies to AI

The result: trials evaluating static AI snapshots generate evidence that does not reflect how the system operates in real-world deployment.

A governance framework for dynamic AI trials¹

AI models pose some unique challenges, in that they require monitoring for ensuring proper function, but peaking into model behavior is a no-go during prospective validation. It is crucial to make the distinction of coexisting monitoring categories since it dictates a governance boundary:

Monitoring as part of trial conduct (independent DMC)

• Exclusive access to between-arm comparisons of clinical endpoints
• Retains authority to pause enrollment; protects trial integrity
• Has no analogue in real-world deployment

A governance framework for dynamic AI trials (cont.)

Monitoring as part of the intervention (AI-QI team, independent of trial investigators)

• Operational metrics: uptime, error logs, alert frequency, human–AI interaction signals
• Protocolized retraining or recalibration triggered by prespecified thresholds
• No access to comparative endpoint data from the trial

The AI-QI team exists during and after the trial; the DMC exists only during the trial.

Risks and mitigations

Main takeaway: the boundary between trial oversight and intervention delivery must be drawn explicitly.

• Trials that conflate the two either freeze the AI (producing unrepresentative evidence) or allow uncontrolled adaptation (threatening validity).
• Nonetheless allowing model monitoring seems to be critical.
• Recent work has attempted to codify how these differences should be accounted for in AI prospective validation

Risks and mitigations: recommendations (Table 1)

Case study 1: detection of pulmonary embolism¹

Design: Multi-center cluster-randomized trial, where hospitals randomized to AI system vs. standard care

• Primary endpoint: all-cause mortality; secondary: serious adverse events (SAEs) related to anti-thrombotic treatment

Monitoring as part of trial conduct (DMC)

• Exclusive access to between-arm differences in deaths and SAEs
• May recommend pausing enrollment if benefit-to-risk profile is unfavorable
• Between-arm comparisons cannot be shared with AI-QI team

Case study 1: detection of pulmonary embolism (cont.)

Monitoring as part of the intervention (AI-QI team)

• Operational metrics: server uptime, image acquisition quality, model outputs, radiologist agreement with alerts
• Prespecified actions: automated failover for downtime; retraining or recalibration if radiologist confirmation rates decline
• No access to comparative clinical endpoint data during the trial

Case study 2: automated discharge summary generation¹

Design: Non-inferiority randomized trial comparing AI-drafted discharge summaries vs. manual documentation

• Primary endpoints: readmission rate, documentation quality (accuracy and completeness assessed by independent reviewers)
• Secondary: time savings, self-reported clinician burnout, cognitive burden metrics
• Effectively open-label: clinicians know whether AI drafted the summary; subjective endpoints increase leakage risk

Monitoring as part of trial conduct (DMC)

• Retains sole responsibility for between-arm comparisons
• Documentation quality assessment may need to remain trial-specific (resource-intensive manual review not feasible in routine deployment, so this remains in this governance boundary)

Case study 2: automated discharge summary generation (cont.)

Monitoring as part of the intervention (AI-QI team)

• Permitted: technical functionality, user training updates, vendor-released model updates (importantly, can allow for updates to improved foundation models)
• Tightly restricted: serious summarization errors must be compartmentalized to AI-QI team only since disclosure to DMC could bias subjective endpoints
• Primary clinical endpoints remain exclusive to the DMC

Even retrospective evaluations of deployed AI tools needs careful design and analysis

Use case: Analysis of Augmented Response Technology (ART) or In-basket messaging, which creates LLM-generated editable draft responses for clinicians and is now widely integrated across many healthcare institutions across the US

Picture credit: Vincent Jeanselme

Prior ART evaluations and study designs

Picture credit: Vincent Jeanselme

Prior ART evaluations and study designs (cont.)

Challenges of prior study designs

• One RCT, pretty small (n=52)
• Pre-post mean comparison: many sources of confounding — e.g., which clinicians opt-in to use the tool?
• Used vs. not-used mean comparison: what type of messages do clinicians most often use the tool for?

Overall, prior work focused on reading and drafting time — but not efficiency-related and other outcomes relevant for clinicians, patients, and healthcare institutions. What is crucial to measure?

ART panel analysis that adjusts for various sources of confounding¹

Picture credit: Vincent Jeanselme

ART panel analysis that adjusts for various sources of confounding (cont.)

• Message-level confounders: message-type
• Primary endpoint: turnaround time (famously, "pajama time")
• Provider-level confounders: share provider-level effects before and after the tool is introduced
• ART-adoption, use for a specific message, and habituation

Other challenges

• Often there are no timely labels (live deployment and monitoring comes with its own challenges) or no single right answer (generative output).
• Selective prediction / abstention — let the model decline; report the risk–coverage tradeoff, not one accuracy. However, simply focusing on calibration is not the best way to determine abstention (learning-to-defer) because of challenges we discussed above
• LLM-as-judge: convenient, scalable, but systematically biased does not evaluate ground truth or an actual end-point that implies positive impact.
• Generative outputs: For radiology report generation, discharge summarization, clinically meaningful end-points are the only way to determine that the tools have positive impact. Famously, traditional semantically focused metrics show very little correlation with expert judgements¹

This afternoon: design a full evaluation pipeline for your usecase

The 2:45 notebook runs end to end on your project — Sections 0–7:

• 0 Project setup → 1 Discrimination → 2 Calibration (overall + subgroup)
• 3 Clinical utility (decision curves) → 4 Leakage & shortcut probes → 5 Model selection
• 6 Identify stages of study designs, what you will measure in each (silent trial, safety/small pilot, prospective evaluation), propose end-points and justify the motivation, use genAI and the internet to identify power analyses, and determine which pilot studies you would need to run to identify appropriate parameters
• 7 Report all designs with justification

Worked example, template, and AI-coding-tool guidance are provided — see the Day 3 & 4 Workshops doc.