Clinic 01

Public/Private Restraint

You have one hour left. The public leaderboard has moved, the private signal looks weaker, and you have to decide whether to trust the jump or stop chasing it.

Back To Clinics Open The Full Track Use The Tracks Route

Situation

Visible Gain, Hidden Risk

The public board says the more complex model won. The hidden signal and weak slice say the story may be brittle.

Your Job

Choose And Defend

Pick the model you would actually ship now, decide whether to keep iterating, and say what evidence would change your mind.

Bad Habit To Avoid

Public Rank = Proof

If your reasoning starts and ends with the visible rank, you missed the clinic.

Situation

You are reviewing the last serious run of a classification task with hidden evaluation.

The current packet says:

• the public leaderboard improved
• the private result prefers a simpler model
• the late-shift slice is weakest exactly where the public winner looks most attractive
• you have time for only one more deliberate move, if any

Artifact Packet

Read this packet before you decide:

model	public ROC AUC	private ROC AUC	public-private gap	late-shift recall
decision_tree	0.781	0.701	0.080	0.52
logistic_baseline	0.752	0.748	0.004	0.71
random_forest	0.768	0.719	0.049	0.63
dummy_prior	0.500	0.500	0.000	0.00

The tempting move is obvious: the decision_tree won the visible board.

The harder question is the real one: is the gain durable enough to trust when the hidden split and weak slice say otherwise?

Decision Prompt

Write the note before you open the reveal.

Your note should answer:

1. Which model would you ship right now?
2. Would you stop or continue?
3. Which single artifact drove the decision most?
4. What evidence would justify changing the decision?

Keep the note short. Four to six sentences is enough.

Strong Reasoning Looks Like

• it names the public winner and still refuses to treat that as proof
• it uses the private result and the weak-slice signal together
• it explains why a stable smaller gain is stronger than a brittle visible jump
• it says what one more iteration would need to target
• it is willing to stop if the next move is not clearly justified

Common Wrong Moves

• selecting the public winner without mentioning the hidden gap
• using complexity as a substitute for evidence
• saying "continue" without naming the exact failure pattern to fix
• treating the weak slice as optional because the overall score looks fine
• changing the split instead of changing the reasoning

Run The Clinic In Browser

The browser runner below prints the same packet and prompt. You can use it as a scratchpad while you write your note.

Validate Your Decision In Browser

Once you have made your decision, use the validator below to enter your answers and get immediate feedback.

Reference Reveal

Open only after you write the note

The reference choice is: - `selected_model = logistic_baseline` - `decision = stop` Why: - the public winner has the largest public-private gap - the stable model is also the private winner - the late-shift slice is materially better for the stable model - with only one hour left, there is no named next move strong enough to justify chasing the visible jump The practical lesson is not "always choose logistic regression." The lesson is: when the hidden evidence and weak slice disagree with the flattering visible score, the right call can be to stop chasing public gain.

What To Do Next

After this clinic:

1. run the matching local clinic example
2. answer the matching Decision Clinics exercise set in AI Academy