⏱️ Slide Timing: 1 min - In this session, - we talk about what is happening right now - the agent does OUR job - load data, build a model, report a number - and we evaluate its work - Same skill, different seat: you are the reviewer, not the builder

⏱️ Slide Timing: 3 min - Participants have probably used a chat assistant for snippets; agents differ because they execute and iterate - Platforms such as Fabric and Azure ML are adding these agents into the notebooks data teams already use - METR measures task length at 50% success; long tasks are exactly where silent mistakes pile up

⏱️ Slide Timing: 3 min - You touch only the two ends: the brief going in, the review coming out - Inside the loop the agent makes dozens of decisions you never see (which columns to drop, which split) - "It ran without errors" is the agent's stopping signal; it is not evidence of correctness

⏱️ Slide Timing: 2 min - Be honest with participants: these gains are real and employers expect them to use agents - Best use: compress the mechanical work so more time goes into judgement - Worst use: outsourcing the judgement itself, which is what the next slide is about > EDA: Exploratory Data Analysis

⏱️ Slide Timing: 4 min - None of these raise an error; the notebook is green all the way down - Every row is a classic data science mistake that a careful reviewer learns to catch - Specification gaming is documented in agent research: agents optimise the signal you give them ❓ Ask: "Which of these would YOU have caught in your last model, without being told where to look?"

⏱️ Slide Timing: 1 min - Same issue as vague project goals: "stop fraud" vs "flag 80% by value in back-testing" - Agent didn't misbehave; it did exactly what was asked - Fix starts with the brief, not with a better agent

⏱️ Slide Timing: 2 min - The goal is stated in business terms on purpose: dollars caught, analyst workload - Watch whether that goal ever reaches the agent; it is not in the one-line brief - The dataset is synthetic (simulated transactions), which matters later

⏱️ Slide Timing: 2 min - Brief and URL are in agent_review/briefs/brief_v1.md: copy and paste, don't improvise - Ask participants to write down every decision the agent makes, as it makes it - If network or licences fail, open the fallback notebook; the review exercise works the same either way

⏱️ Slide Timing: 10 min - Agents vary run to run; some will catch the duplicates. That is a useful discussion too - "would you have known it was right if it HAD caught them?" - Don't correct the agent mid-run; the point is to see unsupervised output - If the agent asks a clarifying question, answer minimally, as a busy manager would

⏱️ Slide Timing: 2 min - These are the numbers from agent_v1_output.ipynb; a live run will be close - Pause here and let it sit: would most managers accept this summary? Many would - "Perfectly balanced" fraud data is itself a red flag: real fraud is rare

⏱️ Slide Timing: 20 min - Hint ladder if a group is stuck at 10 min: - 1: "what does trans_num tell you?" - 2: "count df.duplicated() before the drop" - 3: "how many transactions per card?" - Don't reveal reference_review.ipynb until all groups have reported - Groups that finish early: compute recall by dollar value

⏱️ Slide Timing: 3 min - A classic trap, and it still catches people (and agents) who've seen it before - Stratified split does not help: it balances labels, it doesn't separate copies - Teaching point: check IDs before dropping them; they are your duplicate detector

⏱️ Slide Timing: 3 min - This one is subtler than duplicates: rows are different, the entity is the same - Same pattern elsewhere: patients in medical data, customers in churn, users in recommendations - Ask: "what will be NEW when this model runs in production?" Split on that

⏱️ Slide Timing: 2 min - Read left to right: each fix removes a source of leakage, and recall falls to its honest level - 15 percentage points of the agent's recall came from leakage, not skill - Accuracy tells a smaller story (98.9% → 96.2%) because the baseline rises from 50% to 78%

⏱️ Slide Timing: 3 min - Twist worth stressing: the review didn't kill the project, it made the claim trustworthy - By-value recall stays high because large frauds are easy (amount is the top feature); small ones slip through - Next questions a real team would ask: time-based split, cost of a missed $5 fraud vs a missed $1,000 one

⏱️ Slide Timing: 4 min - Full text in agent_review/briefs/brief_v2.md, with a table of what changed from v1 - Not naming the traps is the point: a good brief works on data you haven't inspected yet - Same structure works for briefing a human contractor; agents just make the cost of a vague brief visible faster

⏱️ Slide Timing: 3 min - If time is short, start the run before the previous slide and come back to it - Expect better but not perfect; point out anything still missed - Compare the two summaries side by side: tone shifts from salesmanship to evidence

⏱️ Slide Timing: 3 min - Checkpoints are the cheapest fix: most failures happen in the first 10 minutes of a run - Part 1 connection: an agent reviewing an agent is just an LLM judge; it needs calibration too - Least privilege is the agent's never-rules: decide what it must never do before it runs

⏱️ Slide Timing: 4 min - Reassure: every "matters more" skill is learnable, and is where good data science training already spends its time - Interviews are shifting: "here is an agent's notebook, what's wrong with it?" is a realistic exercise - Juniors who can review agent output are more valuable than juniors who only produce it

⏱️ Slide Timing: 2 min - Tie both parts together: Part 1 evaluated an agent's answers, Part 2 evaluated an agent's analysis - Common thread: write expectations first, test against ground truth, read the failures - Follow-up idea: give brief v2 to a different agent and compare the reviews

⏱️ Slide Timing: 1 min - Bookmark first: the scikit-learn grouped-data page; it applies to every project with repeated entities - Agent docs change fast; skim for how each tool asks permission before running code

⏱️ Slide Timing: 1 min - Kapoor & Narayanan is the must-read: leakage invalidated results across 17+ scientific fields, no agents required - Dataset origin explains the synthetic patterns (merchant names prefixed "fraud_", bursty cards)

⏱️ Slide Timing: 3 min - Invite questions on either part; common one: "should I trust the agent less than a junior colleague?" - answer: trust both the same way, through evidence and review - Remind participants the notebooks run in Colab with no keys required