Five challenges
Examples, not a plan — ranked by signal value per unit effort. Exposure = unreleased-checkpoint contact: none, metered via a blind endpoint, or internal only.
Enter the harness into live competitions
Score Antigravity on the hidden private split of competitions whose task and data were not public before the training cutoff — the same split used for humans.
Qualifying competitions are entered as they open; where none fits, a free Community Competition can be authored. For existing competitions nothing changes and no contestant is affected.
An evaluation that cannot be contaminated, and one that renews as qualifying competitions open — and where none fits, one is cheap to create. The result is a percentile against thousands of practitioners rather than an abstract rate. Data-science work is long-horizon and tool-using by nature, so this tests sustained tool use rather than a single model call.
Watch for Data-science competitions are not the whole of agentic work, so a strong result here does not license a claim about coding agents generally. Entering under a Google name is a decision, not a technical question.
Read Meta Kaggle Code
The source of every public notebook version, weekly, back to 2015. Competition notebooks carry a score at the end of each history. Already public.
No Kaggle build: read the existing corpus. Each notebook history approximates a human working sequence — false starts, failed features, fixes — with the competition score at the end.
Multi-step human problem-solving with an outcome score attached to each submitted version, at a scale no crowd programme could be paid to reproduce.
Watch for Notebook iteration is not agent iteration — the human had intent we cannot observe, and only public notebooks are included. A corpus to learn the shape of human process from, not trajectories to imitate.
Run agent episodes beyond board games
kaggle-environments already runs autonomous agents and keeps the replay. Change the environments, not the machinery.
kaggle-environments already runs autonomous agents and retains the replay. Reuse the episode machinery; change the environments — tool use, a terminal, a file tree, tasks where agents compete or cooperate under partial information.
Training signal comes from episodes the Antigravity harness itself runs against Kaggle-authored environments and opponents. A contestant’s episode is their trajectory, not on-policy data — hence internal.
Watch for Game-shaped tasks drift from real work. Anchor every environment to an automatic executable check, and rotate tasks, or contestants optimise the environment instead of the capability.
Compete on the harness itself
Fix the checkpoint, expose it behind a metered endpoint, and let thousands compete on context selection, compaction, retry and delegation.
Prerequisite: Kaggle’s container-native Harbor executor is generally available. Then entrants submit a scaffold, not a model. The checkpoint is fixed and its identity hidden, reached only from scoring under a token budget. Scoring is task success against cost, on a fresh set of repositories each round so nobody wins by fitting last month’s distribution.
The distinctive Kaggle play: a competitive population optimising the harness itself — context selection, compaction, retry, delegation — at a scale an internal team cannot reproduce. Winning scaffolds have a direct path into the shipped harness.
Watch for Early-stopping heuristics that abandon hard tasks cheaply — charge unresolved tasks the maximum cost. A scaffold tuned to one checkpoint may not transfer to the next, so the result has a shelf life.
Calibrate reward; collect hard failures
Two linked competitions: calibrate the reward, and collect reproducible failures.
Given a passing agent run, predict a held-out senior-engineer correctness or preference label. Log-loss on held-out labels — the format Kaggle has run for a decade — turns the ~30% test-pass divergence rate into a calibration target.
A permanent bounty for reproducible agentic failures on a rotating hold-out — recovery loops, malformed tool calls, instructions lost at the compaction boundary. Accepted breaks join a private set and are never published, so the benchmark cannot saturate.
Watch for Judge labels are the expensive half; for red-team submissions, reward novelty and reproducibility, never volume.
Where the incentive comes from
Medals first, cash last
Kaggle pays in standing before cash. Use that order: medals and tier progress; compute quota, cheap to give and worth more to the recipient; named authorship, which recruits researchers; and cash only to seed a launch cohort.
Around 20,000 Expert-or-above placements sit across the three tracks, and those accounts do the work — but benchmark and agentic contributions earn no tier progress today, because the backend has no achievement path for them. Making them medal-bearing recruits against status rather than budget.