Improving Gemini & AGY with Kaggle
What the platform already does, what it uniquely has, and some ideas worth running on it.
The thing worth knowing first
Kaggle pushed a Harbor agent-execution starter template this month with an Antigravity adapter already inside it, and a working credential path to their model proxy. No announcement, no UI entry point.
On a local Docker daemon with those credentials, an Antigravity benchmark can be run against a Kaggle-served Gemini today. Which makes which checkpoints that proxy is permitted to serve, and whether the run happens on Kaggle’s executor rather than a laptop, the two open questions that matter.
Observed strengths
Tasks that post-date training
A competition built on an unpublished task did not exist when anything was trained. Score an agent on the same hidden private split as the humans and the result is uncontaminated; new competitions renew the test.
A human baseline on every task
Thousands of practitioners solve the same problem with the same data, so an agent’s result becomes a percentile against people rather than an abstract rate.
Agent episodes already recorded, step by step
kaggle-environments already runs autonomous agents and retains replays plus per-agent logs. The episode and logging machinery exists; tool-using environments are the new layer.
A decade of human process with an outcome score attached
Meta Kaggle Code publishes the source of every public notebook version, weekly, back to 2015. Where a notebook belongs to a competition, its version history is a human working the problem in sequence with a leaderboard score at the end. Not a competition to run; a corpus to read.
A long-horizon domain native to Kaggle
Data-science work already uses terminals, files and tools, with an automatic reward at the end.
A crowd that has already built evals for DeepMind
The AGI hackathon paid $200k for benchmarks rather than models. 1,063 teams entered, and a winning benchmark is live on the platform today, still re-run against each new frontier model.
A status economy that buys expert effort
Medals and tier progress recruit below market. Around 20,000 Expert-or-above placements sit across its three tracks, and benchmark contribution earns none of them anything today — the backend has no achievement path for it.
The ideas
Five challenges Kaggle could run
Examples rather than a plan — shapes of competition the platform could host, ranked by signal value per unit effort. Each says what it does, what it returns, and how it gets gamed. The first needs no new infrastructure and no model access at all.
Review the challenges →If you want the working
How the platform executes a submission
Submissions are code, re-run by Kaggle. In a code competition you submit a notebook the platform re-executes server-side in a fixed container with internet disabled, inside a bounded runtime. Kaggle is already an isolated executor of untrusted code with no egress.
The private leaderboard is a probing firewall. Live scores use a public split; the ranking that counts uses a hidden private split. Pair that with a genuinely post-cutoff task and you get both freshness and resistance to probing.
Agent-shaped execution already exists. The Konwinski Prize fed GitHub issues one at a time to a contestant’s inference server under a 30-minute per-issue budget, through a gateway that never exposes the hidden set. The experimental Standardized Agent Exams let an agent register over HTTP and sit a timed exam with no Kaggle account at all.
Checkpoint exposure, and what cannot run there today
Most of the hard part is already built. Kaggle’s model proxy mints ephemeral per-session tokens with an explicit expiry, enforces a hard budget per participant, and serves inference into an otherwise sealed sandbox — the exact controls a blind-checkpoint competition needs.
Three exposure modes. None — the work never touches an unreleased model; where the corpus, replay and evaluation ideas sit. Metered — entrants call a blind checkpoint from scoring under budget; what a harness competition needs. Internal — the checkpoint runs against Kaggle-sourced tasks on Google hardware, the default for anything pre-release.
A Kaggle notebook gives a contestant no Docker daemon, so container-in-container work cannot run inside a submission — it needs Kaggle’s own benchmark executor. Harbor is the container-execution path, and the choice is telling: it is the Terminal-Bench-lineage framework much of the independent evaluation ecosystem now builds on.
Why environments are not trajectories
Kaggle can supply executable environments in quantity. An environment is a place a rollout can happen. A trajectory is what a checkpoint did inside one, under its tool schema, context policy, compaction and retry behaviour — on-policy and structurally unbuyable, because it is a property of the model and harness rather than of the task.
So Kaggle supplies environments, opponents, fresh uncontaminated tasks and human baselines at a scale nobody else can. The rollouts stay in-house.
Reward validity, as a constraint on all of it
Around three in ten patches that pass a pull request’s own tests diverge from the reference under wider differential testing. That makes test-pass reward suspect enough to calibrate rather than trust, and more task volume does not correct it.
It is a design constraint on every idea on the next page. Where it warrants a competition, that competition is judge calibration.