AgenticVBench · agentic video understanding
Contribute a task.
Co-author the papers.
One or multiple videos, one unambiguous question, and a deterministic scorer. Build one to the standard and it merges. Discuss your idea with our agent first — it takes a couple of minutes and saves you days.
Submit by July 31 to make the first batch
Why join the first batch
Real credit, and people who show up.
Closes July 31, 2026
Your name on the ICLR paper
We submit to ICLR this September. Every task that merges before the deadline goes in, credited to you. Land two and you co-author both the benchmark and the survey.
In person
We give back to the community
We co-host events in San Francisco and at the major ML conferences, together with frontier labs and other AI infrastructure companies. Come learn, share what you are building, and meet the people behind it.
No busywork
One good task is plenty
About 10 hours of focused work, on average. One task that passes review is worth more than ten thin ones.
What makes a task
Hard, long, and impossible to fake.
Long horizon
A real strong-agent attempt takes more than 50 tool-call turns — the answer lives across the whole video, not one lookup.
Hard for the right reason
A strong current agent scores below 0.10; your oracle scores 1.0 and an empty attempt near 0. Difficulty comes from the skill, not trick wording.
Deterministically scored
Pure code grades the answer — no VLM or LLM judge — strict enough that guessing scores about 0.
No shortcuts
Recall of famous footage, on-screen graphics, a single frame, or no media at all must not solve it.
The flow
Four steps, propose-first — all on this page.
- 1Discuss your proposal
Describe your idea to the agent below. It tells you whether the difficulty and long-horizon bar is met, and helps you shape the videos, the prompt, and the rubric before you build.
- 2Build the task
One or more real videos, one unambiguous question, a deterministic verifier, and an oracle that proves it is solvable.
- 3Calibrate with three agents
Run Antigravity, Codex, and Claude Code on your task; gather the rollouts and grade them. Iterate until every agent scores below 0.10 over 50+ tool-call turns.
- 4Open a PR
Submit the bundle: the task prompt, the verifier, and the rollouts + scores for all three agents. A maintainer reviews and merges.
Step 1 · discuss your proposal
Is your idea hard enough? Ask before you build.
Describe the video and the question you have in mind. The agent checks it against the bar — difficulty and long horizon first — and suggests the smallest change that would sharpen it.
Proposal agent · checks difficulty & long-horizon fit
Try one of these:
Steps 2–3 · build and calibrate
Build the task, then calibrate it.
Build in the repo, then run it against real agents and grade them. Iterate the two until every agent is well below the bar — that loop is the work.
Step 2 · build
- Gather one or more real videos and write one unambiguous question.
- Write the oracle solution — it scores 1.0, while an empty attempt scores near 0.
- Write the deterministic verifier: pure code, no VLM or LLM judge.
- Ablate the shortcuts — famous footage, on-screen graphics, or a single frame must not solve it.
Step 3 · calibrate (the loop)
- Run the task with Antigravity, Codex, and Claude Code.
- Save each full rollout and grade it with your verifier.
- If any agent clears 0.10, or solves it in under ~50 turns, tighten the task and re-run.
- Done when all three score below 0.10 over 50+ tool-call turns.
Step 4 · submit
Open a PR before July 31.
Open a pull request adding your task folder — the prompt, the verifier, the oracle, and your calibration bundle (the rollouts and scores for Antigravity, Codex, and Claude Code). A maintainer reviews and merges — and you are on the paper.