AgenticVBenchcommunity

Task examples

One task cleared the bar. Learn from it, and from the ones that didn't.

The bar is a strong agent scoring below 0.10 over a long rollout. Every number here was measured on a real headless-agent run — a positive example that cleared it, and negatives that came in too easy or had ground-truth problems. Click into any task for the full prompt, ground truth, and scorer.

Positive · cleared the bar

This is what a task that meets the standard looks like — a strong agent lands well under 0.10, and only by actually working through the whole video.

Cleared the baragentic_vbench_sports

Sports: three-point timeline reconstruction

Given the full broadcast of 2018 NBA Finals Game 4 (archive.org, 1080p, 155 min), reconstruct the timeline of all 22 made three-pointers: quarter, game clock, shooter, assister.

Strong agent

0.0465

below 0.10 · hard

Negative · fell short, and why

Real, well-built tasks that still missed the bar — a strong agent handled them too easily, or the ground truth needed work. They show the failure patterns to design around.

Too easy — agent scored 0.5agentic_vbench_robotics

Robotics A: pick-and-place action timeline (pair)

Two pick-and-place rollouts (ExylosAi/pick_and_place_sample, 1280×960); report grasp time, release time, and success for each.

Strong agent

0.5

above 0.10 · too easy

Too easy — agent solved it (1.0)agentic_vbench_robotics

Robotics B: five-event motion timeline (open environment)

Same rollout, five motion events (motion_start, grasp, transport_apex, release, motion_end) in an OPEN environment: shell, internet, pip. The agent is told to bring its own tools.

Strong agent

1.0

above 0.10 · too easy

Ground truth needs workagentic_vbench_games

Game QA: weapon clipping bug

A real Witcher 3 clip (verified-glitch clip from the public VideoGameQA-Bench dataset); identify the bug: type from a closed vocabulary, the time window it is visible, and the affected object.

Strong agent

0.333

above 0.10 · too easy