← All examples
Negative · Too easy — agent scored 0.5agentic_vbench_robotics/pickplace-action-timeline-pair
Robotics A: pick-and-place action timeline (pair)
Too easy: a strong agent scored 0.5. A short two-clip rollout can't carry the difficulty.
The ask
Two pick-and-place rollouts (ExylosAi/pick_and_place_sample, 1280×960); report grasp time, release time, and success for each.
Ground truth[logged]
The robot's logged gripper actuator channel plus the logged next.success flag. The dataset's own annotations.json is WRONG (it labels episode_000025 a slip-drop failure; the logs and the video show it succeeds). Never trust dataset video labels; use the logs.
Scorer
Deterministic, pure-stdlib comparison against the logged signals.
The lesson
The agent timed the release to 0.07 s but missed the grasp moment. Comparison pairs of short clips are exempt from the 10–300 min length floor.
Measured, not claimed
- Oracle
- 1.0
- Empty baseline
- 0.0
- Strong agent (Opus 4.8)
- 0.5
- Rollout
- 43 tool-call turns