Game QA: weapon clipping bug
Not usable yet: the single-annotator ground truth missed a real second occurrence the agent found.
The ask
A real Witcher 3 clip (verified-glitch clip from the public VideoGameQA-Bench dataset); identify the bug: type from a closed vocabulary, the time window it is visible, and the affected object.
Ground truth[human-verified]
Hand-annotated (sword clipping into the foreground, 2.4 to 4.3 s). The human-verified tier proved fragile: the agent found a REAL second occurrence of the same glitch (~6 to 8.5 s) that the single-annotator ground truth had missed.
Scorer
Deterministic scoring over bug type, window, and object.
The lesson
Single-annotator ground truth misses real occurrences. Human-verified tasks require 2+ independent annotators, ALL occurrence windows annotated, and any-correct-occurrence scoring.
Measured, not claimed
- Oracle
- 1.0
- Empty baseline
- 0.0
- Strong agent (Opus 4.8)
- 0.333
- Rollout
- 93 tool-call turns