Contribute a task.
Co-author the papers.

A task is three things:

1. A task prompt: what the agent is asked to do.
2. Media sources: one or multiple videos, with optional additional files.
3. A deterministic verifier: checks the agent output and gives a score, 0 to 1.

Think about future smart devices: glasses and earbuds that see and hear what you do, acting as a friend or coach that teaches you volleyball, watches a match with you, or helps you learn guitar. Or security cameras in a robotics company or factory, in the wild (animals, extreme weather), even in space or a virtual 3d world. Some examples:

• Sports analytics: rep counting, form scoring, play or event detection.
• Robotics understanding: did the action succeed, object and state tracking.
• Game understanding: read game state, score, or events from footage.
• Security camera: detect an event, count, or localize it in time.
• Detective and reasoning: who did what and when, inferred across clips.
• Music video understanding: cuts, beats, performers, and how the visuals sync to the music.

And more: if your field has video with a checkable answer, it fits.

Run a strong model under each degraded input; every run must score near the null baseline. They matter for two reasons: they prove the task has no shortcut, and they show which modalities the answer really needs. Is it video, audio, or audio-visual understanding at the seam? If any ablation scores well, fix the task, not the threshold.

1. Single frame only.
2. Video-only and audio-only (for audio-visual tasks): this pair is what separates true audio-visual understanding from single-modality tasks.
3. No media at all: catches recall and guessable schemas.
4. All frames pasted, no tools: proves agency matters.

The ones we keep seeing:

1. The answer key baked into the agent's image. Keep it verifier-side under steps/solve/tests/, where the agent can never read it.
2. Simulated or argued ablation numbers. Every ablation must be a real measured run.
3. Calibrating outside the shipped environment. Local runs are fine only in an isolated env with the image's exact libraries and the same harness and model versions.
4. Padding turn counts with prompt instructions like “use at least 51 turns”. If the task finishes too fast, harden the task.
5. Fields no agent could ever get right (off-camera events, blurred-out scoreboards, hidden state). Verify every ground-truth event is actually observable in the media.
6. Committing thousands of lines of reward dumps. One raw trajectory per agent plus scores.md is the convention.

Two known behaviors. Gemini 3.1 Pro sometimes skips the media entirely and fabricates a confident answer in a handful of tool calls. Gemini 3.5 Flash can silently fall back to Google Search grounding when stuck; this happens server-side at API runtime, so no container network policy can see or block it. Telltales: an answer matching every public box-score field while missing only the fields no public record has, or “I will search for ...” in the output. Countermeasures: state the no-lookup, no-memory rule clearly in the system prompt (this measurably helps), audit every raw trajectory before trusting a score, and disable grounding in the Gemini API if you control the key.

No by default. Omnimodal-understanding scorers must be deterministic code (Python stdlib, or at most a small CV model), strict enough that guessing scores about 0. If you truly find it difficult to design a deterministic verifier for your idea, check with our team first: book a QA / office-hours slot through the calendar link.