How We Taught Video Search to Say "It's Not There"

The Deceptively Simple Ask

We were building semantic search over surveillance footage. The product question sounds trivial: show me the scooter, where's the motorcycle, find the accident.

The hard part isn't finding things that are there. The hard part is confidently returning nothing when the thing isn't there. A search box that always hands back its top-8 "closest" frames is useless for surveillance: if there is no scooter in three hours of footage, the operator needs an empty result set, not eight frames of the least-wrong cars.

So we didn't just need ranking. We needed a calibrated relevance threshold: present objects scoring above a line, absent objects below it, and a gap wide enough that a single cutoff works across every query. Everything below is the story of getting that gap to open up.

The Architecture, In One Breath

Frames don't embed into a searchable text space on their own, so the pipeline is: a vision-language model (VLM) describes each frame group in natural language; a text embedder turns each description into a vector; the vectors go into a vector database; a query embeds and pulls the top-K nearest descriptions for recall; and finally a cross-encoder reranker re-scores each candidate against the original query and drops everything below the threshold.

Those last two stages are the classic retrieve-then-rerank split, and the rerank step is where "it's not there" becomes a first-class answer. But we learned, the hard way, that retrieval and reranking are only as good as the words the VLM chose in the very first step. Most of this story is about that annotation step quietly sabotaging everything downstream.

Act 1: The Bi-Encoder Can't Say No

Our first version was retrieve-only: embed the query, cosine-match against frame descriptions, return the hits above a cosine threshold. It leaked constantly.

"Orange truck" — an object that appears nowhere in the clip — kept returning frames of a white truck, or a red one, at a similarity barely distinguishable from a true hit. Bi-encoders compress a whole sentence into one vector, so "orange truck," "red truck," and "white van" all land in the same neighborhood. There is no threshold that keeps the real red truck while rejecting the phantom orange one, because their scores overlap.

A bi-encoder is a great recall tool and a terrible judge. That's a known result; we just had to feel it before we believed it.

Act 2: The Reranker Opens the Gap

So we added a cross-encoder reranker — a model that reads the query and one candidate description together and emits a single relevance score. Because it attends to both texts jointly, it can notice that "orange" and "white" don't match even when everything else does.

The effect was immediate and large. The gap between a true hit and a phantom went from basically noise to nearly binary. Suddenly a threshold was possible. Two tuning notes that cost us real time:

Serving it was fiddly. The obvious path, a ready-made rerank endpoint, produced a negative present/absent gap because of a broken scoring template. We pivoted to scoring with the model directly inside a thin in-process shim, which reproduced the proven gap. Verify your reranker's score distribution on a known present/absent pair before trusting the endpoint.

Bigger reranker, sharper color. The small (0.6B) reranker confused orange with white — it scored an orange-versus-white-truck pair at 0.77, a near-match. The 4B model scored the same pair near zero. But the larger model was also sharper overall, which pushed some weakly-described true hits below the line, so the threshold had to move down. Reranker size trades color precision against recall margin: pick the size, then set the threshold.

We had a threshold. We shipped it. And then the benchmark turned red for reasons that had nothing to do with retrieval.

Act 3: The Model Has to Actually See It

One query kept failing: the accident. Our clip has a subtle low-speed collision at a busy intersection — one car nudged into the rear of another and ended up rotated across the lane. Ask "where's the accident" and we got nothing, because the VLM's description of those frames never mentioned a collision. You cannot retrieve what was never written down.

This kicked off a model tour:

gemma-4-12b — non-deterministic; often failed to perceive the collision at all.
gemma-4-26b (MoE) — better, still unreliable on the subtle crash.
A hosted flash-tier VLM — perceived well, but broke the reranker (see below).
Qwen2.5-VL-7B — solid, but called an orange van "yellow" and under-described people.
Qwen3-VL-8B — nailed the crash, the colors, and the people. The winner.

Two findings earned their own paragraphs.

The hosted model broke the reranker by being too enthusiastic. It described scenes as per-object enumerations — "X is moving. Y is moving. Z is moving." — and the reranker, seeing every object name spelled out, scored everything around 0.99. The present/absent gap collapsed and absent objects leaked. A description that is great for a human reader can be poison for a cross-encoder.

Perception is prompt-independent up to a point, then it isn't. No amount of prompt-nudging, max-pixel bumps, or frame-tiling got Qwen2.5-VL to call the bright-orange van "orange" — it insisted on "yellow." Qwen3-VL-8B, the same ~8B size class, said "orange" on a full frame with no tricks. Sometimes the fix is the model, not the prompt. But — and this is the whole rest of the story — sometimes it really is the prompt, in ways you would never guess.

Act 4: The Twist — Negations Poison Retrieval

With the best model in place, one query still returned nothing: "person."

This made no sense. In direct testing, Qwen3-VL-8B described people beautifully: several people around the red truck, one in dark clothing standing by an open door, another sitting on the roof of the black car. The people were in the descriptions. So why did searching for "person" return zero hits above the threshold?

We dumped the raw descriptions and counted. Of 125 indexed frames, 114 contained a people-word — but most of them were negations: "there are no pedestrians, nobody is falling, no accident or collision; all vehicles are operating normally."

The VLM, trying to be a diligent analyst, was documenting the absence of things. And those negations contained the exact tokens — "person," "pedestrian," "no accident" — that our queries were searching for. Retrieval would surface a frame that literally said "there is no person," the reranker would correctly score it low, and the genuinely people-rich frames never made it to the top. The absence-statements were out-competing the presence-statements for the same keyword.

The fix was a hard, explicit ban in the prompt: never write what is not there. No "no pedestrians," no "nobody fell," no "normal situation." If something isn't there, don't mention it. Write "person" only when people are actually in the frame.

Re-index, re-run. "Person" went from zero hits to found — the real crowd around the crash surfaced to the top. And "orange truck", the phantom, saw its false score drop from 0.47 to 0 — because the model had stopped narrating "no orange truck here" in frames near the orange van. One prompt paragraph fixed a present-object miss and an absent-object leak at the same time. Negations are retrieval poison; ban them.

Act 5: Attention Is Zero-Sum

There was a subtler version of the same disease. Early on, our prompt was heavily weighted toward accident detection — a big block of instructions about spotting rotated vehicles, collisions, and damage. Qwen3-VL-8B obediently obsessed over the crash and under-described the routine traffic. The minivan that was clearly on screen got one glancing mention, so its reranker score against "minivan" collapsed to 0.03, below threshold. A present object, missed, because the prompt spent the model's attention elsewhere.

We rebalanced. The primary instruction became "enumerate every notable vehicle, with type and exact color, and every person," with accident detection demoted to a shorter, secondary note. Same model, same footage, one prompt restructure: "minivan" went from 0.03 to 0.36, and all the routine objects came back. The accident was still detected — it just no longer starved everything else.

The lesson generalizes past VLMs: a prompt's attention budget is zero-sum. Whatever you emphasize, you de-emphasize everything else. If a class of objects has to be findable, it has to be described, and "described" competes directly with whatever else you told the model to fixate on.

The Result: A Paired Discriminator

We drove the whole thing with a benchmark: a fixed list of present queries that must return at least one hit, and absent queries that must return zero, red until every case passes. The most satisfying entry is a pair, same color and different vehicle type:

"orange van" is found — the real Sprinter crossing the intersection at ~95s, score 0.37.
"orange truck" returns zero hits.

If the reranker can find one and reject the other, you know it is discriminating on meaning, not just on the word "orange." Present cases that now pass include the accident, the minivan (0.03 → 0.36), the person (from zero hits to 0.10), the orange van, and the routine red dump truck, white car, black SUV, blue truck, and red truck. Absent cases — orange truck (previously leaking at 0.47), scooter, bicycle, motorcycle, tractor, elephant, helicopter, fire truck, and boat — all return zero.

What We'd Tell the Next Team

A threshold needs a reranker. Bi-encoder cosine can't cleanly separate "present" from "absent" for near-synonyms. A cross-encoder that reads query and candidate jointly is what makes "it's not there" a reliable answer.
You can't retrieve what wasn't perceived. Before blaming retrieval, dump the descriptions. If the model never wrote "collision," no reranker on earth will find the collision.
Negations are retrieval poison. A model documenting what isn't there fills your index with the exact keywords your queries match. Ban negations explicitly; index only presence.
Prompt attention is zero-sum. Over-emphasize one thing and you silently starve the description of everything else, quietly dropping those objects below your threshold. Balance the prompt around findability.
Build a benchmark with paired present/absent discriminators. "Orange van found versus orange truck rejected" tells you more in one line than a dozen present-only queries. Red-until-green turns fuzzy "seems better" tuning into a decision.

The final system runs a modern ~8B vision-language model as the annotator, a 4B cross-encoder reranker for the threshold, and a bi-encoder plus vector database for recall. But the parts that actually moved the benchmark from red to green weren't the model swaps. They were two paragraphs of prompt: stop describing what isn't there, and describe everything that is.