Corvus ISR Benchmark Reveals Realistic Tracker Performance on Synthetic Scenes

Corvus ISR, known for its wide-area motion imagery (WAMI) exploitation products, recently published a detailed public tracker benchmark that compares two distinct tracker models on an identical, fixed-seed synthetic scene. This approach allows for an apples-to-apples evaluation of tracking algorithms under perfect ground truth conditions, eliminating real-world noise and variability. The benchmark uses seed 1337, with a 20-second warm-up and 120 seconds of measurement per row, ensuring consistency across tests, with sensor models and detection generation identical between models so that only the tracker differs.

The two models under comparison are the v1 ‘greedy nearest-neighbour’ baseline, which employs a simple two-pass greedy association with constant velocity prediction and fixed 2-second coasting, and the more sophisticated v2 ‘confirmed-track auction’. The latter uses a three-tier auction association, velocity-consistency gating, noise-scaled reservation prices, and confidence-decayed coasting—representing the current state-of-the-art in tracking technology. This methodology ensures that the models are tested under identical conditions, providing meaningful insights into their relative performance.

The results demonstrate significant improvements with the v2 model. For a baseline scenario with 150 movers at 2 frames per second, the ID switches per minute dropped from 2,042 to 1,183, a reduction of 42.1%. In a dense scenario with 400 movers, the ID switches decreased from 14,032 to 8,040, a 42.7% improvement. Additional stress conditions, such as frame starvation at 0.5 fps, occlusion at 20%, and degraded detection conditions (1 fps with jitter and low contrast), show reductions of approximately 18% in ID switches, illustrating the robustness of the newer tracker.

It’s important to note that detection rate remains a sensor property and is identical for both models by design. The published failure numbers are not marketing fluff; instead, they highlight that even the most advanced synthetic scenes with perfect ground truth still result in thousands of identity errors per minute under stress. This transparency emphasizes that every future tracker will be benchmarked publicly against the same seed, emphasizing an objective measure of progress rather than selectively showcasing success stories.

From an engineering perspective, the v2 tracker achieves an average processing time of approximately 1.2ms per sensor tick at a 400-mover density, with a worst-case of about 5ms against a 10ms budget. This demonstrates real-time performance within a browser environment. Anyone interested can reproduce it live by opening the demo and pressing ‘Run benchmark,’ with no signup or NDA required. The entire process is built on synthetic data—pixels generated digitally—ensuring a controlled environment for rigorous testing.

The use of perfect ground truth from synthetic scenes is a key methodological advantage, enabling researchers to measure true tracker performance without the confounding factors of real-world noise or partial data. Publishing failure metrics alongside success stories fosters a culture of transparency and continuous improvement. In the end, the fixed-seed benchmark matrix provides a reproducible and scientific basis for assessing tracker advancements, supporting a data-driven approach to algorithm development.

Science-minded readers are encouraged to explore the benchmark themselves and see how the models perform under controlled conditions, helping to further understanding of the challenges in motion tracking and the importance of rigorous testing. The Corvus ISR benchmark exemplifies how transparent, exact measurements can advance the field by providing clear, reproducible results for all to examine.

Powered by Thorsten Meyer AI