Corvus ISR Publishes Synthetic Benchmark Comparing Tracker Models

Corvus ISR tracker benchmark matrix (seed 1337) — The published matrix — every row reproducible. Source: corvusisr.com/benchmark

Corvus ISR, a leader in wide-area motion imagery (WAMI) exploitation, has taken an important step toward transparent evaluation with its public benchmark. This benchmark compares two tracker models on an identical fixed-seed synthetic scene with perfect ground truth, enabling precise measurement of tracking performance without the ambiguities of real-world data. The seed value 1337 ensures reproducibility and fairness, establishing a rigorous foundation for scientific analysis.

The two models under comparison are the v1 “greedy nearest-neighbour,” a simple baseline employing a two-pass greedy association with constant-velocity prediction and fixed 2-second coasting, and the v2 “confirmed-track auction,” a more sophisticated approach featuring three-tier auction association, velocity-consistency gating, and noise-scaled reservation pricing. Both models were tested under identical conditions, with detection capabilities held constant by design. This allows for an apples-to-apples comparison focused solely on tracking algorithm improvements.

Results show significant reductions in ID switches per minute: for 150 movers at 2 fps, v1 recorded 2,042 while v2 achieved only 1,183, a 42.1% improvement. Under higher density (400 movers), switches decreased from 14,032 to 8,040, a 42.7% reduction. Additional metrics under challenging conditions, like frame starvation at 0.5 fps, occlusion at 20%, and degraded visibility with noise and jitter, all demonstrated similar improvements, with reductions around 18%. Importantly, the detection rate was identical for both models, since it depends solely on the sensor properties.

Corvus ISR emphasizes the importance of publishing failure metrics alongside successes. The published numbers reveal that even the more advanced v2 tracker continues to make thousands of identity errors per minute in synthetic scenes with perfect ground truth. The strict ID switch count, which considers every change in track identity—including fragmentations and reacquisitions—serves as a rigorous indicator of tracking robustness. This transparency underscores the importance of measurement over marketing hype, especially in a synthetic environment where ground truth is exact and unambiguous.

From an engineering perspective, v2 runs efficiently, averaging about 1.2 milliseconds per sensor tick at a density of 400 objects, with the worst case around 5 milliseconds—well within real-time constraints. The entire benchmarking framework is accessible through the live demo, where anyone can reproduce the benchmark without signup or NDA, promoting open science and validation. Every row published is a product of independent review and an AI executor trained against a written acceptance contract, ensuring reliability and replicability.

The synthetic nature of these tests, with every pixel generated and no real-world variables involved, allows for perfect ground truth and precise performance metrics. This methodology exemplifies how synthetic scenes can serve as a powerful tool for scientific validation, providing clarity that real-world datasets often cannot match due to their inherent uncertainties. Publishing failure numbers, rather than only successes, aligns with scientific principles of transparency and rigorous measurement. It sets a high standard for future tracker development, where every new model must be benchmarked publicly and fairly against the same synthetic seed.

We invite readers interested in the scientific method of performance evaluation to explore the public benchmark and try reproduce it live. Running the benchmark yourself can provide firsthand understanding of how synthetic data and fixed-seed testing contribute to the advancement of tracking algorithms and the broader field of AI-based motion analysis.