The challenge
The system only sees a 960×540-pixel window of a 4K scene, but has to report objects across the whole frame. It must decide where to point the camera and how far to zoom, and respond within the deadline for every frame in a stream of three frames per second. The development material was only 25 annotated frames from a single scene, while the system was evaluated on new, unseen sequences.
The solution
A multi-stage pipeline: a detector finds candidate objects in the visible window, an image model (DINOv2) with dedicated classification heads decides the object class and separates targets from background, and a tracker keeps objects in global coordinates with motion and scene drift. Camera control combines a coverage tour of the whole scene with targeted zoom on areas of interest.
How it was built
I implemented the majority of the Drone Flyby perception pipeline as part of a team challenge. To stop the detector from only recognising the few examples in the reference scene, I built an object-centric, domain-randomised training dataset: objects were cut out of the reference frames and re-placed with random rotation, size, clipping, partial occlusion and shadows, with a check that rejects examples that are not visible. The detector (YOLO11) was trained on this dataset. The solution was served as a FastAPI service on a GPU, with telemetry and diagnostic captures for debugging against the actual evaluation.
Result
The pipeline worked end to end with a median response time of about 50 ms per frame on a single GPU, comfortably within the evaluator’s runtime budget. Evaluation on new sequences nevertheless exposed poor generalisation: accuracy was low because the models were in practice developed from one small reference scene. Diagnostics on a small, manually annotated set of evaluation frames showed that the new training dataset gave the detector far more hits than the first version, and that class recognition (not detection) became the remaining bottleneck.
Context
The response time was measured on a single GPU during evaluation. The lesson is that a technically well-functioning real-time system is not enough when development data is very limited, and that camera control and coverage must be measured together: strictly rule-compliant camera control removed every camera error but reduced coverage and the result.