UAV Cinematography Constraints Imposed by Visual Target Tracking Study

UAV CINEMATOGRAPHY CONSTRAINTS IMPOSED BY VISUAL TARGET TRACKING Iason Karakostas, Ioannis Mademlis, Nikos Nikolaidis, Ioannis Pitas Dpt. Artificial Intelligence and Information Analysis Computer Science Aristotle University of Thessaloniki Greece This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE)

UAV in Cinematography UAVs have revolutionized aerial cinematography Can replace helicopters, cranes, etc. UAVs support autonomous functionalities based on machine learning and computer vision Autonomous UAVs may visually track and actively follow a specific target of interest We present common UAV target-tracking trajectories and shot types We study the constraints in maximum focal length for successful target tracking This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 2 16/6/2025

UAV in Cinematography We standardized and geometrically modeled a number of common, target-following UAV motion types We identified the compatible shot types We analytically determined the maximal permissible camera focal length, so that 2D visual tracking does not get lost, for each UAV motion type. This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 3 16/6/2025

UAV/Camera Shot Types The desired shot type is defined by the percentage of the video frame that is covered by the target Region Of Interest (ROI) Shot Type Extreme Long Shot (ELS) Very Long Shot (VLS) Long Shot (LS) Medium Shot (MS) Medium Close-Up (MCU) Close-Up (CU) Percentage of ROI < 5 % 5 % 20 % 20 % 40 % 40 % 60 % 60 % - 75 % > 75 % This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 4 16/6/2025

UAV/Camera Motion Types 5 UAV industry- standard camera motion types are detailed and geometrically modeled Lateral Tracking Shot (LTS) Vertical Tracking Shot (VTS) Fly-Over (FLYOVER) Fly-By (FLYBY) Chase/Follow Shot (CHASE) This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 5 16/6/2025

UAV/Camera Motion Types VTS LTS FLYOVER CHASE FLYBY This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 6 16/6/2025

Constraints On Maximum Focal Length 2D visual tracking algorithms assume that the location of the target ROI center varies no more than a threshold ????(in pixels) between successive video frames Focal length ? affects the permissible shot framing types For UAV cinematography it is important to determine the constraints on maximum focal length imposed by the needs of visual target tracking This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 7 16/6/2025

Constraints On Maximum Focal Length In a fully known 3D environment, if the target moves exactly as expected, its next 3D location can been predicted Thus, in this ideal scenario, central composition may always be retained by computing the appropriate LookAt vector at each time instance If target motion deviates from expected, its ROI may be displaced more than ????on the next video frame This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 8 16/6/2025

Constraints On Maximum Focal Length We examine an entire shooting session as a sequence of repeated transitions between the first (? = 0) and the second video frame (? + 1 = 1) The target ROI center is meant to be fixed at the image center for all video frames, assuming accurate target velocity vector estimation at all times To study ????we assume a maximum search radius ????(in pixels) within which the ROI in ? + 1 must lie This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 9 16/6/2025

Constraints On Maximum Focal Length Maximum focal length is calculated based on the camera projection equations ??? + 1 = ?? ? ?? ??? + 1 = ?? ? ?? ??+1,??+1: target/UAV expected position ??, ??: target center (pixel coordinates) ??, ??: image center (pixel coordinates) ??, ??: pixel size (mm) ?1, ?2, ?3: rows of the rotation matrix ?1???+1 ??+1 ?3 ?2 ?3 ???+1 ??+1 ???+1 ??+1 ???+1 ??+1 This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 10 16/6/2025

Constraints On Maximum Focal Length Rotation Matrix the camera axis points directly at the target the unit vector of the k-axis for the Camera Coordinate System(?3), can be obtained from ??+1as follows: ??+1 ??+1 ? ?3= ? ?, ?2 ??+1 ??+1 ??+1 ??+1 ??+1 ??+1 and ?1 = ? = ? ?1 ?1 , ?2= ?1 ?1 ?1= This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 11 16/6/2025

Constraints On Maximum Focal Length Using the limit constraint ??+1= ???? 2 2+ ??? + 1 ?? ????= ??? + 1 ?? Using projection equations, ????is given by: 2 ?????? ?????1+ ? ?? ????= 2+ ?? 2 ???? 3?2 2?3 2?? 2 ????3?? where ?1= ??1?? 1 ??2?? 2 ??3?? 3, ?2= ??1?? 1+ ??2?? 2, ?3= ??2?? 1 ??1?? 2. This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 12 16/6/2025

Constraints On Maximum Focal Length Experimental cases 8 cases for the deviation vector ?? j case 7 case 3 case 5 ??1= [5,0,??3] ??2= [ 5,0,??3] case 2 expected case 1 i ??3= [0,5,??3] ??4= [0, 5,??3] ??5= [5,5,??3] case 6 case 4 case 8 ??6= [ 5, 5,??3] ??7= [ 5,5,??3] ??8= [5, 5,??3] This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 13 16/6/2025

Constraints On Maximum Focal Length Lateral Tracking Shot UAV position is given by ??+1= 0,??2,0? Target position is given by ??+1= framerate Maximum focal length for the LTS is now given by ????=??????????2 ???2 ?, where ? is the camera ??1 ?,??2 ?,??3 ? 2??1 2+ ?? 2??3 2 ?? This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 14 16/6/2025

Constraints On Maximum Focal Length 1000 Variations in target altitude affect all study cases 1 8 when ??3= 0 the projected ROI center will not change in pixel coordinates, if the target approaches or goes away from the UAV. Due to the position of the UAV, target acceleration and deceleration have identical impact on ???? Cases 3, 4 Cases 1, 2, 5 - 8 maximum focal length (mm) 800 600 400 200 0 -10 -5 0 5 10 qt3 (m/s) Variation of ????against ??3for LTS This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 15 16/6/2025

Constraints On Maximum Focal Length Cases 1-2: UAV approaches the target and the maximum focal length decreases, before increasing again as the UAV is flying parallel to the ?-axis. When the drone is positioned far from the target, any change in target velocity corresponds to a small change in the distance between the UAV and the target. Cases 3-4: target deviates from its expected position but remains on the ?-axis ????increases with distance between the UAV and the target. ????slightly increases when the UAV is very close to the target. Cases 5-8: ????depends on the angle between the LookAt vector and the ?-axis: it has lower values when this angle is close to ? (? = 10). 400 maximum focal length (mm) 300 200 Cases 1, 2 Cases 3, 4 Cases 5, 6 Cases 7, 8 100 0 0 5 10 15 20 time (seconds) 2 Variation of ????against ? for FLYBY This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 16 16/6/2025

Constraints On Maximum Focal Length 600 600 1500 Cases 1, 2 Cases 3, 4, 5, 6, 7 ,8 Cases 1, 2 Cases 3, 4 Cases 5, 6, 7, 8 maximum focal length (mm) maximum focal length (mm) maximum focal length (mm) 500 500 400 400 1000 300 300 500 200 200 100 100 Cases 1, 2, 3, 4 Cases 5, 6, 7, 8 0 0 0 0 10 20 30 40 50 60 20 40 60 80 100 0 5 10 15 20 xt1 (m) xt3 (m) time (seconds) CHASE FLYOVER VTS This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 17 16/6/2025

Conclusions Industry-standard target-tracking UAV/camera motion types have been formalized and geometrically modelled Maximum focal length constraints for computer vision-assisted UAV physical target following have been extracted The derived formulas can be readily employed as low-level rules in intelligent UAV shooting and cinematography planning systems This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 18 16/6/2025

Acknowledgement The research leading to these results has received funding from the European Union s European Union Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE). This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE) 19 16/6/2025

Thank you! Q&A This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No 731667 (MULTIDRONE)