Abstract: An automated air-to-air refueling (A3R) system is usable with a tanker aircraft having a refueling boom. The system includes a camera connected to the tanker in proximity to the refueling boom which outputs a video stream of the boom and a fuel-receiving aircraft/receiver during an aerial refueling process. The system also includes a human-machine interface (“HMI”) located aboard the tanker, and an electronic control unit (“ECU”) in communication with the camera and HMI. The ECU identifies keypoints on the receiver indicative of flight control surfaces thereof, tracks corresponding positions of the flight control surfaces in real-time, and predicts a change in position of the receiver in free space as a predicted 3D position using the corresponding positions. The HMI also outputs a directional indicator indicative of the predicted 3D position, e.g., as a graphical overlay to a display screen.

Abstract: A method for performing automated refueling operations includes receiving a two-dimensional (2D) image of a device associated with in-flight refueling operations between a receiver aircraft and a tanker aircraft, identifying keypoints on the device in the 2D image, projecting the keypoints to three-dimensional (3D) space to produce 3D keypoints, comparing the 3D keypoints to a previously stored 3D model of the device to produce a confidence value, and automatically controlling the refueling operations in response to the confidence value.

Abstract: A system for calibrating a camera for use aboard a fuel-supplying aircraft (“tanker”) for an automated air-to-air refueling (“A3R”) process in which the tanker offloads aviation fuel to a fuel-receiving aircraft (“receiver”) includes an electronic control unit having a processor and a non-transitory computer-readable storage medium on which is recorded instructions for calibrating the camera. Executing the instructions causes the processor to perform a corresponding method, i.e., to receive boom position data indicative a true position of the boom in free space, and real-time image data from the camera. The instructions also cause the processor to use the boom position data and a 3D model of the boom to project a digital representation of the boom into the image space. The processor then minimizes a difference between the projected and real images of the boom to thereby calibrate the camera.

Abstract: A system for calibrating a camera for use aboard a fuel-supplying aircraft (“tanker”) for an automated air-to-air refueling (“A3R”) process in which the tanker offloads aviation fuel to a fuel-receiving aircraft (“receiver”) includes an electronic control unit having a processor and a non-transitory computer-readable storage medium on which is recorded instructions for calibrating the camera. Executing the instructions causes the processor to perform a corresponding method, i.e., to receive boom position data indicative a true position of the boom in free space, and real-time image data from the camera. The instructions also cause the processor to use the boom position data and a 3D model of the boom to project a digital representation of the boom into the image space. The processor then minimizes a difference between the projected and real images of the boom to thereby calibrate the camera.

Abstract: An automated air-to-air refueling (A3R) system is usable with a tanker aircraft having a refueling boom. The system includes a camera connected to the tanker in proximity to the refueling boom which outputs a video stream of the boom and a fuel-receiving aircraft/receiver during an aerial refueling process. The system also includes a human-machine interface (“HMI”) located aboard the tanker, and an electronic control unit (“ECU”) in communication with the camera and HMI. The ECU identifies keypoints on the receiver indicative of flight control surfaces thereof, tracks corresponding positions of the flight control surfaces in real-time, and predicts a change in position of the receiver in free space as a predicted 3D position using the corresponding positions. The HMI also outputs a directional indicator indicative of the predicted 3D position, e.g., as a graphical overlay to a display screen.