Abstract: Methods, apparatuses and systems of training a neural network using in a vision-based tracking system are disclosed. A two-dimensional (2D) image of at least a portion of an object is received via a camera that is fixed, relative to the object. Subsequently, a keypoint detector predicts a set of keypoints on the object in the 2D image, generating predicted 2D keypoints. These predicted 2D keypoints are then projected into three-dimensional (3D) space, and keypoint depth information is added to generate predicted 3D keypoints. To enhance the training process, a 3D model of the object is utilized. Known rotational and translational information of the object in the 2D image is incorporated to known 3D model keypoints, resulting in transformed 3D model keypoints. Following this, a comparison between predicted 3D keypoints and transformed 3D model keypoints is made to calculate a loss value. The training process is further refined using an optimizer, minimizing the loss value during a training period.

Abstract: Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Abstract: Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a camera, a refueling boom, a camera configured to generate a two-dimensional (2D) image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code is executable by the processor to perform operations including receiving the two-dimensional (2D) image from the camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and the corresponding three-dimensional (3D) space 3D keypoints, optimizing 3D keypoints associated with moveable components of the refueling boom in response to a plurality of boom control parameters to produce optimized 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Abstract: Methods, apparatuses and systems for extrinsic parameter optimization used in image-based pose estimation of a boom are disclosed. The method includes generating a 3D model of the boom, generating 3D keypoints using the generated 3D model of the object (206) and generating predicted 2D keypoints from the generated 3D keypoints and at least one initial extrinsic parameter. The value of the initial extrinsic parameter is optimized using an iterative process such that the variation between the predicted 2D keypoints and location of the boom is minimized.

Abstract: Methods, apparatuses and systems of training a neural network using in a vision-based tracking system are disclosed. A two-dimensional (2D) image of at least a portion of an object is received via a camera that is fixed, relative to the object. Subsequently, a keypoint detector predicts a set of keypoints on the object in the 2D image, generating predicted 2D keypoints. These predicted 2D keypoints are then projected into three-dimensional (3D) space, and keypoint depth information is added to generate predicted 3D keypoints. To enhance the training process, a 3D model of the object is utilized. Known rotational and translational information of the object in the 2D image is incorporated to known 3D model keypoints, resulting in transformed 3D model keypoints. Following this, a comparison between predicted 3D keypoints and transformed 3D model keypoints is made to calculate a loss value. The training process is further refined using an optimizer, minimizing the loss value during a training period.

Abstract: Disclosed herein are methods, systems, and aircraft for verifying performing automated refueling data. A method includes receiving a two-dimensional (2D) image from a camera, cropping the 2D image based on predefined feature areas of interest of a target object to produce a plurality of cropped images, resizing one or more of the cropped images responsive to the target object being greater than a threshold distance from the camera to produce one or more resized images, determining 2D keypoints of the target object within the one or more resized images or the plurality of cropped images, estimating a 6 degrees-of-freedom (6DOF) pose based on the 2D keypoints and a three-dimensional (3D) model of the target object to produce an estimated 6DOF pose, and outputting the 6DOF pose.

Abstract: Methods, apparatuses and systems of training neural networks for predictive tasks are disclosed. The method includes training a neural network within a closed-loop framework, where the focus is on predicting target tasks for an object. The method identifies situations and inputs (i.e., augmentations) that lead to higher errors in the neural network's predictions by calculating a loss value between predicted target tasks and known target tasks during a baseline training period. An optimizer uses the loss value to predict correlations between augmentations and loss values and identifies difficult augmentations. The difficult augmentations are then intentionally emphasized during subsequent training periods, to increase the likelihood that the neural network is trained on difficult augmentations to enhance the neural network's capacity to adapt and learn from error-prone conditions.

Abstract: Methods, apparatuses and systems of estimating uncertainty in a vision-based tracking system are disclosed. The method includes receiving a two-dimensional (2D) image of at least a portion of a first object via a camera on a second object. A set of keypoints are predicted on the first object in the 2D image by each one of a plurality of keypoint detectors (i.e., neural networks), organized into an ensemble. A three-dimensional (3D) pose is predicted for each one of the plurality of keypoint detectors from the corresponding set of keypoints. Additionally, the method includes deriving a measure of variation between each one of the 3D poses of the plurality of keypoint detectors and computing a Euclidean norm of the measure of variation to produce an uncertainty value. A process between the first object and the second object can be controlled in response to the calculated uncertainty value.

Abstract: Disclosed herein are methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a camera system configured to generate a two-dimensional (2D) image of a receiving aircraft and a depth image of the receiving aircraft, a processor, and non-transitory computer readable storage media storing code. The code is executable by the processor to perform operations including detecting the receiving aircraft within the 2D image based on 2D image values, determining 2D keypoints of the receiving aircraft located within the 2D image based on the depth image, the 2D image values, and a previously determined keypoint model, determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints, and outputting a position of at least a component of the receiving aircraft based on the 6DOF pose.

Abstract: Disclosed herein are methods, systems, and aircraft for performing image analysis for aiding refueling operations. A method includes a method includes receiving a two-dimensional (2D) image from a camera of a first device, determining 2D keypoints of a second device located within the 2D image based on a predefined point model of a generalized version of the second device, determining a 6 degree-of-freedom (6DOF) pose of the second device using the 2D keypoints, and outputting an estimated position of at least a component of the second device based on the 6DOF pose.

Abstract: Disclosed herein are methods, systems, and aircraft for verifying performing automated refueling data. A method includes receiving a two-dimensional (2D) image from a camera, determining 2D keypoints of a target object located within the 2D image based on a predefined model of the target object, estimating a 6DOF pose based on the 2D keypoints and a three-dimensional model of the target object, generating an uncertainty value of the 6DOF pose, and outputting the uncertainty value of the 6DOF pose.

Abstract: Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.