Abstract: Training an encoder is provided. The method comprises inputting a current state of a number of aircraft into a recurrent layer of a neural network, wherein the current state comprises a reduced state in which a value of a specified parameter is missing. An action applied to the aircraft is input into the recurrent layer concurrently with the current state. The recurrent layer learns a value for the parameter missing from current state, and the output of the recurrent layer is input into a number of fully connected hidden layers. The hidden layers, according to the current state, learned value, and current action, determine a residual output that comprises an incremental difference in the state of the aircraft resulting from the current action.

Abstract: A system is provided. The system includes a first platform including a first platform level agent configured to direct one or more actions of the first platform based on at least one of a selected target or a selected goal. The system also includes a computer system in communication with the first platform level agent. The computer system programmed to a) execute a supervisor level agent configured to select at least one of a target or a goal for one or more platforms including the first platform, b) receive targeting information including one or more targets, c) receive platform information for the one or more platforms, d) select, by the supervisor level agent, a target of the one or more targets based on the target information and the platform information, and e) transmit, to the first platform level agent, the selected target.

Abstract: Disclosed herein are methods, systems, and aircraft for performing image analysis for aiding refueling operations. A method includes receiving a 2D image from a camera, determining a domain score for the 2D image based on previously defined training data, and sending the 2D image to the vision position estimation system in response to the domain score being greater than a predefined threshold, thus creating a sent 2D image.

Abstract: A method includes receiving a two-dimensional (2D) image from a camera, predicting 2D keypoints of a target object within the 2D image based on a previously trained ensemble of neural networks, and estimating 6 degree-of-freedom (6DOF) position (pose) of the target object using the 2D keypoints using a perspective-n-point (PnP) optimization technique to create 6DOF pose parameters for each neural network in the ensemble. The method combines the result into a single estimate of 6DOF pose parameters. The method also includes determining an uncertainty score based on a first uncertainty value and a second uncertainty value, and outputting the 6DOF pose parameters in response to the uncertainty score being within a predefined threshold.

Abstract: A computer-implemented method for predicting behavior of aircraft is provided. The method comprises inputting a current state of a number of aircraft into a number of hidden layers of a neural network, wherein the neural network is fully connected. An action applied to the aircraft is input into the hidden layers concurrently with the current state. The hidden layers, according to the current state and current action, determine a residual output that comprises an incremental difference in the state of the aircraft resulting from the current action. A skip connection feeds forward the current state of the aircraft, and the residual output is added to the current state to determine a next state of the aircraft.

Abstract: A pose estimation system comprising a computer system and a pose estimator. The pose estimator is configured to estimate initial point cloud colors for points in a colored point cloud of a surface of an object using frames in a video of the object and initial pose estimates for the object in the frames; adjust the initial pose estimates using the frames and the colored point cloud to form updated pose estimates; and determine updated point cloud colors for the points in the colored point cloud using the frames in the video and the updated pose estimates. The pose estimator is configured to repeat adjusting the updated pose estimates using the frames and the colored point cloud and determining the updated point cloud colors for the points using the frames in the video and the updated pose estimates until the updated pose estimates meet a threshold.

Abstract: Methods, apparatuses and systems of training a neural network using in a vision-based tracking system are disclosed. A two-dimensional (2D) image of at least a portion of an object is received via a camera that is fixed, relative to the object. Subsequently, a keypoint detector predicts a set of keypoints on the object in the 2D image, generating predicted 2D keypoints. These predicted 2D keypoints are then projected into three-dimensional (3D) space, and keypoint depth information is added to generate predicted 3D keypoints. To enhance the training process, a 3D model of the object is utilized. Known rotational and translational information of the object in the 2D image is incorporated to known 3D model keypoints, resulting in transformed 3D model keypoints. Following this, a comparison between predicted 3D keypoints and transformed 3D model keypoints is made to calculate a loss value. The training process is further refined using an optimizer, minimizing the loss value during a training period.

Abstract: A computerized system configured to execute a multi-agent machine learning model for controlling a plurality of vehicles in a multi-vehicle autonomous control session in a multi-vehicle environment is disclosed. Multi-modal neural network agents of the model each control a corresponding autonomous vehicle in the session. The agents receive image data and parameter data, input the image data to an image feature extractor to produce an image feature vector, input the parameter data to a parameter data feature extractor to produce a parameter data feature vector, produce a joint latent representation of the image data and parameter data, and input the joint latent representation to an actor model neural network, to generate a selected action for the autonomous vehicle. The multi-agent machine learning model is configured to control each autonomous vehicle in the session according to the corresponding selected action for each autonomous vehicle.

Abstract: Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Abstract: A method of monitoring an assembly process is disclosed herein. The method includes obtaining an event model for each of a plurality of objects in the assembly process with the event model for each of the plurality of objects including a predetermined time frame for a change in presence to occur. The method includes collecting an image sequence of the assembly process for monitoring and identifying if a change in presence for each of the plurality of objects occurred with a detector model. The method further includes reviewing the event model for each of the plurality of objects to determine if the predetermined time frame lapsed without the change in presence of a corresponding one of the plurality of objects being identified and issuing an alert if the predetermined time frame lapsed without the presence of a corresponding one of the plurality of objects being identified.

Abstract: A vehicle management system comprising a computer system and an agent. The agent comprises a machine learning model and a rule system. The machine learning model system is trained to receive observations for the vehicle system and select a behavior for the vehicle system in response to receiving the observations. The rule system is configured to select a set of actions to execute the behavior for the vehicle system in response to a selection of the behavior by the machine learning model system.

Abstract: A method of generating a detector includes obtaining a first training dataset having a first image sequence with a first set of object tags identifying at least one first object class in a corresponding image. A first set of ground truth tags is obtained based on a ground truth timeline identifying when the at least one first object class appeared in the first image sequence. Images from the first training dataset are discarded by either identifying object tags by class from the first set of object tags without a corresponding ground truth tag from the first set of ground truth tags or identifying object tags by class from the first set of ground truth tags without a corresponding object tag from the first set of object tags to generate a first verified training dataset. A first parts-level detector is trained based on the first verified training dataset.

Abstract: An example includes a method of training a first computational model to emulate a second computational model. The method includes using the first computational model to generate a first output in response to receiving an input and selecting a reward based on whether a difference between the first output and a second output is less than a threshold. The second output is generated by the second computational model in response to receiving the input. The method further includes updating the first computational model using the reward.

Abstract: Training adversarial aircraft controllers is provided. The method comprises inputting current observed states of a number of aircraft into a world model encoder, wherein each current state represents a state of a different aircraft, and wherein each current state comprises a missing parameter value. A number of adversarial control actions for the aircraft are input into the world model encoder concurrently with the current observed state, wherein the adversarial control actions are generated by competing neural network controllers. The world model encoder generates a learned observation from the current observed states and adversarial control actions, wherein the learned observation represents the missing parameter value from the current observed states. The learned observation and current observed states are input into the competing neural network controllers, wherein each current observed state is fed into a respective controller.

Abstract: Aspects of the disclosure provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation). A video stream comprising a plurality of video frames each showing an aircraft to be refueled, is received from a single camera. An initial position estimate is determined for the aircraft for the plurality of video frames, generating an estimated flight history for the aircraft. The estimated flight history for the aircraft is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. The position of a fuel receptacle on the aircraft is determined, based on the refined position estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples may use a deep learning neural network (NN) or optimization (e.g., bundle adjustment) to determine the refined position estimate from the estimated flight history.

Abstract: Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a camera, a refueling boom, a camera configured to generate a two-dimensional (2D) image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code is executable by the processor to perform operations including receiving the two-dimensional (2D) image from the camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and the corresponding three-dimensional (3D) space 3D keypoints, optimizing 3D keypoints associated with moveable components of the refueling boom in response to a plurality of boom control parameters to produce optimized 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Abstract: Methods, apparatuses and systems for extrinsic parameter optimization used in image-based pose estimation of a boom are disclosed. The method includes generating a 3D model of the boom, generating 3D keypoints using the generated 3D model of the object (206) and generating predicted 2D keypoints from the generated 3D keypoints and at least one initial extrinsic parameter. The value of the initial extrinsic parameter is optimized using an iterative process such that the variation between the predicted 2D keypoints and location of the boom is minimized.

Abstract: Methods, apparatuses and systems of training a neural network using in a vision-based tracking system are disclosed. A two-dimensional (2D) image of at least a portion of an object is received via a camera that is fixed, relative to the object. Subsequently, a keypoint detector predicts a set of keypoints on the object in the 2D image, generating predicted 2D keypoints. These predicted 2D keypoints are then projected into three-dimensional (3D) space, and keypoint depth information is added to generate predicted 3D keypoints. To enhance the training process, a 3D model of the object is utilized. Known rotational and translational information of the object in the 2D image is incorporated to known 3D model keypoints, resulting in transformed 3D model keypoints. Following this, a comparison between predicted 3D keypoints and transformed 3D model keypoints is made to calculate a loss value. The training process is further refined using an optimizer, minimizing the loss value during a training period.

Abstract: Disclosed herein are methods, systems, and aircraft for verifying performing automated refueling data. A method includes receiving a two-dimensional (2D) image from a camera, cropping the 2D image based on predefined feature areas of interest of a target object to produce a plurality of cropped images, resizing one or more of the cropped images responsive to the target object being greater than a threshold distance from the camera to produce one or more resized images, determining 2D keypoints of the target object within the one or more resized images or the plurality of cropped images, estimating a 6 degrees-of-freedom (6DOF) pose based on the 2D keypoints and a three-dimensional (3D) model of the target object to produce an estimated 6DOF pose, and outputting the 6DOF pose.

Abstract: Methods, apparatuses and systems of training neural networks for predictive tasks are disclosed. The method includes training a neural network within a closed-loop framework, where the focus is on predicting target tasks for an object. The method identifies situations and inputs (i.e., augmentations) that lead to higher errors in the neural network's predictions by calculating a loss value between predicted target tasks and known target tasks during a baseline training period. An optimizer uses the loss value to predict correlations between augmentations and loss values and identifies difficult augmentations. The difficult augmentations are then intentionally emphasized during subsequent training periods, to increase the likelihood that the neural network is trained on difficult augmentations to enhance the neural network's capacity to adapt and learn from error-prone conditions.