Abstract: Disclosed herein are methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a camera system configured to generate a two-dimensional (2D) image of a receiving aircraft and a depth image of the receiving aircraft, a processor, and non-transitory computer readable storage media storing code. The code is executable by the processor to perform operations including detecting the receiving aircraft within the 2D image based on 2D image values, determining 2D keypoints of the receiving aircraft located within the 2D image based on the depth image, the 2D image values, and a previously determined keypoint model, determining a 6 degree-of-freedom (6DOF) pose using the 2D keypoints and corresponding three-dimensional (3D) keypoints, and outputting a position of at least a component of the receiving aircraft based on the 6DOF pose.

Abstract: Disclosed herein are methods, systems, and aircraft for performing image analysis for aiding refueling operations. A method includes a method includes receiving a two-dimensional (2D) image from a camera of a first device, determining 2D keypoints of a second device located within the 2D image based on a predefined point model of a generalized version of the second device, determining a 6 degree-of-freedom (6DOF) pose of the second device using the 2D keypoints, and outputting an estimated position of at least a component of the second device based on the 6DOF pose.

Abstract: A method of monitoring an assembly process is disclosed herein. The method includes obtaining an event model for each of a plurality of objects in the assembly process with the event model for each of the plurality of objects including a predetermined time frame for a change in presence to occur. The method includes collecting an image sequence of the assembly process for monitoring and identifying if a change in presence for each of the plurality of objects occurred with a detector model. The method further includes reviewing the event model for each of the plurality of objects to determine if the predetermined time frame lapsed without the change in presence of a corresponding one of the plurality of objects being identified and issuing an alert if the predetermined time frame lapsed without the presence of a corresponding one of the plurality of objects being identified.

Abstract: A method of generating a detector is disclosed herein. The method includes obtaining a first training dataset including a first set of tagged images identifying a first object and obtaining a second training dataset including a second set of tagged images identifying a second object. A first parts-level detector is trained based on the first training dataset and a second parts-level detector is trained based on the second training dataset. A unified detector is trained based on the first training dataset and the second training dataset.

Abstract: Aspects of the disclosure provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation). A video stream comprising a plurality of video frames each showing an aircraft to be refueled, is received from a single camera. An initial position estimate is determined for the aircraft for the plurality of video frames, generating an estimated flight history for the aircraft. The estimated flight history for the aircraft is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. The position of a fuel receptacle on the aircraft is determined, based on the refined position estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples may use a deep learning neural network (NN) or optimization (e.g., bundle adjustment) to determine the refined position estimate from the estimated flight history.

Abstract: A vehicle management system comprising a computer system and an agent. The agent comprises a machine learning model and a rule system. The machine learning model system is trained to receive observations for the vehicle system and select a behavior for the vehicle system in response to receiving the observations. The rule system is configured to select a set of actions to execute the behavior for the vehicle system in response to a selection of the behavior by the machine learning model system.

Abstract: Disclosed herein are methods, systems, and aircraft for verifying performing automated refueling data. A method includes receiving a two-dimensional (2D) image from a camera, determining 2D keypoints of a target object located within the 2D image based on a predefined model of the target object, estimating a 6DOF pose based on the 2D keypoints and a three-dimensional model of the target object, generating an uncertainty value of the 6DOF pose, and outputting the uncertainty value of the 6DOF pose.

Abstract: Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Abstract: Aspects of the disclosure provide fuel receptacle and boom tip position and pose estimation for aerial refueling. A video frame is received and within the video frame, aircraft keypoints for an aircraft to be refueled are determined. Based on at least the aircraft keypoints, a position and pose of a fuel receptacle on the aircraft is determined. Within the video frame, a boom tip keypoint for a boom tip of an aerial refueling boom is also determined. Based on at least the boom tip keypoint, a position and pose of the boom tip is determined. Based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, the aerial refueling boom is controlled to engage the fuel receptacle. Some examples overlay projections of an aircraft model on displayed video for a human observer.

Abstract: A vehicle behavior system comprises a computer system, an observation processor, and neural networks. The observation processor and the neural networks are located in the computer system. The observation processor is configured to receive observations for a vehicle system. The observations are for a current time. The observation processor is configured to extract features from the observations. The neural networks are configured to receive the features extracted from the observations and estimate a behavior for the vehicle system for time steps in response to receiving features extracted from the observations processed by the observation processor. Each of the neural networks is trained to estimate the behavior for the vehicle system for a different time step in the time steps.

Abstract: A local agent of a multi-agent reinforcement learning (MARL) system is disclosed, the local agent comprising a MARL network comprising at least one local hidden layer responsive to a plurality of local observations. A transmitter is configured to transmit an output of the local hidden layer to at least one remote agent, and a receiver is configured to receive an output of a remote hidden layer from the at least one remote agent. A combiner module is configured to combine the local hidden layer output with the remote hidden layer output to generate a combined hidden layer output, wherein the MARL network is configured to process the combined hidden layer output to generate at least one action value for the local agent.

Abstract: Apparatuses, systems, and methods dynamically model intrinsic parameters of a camera. Methods include: collecting, using a camera having a focus motor, calibration data at a series of discrete focus motor positions; generating, from the calibration data, a set of constant point intrinsic parameters; determining, from the set of constant point intrinsic parameters, a subset of intrinsic parameters to model dynamically; performing, for each intrinsic parameter of the subset of intrinsic parameters, a fit of the point intrinsic parameter values against focus motor positions; generating a model of the intrinsic parameters for the camera based, at least in part, on the fit of the point intrinsic parameter values against the focus motor positions; and determining a position of a fiducial marker within a field of view of the camera based, at least in part, on the model of the intrinsic parameters for the camera.

Abstract: Aspects of the disclosure provide fuel receptacle position/pose estimation for aerial refueling (derived from aircraft position and pose estimation). A video frame, showing an aircraft to be refueled, is received from a single camera. An initial position/pose estimate is determined for the aircraft, which is used to generating an initial rendering of an aircraft model. The video frame and the initial rendering are used to determining refinement parameters (e.g., a translation refinement and a rotational refinement) for the initial position/pose estimate, providing a refined position/pose estimate for the aircraft. The position/pose of a fuel receptacle on the aircraft is determined, based on the refined position/pose estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples extract features from the aircraft in the video frame and the aircraft model rendering, and use a deep learning neural network (NN) to determine the refinement parameters.

Abstract: Apparatuses and methods train a model and then use the trained model to determine a global three dimensional (3D) position and orientation of a fiduciary marker. In the context of an apparatus for training a model, a wider field-of-view sensor is configured to acquire a static image of a space in which the fiducial marker is disposed and a narrower field-of-view sensor is configured to acquire a plurality of images of at least a portion of the fiducial marker. The apparatus also includes a pan-tilt unit configured to controllably alter pan and tilt angles of the narrower field-of-view sensor during image acquisition. The apparatus further includes a control system configured to determine a transformation of position and orientation information determined from the images acquired by the narrower field-of-view sensor to a coordinate system for the space for which the static image is acquired by the wider field-of-view sensor.

Abstract: A method, apparatus and computer program product are provided to generate a model of one or more objects relative to a vehicle. In the context of a method, radar information is received in the form of in-phase quadrature (IQ) data and the IQ data is converted to one or more first range-doppler maps. The method further includes evaluating the one or more first range-doppler maps with a machine learning model to generate the model that captures the detection of the one or more objects relative to the vehicle. A corresponding apparatus and computer program product are also provided.

Abstract: Described is a system for controlling multiple autonomous platforms. A training process is performed to produce a trained learning agent in a simulation environment. In each episode, each controlled platform is assigned to one target platform that produces an observation. A learning agent processes the observation using a deep learning network and produces an action corresponding to each controlled platform until an action has been produced for each controlled platform. A reward value is obtained corresponding to the episode. The trained learning agent is executed to control each autonomous platform, where the trained agent receives one or more observations from one or more platform sensors and produces an action based on the one or more observations. The action is then used to control one or more platform actuators.

Abstract: An example includes a method for training an agent to control an aircraft. The method includes: selecting, by the agent, first actions for the aircraft to perform within a first environment respectively during first time intervals based on first states of the first environment during the first time intervals, updating the agent based on first rewards that correspond respectively to the first states, selecting, by the agent, second actions for the aircraft to perform within a second environment respectively during second time intervals based on second states of the second environment during the second time intervals, and updating the agent based on second rewards that correspond respectively to the second states. At least one first rule of the first environment is different from at least one rule of the second environment.

Abstract: An example includes a method for training an agent to control an aircraft. The method includes: selecting, by the agent, first actions for the aircraft to perform within a first environment respectively during first time intervals based on first states of the first environment during the first time intervals, updating the agent based on first rewards that correspond respectively to the first states, selecting, by the agent, second actions for the aircraft to perform within a second environment respectively during second time intervals based on second states of the second environment during the second time intervals, and updating the agent based on second rewards that correspond respectively to the second states. At least one first rule of the first environment is different from at least one rule of the second environment.

Abstract: Apparatuses and methods determine the three-dimensional position and orientation of a fiducial marker and tracking the three-dimensional position and orientation across different fields-of-view. Methods include: capturing an image of a first space in which the fiducial marker is disposed with a first sensor having a first field-of-view; determining the three-dimensional location and orientation of the fiducial marker within the first space based on the image of the first space in which the fiducial marker is disposed; capturing an image of a second space in which the fiducial marker is disposed with a second sensor having a second field-of-view; calculating pan and tilt information for the second sensor to move the second field-of-view of the second sensor to acquire an image of the fiducial marker; and determining the three-dimensional location and orientation of the fiducial marker within the second space based on the image of the second space.

Abstract: Described is a system for onboard, real-time, activity detection and classification. During operation, the system detects one or more objects in a scene using a mobile platform and tracks each of the objects as the objects move in the scene to generate tracks for each object. The tracks are transformed using a fuzzy-logic based mapping to semantics that define group activities of the one or more objects in the scene. Finally, a state machine is used to determine whether the defined group activities are normal or abnormal phases of a predetermined group operation.