Abstract: Implementations disclosed herein relate to mitigating the reality gap through feature-level domain adaptation in training of a vision-based robotic action machine learning (ML) model. Implementations mitigate the reality gap through utilization of embedding consistency losses and/or action consistency losses during training of the action ML model.

Abstract: Some implementations relate to using trained robotic action ML models in controlling a robot to perform a robotic task. Some versions of those implementations include (a) a first modality robotic action ML model that is used to generate, based on processing first modality sensor data instances, first predicted action outputs for the robotic task and (b) a second modality robotic action ML model that is used to generate, in parallel and based on processing second modality sensor data instances, second predicted action outputs for the robotic task. In some of those versions, respective weights for each pair of the first and second predicted action outputs are dynamically determined based on analysis of embeddings generated in generating the first and second predicted action outputs. A final predicted action output, for controlling the robot, is determined based on the weights.

Abstract: Implementations disclosed herein relate to mitigating the reality gap through feature-level domain adaptation in training of a vision-based robotic action machine learning (ML) model. Implementations mitigate the reality gap through utilization of embedding consistency losses and/or action consistency losses during training of the action ML model.