Abstract: System and method are disclosed for training a generative adversarial network pipeline that can produce realistic artificial depth images useful as training data for deep learning networks used for robotic tasks. A generator network receives a random noise vector and a computer aided design (CAD) generated depth image and generates an artificial depth image. A discriminator network receives either the artificial depth image or a real depth image in alternation, and outputs a predicted label indicating a discriminator decision as to whether the input is the real depth image or the artificial depth image. Training of the generator network is performed in tandem with the discriminator network as a generative adversarial network. A generator network cost function minimizes correctly predicted labels, and a discriminator cost function maximizes correctly predicted labels.

Abstract: A computer-implemented method for building an object detection module uses mesh representations of objects belonging to specified object classes of interest to render images by a physics-based simulator. Each rendered image captures a simulated environment containing objects belonging to multiple object classes of interest placed in a bin or on a table. The rendered images are generated by randomizing a set of parameters by the simulator to render a range of simulated environments. The randomized parameters include environmental and sensor-based parameters. A label is generated for each rendered image, which includes a two-dimensional representation indicative of location and object classes of objects in that rendered image frame. Each rendered image and the respective label constitute a data sample of a synthetic training dataset. A deep learning model is trained using the synthetic training dataset to output object classes from an input image of a real-world physical environment.

Abstract: An autonomous system can detect out-of-distribution (OOD) data in robotic grasping systems, based on evaluating image inputs of the robotic grasping systems. Furthermore, the system makes various decisions based on detecting the OOD data, so as to avoid inefficient or hazardous situations or other negative consequences (e.g., damage to products). For example, the system can determine whether a suction-based gripper is optimal for grasping objects in a given scene, based at least in part on determining whether an image defines OOD data.

Abstract: Fully flexible kitting processes can be automated by generating pick and place motions for multi-robot, multi-gripper, robotic systems.

Abstract: A covariate shift generally refers to the change of the distribution of the input data (e.g., noise distribution) between the training and inference regimes. Such covariate shifts can degrade the performance grasping neural networks, and thus robotic grasping operations. As described herein, an output of a grasp neural network can be transformed, so as to determine appropriate locations on a given object for a robot or autonomous machine to grasp.

Abstract: In some cases, grasp point algorithms can be implemented so as to compute grasp points on an object that enable a stable grasp. It is recognized herein, however, that in practice a robot in motion can drop the object or otherwise have grasp issues when the object is grasped at the computed stable grasp points. Path constraints that can differ based on a given object are generated while generating the trajectory for a robot, so as to ensure that a grasp remains stable throughout the motion of the robot.

Abstract: It is recognized It is recognized herein that current approaches to robotic picking lack efficiency and capabilities. In particular, current approaches often do not properly or efficiently estimate the pose of bins, due to various technical challenges in doing so, which can impact grasp computations and overall performance of a given robot. The pose of the bin can be determined or estimated based on depth images. Such bin pose estimation can be performed during runtime of a given robot, such that grasping can be enhanced due to the bin pose estimations.

Abstract: It is recognized herein that robots or autonomous systems can lose time when computing grasp scores for empty bins. Further, when grasps are attempted on empty bins, for instance due to the related grasp score computations, the robot can lose additional time through being used unnecessarily to attempt the grasp. Such usage can wear on the robot, or damage the robot, in some cases. An autonomous system can classify or determine whether a bin contains an object or is empty, for example, such that a grasp computation is not performed when the bin is empty. In some examples, a system classifies a given bin at runtime before each grasp computation is performed. Thus, systems described herein can avoid performing unnecessary grasp computations, thereby conserving processing time and overheard, among addressing other technical problems.

Abstract: A method for automatically generating a bill of process in a manufacturing system comprising: receiving design information representative of a product to be produced; iteratively performing simulations of the manufacturing system; identifying manufacturing actions based on the simulations; optimizing the identified manufacturing actions to efficiently produce the product to be produced; generating, by the manufacturing system, a bill of process for producing the product. Simulations may be performed using a digital twin of the product being produced and a digital twin of the environment. System actions are optimized using a reinforcement learning technique to automatically produce a bill of process based on the design information of the product and task specifications.

Abstract: A computer-implemented method for designing execution of a process by a robotic cell includes obtaining a process goal and one or more process constraints. The method includes accessing a library of constructs and a library of skills. Each construct includes a digital representation of a component of the robotic cell or a geometric transformation of the robotic cell. Each skill includes a functional description for using a robot of the robotic cell to interact with a physical environment to perform a skill objective. The method uses a simulation engine to simulate a multiplicity of designs, wherein each design is characterized by a combination of constructs and skills to achieve the process goal, and determine a set of feasible designs that meet the one or more process constraints. The method includes outputting recommended designs from the set of feasible designs.