Abstract: Described herein are methods and systems for providing virtual presence for telerobotics in a dynamic scene. A sensor captures frames of a scene comprising one or more objects. A computing device generates a set of feature points corresponding to objects in the scene and matches the set of feature points to 3D points in a map of the scene. The computing device generates a dense mesh of the scene and the objects using the matched feature points and transmits the dense mesh the frame to a remote viewing device. The remote viewing device generates a 3D representation of the scene and the objects for display to a user and receives commands from the user corresponding to interaction with the 3D representation of the scene. The remote viewing device transmits the commands to a robot device that executes the commands to perform operations on the objects in the scene.

Abstract: The present invention relates to a plug and socket set for use in a mandibular advancement splint in the treatment of snoring or sleep apnoea, the plug having a head for engagement with a mandibular abutment and the head having two prongs projecting therefrom, the socket having a substantially blind aperture configured to receive the two prongs with at least one clearance, and a crosspiece that spans the aperture for an interengagement with the two prongs, wherein the two prongs can interengage with the crosspiece to form a snap fit connection whereby a first surface and a second surface are substantially sealably mated with one another.

Abstract: Described herein are methods and systems for remote visualization of real-time three-dimensional (3D) facial animation with synchronized voice. A sensor captures frames of a face of a person, each frame comprising color images of the face, depth maps of the face, voice data associated with the person, and a timestamp. The sensor generates a 3D face model of the person using the depth maps. A computing device receives the frames of the face and the 3D face model. The computing device preprocesses the 3D face model. For each frame, the computing device: detects facial landmarks using the color images; matches the 3D face model to the depth maps using non-rigid registration; updates a texture on a front part of the 3D face model using the color images; synchronizes the 3D face model with a segment of the voice data using the timestamp; and transmits the synchronized 3D face model and voice data to a remote device.

Abstract: Described herein are methods and systems for generating multiple maps during object scanning for 3D object reconstruction. A sensor device captures RGB images and depth maps of objects in a scene. A computing device receives the RGB images and the depth maps from the sensor device. The computing device creates a first map using at least a portion of the depth maps, a second map using at least a portion of the depth maps, and a third map using at least a portion of the depth maps. The computing device finds key point matches among the first map, the second map, and the third map. The computing device performs bundle adjustment on the first map, the second map, and the third map using the matched key points to generate a final map. The computing device generates a 3D mesh of the object using the final map.

Abstract: Described herein are methods and systems for three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources. A sensor coupled to a mobile device captures (i) depth maps of a physical object, the depth maps including pose information, and (ii) color images of the object. An edge cloud device coupled to the mobile device via a 5G connection receives the depth maps and the color images. The edge cloud device generates a new 3D model of the object based on the depth maps and color images, when a 3D model of the object has not been generated. The edge cloud device updates an existing 3D model of the object based on the depth maps and color images, when a 3D model of the object has previously been generated. The edge cloud device transmits the new 3D model or the updated 3D model to the mobile device.

Abstract: Described herein are methods and systems for remote visualization of real-time three-dimensional (3D) facial animation with synchronized voice. A sensor captures frames of a face of a person, each frame comprising color images of the face, depth maps of the face, voice data associated with the person, and a timestamp. The sensor generates a 3D face model of the person using the depth maps. A computing device receives the frames of the face and the 3D face model. The computing device preprocesses the 3D face model. For each frame, the computing device: detects facial landmarks using the color images; matches the 3D face model to the depth maps using non-rigid registration; updates a texture on a front part of the 3D face model using the color images; synchronizes the 3D face model with a segment of the voice data using the timestamp; and transmits the synchronized 3D face model and voice data to a remote device.

Abstract: Described herein are methods and systems for providing virtual presence for telerobotics in a dynamic scene. A sensor captures frames of a scene comprising one or more objects. A computing device generates a set of feature points corresponding to objects in the scene and matches the set of feature points to 3D points in a map of the scene. The computing device generates a dense mesh of the scene and the objects using the matched feature points and transmits the dense mesh the frame to a remote viewing device. The remote viewing device generates a 3D representation of the scene and the objects for display to a user and receives commands from the user corresponding to interaction with the 3D representation of the scene. The remote viewing device transmits the commands to a robot device that executes the commands to perform operations on the objects in the scene.

Abstract: Described herein are methods and systems for remote visualization of three-dimensional (3D) animation. A sensor of a mobile device captures scans of non-rigid objects in a scene, each scan comprising a depth map and a color image. A server receives a first set of scans from the mobile device and reconstructs an initial model of the non-rigid objects using the first set of scans. The server receives a second set of scans. For each scan in the second set of one or more scans, the server determines an initial alignment between the depth map and the initial model. The server converts the depth map into a coordinate system of the initial model, and determines a displacement between the depth map and the initial model. The server deforms the initial model to the depth map using the displacement, and applies a texture to at least a portion of the deformed model.

Abstract: Described herein are methods and systems for keyframe-based object scanning and tracking. A sensor device captures images of objects in a scene. For each image, a computing device labels each of at least a plurality of pixels in the image, tracks at least one region of the labeled image to determine an estimate of a current pose of at least one object, validates the estimate of the current pose of the at least one object, selects the labeled image as a keyframe based upon validation of the estimate of the current pose, and updates a volumetric model comprising the at least one object using the keyframe. The computing device generates a final 3D model of the at least one object based upon the updated volumetric model.

Abstract: Described are methods, systems, and apparatuses for 3D vision processing using an IP block. A vision processing module comprises an integrated circuit that performs one or more 3D vision processing algorithms and a plurality of controllers that couple the integrated circuit to each of: a sensor device, a processor, a memory module, and a network interface. The vision processing module receives image data from the sensor device, the image data corresponding to one or more images captured by the sensor device. The vision processing module executes one or more of the 3D vision processing algorithms using at least a portion of the image data as input. The vision processing module transmits an output from execution of one or more of the 3D vision processing algorithms to at least one of: the processor, the memory module, or the network interface.

Abstract: Described herein are methods and systems for generating multiple maps during object scanning for 3D object reconstruction. A sensor device captures RGB images and depth maps of objects in a scene. A computing device receives the RGB images and the depth maps from the sensor device. The computing device creates a first map using at least a portion of the depth maps, a second map using at least a portion of the depth maps, and a third map using at least a portion of the depth maps. The computing device finds key point matches among the first map, the second map, and the third map. The computing device performs bundle adjustment on the first map, the second map, and the third map using the matched key points to generate a final map. The computing device generates a 3D mesh of the object using the final map.

Abstract: Described are methods and systems for real-time remote avatar creation and animation control. A sensor device captures images of non-rigid objects in a scene. A server coupled to the sensor device generates an initial 3D model for each of the non-rigid objects in the scene using the images. The server detects landmark points on the non-rigid objects using the initial 3D model. The server generates a control point animation map for the 3D model using the detected points. The server applies the animation map to the 3D model to generate a mapped 3D model. A viewer coupled to the server receives (i) the mapped 3D model and (ii) tracking information associated with the objects, including a model pose and deformation of the landmark points. The viewer modifies the mapped 3D model using the tracking information, and renders a video stream on the viewer using the modified 3D model.

Abstract: Described herein are methods and systems for three-dimensional (3D) object capture and object reconstruction using edge cloud computing resources. A sensor coupled to a mobile device captures (i) depth maps of a physical object, the depth maps including pose information, and (ii) color images of the object. An edge cloud device coupled to the mobile device via a 5G connection receives the depth maps and the color images. The edge cloud device generates a new 3D model of the object based on the depth maps and color images, when a 3D model of the object has not been generated. The edge cloud device updates an existing 3D model of the object based on the depth maps and color images, when a 3D model of the object has previously been generated. The edge cloud device transmits the new 3D model or the updated 3D model to the mobile device.

Abstract: Described herein are methods and systems for remote visualization of three-dimensional (3D) animation. A sensor of a mobile device captures scans of non-rigid objects in a scene, each scan comprising a depth map and a color image. A server receives a first set of scans from the mobile device and reconstructs an initial model of the non-rigid objects using the first set of scans. The server receives a second set of scans. For each scan in the second set of one or more scans, the server determines an initial alignment between the depth map and the initial model. The server converts the depth map into a coordinate system of the initial model, and determines a displacement between the depth map and the initial model. The server deforms the initial model to the depth map using the displacement, and applies a texture to at least a portion of the deformed model.

Abstract: Described herein are methods and systems for keyframe-based object scanning and tracking. A sensor device captures images of objects in a scene. For each image, a computing device labels each of at least a plurality of pixels in the image, tracks at least one region of the labeled image to determine an estimate of a current pose of at least one object, validates the estimate of the current pose of the at least one object, selects the labeled image as a keyframe based upon validation of the estimate of the current pose, and updates a volumetric model comprising the at least one object using the keyframe. The computing device generates a final 3D model of the at least one object based upon the updated volumetric model.

Abstract: Described are methods, systems, and apparatuses for 3D vision processing using an IP block. A vision processing module comprises an integrated circuit that performs one or more 3D vision processing algorithms and a plurality of controllers that couple the integrated circuit to each of: a sensor device, a processor, a memory module, and a network interface. The vision processing module receives image data from the sensor device, the image data corresponding to one or more images captured by the sensor device. The vision processing module executes one or more of the 3D vision processing algorithms using at least a portion of the image data as input. The vision processing module transmits an output from execution of one or more of the 3D vision processing algorithms to at least one of: the processor, the memory module, or the network interface.

Abstract: Described are methods and systems for generating a video stream of a scene including one or more objects. A sensor captures images of objects in a scene. A server coupled to the sensor, for each image, generates an initial 3D model for the objects and an initial 3D model of the scene. The server, for each image, captures pose information of the sensor as the sensor moves in relation to the scene or as the objects move in relation to the sensor. A viewing device receives the models and the pose information from the server. The viewing device captures pose information of the viewing device as the viewing device moves in relation to the scene. The viewing device renders a video stream on a display element using the received 3D models and at least one of the pose information of the sensor or the pose information of the viewing device.

Abstract: Described are methods and systems for real-time remote avatar creation and animation control. A sensor device captures images of non-rigid objects in a scene. A server coupled to the sensor device generates an initial 3D model for each of the non-rigid objects in the scene using the images. The server detects landmark points on the non-rigid objects using the initial 3D model. The server generates a control point animation map for the 3D model using the detected points. The server applies the animation map to the 3D model to generate a mapped 3D model. A viewer coupled to the server receives (i) the mapped 3D model and (ii) tracking information associated with the objects, including a model pose and deformation of the landmark points. The viewer modifies the mapped 3D model using the tracking information, and renders a video stream on the viewer using the modified 3D model.

Abstract: Described are methods and systems for enhancing depth sensor-based 3D geometry reconstruction with photogrammetry. A 3D sensor captures scans of a physical object, including related pose information and HD images corresponding to each scan. For each scan, a computing device generates an initial 3D model of the physical object, the initial model having missing sections. The computing device detects the missing sections in a 3D point cloud associated with the initial model and projects the missing sections in the point cloud to a corresponding HD image. The computing device generates image segments of the corresponding HD image that match the missing sections in the point cloud, generates a point cloud structure for each of the image segments of the corresponding HD image, and merges the initial model and the generated point cloud structures to generate a final 3D model with the missing sections filled in.