Abstract: A method includes receiving an input comprising multi-modal inputs such as text, audio, video, or context information from a client device associated with a user, assigning a task associated with the input to a server among a plurality of servers, determining a context response corresponding to the input based on the input and interaction history between the computing system and the user, generating meta data specifying expressions, emotions, and non-verbal and verbal gestures associated with the context response by querying a trained behavior knowledge graph, generating media content output based on the determined context response and the generated meta data, the media content output comprising of text, audio, and visual information corresponding to the determined context response in the expressions, the emotions, and the non-verbal and verbal gestures specified by the meta data, sending instructions for presenting the generated media content output to the user to the client device.

Abstract: A system and method for a master platform includes receiving first pose data associated with an image sensor of a first device, and a first semantic map generated by the first device, the first semantic map including a simplified object representation in a coordinate space of the first device. The master platform also receives second pose data associated with an image sensor of a second device, and a second semantic map generated by the second device, the second semantic map including a simplified object representation in a coordinate space of the second device. A shared simplified object representation common to the first and semantic maps is identified. The master platform further combines the first semantic and second semantic maps based on the first and second pose data. The first pose data, first semantic map, second pose data, and second semantic map are associated with a common time interval.

Abstract: An untethered apparatus for performing inside-out device tracking based on visual-inertial simultaneous location and mapping (SLAM) includes a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, an inertial measurement unit (IMU) sensor configured to collect IMU data associated with motion of the apparatus at a predetermined interval, a processor and a memory. The memory contains instructions, which when executed by the processor, cause the apparatus to accumulate DVS sensor output over a sliding time window, the sliding time window including the predetermined interval, apply a motion correction to the accumulated DVS sensor output, the motion correction based on the IMU data collected over the predetermined interval, generate an event-frame histogram of DVS sensor events based on the motion correction, and provide the event-frame histogram of the DVS sensor events and the IMU data to a visual inertial SLAM pipeline.

Abstract: An apparatus includes a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, and a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data. The apparatus further includes a hybrid feature handler configured to receive, as an input, one or more of a DVS output or a CMOS image sensor output, and provide tracked features to a visual-inertial simultaneous location and mapping (SLAM) pipeline performing inside-out device tracking, and a sensor scheduler configured to switch off the CMOS image sensor based on a current value of one or more CMOS control factors.

Abstract: A method for semi-dense depth estimation includes receiving, at an electronic device, a control signal of a speckle pattern projector (SPP and receiving from each sensor of a dynamic vision sensor (DVS) stereo pair, an event stream of pixel intensity change data, wherein the event stream is time-synchronized with the control signal of the SPP. The method further includes performing projected light filtering on the event stream of pixel intensity change data for each sensor of the DVS stereo pair, to generate synthesized event image data, the synthesized event image data having one or more channels, each channel based on an isolated portion of the event stream of pixel intensity change data and performing stereo matching on at least one channel of the synthesized event image data for each sensor of the DVS stereo pair to generate a depth map for at least a portion of the field of view.

Abstract: A retriever for retrieving objects, including an elongated arm; a container on one end of the arm having an opening providing access to an interior of the container; and an actuator on the other end of the arm. Operation of the actuator causes movement of a scraper plate between an open position and a closed position. When in the open position, the scraper plate is spaced from the opening. When in the closed position, the scraper plate contacts the container at the opening. If an object is located in the space between the scraper plate and the opening, then movement of the scraper plate between the open and closed positions pushes the object into the interior of the container. The object can then slide through a channel along the arm, and be retrieved by a user from an opening in the channel near the trigger.

Abstract: An untethered apparatus for performing inside-out device tracking based on visual-inertial simultaneous location and mapping (SLAM) includes a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, an inertial measurement unit (IMU) sensor configured to collect IMU data associated with motion of the apparatus at a predetermined interval, a processor and a memory. The memory contains instructions, which when executed by the processor, cause the apparatus to accumulate DVS sensor output over a sliding time window, the sliding time window including the predetermined interval, apply a motion correction to the accumulated DVS sensor output, the motion correction based on the IMU data collected over the predetermined interval, generate an event-frame histogram of DVS sensor events based on the motion correction, and provide the event-frame histogram of the DVS sensor events and the IMU data to a visual inertial SLAM pipeline.

Abstract: A system and method for a master platform includes receiving first pose data associated with an image sensor of a first device, and a first semantic map generated by the first device, the first semantic map including a simplified object representation in a coordinate space of the first device. The master platform also receives second pose data associated with an image sensor of a second device, and a second semantic map generated by the second device, the second semantic map including a simplified object representation in a coordinate space of the second device. A shared simplified object representation common to the first and semantic maps is identified. The master platform further combines the first semantic and second semantic maps based on the first and second pose data. The first pose data, first semantic map, second pose data, and second semantic map are associated with a common time interval.

Abstract: A system for determining and tracking camera pose includes a dynamic vision sensor (DVS) configured to generate a current DVS image, an inertial measurement unit (IMU) configured to generate inertial data, and a memory. The memory is configured to store a 3-dimensional (3D) map of a known 3D environment. The system may also include a processor coupled to the memory. The processor is configured to initiate operations including determining a current camera pose for the DVS based on the current DVS image, the inertial data, the 3D map, and a prior camera pose.

Abstract: An apparatus includes a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, a CMOS image sensor configured to output frames of image data, an inertial measurement unit (IMU), a processor and a memory. The memory contains instructions, which when executed by the processor, cause the apparatus to generate a semantic segmentation of a time-stamped frame, which is based on one or more of an output of the CMOS image sensor, or a synthesized event frame based on an output from the DVS and an output from the IMU over a time interval. The semantic segmentation includes a semantic label associated with a region of the time-stamped frame. When executed, the instructions further cause the apparatus to determine, based on the semantic segmentation, a simplified object representation in a coordinate space, and update a stable semantic map based on the simplified object representation.

Abstract: A method of camera pose and plane estimation may include detecting a marker within a 3-dimensional (3D) environment by detecting, using a Dynamic Vision Sensor (DVS), a first plurality of light sources arranged in a known shape and blinking at a first frequency, wherein the known shape corresponds to the marker, determining an orientation and an identity of the marker based upon detecting, using the DVS, a second plurality of light sources corresponding to the marker and blinking at a second frequency different from the first frequency. A camera pose for the DVS may be determined based upon the known shape, the orientation, and the identity of the marker using the processor.

Abstract: An apparatus for catching bugs is provided which includes a catch box having an interior region and an elongated handle attached to the catch box. The elongated handle includes a flexible tube member and a flexible rod member disposed within the flexible tube member. The apparatus includes a trap adapted to move back and forth within the interior region of the catch box. The flexible tube member is attached to the catch box and the flexible rod member is attached to the trap. The flexible rod member is adapted to provide back and forth movement of the trap within the interior region of the catch box to thereby define a closed chamber in the interior of the box for trapping bugs.

Abstract: A method for semi-dense depth estimation includes receiving, at an electronic device, a control signal of a speckle pattern projector (SPP and receiving from each sensor of a dynamic vision sensor (DVS) stereo pair, an event stream of pixel intensity change data, wherein the event stream is time-synchronized with the control signal of the SPP. The method further includes performing projected light filtering on the event stream of pixel intensity change data for each sensor of the DVS stereo pair, to generate synthesized event image data, the synthesized event image data having one or more channels, each channel based on an isolated portion of the event stream of pixel intensity change data and performing stereo matching on at least one channel of the synthesized event image data for each sensor of the DVS stereo pair to generate a depth map for at least a portion of the field of view.

Abstract: An apparatus includes a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, and a complementary metal-oxide-semiconductor (CMOS) image sensor configured to output frames of image data. The apparatus further includes a hybrid feature handler configured to receive, as an input, one or more of a DVS output or a CMOS image sensor output, and provide tracked features to a visual-inertial simultaneous location and mapping (SLAM) pipeline performing inside-out device tracking, and a sensor scheduler configured to switch off the CMOS image sensor based on a current value of one or more CMOS control factors.

Abstract: An apparatus includes a dynamic vision sensor (DVS) configured to output an asynchronous stream of sensor event data, a CMOS image sensor configured to output frames of image data, an inertial measurement unit (IMU), a processor and a memory. The memory contains instructions, which when executed by the processor, cause the apparatus to generate a semantic segmentation of a time-stamped frame, which is based on one or more of an output of the CMOS image sensor, or a synthesized event frame based on an output from the DVS and an output from the IMU over a time interval. The semantic segmentation includes a semantic label associated with a region of the time-stamped frame. When executed, the instructions further cause the apparatus to determine, based on the semantic segmentation, a simplified object representation in a coordinate space, and update a stable semantic map based on the simplified object representation.

Abstract: A retriever for retrieving objects, including an elongated arm; a container on one end of the arm having an opening providing access to an interior of the container; and an actuator on the other end of the arm. Operation of the actuator causes movement of a scraper plate between an open position and a closed position. When in the open position, the scraper plate is spaced from the opening. When in the closed position, the scraper plate contacts the container at the opening. If an object is located in the space between the scraper plate and the opening, then movement of the scraper plate between the open and closed positions pushes the object into the interior of the container. The object can then slide through a channel along the arm, and be retrieved by a user from an opening in the channel near the trigger.

Abstract: A system for determining and tracking camera pose includes a dynamic vision sensor (DVS) configured to generate a current DVS image, an inertial measurement unit (IMU) configured to generate inertial data, and a memory. The memory is configured to store a 3-dimensional (3D) map of a known 3D environment. The system may also include a processor coupled to the memory. The processor is configured to initiate operations including determining a current camera pose for the DVS based on the current DVS image, the inertial data, the 3D map, and a prior camera pose.

Abstract: A method of camera pose and plane estimation may include detecting a marker within a 3-dimensional (3D) environment by detecting, using a Dynamic Vision Sensor (DVS), a first plurality of light sources arranged in a known shape and blinking at a first frequency, wherein the known shape corresponds to the marker, determining an orientation and an identity of the marker based upon detecting, using the DVS, a second plurality of light sources corresponding to the marker and blinking at a second frequency different from the first frequency. A camera pose for the DVS may be determined based upon the known shape, the orientation, and the identity of the marker using the processor.

Abstract: An apparatus for catching bugs is provided which includes a catch box having an interior region and an elongated handle attached to the catch box. The elongated handle includes a flexible tube member and a flexible rod member disposed within the flexible tube member. The apparatus includes a trap adapted to move back and forth within the interior region of the catch box. The flexible tube member is attached to the catch box and the flexible rod member is attached to the trap. The flexible rod member is adapted to provide back and forth movement of the trap within the interior region of the catch box to thereby define a closed chamber in the interior of the box for trapping bugs.

Abstract: A roof tool having a movable blade for removing roofing materials from a roof includes a carriage and a handle assembly pivotally mounted at a first end to the carriage. A blade is mounted adjacent the first end of the handle assembly for movement with the handle assembly when the handle assembly is pivotally moved relative to the carriage. Movement of a second end of the handle assembly causes leveraged movement of the blade relative to the carriage.