Patents by Inventor Julian Straub
Julian Straub has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Patent number: 11625862Abstract: In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.Type: GrantFiled: October 16, 2020Date of Patent: April 11, 2023Assignee: Meta Platforms Technologies, LLCInventors: Michael Goesele, Julian Straub, Thomas John Whelan, Richard Andrew Newcombe, Steven John Lovegrove
-
Patent number: 11042749Abstract: The disclosed computer-implemented method may include receiving, from devices in an environment, real-time data associated with the environment and determining, from the real-time data, current object data for the environment. The current object data may include both state data and relationship data for objects in the environment. The method may also include determining object deltas between the current object data and prior object data from an event graph. The prior object data may include prior state data and prior relationship data for the objects. The method may include detecting an unknown state for one of the objects, inferring a state for the object based on the event graph, and updating the event graph based on the object deltas and the inferred state. The method may further include sending updated event graph data to the devices. Various other methods, systems, and computer-readable media are also disclosed.Type: GrantFiled: March 18, 2020Date of Patent: June 22, 2021Assignee: Facebook Technologies, LLCInventors: Richard Andrew Newcombe, Jakob Julian Engel, Julian Straub, Thomas John Whelan, Steven John Lovegrove, Yuheng Ren
-
Patent number: 10930077Abstract: The disclosed computer-implemented method may include determining a local position and a local orientation of a local device in an environment and receiving, by the local device and from a mapping system, object data for objects within the environment. The object data may include position data and orientation data for the objects and relationship data between the objects. The method may also include deriving, based on the object data received from the mapping system, and the local position and orientation of the local device, a contextual rendering of the objects that provides contextual data that modifies a user's view of the environment. The method may include displaying, using the local device, the contextual rendering of at least one of the plurality of objects to modify the user's view of the environment. Various other methods, systems, and computer-readable media are also disclosed.Type: GrantFiled: September 14, 2018Date of Patent: February 23, 2021Assignee: Facebook Technologies, LLCInventors: Richard Andrew Newcombe, Jakob Julian Engel, Julian Straub, Thomas John Whelan, Steven John Lovegrove, Yuheng Ren
-
Publication number: 20210035329Abstract: In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.Type: ApplicationFiled: October 16, 2020Publication date: February 4, 2021Inventors: Michael Goesele, Julian Straub, Thomas John Whelan, Richard Andrew Newcombe, Steven John Lovegrove
-
Patent number: 10839560Abstract: In one embodiment, a method includes accessing a digital image captured by a camera that is connected to a machine-detectable object, detecting a reflection of the machine-detectable object in the digital image, computing, in response to the detection, a plane that is coincident with a reflective surface associated with the reflection, determining a boundary of the reflective surface in the plane based on at least one of a plurality of cues, and storing information associated with the reflective surface, where the information includes a pose of the reflective surface and the boundary of the reflective surface in a 3D model of a physical environment, and where the information associated with the reflective surface and the 3D model are configured to be used to render a reconstruction of the physical environment.Type: GrantFiled: February 26, 2019Date of Patent: November 17, 2020Assignee: Facebook Technologies, LLCInventors: Michael Goesele, Julian Straub, Thomas John Whelan, Richard Andrew Newcombe, Steven John Lovegrove
-
Publication number: 20200218898Abstract: The disclosed computer-implemented method may include receiving, from devices in an environment, real-time data associated with the environment and determining, from the real-time data, current object data for the environment. The current object data may include both state data and relationship data for objects in the environment. The method may also include determining object deltas between the current object data and prior object data from an event graph. The prior object data may include prior state data and prior relationship data for the objects. The method may include detecting an unknown state for one of the objects, inferring a state for the object based on the event graph, and updating the event graph based on the object deltas and the inferred state. The method may further include sending updated event graph data to the devices. Various other methods, systems, and computer-readable media are also disclosed.Type: ApplicationFiled: March 18, 2020Publication date: July 9, 2020Inventors: Richard Andrew Newcombe, Jakob Julian Engel, Julian Straub, Thomas John Whelan, Steven John Lovegrove, Yuheng Ren
-
Patent number: 10635905Abstract: The disclosed computer-implemented method may include receiving, from devices in an environment, real-time data associated with the environment. The method may also include determining, from the real-time data, current mapping and object data. The current mapping data may include coordinate data for the environment and the current object data may include both state data and relationship data for objects in the environment. The method may also include determining mapping deltas between the current mapping data and baseline map data and determining object deltas between the current object data and an event graph. The event graph may include prior state data and prior relationship data for objects. The method may also include updating the baseline map data and the event graph based on the deltas and sending updated baseline map data and event graph data to the devices. Various other methods, systems, and computer-readable media are also disclosed.Type: GrantFiled: September 14, 2018Date of Patent: April 28, 2020Assignee: Facebook Technologies, LLCInventors: Richard Andrew Newcombe, Jakob Julian Engel, Julian Straub, Thomas John Whelan, Steven John Lovegrove, Yuheng Ren
-
Publication number: 20200089953Abstract: The disclosed computer-implemented method may include receiving, from devices in an environment, real-time data associated with the environment. The method may also include determining, from the real-time data, current mapping and object data. The current mapping data may include coordinate data for the environment and the current object data may include both state data and relationship data for objects in the environment. The method may also include determining mapping deltas between the current mapping data and baseline map data and determining object deltas between the current object data and an event graph. The event graph may include prior state data and prior relationship data for objects. The method may also include updating the baseline map data and the event graph based on the deltas and sending updated baseline map data and event graph data to the devices. Various other methods, systems, and computer-readable media are also disclosed.Type: ApplicationFiled: September 14, 2018Publication date: March 19, 2020Inventors: Richard Andrew Newcombe, Jakob Julian Engel, Julian Straub, Thomas John Whelan, Steven John Lovegrove, Yuheng Ren
-
Patent number: 10529128Abstract: A sensor apparatus may include an array of protruding members that each extend outward in a different radial direction from a central axis, each protruding member including a projector that projects structured light into a local environment, one or more cameras that capture reflections of the structured light from the local environment, and another camera that captures visible-spectrum light from the local environment. The sensor apparatus may also include one or more localization devices for determining a location of the sensor apparatus. Various other apparatuses, systems, and methods are also disclosed.Type: GrantFiled: April 27, 2018Date of Patent: January 7, 2020Assignee: Facebook Technologies, LLCInventors: Yuheng Ren, Julian Straub, Shobhit Verma, Richard Andrew Newcombe, Renzo De Nardi
-
Patent number: 10237481Abstract: An imaging device operates as an event camera. The device includes an event sensor and a controller. The sensor comprises a plurality of photodiodes that asynchronously output data values corresponding to relative intensity changes within a local area. The controller populates an event matrix based in part on data values asynchronously received from the sensor and positions of photodiodes associated with the received data values over a first time period. The controller populates a change matrix based in part on a threshold intensity value and the photodiodes associated with the received data values over the first time period, and generates an image for the first time period using the event matrix and the change matrix.Type: GrantFiled: April 18, 2017Date of Patent: March 19, 2019Assignee: Facebook Technologies, LLCInventors: Richard Andrew Newcombe, Michael Hall, Xinqiao Liu, Steven John Lovegrove, Julian Straub
-
Publication number: 20180302562Abstract: An imaging device operates as an event camera. The device includes an event sensor and a controller. The sensor comprises a plurality of photodiodes that asynchronously output data values corresponding to relative intensity changes within a local area. The controller populates an event matrix based in part on data values asynchronously received from the sensor and positions of photodiodes associated with the received data values over a first time period. The controller populates a change matrix based in part on a threshold intensity value and the photodiodes associated with the received data values over the first time period, and generates an image for the first time period using the event matrix and the change matrix.Type: ApplicationFiled: April 18, 2017Publication date: October 18, 2018Inventors: Richard Andrew Newcombe, Michael Hall, Xinqiao Liu, Steven John Lovegrove, Julian Straub
-
Patent number: 9665791Abstract: In one embodiment, a method of identifying the dominant orientations of a scene comprises representing a scene as a plurality of directional vectors. The scene may comprise a three-dimensional representation of a scene, and the plurality of directional vectors may comprise a plurality of surface normals. The method further comprises determining, based on the plurality of directional vectors, a plurality of orientations describing the scene. The determined plurality of orientations explains the directionality of the plurality of directional vectors. In certain embodiments, the plurality of orientations may have independent axes of rotation. The plurality of orientations may be determined by representing the plurality of directional vectors as lying on a mathematical representation of a sphere, and inferring the parameters of a statistical model to adapt the plurality of orientations to explain the positioning of the plurality of directional vectors lying on the mathematical representation of the sphere.Type: GrantFiled: April 3, 2015Date of Patent: May 30, 2017Assignee: Massachusetts Institute of TechnologyInventors: Julian Straub, Guy Rosman, Oren Freifeld, John J. Leonard, John W. Fisher, III
-
Publication number: 20150286893Abstract: In one embodiment, a method of identifying the dominant orientations of a scene comprises representing a scene as a plurality of directional vectors. The scene may comprise a three-dimensional representation of a scene, and the plurality of directional vectors may comprise a plurality of surface normals. The method further comprises determining, based on the plurality of directional vectors, a plurality of orientations describing the scene. The determined plurality of orientations explains the directionality of the plurality of directional vectors. In certain embodiments, the plurality of orientations may have independent axes of rotation. The plurality of orientations may be determined by representing the plurality of directional vectors as lying on a mathematical representation of a sphere, and inferring the parameters of a statistical model to adapt the plurality of orientations to explain the positioning of the plurality of directional vectors lying on the mathematical representation of the sphere.Type: ApplicationFiled: April 3, 2015Publication date: October 8, 2015Inventors: Julian Straub, Guy Rosman, Oren Freifeld, John J. Leonard, John W. Fisher, III