Patents Assigned to Matterport, Inc.
  • Publication number: 20210264609
    Abstract: Systems and methods for user guided iterative frame and scene segmentation are disclosed herein. The systems and methods can rely on overtraining a segmentation network on a frame. A disclosed method includes selecting a frame from a scene and generating a frame segmentation using the frame and a segmentation network. The method also includes displaying the frame and frame segmentation overlain on the frame, receiving a correction input on the frame, and training the segmentation network using the correction input. The method includes overtraining the segmentation network for the scene by iterating the above steps on the same frame or a series of frames from the scene.
    Type: Application
    Filed: May 11, 2021
    Publication date: August 26, 2021
    Applicant: Matterport, Inc.
    Inventor: Gary Bradski
  • Patent number: 11094117
    Abstract: Systems and techniques for processing and/or transmitting three-dimensional (3D) data are presented. A partitioning component receives captured 3D data associated with a 3D model of an interior environment and partitions the captured 3D data into at least one data chunk associated with at least a first level of detail and a second level of detail. A data component stores 3D data including at least the first level of detail and the second level of detail for the at least one data chunk. An output component transmits a portion of data from the at least one data chunk that is associated with the first level of detail or the second level of detail to a remote client device based on information associated with the first level of detail and the second level of detail.
    Type: Grant
    Filed: March 10, 2020
    Date of Patent: August 17, 2021
    Assignee: Matterport, Inc.
    Inventors: Matthew Tschudy Bell, David Alan Gausebeck, Gregory William Coombe, Daniel Ford
  • Patent number: 11094137
    Abstract: The disclosed subject matter is directed to employing machine learning models configured to predict 3D data from 2D images using deep learning techniques to derive 3D data for the 2D images. In some embodiments, a system is described comprising a memory that stores computer executable components, and a processor that executes the computer executable components stored in the memory. The computer executable components comprise a reception component configured to receive two-dimensional images, and a three-dimensional data derivation component configured to employ one or more three-dimensional data from two-dimensional data (3D-from-2D) neural network models to derive three-dimensional data for the two-dimensional images.
    Type: Grant
    Filed: September 25, 2018
    Date of Patent: August 17, 2021
    Assignee: Matterport, Inc.
    Inventors: David Alan Gausebeck, Matthew Tschudy Bell, Waleed K. Abdulla, Peter Kyuhee Hahn
  • Patent number: 11080861
    Abstract: Systems and methods for frame and scene segmentation are disclosed herein. A disclosed method includes providing a frame of a scene. The scene includes a scene background. The method also includes providing a model of the scene background. The method also includes determining a frame background using the model and subtracting the frame background from the frame to obtain an approximate segmentation. The method also includes training a segmentation network using the approximate segmentation.
    Type: Grant
    Filed: May 14, 2019
    Date of Patent: August 3, 2021
    Assignee: Matterport, Inc.
    Inventors: Gary Bradski, Ethan Rublee
  • Patent number: 11080884
    Abstract: A trained network for point tracking includes an input layer configured to receive an encoding of an image. The image is of a locale or object on which the network has been trained. The network also includes a set of internal weights which encode information associated with the locale or object, and a tracked point therein or thereon. The network also includes an output layer configured to provide an output based on the image as received at the input layer and the set of internal weights. The output layer includes a point tracking node that tracks the tracked point in the image. The point tracking node can track the point by generating coordinates for the tracked point in an input image of the locale or object. Methods of specifying and training the network using a three-dimensional model of the locale or object are also disclosed.
    Type: Grant
    Filed: May 15, 2019
    Date of Patent: August 3, 2021
    Assignee: Matterport, Inc.
    Inventors: Gary Bradski, Gholamreza Amayeh, Mona Fathollahi, Ethan Rublee, Grace Vesom, William Nguyen
  • Patent number: 11069117
    Abstract: Systems and methods for generating three-dimensional models having regions of various resolutions are provided. In particular, imagery data can be captured and utilized to generate three-dimensional models. Regions of texture can be mapped to regions of a three-dimensional model when rendered. Resolutions of texture can be selectively altered and regions of texture can be selectively segmented to reduce texture memory cost. Texture can be algorithmically generated based on alternative texturing techniques. Models can be rendered having regions at various resolutions.
    Type: Grant
    Filed: May 6, 2019
    Date of Patent: July 20, 2021
    Assignee: Matterport, Inc.
    Inventors: Daniel Ford, Matthew Tschudy Bell, David Alan Gausebeck, Mykhaylo Kurinnyy
  • Patent number: 11062509
    Abstract: The present disclosure concerns a methodology that allows a user to “orbit” around a model on a specific axis of rotation and view an orthographic floor plan of the model. A user may view and “walk through” the model while staying at a specific height above the ground with smooth transitions between orbiting, floor plan, and walking modes.
    Type: Grant
    Filed: April 17, 2019
    Date of Patent: July 13, 2021
    Assignee: Matterport, Inc.
    Inventors: Matthew Bell, Michael Beebe
  • Patent number: 11004203
    Abstract: Systems and methods for user guided iterative frame and scene segmentation are disclosed herein. The systems and methods can rely on overtraining a segmentation network on a frame. A disclosed method includes selecting a frame from a scene and generating a frame segmentation using the frame and a segmentation network. The method also includes displaying the frame and frame segmentation overlaid on the frame, receiving a correction input on the frame, and training the segmentation network using the correction input. The method includes overtraining the segmentation network for the scene by iterating the above steps on the same frame or a series of frames from the scene.
    Type: Grant
    Filed: May 14, 2019
    Date of Patent: May 11, 2021
    Assignee: Matterport, Inc.
    Inventor: Gary Bradski
  • Patent number: 10997448
    Abstract: Systems and methods for registering arbitrary visual features for use as fiducial elements are disclosed. An example method includes aligning a geometric reference object and a visual feature and capturing an image of the reference object and feature. The method also includes identifying, in the image of the object and the visual feature, a set of at least four non-colinear feature points in the visual feature. The method also includes deriving, from the image, a coordinate system using the geometric object. The method also comprises providing a set of measures to each of the points in the set of at least four non-colinear feature points using the coordinate system. The measures can then be saved in a memory to represent the registered visual feature and serve as the basis for using the registered visual feature as a fiducial element.
    Type: Grant
    Filed: May 15, 2019
    Date of Patent: May 4, 2021
    Assignee: Matterport, Inc.
    Inventors: Gary Bradski, Gholamreza Amayeh, Mona Fathollahi, Ethan Rublee, Grace Vesom, William Nguyen
  • Patent number: 10989816
    Abstract: Systems, computer-implemented methods, apparatus and/or computer program products are provided that facilitate improving the accuracy of global positioning system (GPS) coordinates of indoor photos. The disclosed subject matter further provides systems, computer-implemented methods, apparatus and/or computer program products that facilitate generating exterior photos of structures based on GPS coordinates of indoor photos.
    Type: Grant
    Filed: February 8, 2019
    Date of Patent: April 27, 2021
    Assignee: Matterport, Inc.
    Inventors: Gunnar Hovden, Scott Adams
  • Patent number: 10984244
    Abstract: Techniques are provided for increasing the accuracy of automated classifications produced by a machine learning engine. Specifically, the classification produced by a machine learning engine for one photo-realistic image is adjusted based on the classifications produced by the machine learning engine for other photo-realistic images that correspond to the same portion of a 3D model that has been generated based on the photo-realistic images. Techniques are also provided for using the classifications of the photo-realistic images that were used to create a 3D model to automatically classify portions of the 3D model. The classifications assigned to the various portions of the 3D model in this manner may also be used as a factor for automatically segmenting the 3D model.
    Type: Grant
    Filed: January 14, 2020
    Date of Patent: April 20, 2021
    Assignee: Matterport, Inc.
    Inventors: Gunnar Hovden, Mykhaylo Kurinnyy
  • Patent number: 10909758
    Abstract: Systems and methods for generating three-dimensional models with correlated three-dimensional and two dimensional imagery data are provided. In particular, imagery data can be captured in two dimensions and three dimensions. Imagery data can be transformed into models. Two-dimensional data and three-dimensional data can be correlated within models. Two-dimensional data can be selected for display within a three-dimensional model. Modifications can be made to the three-dimensional model and can be displayed within a three-dimensional model or within two-dimensional data. Models can transition between two dimensional imagery data and three dimensional imagery data.
    Type: Grant
    Filed: October 9, 2018
    Date of Patent: February 2, 2021
    Assignee: Matterport, Inc.
    Inventors: Matthew Tschudy Bell, David Alan Gausebeck, Gregory William Coombe, Daniel Ford, William John Brown
  • Patent number: 10909770
    Abstract: Systems and methods for building a three-dimensional composite scene are disclosed. Certain embodiments of the systems and methods may include the use of a three-dimensional capture device that captures a plurality of three-dimensional images of an environment. Some embodiments may further include elements concerning aligning and/or mapping the captured images. Various embodiments may further include elements concerning reconstructing the environment from which the images were captured. The methods disclosed herein may be performed by a program embodied on a non-transitory computer-readable storage medium when executed the program is executed a processor.
    Type: Grant
    Filed: November 1, 2013
    Date of Patent: February 2, 2021
    Assignee: Matterport, Inc.
    Inventors: Matthew Bell, David Gausebeck, Michael Beebe
  • Publication number: 20200388072
    Abstract: Systems and techniques for processing and/or transmitting three-dimensional (3D) data are presented. A partitioning component receives captured 3D data associated with a 3D model of an interior environment and partitions the captured 3D data into at least one data chunk associated with at least a first level of detail and a second level of detail. A data component stores 3D data including at least the first level of detail and the second level of detail for the at least one data chunk. An output component transmits a portion of data from the at least one data chunk that is associated with the first level of detail or the second level of detail to a remote client device based on information associated with the first level of detail and the second level of detail.
    Type: Application
    Filed: March 10, 2020
    Publication date: December 10, 2020
    Applicant: Matterport, Inc.
    Inventors: Matthew Tschudy Bell, David Alan Gausebeck, Gregory William Coombe, Daniel Ford
  • Patent number: 10848731
    Abstract: This application generally relates to capturing and aligning panoramic image and depth data. In one embodiment, a device is provided that comprises a housing and a plurality of cameras configured to capture two-dimensional images, wherein the cameras are arranged at different positions on the housing and have different azimuth orientations relative to a center point such that the cameras have a collective field-of-view spanning up to 360° horizontally. The device further comprises a plurality of depth detection components configured to capture depth data, wherein the depth detection components are arranged at different positions on the housing and have different azimuth orientations relative to the center point such that the depth detection components have the collective field-of-view spanning up to 360° horizontally.
    Type: Grant
    Filed: January 26, 2017
    Date of Patent: November 24, 2020
    Assignee: Matterport, Inc.
    Inventors: Kyle Simek, David Gausebeck, Matthew Tschudy Bell
  • Publication number: 20200364482
    Abstract: Systems and methods for registering arbitrary visual features for use as fiducial elements are disclosed. An example method includes aligning a geometric reference object and a visual feature and capturing an image of the reference object and feature. The method also includes identifying, in the image of the object and the visual feature, a set of at least four non-colinear feature points in the visual feature. The method also includes deriving, from the image, a coordinate system using the geometric object. The method also comprises providing a set of measures to each of the points in the set of at least four non-colinear feature points using the coordinate system. The measures can then be saved in a memory to represent the registered visual feature and serve as the basis for using the registered visual feature as a fiducial element.
    Type: Application
    Filed: May 15, 2019
    Publication date: November 19, 2020
    Applicant: Matterport, Inc.
    Inventors: Gary Bradski, Gholamreza Amayeh, Mona Fathollahi, Ethan Rublee, Grace Vesom, William Nguyen
  • Publication number: 20200364878
    Abstract: Systems and methods for frame and scene segmentation are disclosed herein. One method includes associating a first primary element from a first frame with a background tag, associating a second primary element from the first frame with a subject tag, generating a background texture using the first primary element, generating a foreground texture using the second primary element, and combining the background texture and the foreground texture into a synthesized frame. The method also includes training a segmentation network using the background tag, the foreground tag, and the synthesized frame.
    Type: Application
    Filed: May 14, 2019
    Publication date: November 19, 2020
    Applicant: Matterport, Inc.
    Inventors: Gary Bradski, Prasanna Krishnasamy, Mona Fathollahi, Michael Tetelman
  • Publication number: 20200364913
    Abstract: Systems and methods for user guided iterative frame segmentation are disclosed herein. A disclosed method includes providing a ground truth segmentation, synthesizing a failed segmentation from the ground truth segmentation, synthesizing a correction input for the failed segmentation using the ground truth segmentation, and conducting a supervised training routine for the segmentation network. The routine uses the failed segmentation and correction input as a segmentation network input and the ground truth segmentation as a supervisory output.
    Type: Application
    Filed: May 14, 2019
    Publication date: November 19, 2020
    Applicant: Matterport, Inc.
    Inventor: Gary Bradski
  • Publication number: 20200364895
    Abstract: A trained network for point tracking includes an input layer configured to receive an encoding of an image. The image is of a locale or object on which the network has been trained. The network also includes a set of internal weights which encode information associated with the locale or object, and a tracked point therein or thereon. The network also includes an output layer configured to provide an output based on the image as received at the input layer and the set of internal weights. The output layer includes a point tracking node that tracks the tracked point in the image. The point tracking node can track the point by generating coordinates for the tracked point in an input image of the locale or object. Methods of specifying and training the network using a three-dimensional model of the locale or object are also disclosed.
    Type: Application
    Filed: May 15, 2019
    Publication date: November 19, 2020
    Applicant: Matterport, Inc.
    Inventors: Gary Bradski, Gholamreza Amayeh, Mona Fathollahi, Ethan Rublee, Grace Vesom, William Nguyen
  • Publication number: 20200364873
    Abstract: Methods and systems regarding importance sampling for the modification of a training procedure used to train a segmentation network are disclosed herein. A disclosed method includes segmenting an image using a trainable directed graph to generate a segmentation, displaying the segmentation, receiving a first selection directed to the segmentation, and modifying a training procedure for the trainable directed graph using the first selection. In a more specific method, the training procedure alters a set of trainable values associated with the trainable directed graph based on a delta between the segmentation and a ground truth segmentation, the first selection is spatially indicative with respect to the segmentation, and the delta is calculated based on the first selection.
    Type: Application
    Filed: May 14, 2019
    Publication date: November 19, 2020
    Applicant: Matterport, Inc.
    Inventors: Gary Bradski, Ethan Rublee, Mona Fathollahi, Michael Tetelman, Ian Meeder, Varsha Vivek, William Nguyen