Automated determination of image acquisition locations in building interiors using determined room shapes
Techniques are described for computing devices to perform automated operations to determine the acquisition locations of images, such as within a building interior based on automatically determined shapes of rooms of the building, and for using the determined image acquisition location information in further automated manners. The image may be a panorama image or of another type (e.g., a rectilinear perspective image) and acquired at an acquisition location in a multi-room building's interior, and the determined acquisition location for such an image may be at least a location on the building's floor plan and optionally an orientation/direction for at least a part of the image—in addition, the automated image acquisition location determination may be further performed without having or using information from any depth sensors or other distance-measuring devices about distances from an image's acquisition location to walls or other objects in the surrounding building.
Latest Zillow, Inc. Patents:
- Automatic updating of real estate database
- Automated analysis of image contents to determine the acquisition location of the image
- Automated room shape determination using visual data of multiple captured in-room images
- Systems and methods for electronic signature
- Generating floor maps for buildings from automated analysis of visual data of the buildings' interiors
This application claims the benefit of U.S. Provisional Patent Application No. 63/117,372, filed Nov. 23, 2020 and entitled “Automated Determination Of Image Acquisition Locations In Building Interiors Using Determined Room Shapes,” which is hereby incorporated by reference in its entirety.
TECHNICAL FIELDThe following disclosure relates generally to techniques for automatically determining the acquisition locations of images in building interiors based on automatically determined room shapes, and for subsequently using the determined acquisition location information in one or more manners, such as to locate an image of an interior of a room in a building on a floor plan of the building and to use the image location to improve navigation of the building.
BACKGROUNDIn various fields and circumstances, such as architectural analysis, property inspection, real estate acquisition and development, remodeling and improvement services, general contracting and other circumstances, it may be desirable to view information about the interior of a house, office, or other building without having to physically travel to and enter the building, including to determine actual as-built information about the building rather than design information from before the building is constructed. However, it can be difficult to effectively capture, represent and use such building interior information, including to display visual information captured within building interiors to users at remote locations (e.g., to enable a user to fully understand the layout and other details of the interior, including to control the display in a user-selected manner). In addition, while a floor plan of a building may provide some information about layout and other details of a building interior, such use of floor plans has some drawbacks in certain situations, including that floor plans can be difficult to construct and maintain, to accurately scale and populate with information about room interiors, to visualize and otherwise use, etc.
The present disclosure describes techniques for using computing devices to perform automated operations related to determining the acquisition locations of images, such as within a building interior based on automatically determined shapes of rooms of the building, and for subsequently using the determined image acquisition location information in one or more further automated manners. Images to be analyzed may include panorama images or other images (e.g., rectilinear perspective images) that are acquired at acquisition locations in an interior of a multi-room building (e.g., a house, office, etc.), referred to generally herein as ‘target images’, and the determined image acquisition location information for such a target image includes at least a location on a floor plan of the building and in some situations further includes an orientation or other direction information for at least a part of the target image—in addition, in at least some such embodiments, the automated image acquisition location determination is further performed without having or using information from any depth sensors or other distance-measuring devices about distances from a target image's acquisition location to walls or other objects in the surrounding building. The determined image acquisition location information for one or more target images acquired for a building may be further used in various manners in various embodiments, such as in conjunction with a corresponding building floor plan and/or other generated mapping-related information for the building, including for controlling navigation of mobile devices (e.g., autonomous vehicles), for display or other presentation over one or more computer networks on one or more client devices in corresponding GUIs (graphical user interfaces), etc. Additional details are included below regarding the automated determination and use of image acquisition location information, and some or all of the techniques described herein may be performed via automated operations of an Image Location Determination Manager (“ILDM”) system in at least some embodiments, as discussed further below.
As noted above, automated operations of an ILDM system may include determining the acquisition location and optionally orientation of a target image that is captured in a room of a house or other building (or in another defined area), by using automatically determined information about shapes of rooms of the building (or shapes of multiple other defined areas that are candidates for the image's acquisition location)—a combination of acquisition location and orientation for a target image is referred to at times herein as an ‘acquisition position’ or merely ‘position’ of the target image. For example, a building floor plan having associated room shape information for rooms of the building may be used in at least some embodiments in the automated determination of a target image's acquisition location within the building—in at least some such situations, 2D and/or 3D room shapes for the rooms shown on the floor plan (or for the other defined areas) may have been previously automatically determined (as discussed further below), while in other situations some or all of the room shapes of the rooms (or other shapes of defined areas) may be determined in other manners (e.g., automatically determined concurrently at a time of determining a target image's acquisition location, determined previously or concurrently based at least in part on manual input by one or more users, etc.). A building floor plan with associated room shape information may have various forms in various embodiments, such as a 2D (two-dimensional) floor map of the building (e.g., an orthographic top view or other overhead view of a schematic floor map that does not include or display height information) and/or a 3D (three-dimensional) or 2.5D (two and a half-dimensional) floor map model of the building that does display height information. In at least some embodiments, the automated determination of a target image's acquisition location within a multi-room building may generally include retrieving determined room shapes for some or all of the building's rooms, and identifying one of those rooms whose determined room shape best matches a room shape for the target image that is estimated from the visual contents of the target image and is placed (e.g., sized and rotated) to match that identified room's determined room shape, including comparing aspects of the target image's visual contents to the determined room shape of that identified room to determine at least a location within that identified room at which the target image was acquired.
As part of automated operations for identifying an estimated room shape for a room enclosing a target image based at least in part on the visual contents of the target image, the described techniques may, in at least some embodiments, include using one or more trained neural networks or other techniques to estimate a 3D room shape shown in the target image. As non-exclusive examples, such 3D room shape estimation may include one or more of the following: using a trained convolutional neural network or other analysis technique to take the target image as input and to estimate a 3D point cloud of the walls and other surfaces of the enclosing room from the visual contents of the target image and/or to estimate 3D walls and other planar surfaces of the enclosing room from the visual contents of the target image; using a trained neural network or other analysis technique to take the target image as input and to estimate wireframe structural lines of the enclosing room from the visual contents of the target image (e.g., structural lines to show one or more of borders between walls, borders between walls and ceiling, borders between walls and floor, outlines of doorways and/or other inter-room wall openings, outlines of windows, etc.); using a trained neural network or other analysis technique to detect wall structural elements (e.g., windows and/or sky-lights; passages into and/or out of the room, such as doorways and other openings in walls, stairs, hallways, etc.; borders between adjacent walls; borders between walls and a floor; borders between walls and a ceiling; corners (or solid geometry vertices) where at least three surfaces or planes meet; etc.) in the visual contents of the target image and to optionally detect other fixed structural elements (e.g., countertops, bath tubs, sinks, islands, fireplaces, etc.) and to generate 3D bounding boxes for the detected elements and to optionally further determine object type tags (e.g., window, doorway, etc.) for those elements; using a trained neural network or other analysis technique to determine a room type tag for the enclosing room (e.g., living room, bedroom, bathroom, kitchen, etc.) from the visual contents of the target image; using a trained deep neural network or other analysis technique to generate an embedding vector that encodes information about elements and other features of the enclosing room, such as element types and locations, room type and shape and size, etc. (e.g., in a concise format that is not easily discernible to a human reader); etc. In some embodiments, acquisition metadata for the target image may be further used as part of determining an estimated room shape for a room enclosing a target image, such as by using data from IMU (internal measurement unit) sensors on the acquiring camera or other associated device as part of performing a SLAM (Simultaneous Localization And Mapping) and/or SfM (Structure from Motion) and/or MVS (multiple-view stereovision) analysis, as discussed further below. Additional details are included below regarding automated operations that may be performed by the ILDM system in at least some embodiments for identifying an estimated room shape for a room enclosing a target image based at least in part on the visual content of the target image.
Once a target image's estimated room shape for an enclosing room is automatically identified, it may be compared to other candidate room shapes (e.g., the previously determined room shapes of some or all rooms for a building) in order to automatically determine a room shape from the candidates that best matches the target image's estimated room shape, with the room having such a matching determined room shape referred to herein as a ‘target’ room within the building in which the target image's acquisition location occurs. Determination of one or more potentially matching candidate room shapes, including a particular placement (i.e., location and rotation) of the target image's estimated room shape relative to that of the candidate room shape, may include considering and evaluating multiple alternative shape matches each having different such placements of the target image's estimated room shape relative to that of one or more candidate room shapes. For example, in at least some embodiments, one or more corners of the target image's estimated room shape may be successively or simultaneously matched to one or more corners of a candidate room shape (e.g., to separately match each corner of the target image's estimated room shape to each corner of each candidate room shape, to separately match each pair of two corners of the target image's estimated room shape to each pair of two other corners of each candidate room shape, etc.), along with using vanishing line angle information for the target image and a candidate room shape (or an existing image with a specified position within that candidate room shape) to determine wall alignment options for the two room shapes with a matching corner—thus, if all alternatives are considered for the target image's estimated room shape and a particular candidate room shape, and there are M predicted room corners in the target image's estimated room shape and N room corners in the candidate room shape and 2 vanishing line angles for each of the target image and the candidate room shape (or for an existing image with a specified position within that candidate room shape), there will be M*N*4 shape matching alternatives for the target image's estimated room shape and that candidate room shape, each providing a possible acquisition location and optionally orientation for the target image according to that shape match alternative, and with such various shape matching alternatives further being considered for each candidate room shape.
In addition, in at least some embodiments, additional automated operations may be performed to determine one or more preferred candidate room shapes to compare to the estimated room shape for the target image (e.g., by ranking some or all of the candidate room shapes, and in some embodiments to compare the preferred candidate(s) to the target image's estimated room shape before or instead of other non-preferred candidates), such as by using one or more types of information to identified the preferred candidate(s) as follows: by matching determined room type tags for the target image's estimated room shape to room type tags for the candidate room shapes; by using information about one or more floor plan rooms associated with one or more other images acquired in the building successively before and/or after the target image in order to identify the same or adjacent rooms in the floor plan as possible acquisition locations of the target image; by comparing the visual contents of the target image pairwise to the visual contents of existing images with specified positions within the room shapes of the floor plan (e.g., by a trained neural network) in order to determine matching features in the pair of images that enable the target image's position to be determined relative to the specified position of a paired existing image (e.g., with a degree of match that exceeds a first defined matching threshold); by comparing the visual contents of the entire target image pairwise to the visual contents of entire existing images with specified positions within the room shapes of the floor plan (e.g., by a trained neural network) in order to determine similarity from co-visible areas in the pair of images that enable the target image's position to be determined relative to the specified position of a paired existing image (e.g., with a degree of similarity that exceeds a second defined matching threshold); by using movement tracking information for the target image's camera (e.g., using SLAM) to determine spatial relation information for the target image, such as approximate 3D shape information for a surrounding environment, approximate 3D planar surfaces in the surrounding environment, camera travel path information in the enclosing room, etc.; by using movement information for the target image's camera (e.g., from one or more IMU sensors of the camera or an associated device) to determine, with respect to a comparison of the visual contents of the target image pairwise with the visual contents of existing images with specified positions within the room shapes of the floor plan (e.g., by a trained neural network), multiple directions between the pair of images to assist in determining the target image's position relative to the specified position of a paired existing image (e.g., with a degree of similarity in the multiple directions that exceeds a third defined matching threshold); etc.
In order to select between shape matching alternatives for the target image's estimated room shape and at least one candidate room shape, one or more matching scores can be considered and combined (if more than one is considered) for each shape matching alternative, such as via weighting or a trained machine learning model for the combining, with non-exclusive examples of such matching scores including the following: scoring the distance between the locations of each corner for the target image's estimated room shape and for the candidate room shape for that shape matching alternative, using a particular placement of the target image's estimated room shape for that shape matching alternative and after re-projecting the candidate room shape into the target image space, and with smaller distances identifying better matches; scoring the distance between wireframe structural lines for the target image's estimated room shape and for the candidate room shape for that shape matching alternative, using a particular placement of the target image's estimated room shape for that shape matching alternative and after re-projecting the candidate room shape's wireframe structural lines into the target image space, and with smaller distances identifying better matches; scoring the distance between the locations of each object 3D bounding box for the target image's estimated room shape and for the candidate room shape for that shape matching alternative, using a particular placement of the target image's estimated room shape for that shape matching alternative and after re-projecting the object 3D bounding boxes for the candidate room shape into the target image space, such as by using an intersection-over-union distance measurement and with smaller distances identifying better matches; scoring the distance between the locations of structural walls for the target image's estimated room shape and for the candidate room shape for that shape matching alternative, using a particular placement of the target image's estimated room shape for that shape matching alternative and after re-projecting the structural walls for the candidate room shape into the target image space, and with smaller distances identifying better matches; scoring the difference between two determinations of directions/angles between the acquisition position of the target image for that shape matching alternative and the specified position of an existing image in the candidate room shape, such as by using the determined location and orientation of the acquisition location for that shape matching alternative relative to the existing image's specified position as a first direction/angle, and using a visual contents comparison of the target image and existing image (e.g., using an SfM analysis; using results of a trained convolutional neural network; etc.) to provide a second direction/angle, and with smaller differences identifying better matches; scoring the difference between the overall visual contents of the target image and of another image in the room with the candidate room shape for that shape matching alternative (e.g., an additional image acquired in that room at a same time as the target image, such as within a defined time period of minutes and/or hours; an existing image previously acquired in that room and with a specified position in that room; etc., and as determined by a trained convolutional neural network or in another manner), and with smaller differences identifying better matches; scoring the difference between a first generated image embedding vector for the target image and a second generated image embedding vector for another image in the room with the candidate room shape for that shape matching alternative (e.g., an additional image acquired in that room at a same time as the target image, such as within a defined time period of minutes and/or hours; an existing image previously acquired in that room and with a specified position in that room; etc.), and with smaller differences identifying better matches; etc.
In addition, after determining an initial acquisition position of the target image for each shape matching alternative in the manner noted above, at least some embodiments include performing one or more additional automated operations to update and refine the initial acquisition position for one or more shape matching alternatives (e.g., all shape matching alternatives; one or more best matches according to the combination of the matching scores, or based on a single matching score if only one is used; etc.), such as one or more of the following non-exclusive list: using the scored distances between the locations of each corner for the target image's estimated room shape and for the candidate room shape for that shape matching alternative using a particular placement of the target image's estimated room shape for that shape matching alternative, such as by identifying corner pairs of the two room shapes that are within a first defined distance threshold (referred to herein as ‘corner inliers’) and using a weighted least squares regression to refine the acquisition position of the target image (e.g., using confidence scores generated for each corner inlier, such as by a trained neural network, as weights for the regression); using the scored distances between the locations of wireframe structural lines for the target image's estimated room shape and for the candidate room shape for that shape matching alternative using a particular placement of the target image's estimated room shape for that shape matching alternative, such as by identifying horizontal structural line pairs of the two room shapes that are within a second defined distance threshold (referred to herein as ‘line inliers’) and using a weighted least squares regression to refine the acquisition position of the target image (e.g., using confidence scores generated for each line inlier, such as by a trained neural network, as weights for the regression); performing a differentiable rendering optimization using image normal/orthogonal direction predictions, by rendering pixel-level surface normal information for the target image's estimated room shape using the initial acquisition position for that shape matching alternative (e.g., by using a trained neural network), and comparing those rendered pixel-level surface normal values to other pixel-level surface normal values estimated for an existing image with a specified position in the candidate room shape to determine a difference-based cost value, such as by iterating until the cost value reaches a local minimum; etc.
Furthermore, in at least some embodiments, the determined initial and/or updated acquisition position of a target image may be further refined in conjunction with other determined acquisition positions of other target images acquired in the building, such as to reach a global optimization for all of those target images (including to optionally use additional information about relative directions between and/or positions of some or all of those target images, such as may be reflected by generated inter-target image links, as discussed in greater detail elsewhere herein). Additional details are included below regarding automated operations that may be performed by the ILDM system in at least some embodiments for comparing a target image's estimated room shape for an enclosing room to other candidate room shapes in order to determine one or more matching target rooms of the building (e.g., a single best-match target room), including to determine the acquisition position of the target image within each of the matching target rooms.
In addition, in some embodiments, an image's estimated room shape may be matched to other room shapes without using a floor plan. For example, in at least some embodiments, the candidate room shapes are generated using estimated room shapes for other images acquired in one or more buildings, such as for other additional images that are concurrently captured (e.g., within a defined time period, such as one or more minutes or hours) in the same building as the target image. In such embodiments, the target image's position may be determined relative to that of one or more other additional images, including in some embodiments to do so dynamically while the target image and additional images are being captured, including to automatically generate a partial or full floor plan for the building using the estimated room shapes for the target image and other additional images.
Furthermore, in some embodiments for an image captured in a defined area other than a room, the image's estimated shape of that defined area may be matched to other defined areas' shapes in various manners. For example, in at least some embodiments, a target image (and optionally additional images) may be acquired outside of one or more buildings, such as in one of multiple separate areas of one or more properties (e.g., for a house, a garden, patio, deck, back yard, side yard, front yard, pool, carport, dock, etc.) that each has a previously or concurrently determined area shape (e.g., a 3D shape, a 2D shape, etc.)—if so, the acquisition position of the target image (and optionally of the additional images) may similarly be automatically determined using such other defined areas' shapes as the candidate ‘room’ shapes.
The automated determination by the ILDM system of the acquisition location of a target image taken in a room may further include additional operations in some embodiments, and corresponding additional details are included below, including with respect to the examples of
The described techniques provide various benefits in various embodiments, including to allow floor plans of multi-room buildings and other structures to be automatically augmented with information about acquisition locations at which images are acquired in the buildings or other structures, including without having or using information from depth sensors or other distance-measuring devices about distances from images' acquisition locations to walls or other objects in a surrounding building or other structure. Furthermore, such automated techniques allow such image acquisition location information to be determined more quickly than previously existing techniques, and in at least some embodiments with greater accuracy, including by using information acquired from the actual building environment (rather than from plans on how the building should theoretically be constructed), as well as enabling the capture of changes to structural elements that occur after a building is initially constructed. Such described techniques further provide benefits in allowing improved automated navigation of a building by mobile devices (e.g., semi-autonomous or fully-autonomous vehicles), based at least in part on the determined acquisition locations of images, including to significantly reduce computing power and time used to attempt to otherwise learn a building's layout. In addition, in some embodiments the described techniques may be used to provide an improved GUI in which a user may more accurately and quickly obtain information about a building's interior (e.g., for use in navigating that interior), including in response to search requests, as part of providing personalized information to the user, as part of providing value estimates and/or other information about a building to a user, etc. Various other benefits are also provided by the described techniques, some of which are further described elsewhere herein.
As noted above, in at least some embodiments and situations, some or all of the images acquired for a building may be panorama images that are each acquired at one of multiple acquisition locations in or around the building, such as to generate a panorama image at each such acquisition location from one or more of a video captured at that acquisition location (e.g., a 360° video taken from a smartphone or other mobile device held by a user turning at that acquisition location), or multiple images captured in multiple directions from the acquisition location (e.g., from a smartphone or other mobile device held by a user turning at that acquisition location; from automated rotation of a device at that acquisition location, such as on a tripod at that acquisition location; etc.), or a simultaneous capture of all the image information for a particular acquisition location (e.g., using one or more fisheye lenses), etc. It will be appreciated that such a panorama image may in some situations be represented in a spherical coordinate system and provide up to 360° coverage around horizontal and/or vertical axes (e.g., 360° of coverage along a horizontal plane and around a vertical axis), while in other embodiments the acquired panorama images or other images may include less than 360° of vertical coverage (e.g., for images with a width exceeding a height by more than a typical aspect ratio, such as at or exceeding 21:9 or 16:9 or 3:2 or 7:5 or 4:3 or 5:4 or 1:1, including for so-called ‘ultrawide’ lenses and resulting ultrawide images). In addition, it will be appreciated that a user viewing such a panorama image (or other image with sufficient horizontal and/or vertical coverage that only a portion of the image is displayed at any given time) may be permitted to move the viewing direction within the panorama image to different orientations to cause different subset images (or “views”) to be rendered within the panorama image, and that such a panorama image may in some situations be represented in a spherical coordinate system (including, if the panorama image is represented in a spherical coordinate system and a particular view is being rendered, to convert the image being rendered into a planar coordinate system, such as for a perspective image view before it is displayed). Furthermore, acquisition metadata regarding the capture of such panorama images may be obtained and used in various manners, such as data acquired from IMU (inertial measurement unit) sensors or other sensors of a mobile device as it is carried by a user or otherwise moved between acquisition locations—non-exclusive examples of such acquisition metadata may include one or more of acquisition time; acquisition location, such as GPS coordinates or other indication of location; acquisition direction and/or orientation; relative or absolute order of acquisition for multiple images acquired for a building or that are otherwise associated; etc., and such acquisition metadata may further optionally be used as part of determining the images' acquisition locations in at least some embodiments and situations, as discussed further below. Additional details are included below regarding automated operations of device(s) implementing an Image Capture and Analysis (ICA) system involved in acquiring images and optionally acquisition metadata, including with respect to
As is also noted above, shapes of rooms of a building may be automatically determined in various manners in various embodiments, including at a time before automated determination of a particular image's acquisition location within the building. For example, in at least some embodiments, a Mapping Information Generation Manager (MIGM) system may analyze various images acquired in and around a building in order to automatically determine room shapes of the building's rooms (e.g., 3D room shapes, 2D room shapes, etc.) and to automatically generate a floor plan for the building. As one example, if multiple images are acquired within a particular room, those images may be analyzed to determine a 3D shape of the room in the building (e.g., to reflect the geometry of the surrounding structural elements of the building)—the analysis may include, for example, automated operations to ‘register’ the camera positions for the images in a common frame of refence so as to ‘align’ the images and to estimate 3D locations and shapes of objects in the room, such as by determining features visible in the content of such images (e.g., to determine the direction and/or orientation of the acquisition device when it took particular images, a path through the room traveled by the acquisition device, etc., such as by using SLAM techniques for multiple video frame images and/or other SfM techniques for a ‘dense’ set of images that are separated by at most a defined distance (such as 6 feet) to generate a 3D point cloud for the room including 3D points along walls of the room and at least some of the ceiling and floor of the room and optionally with 3D points corresponding to other objects in the room, etc.) and/or by determining and aggregating information about planes for detected features and normal (orthogonal) directions to those planes to identify planar surfaces for likely locations of walls and other surfaces of the room and to connect the various likely wall locations (e.g., using one or more constraints, such as having 90° angles between walls and/or between walls and the floor, as part of the so-called ‘Manhattan world assumption’) and form an estimated room shape for the room. After determining the estimated room shapes of the rooms in the building, the automated operations may, in at least some embodiments, further include positioning the multiple room shapes together to form a floor plan and/or other related mapping information for the building, such as by connecting the various room shapes. Additional details are included below regarding automated operations of device(s) implementing an MIGM system involved in determining room shapes and combining room shapes to generate a floor plan, including with respect to
For illustrative purposes, some embodiments are described below in which specific types of information are acquired, used and/or presented in specific ways for specific types of structures and by using specific types of devices—however, it will be understood that the described techniques may be used in other manners in other embodiments, and that the invention is thus not limited to the exemplary details provided. As one non-exclusive example, while floor plans may be generated for houses that do not include detailed measurements for particular rooms or for the overall houses, it will be appreciated that other types of floor plans or other mapping information may be similarly generated in other embodiments, including for buildings (or other structures or layouts) separate from houses. As another non-exclusive example, while floor plans for houses or other buildings may be used for display to assist viewers in navigating the buildings, generated mapping information may be used in other manners in other embodiments. In addition, the term “building” refers herein to any partially or fully enclosed structure, typically but not necessarily encompassing one or more rooms that visually or otherwise divide the interior space of the structure—non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), etc. The term “acquire” or “capture” as used herein with reference to a building interior, acquisition location, or other location (unless context clearly indicates otherwise) may refer to any recording, storage, or logging of media, sensor data, and/or other information related to spatial and/or visual characteristics and/or otherwise perceivable characteristics of the building interior or subsets thereof, such as by a recording device or by another device that receives information from the recording device. As used herein, the term “panorama image” may refer to a visual representation that is based on, includes or is separable into multiple discrete component images originating from a substantially similar physical location in different directions and that depicts a larger field of view than any of the discrete component images depict individually, including images with a sufficiently wide-angle view from a physical location to include angles beyond that perceivable from a person's gaze in a single direction (e.g., greater than 120° or 150° or 180°, etc.). The term “sequence” of acquisition locations, as used herein, refers generally to two or more acquisition locations that are each visited at least once in a corresponding order, whether or not other non-acquisition locations are visited between them, and whether or not the visits to the acquisition locations occur during a single continuous period of time or at multiple different times, or by a single user and/or device or by multiple different users and/or devices. In addition, various details are provided in the drawings and text for exemplary purposes, but are not intended to limit the scope of the invention. For example, sizes and relative positions of elements in the drawings are not necessarily drawn to scale, with some details omitted and/or provided with greater prominence (e.g., via size and positioning) to enhance legibility and/or clarity. Furthermore, identical reference numbers may be used in the drawings to identify similar elements or acts.
One or more users (not shown) of one or more client computing devices 175 may further interact over one or more computer networks 170 with the ILDM system 140 and optionally the ICA system and/or MIGM system, such as to assist in determining acquisition locations of one or more target images and obtaining corresponding determined acquisition location information, and/or to obtain and optionally interact with a generated floor plan on which one or more additional images have been located, and/or to obtain and optionally interact with additional information such as one or more associated existing images (e.g., to change between a floor plan view and a view of a particular image at an acquisition location within or near the floor plan; to change the horizontal and/or vertical viewing direction from which a corresponding view of a panorama image is displayed, such as to determine a portion of a panorama image to which a current user viewing direction is directed, etc.). In addition, while not illustrated in
In the depicted computing environment of
In the example of
In operation, a user associated with the mobile device arrives at a first acquisition location 210A within a first room of the building interior (in this example, an entryway from an external door 190-1 to the living room), and captures a view of a portion of the building interior that is visible from that acquisition location 210A (e.g., some or all of the first room, and optionally small portions of one or more other adjacent or nearby rooms, such as through doorways, halls, stairways or other connecting passages from the first room) as the mobile device is rotated around a vertical axis at the first acquisition location (e.g., with the user turning his or her body in a circle while holding the mobile device stationary relative to the user's body). The actions of the user and/or the mobile device may be controlled or facilitated via use of one or more programs executing on the mobile device, such as ICA application system 155 and/or optional browser 162, control system 147 to manage I/O (input/output) and/or communications and/or networking for the device 185 (e.g., to receive instructions from and present information to the user), etc., and the view capture may be performed by recording a video at location 210A and/or taking a succession of one or more images at location 210A, including to capture visual information depicting a number of objects or other elements (e.g., structural details) that may be visible in images (e.g., video frames) captured from the acquisition location. In the example of
After the first acquisition location 210A has been adequately captured (e.g., by a full rotation of the mobile device), the user and/or device 185 may proceed to a next acquisition location (such as acquisition location 210B), optionally recording movement data during movement between the acquisition locations, such as video and/or other data from the hardware components (e.g., from one or more IMUs, from the camera, etc.). At the next acquisition location, the mobile device may similarly capture one or more images from that acquisition location. This process may repeat from some or all rooms of the building and optionally external to the building, as illustrated for acquisition locations 210C-210M. The acquired video and/or other images for each acquisition location are further analyzed to generate a panorama image for each of acquisition locations 210A-210M, including in some embodiments to stitch together multiple constituent images to create a panorama image and/or to match objects and other elements in different images. In addition to generating such panorama images, further analysis may be performed in at least some embodiments in order to ‘link’ at least some of the panoramas and their acquisition locations together (with some corresponding lines 215 between example acquisition locations 210A-210C being shown for the sake of illustration), such as to determine relative positional information between pairs of acquisition locations that are visible to each other, to store corresponding inter-panorama links (e.g., links 215-AB, 215-BC and 215-AC between acquisition locations 210A and 210B, 210B and 210C, and 210A and 210C, respectively), and in some embodiments and situations to further link at least some acquisition locations that are not visible to each other (e.g., a link 215-BE, not shown, between acquisition locations 210B and 210E).
Additional details related to embodiments of generating and using linking information between panorama images, including using travel path information and/or elements or other features visible in multiple images, are included in U.S. Non-Provisional patent application Ser. No. 17/064,601, filed Oct. 7, 2020 and entitled “Connecting And Using Building Data Acquired From Mobile Devices” (which includes disclosure of an example BICA system that is generally directed to obtaining and using linking information to inter-connect multiple panorama images captured within one or more buildings or other structures); in U.S. Non-Provisional patent application Ser. No. 17/080,604, filed Oct. 26, 2020 and entitled “Generating Floor Maps For Buildings From Automated Analysis Of Visual Data Of The Buildings' Interiors”; and in U.S. Provisional Patent Application No. 63/035,619, filed Jun. 5, 2020 and entitled “Automated Generation On Mobile Devices Of Panorama Images For Buildings Locations And Subsequent Use”; each of which is incorporated herein by reference in its entirety.
Various details are provided with respect to
In particular,
In particular,
While not illustrated in
Additional details related to embodiments of a system providing at least some such functionality of an MIGM system or related system for generating floor plans and associated information and/or presenting floor plans and associated information are included in U.S. Non-Provisional patent application Ser. No. 16/190,162, filed Nov. 14, 2018 and entitled “Automated Mapping Information Generation From Inter-Connected Images” (which includes disclosure of an example Floor Map Generation Manager, or FMGM, system that is generally directed to automated operations for generating and displaying a floor map or other floor plan of a building using images acquired in and around the building); in U.S. Non-Provisional patent application Ser. No. 16/681,787, filed Nov. 12, 2019 and entitled “Presenting Integrated Building Information Using Three-Dimensional Building Models” (which includes disclosure of an example FMGM system that is generally directed to automated operations for displaying a floor map or other floor plan of a building and associated information); in U.S. Non-Provisional patent application Ser. No. 16/841,581, filed Apr. 6, 2020 and entitled “Providing Simulated Lighting Information For Three-Dimensional Building Models” (which includes disclosure of an example FMGM system that is generally directed to automated operations for displaying a floor map or other floor plan of a building and associated information); in U.S. Non-Provisional patent application Ser. No. 17/080,604, filed Oct. 26, 2020 and entitled “Generating Floor Maps For Buildings From Automated Analysis Of Visual Data Of The Buildings' Interiors” (which includes disclosure of an example Video-To-Floor Map, or VTFM, system that is generally directed to automated operations for generating a floor map or other floor plan of a building using video data acquired in and around the building); in U.S. Provisional Patent Application No. 63/035,619, filed Jun. 5, 2020 and entitled “Automated Generation On Mobile Devices Of Panorama Images For Buildings Locations And Subsequent Use”; in U.S. Non-Provisional patent application Ser. No. 17/069,800, filed Oct. 13, 2020 and entitled “Automated Tools For Generating Building Mapping Information”; and in U.S. Non-Provisional patent application Ser. No. 16/807,135, filed Mar. 2, 2020 and entitled “Automated Tools For Generating Mapping Information For Buildings” (which includes disclosure of an example MIGM system that is generally directed to automated operations for generating a floor map or other floor plan of a building using images acquired in and around the building); each of which is incorporated herein by reference in its entirety. In addition, further details related to embodiments of a system providing at least some such functionality of a system for using acquired images and/or generated floor plans are included in U.S. Non-Provisional patent application Ser. No. 17/185,793, filed Feb. 25, 2021 and entitled “Automated Usability Assessment Of Buildings Using Visual Data Of Captured In-Room Images” (which includes disclosure of an example Building Usability Assessment Manager, or BUAM, system that is generally directed to automated operations for analyzing visual data from images captured in rooms of a building to assess room layout and other usability information for the building's rooms and optionally for the overall building, and to subsequently using the assessed usability information in one or more further automated manners); which is incorporated herein by reference in its entirety.
As illustrated in
In particular,
Various details have been provided with respect to
As a non-exclusive example embodiment, the automated operations of the ILDM system may include the following actions. Begin with one or more target images with RGB visual data (but no separate depth data), optionally with further acquisition metadata for one or more of the target images that may include image capture time stamps, camera IMU signals, camera SLAM-based tracking data, and/or image room tags (e.g., supplied by a user who captured a target image for its enclosing room). In addition, begin with a floor plan in 2D or 3D using a vector format and optionally having existing images at specified positions and in different image projections from those of any of the target images, and/or with additional images in different image projections from those of any of the target images with optional further acquisition metadata of the same types.
The automated operations may include doing pre-processing on the target image(s) and optionally the additional image(s) and optionally the existing images to solve camera intrinsic and extrinsic if needed, such as to detect image vanishing lines and vanishing point, extract (if perspective image) camera focal length and field of view angle, solve camera roll pitch yaw angles relative to vanishing lines presented in the images, and re-project image into spherical space (with new camera pose leveled relative to the floor plane).
The automated operations may further include generating geometry predictions for each target image and optionally each additional image and optionally each existing image, including the following: estimating room shape geometry of the indoor structure in which the camera is located (e.g., using a convolutional-neural-network-based room shape estimator, such as HorizonNet and DuLaNet, to approximate room shape geometry to 3D shapes with uniform room height, with the camera can be found in the origin of this shape); optionally using an image structural wireframe estimator (e.g., LCNN) to predict image structural lines and projecting these lines in image 3D space as room corner candidates; using an object detection algorithm on the image to generate 2D object bounding boxes with labels and object image descriptor and then ray casting 2D image bounding boxes onto previously estimated 3D room shapes and generating footprints of 3D objects to represent their spatial information of objects, as well as using 3D bounding box generation algorithms; optionally generating image embedding vectors (e.g., using deep neural networks models) for later use in comparing image content similarities and image overlaps; and optionally tagging the image with one or more room types (e.g., bedroom, kitchen, etc.).
The automated operations may further include generating image-to-image relations between each target image to one or more additional images and/or existing image, including the following: optionally using a feature-based image matching algorithm between the pair of images, such as SfM to solve image angular connections or pairwise image location information (e.g., which direction in image A is connecting to which direction in image B); and optionally using a deep learning-based image co-visibility algorithm between the pair of images to determine image content similarity (e.g., for later use with an assumption that images sharing high co-visibility scores have a high chance to be close to each other spatially).
The automated operations may further include retrieving a set of room shapes candidates on which to attempt to localize each target image in order to determine a precise acquisition location of the target image—the room shape candidates may be obtained from existing room shapes associated with a floor plan and/or room shapes estimated for a set of spatially-related additional images. Various heuristics may be used to generate binary relations between a pair of a target image and an additional image or between a target image and an area in existing floor plan, including the following: use similarity/overlaps between room type tags for the target image and paired image/area (if available, such as by created by automated image classification algorithm and/or photographer and/or subsequent annotator) to aggregate a list of preferred candidate room shapes; use the temporal relation between images (if image capture time stamp metadata is available) to retrieve a set of temporally-related additional images; use a feature-matching-based image alignment algorithm to generate pairwise or groupwise image co-relations (e.g., image relative angle or binary image co-relation); use a neural-network-based image comparison algorithm to generate pairwise image to image co-relation; to use IMU metadata collected during the image capture process (if available) to give image angular connections; and use SLAM-based camera tracking algorithm (if SLAM data available) to produce image spatial relation.
The automated operations may further include performing geometry matching for each target image to one or more candidate room shapes, to match the target image's estimated room shape to a corresponding determined room shape for a room on a floor plan or to a corresponding estimated room shape for an additional image, and localize a target image to a single room shape (e.g., to produce one or more camera pose acquisition positions for the target image, optionally along with a confidence score for each camera pose). The automated operations generally include the following: proposing a number of shape matching options (which is based on target image camera pose in the candidate room shape space); compute a score for each of the proposed camera poses (proposed shape matching position); select the camera pose with the highest score or use threshold to pick multiple camera poses; and refine the one or more selected camera poses.
The proposing of the various shape matching options may include assuming that 2 room shapes have the same scale if they are captured by the same camera at the same height (such as for one or more target images and one or more additional images that are concurrently captured during the same period of time). Corners of the room shapes are used to generate a collection of corner snapping options (alternative shape matches) between the target image's existing room shape and candidate room shape, with different shape orientations. The shape orientations are generated by snapping the horizontal vanishing angle of target image to the vanishing angle of paired additional or existing image or candidate room shape. So, if there are M predicted room corners in target image, N room corners in candidate room shape, and 4 vanishing directions from the target image and the paired additional or existing image, M*N*4 camera poses are proposed for the target image. When 2 images are captured with inconsistent camera height, a camera pose can be proposed by selecting 2 control corners from each shape, and using that to generate proposed scale and xyz, with the vanishing angle alignment used to correct the proposed camera angle.
The computing of a score for each of the proposed camera poses (proposed shape matching position) may include combining multiple individual scores given the proposed camera pose (e.g., taking the weighted sum of each individual score, extracting a descriptor from each of these terms and use machine learning model to generate the final score, etc.). Individual scores may include one or more of the following: a corner re-projection score, in which the candidate room shape is re-projected into the target image space, the projected room corners from candidate room shape are compared with room corners from original target image existing room shape, and each target room corner is matched with its nearest candidate room shape corner, using the distance of each matching corner pair and the number of matches to generate the corner re-projection score (e.g., with the closer the match, the higher the score); a wireframe structural line re-projection score, in which the candidate room shape's structural lines are re-projected into the target image space, the projected structural lines from the candidate room shape are compared with the structural lines from the target image estimated room shape, and each target image structural line is matched with its nearest candidate room shape structural line, using the distance of each matching structural line pair and the number of matches to generate the wireframe structural line re-projection score (e.g., with the closer the match, the higher the score); a structural wall element object re-projection score, in which the candidate room shape's 3D object bounding boxes from the candidate room shape are re-projected into the target image estimated room space, the projected object bounding boxes from the candidate room shape are compared with the object bounding boxes from the target image estimated room shape, and each target image object bounding box is matched with its nearest candidate room shape object bounding box, using the distance of each matching object bounding box pair based on an intersection-over-union and the consistency of object type tags; an image angular score, in which the departure/landing angle starting from target image to additional/existing image is generated, in which a separate departure/landing angle is also generated for each pair of images using a different technique (e.g., SfM, convolutional neural network, etc.), and in which the score is computed by comparing these 2 sets of angles (e.g., with the bigger the discrepancy, the more penalty in this score); an image content matching score, in which the image content similarity for a given image pair is generated (e.g., using a convolutional neural network); and a shape-based boundary intersection score, in which structural walls of the candidate room shape are re-projected in the 3D space of target image, and the mismatch between the structural walls of the projected room shape and of the target image estimated room shape are used to evaluate the proposed camera pose.
The refining of the one or more selected camera poses may include using an initial camera pose for the target image from the previous operations (e.g., using corner point matching), and refining the camera pose using one or a combination of multiple steps. The steps may include one or more of the following: performing an alignment using corner inliers, in which a distance threshold is used to filter all the matching pairs from the previous corner matching operations within a certain re-projection image distance (with the resulting corner pairs called corner inliers), and weighted least squares is used to find the best camera position xyz, with confidence scores from the predicted corners of the target image's estimated room shape (e.g., as generated by a neural network model) used as weights in the weighted least square regression to generate a more accurate camera position than the previous camera pose; performing an alignment using line matching of wireframe structural line predictions for the target image and for the candidate room shape (e.g., between horizontal lines on the floor), such as with a distance threshold used to filter all the matching lines from the previous line matching operations within a certain re-projection image distance (with the resulting line pairs called line inliers), and weighted least squares used to find the best camera position xyz, with confidence scores from the predicted structural lines of the target image's estimated room shape (e.g., as generated by a neural network model) used as weights in the weighted least square regression to generate a more accurate camera position than the previous camera pose; and performing a differentiable rendering optimization method using image normal predictions, where camera pose is optimized for a lower cost function value, by rendering the pixel-level surface normal information for the candidate room shape in the target image space starting from an initial camera pose guess, comparing the rendered surface normal with surface normal estimated from the target image in its image space (e.g., using a neural-network-based method like Taskonomy), and computing a cost value, to optimize camera pose by iteration until the cost value reaches a local minimum.
Various details have been provided with respect to this example non-exclusive embodiment, but it will be appreciated that the provided details are included for illustrative purposes, and other embodiments may be performed in other manners without some or all such details.
The server computing system(s) 300 and executing ILDM system 340 may communicate with other computing systems and devices via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), such as user client computing devices 390 (e.g., used to view floor plans, associated images and/or other related information), ICA and MIGM server computing system(s) 380, one or more mobile image acquisition devices 360, optionally other navigable devices 395 that receive and use floor plans and determined image acquisition locations and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices), and optionally other computing systems that are not shown (e.g., used to store and provide additional information related to buildings; used to capture building interior data; used to store and provide information to client computing devices, such as additional supplemental information associated with images and their encompassing buildings or other surrounding environment; etc.).
In the illustrated embodiment, an embodiment of the ILDM system 340 executes in memory 330 in order to perform at least some of the described techniques, such as by using the processor(s) 305 to execute software instructions of the system 340 in a manner that configures the processor(s) 305 and computing system(s) 300 to perform automated operations that implement those described techniques. The illustrated embodiment of the ILDM system may include one or more components, not shown, to each perform portions of the functionality of the ILDM system, and the memory may further optionally execute one or more other programs 335—as one specific example, copies of the ICA and/or MIGM systems may execute as one of the other programs 335 in at least some embodiments, such as instead of or in addition to the ICA system 387 and MIGM system 388 on the server computing system(s) 380. The ILDM system 340 may further, during its operation, store and/or retrieve various types of data on storage 320 (e.g., in one or more databases or other data structures), such as various types of floor plan information and other building mapping information 326 (e.g., generated and saved 2D floor plans and positions of wall elements and other elements on those floor plans, generated and saved 2.5D and/or 3D model floor plans that include height information, building and room dimensions for use with associated floor plans, existing images with specified positions, annotation information, etc.), information 321 about target images whose acquisition locations are to be determined and associated information 325 about such determined acquisition locations, information 323 about estimated 3D room shapes and wall element information for target images (e.g., as generated by the ILDM system during its automated operations), user information 328 about users of client computing devices 390 and/or operator users of mobile devices 360 who interact with the ILDM system, and optionally various other types of additional information 329. The ICA system 387 and/or MIGM system 388 may similarly store and/or retrieve various types of data on storage 385 (e.g., in one or more databases or other data structures) during their operation and provide some or all such information to the ILDM system 340 for its use (whether in a push and/or pull manner), such as images 393 (e.g., acquired 360° panorama images) and inter-image directional link information 396 that is generated by the ICA system and used by the MIGM system to generate floor plans, resulting floor plan information and optionally other building mapping information 391 (e.g., similar to or the same as information 326) that is generated by the MIGM system, additional information that is generated by the MIGM system as part of generating the floor plans such as determined room shapes 392 and image location information 394, and optionally various types of additional information 397 (e.g., various analytical information related to presentation or other use of one or more building interiors or other environments captured by an ICA system).
Some or all of the user client computing devices 390 (e.g., mobile devices), mobile image acquisition devices 360, other navigable devices 395 and other computing systems may similarly include some or all of the same types of components illustrated for server computing systems 300 and 380. As one non-limiting example, the mobile image acquisition devices 360 are each shown to include one or more hardware CPU(s) 361, I/O components 362, storage 365, imaging system 364, IMU hardware sensors 369, and memory 367, with one or both of a browser and one or more client applications 368 (e.g., an application specific to the ILDM system and/or ICA system) executing within memory 367, such as to participate in communication with the ILDM system 340, ICA system 387 and/or other computing systems. While particular components are not illustrated for the other navigable devices 395 or client computing systems 390, it will be appreciated that they may include similar and/or additional components.
It will also be appreciated that computing systems 300 and 380 and the other systems and devices included within
It will also be appreciated that, while various items are illustrated as being stored in memory or on storage while being used, these items or portions of them may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments some or all of the software components and/or systems may execute in memory on another device and communicate with the illustrated computing systems via inter-computer communication. Thus, in some embodiments, some or all of the described techniques may be performed by hardware means that include one or more processors and/or memory and/or storage when configured by one or more software programs (e.g., by the ILDM system 340 executing on server computing systems 300) and/or data structures, such as by execution of software instructions of the one or more software programs and/or by storage of such software instructions and/or data structures, and such as to perform algorithms as described in the flow charts and other disclosure herein. Furthermore, in some embodiments, some or all of the systems and/or components may be implemented or provided in other manners, such as by consisting of one or more means that are implemented partially or fully in firmware and/or hardware (e.g., rather than as a means implemented in whole or in part by software instructions that configure a particular CPU or other processor), including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and/or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage mediums, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), a network storage device, or a portable media article (e.g., a DVD disk, a CD disk, an optical disk, a flash memory device, etc.) to be read by an appropriate drive or via an appropriate connection. The systems, components and data structures may also in some embodiments be transmitted via generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal) on a variety of computer-readable transmission mediums, including wireless-based and wired/cable-based mediums, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). Such computer program products may also take other forms in other embodiments. Accordingly, embodiments of the present disclosure may be practiced with other computer system configurations.
The illustrated embodiment of the routine begins at block 405, where instructions or information are received. At block 410, the routine determines whether the received instructions or information indicate to acquire data representing a building interior, and if not continues to block 490. Otherwise, the routine proceeds to block 412 to receive an indication (e.g., from a user of a mobile image acquisition device) to begin the image acquisition process at a first acquisition location. After block 412, the routine proceeds to block 415 in order to perform acquisition location image acquisition activities in order to acquire a 360° panorama image for the acquisition location in the interior of the target building of interest, such as to provide horizontal coverage of at least 360° around a vertical axis. The routine may also optionally obtain annotation and/or other information from a user regarding the acquisition location and/or the surrounding environment, such as for later use in presentation of information regarding that acquisition location and/or surrounding environment.
After block 415 is completed, the routine continues to block 420 to determine if there are more acquisition locations at which to acquire images, such as based on corresponding information provided by the user of the mobile device. If so, the routine continues to block 422 to optionally initiate the capture of linking information (such as acceleration data, visual data, etc.) during movement of the mobile device along a travel path away from the current acquisition location and towards a next acquisition location within the building interior. As described elsewhere herein, the captured linking information may include additional sensor data (e.g., from one or more IMU, or inertial measurement units, on the mobile device or otherwise carried by the user) and/or additional visual information (e.g., panorama images, other types of images, panoramic or non-panoramic video, etc.) recorded during such movement. Initiating the capture of such linking information may be performed in response to an explicit indication from a user of the mobile device or based on one or more automated analyses of information recorded from the mobile device. In addition, the routine may further optionally monitor the motion of the mobile device in some embodiments during movement to the next acquisition location, and provide one or more guidance cues to the user regarding the motion of the mobile device, quality of the sensor data and/or visual information being captured, associated lighting/environmental conditions, advisability of capturing a next acquisition location, and any other suitable aspects of capturing the linking information. Similarly, the routine may optionally obtain annotation and/or other information from the user regarding the travel path, such as for later use in presentation of information regarding that travel path or a resulting inter-panorama image connection link. In block 424, the routine determines that the mobile device has arrived at the next acquisition location (e.g., based on an indication from the user, based on the forward movement of the user stopping for at least a predefined amount of time, etc.), for use as the new current acquisition location, and returns to block 415 in order to perform the acquisition location image acquisition activities for the new current acquisition location.
If it is instead determined in block 420 that there are not any more acquisition locations at which to acquire image information for the current building or other structure, the routine proceeds to block 425 to optionally analyze the acquisition location information for the building or other structure, such as to identify possible additional coverage (and/or other information) to acquire within the building interior. For example, the ICA system may provide one or more notifications to the user regarding the information acquired during capture of the multiple acquisition locations and optionally corresponding linking information, such as if it determines that one or more segments of the recorded information are of insufficient or undesirable quality, or do not appear to provide complete coverage of the building. After block 425, the routine continues to block 435 to optionally preprocess the acquired 360° panorama images before their subsequent use for generating related mapping information (e.g., to place them in a spherical format, to determine vanishing lines and vanishing points for the images, etc.). In block 477, the images and any associated generated or obtained information is stored for later use.
If it is instead determined in block 410 that the instructions or other information recited in block 405 are not to acquire images and other data representing a building interior, the routine continues instead to block 490 to perform any other indicated operations as appropriate, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device who captures one or more building interiors, an operator user of the ICA system, etc.), to obtain and store other information about users of the system, to respond to requests for generated and stored information, etc.
Following blocks 477 or 490, the routine proceeds to block 495 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 405 to await additional instructions or information, and if not proceeds to step 499 and ends.
The illustrated embodiment of the routine begins at block 505, where information or instructions are received. The routine continues to block 510 to determine whether the instructions received in block 505 indicate to generate mapping information for an indicated building, and if so the routine continues to perform blocks 515-588 to do so, and otherwise continues to block 590.
In block 515, the routine determines whether image information is already available for the building, or if such information instead needs to be acquired. If it is determined in block 515 that the information needs to be acquired, the routine continues to block 520 to acquire such information, optionally waiting for one or more users or devices to move throughout the building and acquire panoramas or other images at multiple acquisition locations in multiple rooms of the building, and to optionally further analyze the images and/or metadata information about their acquisition to interconnect the images, as discussed in greater detail elsewhere herein—
After blocks 520 or 530, the routine continues to block 535 to optionally obtain additional information about the building, such as from activities performed during acquisition and optionally analysis of the images, and/or from one or more external sources (e.g., online databases, information provided by one or more end users, etc.)—such additional information may include, for example, exterior dimensions and/or shape of the building, additional images and/or annotation information acquired corresponding to particular locations within the building (optionally for locations different from acquisition locations of the acquired panorama or other images), additional images and/or annotation information acquired corresponding to particular locations external to the building (e.g., surrounding the building), etc.
After block 535, the routine continues to block 550 to determine, for each room inside the building with one or more acquisition locations and associated acquired images, a room shape of the room from data in the image(s) taken inside the room, and optionally a specified position within the room of its acquisition location(s), such as in an automated manner. The operations of block 550 may further include using visual data in the images and/or the acquisition metadata for them to determine, for each room in the building, any connecting passages in or out of the room (e.g., in an automated manner), and any wall elements in the room and their positions (e.g., in an automated manner), such as for windows, inter-wall borders, etc. The operations of block 550 further include using some or all of the other information determined in block 550 to determine estimated room shapes of the rooms. After block 550, the routine continues to block 565, where it uses the determined room shapes to create an initial 2D floor plan, such as by connecting inter-room passages in their respective rooms, by optionally positioning room shapes around determined acquisition location positions of the images (e.g., if the acquisition location positions are inter-connected), and by optionally applying one or more constraints or optimizations. Such a floor plan may include, for example, relative position and shape information for the various rooms without providing any actual dimension information for the individual rooms or building as a whole, and may further include multiple linked or associated sub-maps (e.g., to reflect different stories, levels, sections, etc.) of the building. The routine further associates positions of the doors, wall openings and other identified wall elements on the floor plan.
After block 565, the routine optionally performs one or more steps 575-580 to determine and associate additional information with the floor plan. In block 575, the routine optionally estimates the dimensions of some or all of the rooms, such as from analysis of images and/or their acquisition metadata or from overall dimension information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan—it will be appreciated that if sufficiently detailed dimension information were available, architectural drawings, blue prints, etc. may be generated from the floor plan. After block 575, the routine continues to block 580 to optionally associate further information with the floor plan (e.g., with particular rooms or other locations within the building), such as additional existing images with specified positions and/or annotation information. In block 585, the routine further estimates heights of walls in some or all rooms, such as from analysis of images and optionally sizes of known objects in the images, as well as height information about a camera when the images were acquired, and further uses such information to generate a 3D computer model floor plan of the building, with the 2D and 3D floor plans being associated with each other.
After block 585, the routine continues to block 588 to store the generated mapping information and optionally other generated information, and to optionally further use the generated mapping information, such as to provide the generated 2D floor plan and/or 3D computer model floor plan for display on one or more client devices, provide that generated information to one or more other devices for use in automating navigation of those devices and/or associated vehicles or other entities, etc.
If it is instead determined in block 510 that the information or instructions received in block 505 are not to generate mapping information for an indicated building, the routine continues instead to block 590 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to requests for previously generated floor plans and/or other generated information (e.g., requests for such information for use by an ILDM system, requests for such information for display on one or more client devices, requests for such information to provide it to one or more other devices for use in automated navigation, etc.), obtaining and storing information about buildings for use in later operations (e.g., information about dimensions, numbers or types of rooms, total square footage, adjacent or nearby other buildings, adjacent or nearby vegetation, exterior images, etc.), etc.
After blocks 588 or 590, the routine continues to block 595 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 505 to wait for and receive additional instructions or information, and otherwise continues to block 599 and ends.
The illustrated embodiment of the routine begins at block 605, where information or instructions are received. The routine continues to block 610 to determine whether the instructions received in block 605 indicate to determine the acquisition position of a target image for an indicated building, and if so the routine continues to perform blocks 612-688 to do so, and otherwise continues to block 690.
In block 612, the routine determines whether to use room shapes from a floor plan for the building or from additional images acquired for the building (e.g., concurrently during the same time period as the target image). If from additional images, the routine continues to block 617 to retrieve those multiple additional images and to add them to an image group for subsequent 3D room shape estimate determination (unless room shapes have already been estimated for the additional images, in which case those estimated room shapes are retrieved), and otherwise continues to block 615 to retrieve a floor plan for the building that has associated information about the rooms' 3D room shapes and room types (and optionally including additional information about wall structural elements' 3D bounding boxes and/or object types, and/or about wireframe structural lines of the room shapes, etc.) and optionally information about existing associated images with specified positions on the floor plan and to select the room shapes as the candidates for enclosing the target image's acquisition position.
After blocks 615 or 617, the routine continues to block 619, where it obtains the target image and optionally acquisition metadata for the target image (e.g., one or more of acquisition time, acquisition order with respect to other images acquired for the building, IMU data, SLAM-based tracking data, room type tag, etc.), such as to receive the information in block 605 or to otherwise retrieve stored information, and adds the target image to the image group for estimated room shape determination. In block 620, the routine then analyzes the visual contents of each image in the image group to determine vanishing lines and vanishing point information for those images, and similarly does so for any existing images with specified positions on the floor plan if not already performed. In block 625, the routine then analyzes the visual contents of each image in the image group to estimate 3D room shape geometry of the room enclosing the image, including to determine 3D bounding boxes for structural wall elements features, to optionally estimate wireframe structural room border lines, to optionally generate a room type tag, to optionally generate object type tags for structural wall elements, and to optionally generate an image embedding vector to encode information about image features and/or attributes for the image.
After block 625, the routine continues to block 640 to determine whether the candidate room shapes are from additional images or from the floor plan, and if the former proceeds to block 645 to select the estimated room shapes for the additional images from block 625 to be used as the candidates for enclosing the target image's acquisition location. After block 645, or if it is instead determined in block 640 that the candidate room shapes are from the floor plan, the routine continues to block 650, where, if the image group includes additional images and/or if there are existing images with specified positions on the floor plan, it optionally performs operations including determining relationships for each pair of the target image with one of those other images, such as to use feature matching of the visual contents of the images of the pair to determine angular connections between the images, and/or to determine the overall image similarity of the images of the user using co-visible elements of the images. After block 650, the routine continues to block 655, where it optionally uses acquisition metadata for the target image to identify one or more preferred room shape candidates, such as based on one or more of room type tag matching, target image acquisition order, etc. In block 660, the routine then proposes matches of the target image's estimated room shape to one or more room shape candidates (using preferred candidates if available, such as before or instead of other candidates), including matching corners and using vanishing lines to generate multiple possible alternative shape matches with different candidate room shape placements for each room shape candidate. In block 665, the routine then generates a matching score for each proposed match, such as by using one or more scoring criteria and combining their scores if multiple are used, with the scoring criteria optionally being based on one or more of corner re-projection, object bounding box re-projection, image angular information, image similarity, and shape-based boundary intersection, with one or more best matches being selected based on the scores and used to generate the initial determination of the target image acquisition position (e.g., acquisition location and optionally orientation) within the room shape of the one or more best matches.
In block 670, the routine then generates an updated determination of the target image acquisition position within the room shape of the one or more best matches, such as by using one or more of alignment based on corner inliers, alignment based on wireframe structural line inliers, and differentiable rendering using image normal information. In block 685, the routine then optionally optimizes the updated determination of the target image acquisition position within the room shape of the one or more best matches in conjunction with other related acquisition position determinations for other images acquired for the building, such as to use additional information related to relationships between the images to adjust their final acquisition positions. After block 685, the routine continues to block 688 to store the information that was determined and generated in blocks 612-685, and to optionally display the determined image acquisition location information for the image in its enclosing room on the floor plan (or floor plan excerpt), although in other embodiments the determined information may be used in other manners (e.g., for automated navigation of one or more devices).
If it is instead determined in block 610 that the information or instructions received in block 605 are not to determine the acquisition position of a target image, the routine continues instead to block 690 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to requests for previously determined image acquisition position information and/or for associated target images (e.g., requests for such information for display on one or more client devices, requests for such information to provide it to one or more other devices for use in automated navigation, etc.), obtaining and storing information about buildings for use in later operations (e.g., information about floor plans and associated wall element positions for rooms in the floor plan, etc.), etc.
After blocks 688 or 690, the routine continues to block 695 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue, the routine returns to block 605 to wait for and receive additional instructions or information, and otherwise continues to block 699 and ends.
The illustrated embodiment of the routine begins at block 705, where instructions or information are received. At block 710, the routine determines whether the received instructions or information indicate to display or otherwise present information representing a building interior, and if not continues to block 790. Otherwise, the routine proceeds to block 712 to retrieve a floor plan and/or other generated mapping information for the building and optionally indications of associated linked information for the building interior and/or a surrounding location external to the building, and selects an initial view of the retrieved information (e.g., a view of the floor plan). In block 715, the routine then displays or otherwise presents the current view of the retrieved information, and waits in block 717 for a user selection. After a user selection in block 717, if it is determined in block 720 that the user selection corresponds to the current building area (e.g., to change the current view), the routine continues to block 722 to update the current view in accordance with the user selection, and then returns to block 715 to update the displayed or otherwise presented information accordingly. The user selection and corresponding updating of the current view may include, for example, displaying or otherwise presenting a piece of associated linked information that the user selects (e.g., a particular image associated with a displayed visual indication of a determined acquisition location), changing how the current view is displayed (e.g., zooming in or out; rotating information if appropriate; selecting a new portion of the floor plan to be displayed or otherwise presented, such as with some or all of the new portion not being previously visible, or instead with the new portion being a subset of the previously visible information; etc.).
If it is instead determined in block 710 that the instructions or other information received in block 705 are not to present information representing a building interior, the routine continues instead to block 790 to perform any other indicated operations as appropriate, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device who captures one or more building interiors, an operator user of the ILDM system, etc.), to obtain and store other information about users of the system, to respond to requests for generated and stored information, etc.
Following block 790, or if it is determined in block 720 that the user selection does not correspond to the current building area, the routine proceeds to block 795 to determine whether to continue, such as until an explicit indication to terminate is received, or instead only if an explicit indication to continue is received. If it is determined to continue (e.g., if the user made a selection in block 717 related to a new location to present), the routine returns to block 705 to await additional instructions or information (or to continue on to block 712 if the user made a selection in block 717 related to a new location to present), and if not proceeds to step 799 and ends.
Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be appreciated that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions. It will be further appreciated that in some implementations the functionality provided by the routines discussed above may be provided in alternative ways, such as being split among more routines or consolidated into fewer routines. Similarly, in some implementations illustrated routines may provide more or less functionality than is described, such as when other illustrated routines instead lack or include such functionality respectively, or when the amount of functionality that is provided is altered. In addition, while various operations may be illustrated as being performed in a particular manner (e.g., in serial or in parallel, or synchronous or asynchronous) and/or in a particular order, in other implementations the operations may be performed in other orders and in other manners. Any data structures discussed above may also be structured in different manners, such as by having a single data structure split into multiple data structures and/or by having multiple data structures consolidated into a single data structure. Similarly, in some implementations illustrated data structures may store more or less information than is described, such as when other illustrated data structures instead lack or include such information respectively, or when the amount or types of information that is stored is altered.
From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention. Accordingly, the invention is not limited except as by corresponding claims and the elements recited by those claims. In addition, while certain aspects of the invention may be presented in certain claim forms at certain times, the inventors contemplate the various aspects of the invention in any available claim form. For example, while only some aspects of the invention may be recited as being embodied in a computer-readable medium at particular times, other aspects may likewise be so embodied.
Claims
1. A computer-implemented method comprising:
- receiving, by one or more computing systems, a target image acquired in a building that has multiple rooms, wherein the target image is a panorama image including 360 degrees of horizontal visual coverage around a vertical axis;
- determining, by the one or more computing systems, an acquisition location of the target image on a floor plan of the building, including: determining, by the one or more computing systems, three-dimensional (“3D”) room shapes for the multiple rooms, wherein each determined 3D room shape identifies at least walls and corners of an associated one of the multiple rooms; generating, by the one or more computing systems and based at least in part on analyzing visual contents of the target image, an estimated 3D room shape for the target image that identifies at least walls and corners visible in the target image, and an indicated position within the estimated 3D room shape of the target image, and information about vanishing line angles for the target image that are based at least in part on the walls visible in the target image; determining, by the one or more computing systems, a plurality of alternative shape matches for the estimated 3D room shape that each includes a placement of one of the determined 3D room shapes relative to the estimated 3D room shape, the placement using the vanishing line angles for the target image and additional vanishing line angles associated with the one determined 3D room shape to match at least one wall and corner of the estimated 3D room shape to at least one other wall and corner of the one determined 3D room shape; selecting, by the one or more computing systems, one of the plurality of alternative shape matches based at least in part on a measurement of a fit for the selected one alternative shape match of the walls and corners of the estimated 3D room shape to the walls and corners of its one determined 3D room shape using the placement for the selected one alternative shape match, and identifying the room of the multiple rooms whose determined room shape is the one determined 3D room shape for the selected one alternative shape match; locating, by the one or more computing systems and within the identified room by using the placement for the selected one alternative shape match of the one determined 3D room shape relative to the estimated 3D room shape, the indicated position within the estimated 3D room shape on that placement, and identifying the acquisition location of the target image on the floor plan based on the located indicated position within the identified room; and
- displaying, by the one or more computing systems, the floor plan of the building with a visual indication of the determined acquisition location of the target image in the identified room.
2. The computer-implemented method of claim 1 wherein the determining of the 3D room shapes for the multiple rooms includes obtaining two-dimensional (“2D”) outlines of the multiple rooms from a floor plan for the building, and generating the 3D room shapes for the multiple rooms based at least in part on the obtained 2D outlines and on information about heights of walls of the multiple rooms.
3. The computer-implemented method of claim 1 wherein the determining of the acquisition location of the target image on the floor plan includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location that identifies a direction in at least two dimensions of at least a starting view from the determined acquisition location, and wherein the displaying of the floor plan further includes displaying, by the one or more computing systems and on a client device, a visual indication of the determined orientation at the determined acquisition location.
4. A computer-implemented method comprising:
- receiving, by one or more computing systems, a target image acquired in a building that has multiple rooms, wherein the target image is a panorama image having visual coverage of multiple surrounding walls;
- determining, by the one or more computing systems, an acquisition location of the target image on a floor plan of the building, including: obtaining, by the one or more computing systems, determined three-dimensional (“3D”) room shapes for the multiple rooms; generating, by the one or more computing systems and based at least in part on analyzing visual contents of the target image, an estimated 3D room shape visible in the target image and an indicated position within the estimated 3D room shape of the target image; selecting, by the one or more computing systems, one of a plurality of alternative shape matches for the estimated 3D room shape based at least in part on a degree of match determined for the selected one alternative shape match, wherein the alternative shape matches each includes a placement of one of the determined 3D room shapes relative to the estimated 3D room shape and has a determined degree of match that is based at least in part on a fit in the placement of the one determined 3D room shape to the estimated 3D room shape, and wherein the one determined 3D room shape for the selected one alternative shape match is for an identified room of the multiple rooms; locating, by the one or more computing systems, and within the identified room by using the placement for the selected one alternative shape match of the determined 3D room shape relative to the estimated 3D room shape, the indicated position within the estimated 3D room shape on that placement, and identifying the acquisition location of the target image on the floor plan based on the located indicated position within the identified room; and
- providing, by the one or more computing systems, information about the determined acquisition location of the target image in the identified room, to enable display of the determined acquisition location on the floor plan of the building.
5. The computer-implemented method of claim 4 wherein the determining of the acquisition location of the target image on the floor plan includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location that identifies a direction in at least two dimensions of at least a starting view from the determined acquisition location, and wherein the providing of the information includes presenting, by the one or more computing systems and on a client device, a display of at least some of the floor plan of the building that includes the identified room and that is overlaid with visual indications of the determined acquisition location and of the determined orientation.
6. The computer-implemented method of claim 4 wherein the generating of the estimated 3D room shape includes determining, by the one or more computing systems, room shape information of a surrounding room around the acquisition location of the target image that includes at least one of a 3D point cloud modeling at least some walls of the surrounding room, or planar surfaces representing the at least some walls of the surrounding room, or wireframe structural lines representing boundaries of the at least some walls of the surrounding room, and that further includes structural wall elements of the surrounding room including at least windows and doorways and non-doorway wall openings, and using the determined room shape information for the estimated 3D room shape.
7. The computer-implemented method of claim 6 wherein the generating of the estimated 3D room shape includes using information about an acquisition time of the target image and further includes determining, by the one or more computing systems, additional room shape information that includes at least one of 3D bounding boxes around the structural wall elements, or at least one room type tag for the surrounding room, or at least one object type tag for each of the structural wall elements, or at least one embedding vector that is generated for the target image by a trained neural network to encode information about features of the surrounding room.
8. The computer-implemented method of claim 6 wherein the determining of the room shape information of the surrounding room includes analyzing, by the one or more computing systems and using one or more trained neural networks, the visual contents of the target image.
9. The computer-implemented method of claim 4 wherein the determining of the acquisition location of the target image further includes determining, by the one or more computing systems, the plurality of alternative shape matches for the estimated 3D room shape, including:
- selecting, by the one or more computing systems, one or more of the determined 3D room shapes as candidate room shapes for matching to the estimated 3D room shape; and
- generating, by the one or more computing systems, and for each of the selected candidate room shapes, one or more of the plurality of alternative shape matches by, for each of the one or more alternative shape matches, determining the placement of that selected candidate room shape relative to the estimated 3D room shape by matching one or more corners of that selected candidate room shape with one or more corners of the estimated 3D room shape and by using vanishing line angle information for the target image to align one or more walls of that selected candidate room shape with one or more walls of the estimated 3D room shape and by adjusting sizes of at least one of the estimated 3D room shape or that selected candidate room shape to reflect each other.
10. The computer-implemented method of claim 9 wherein the selecting of the one or more determined 3D room shapes includes using information about an acquisition time of the target image and further includes identifying, by the one or more computing systems, a preferred subset of the determined 3D room shapes to use as the selected candidate room shapes, including using at least one of matching room type tags for the estimated 3D room shape and the selected candidate room shapes, or one or more determined 3D room shapes for one or more rooms of the building associated with one or more additional images acquired before and/or after the target image in the building, or one or more determined 3D room shapes for one or more rooms of the building associated with one or more additional images having features that match those of the target image above a first defined matching threshold, or one or more determined 3D room shapes for one or more rooms of the building associated with one or more additional images having an overall similarity to the target image via co-visible areas that match above a second defined matching threshold, or one or more determined 3D room shapes for one or more rooms of the building associated with one or more additional images for which multiple determined directions between each pair of the target image and one of the additional images match above a third defined matching threshold, with one of the multiple determined directions between each pair being based on visual contents of images of the pair and with another of the multiple determined directions between each pair being based at least in part on acquisition metadata from acquiring the target image.
11. The computer-implemented method of claim 9 wherein the selecting of the one or more determined 3D room shapes includes selecting each of the determined 3D room shapes as candidate room shapes, and wherein the generating of the one or more alternative shape matches for each of the selected candidate room shapes includes generating a separate alternative shape match for each combination of a corner of that selected candidate room shape and a corner of the estimated 3D room shape and a vanishing line angle for the target image.
12. The computer-implemented method of claim 9 wherein the selecting of one of the alternative shape matches for the estimated 3D room shape based at least in part on its determined degree of match includes determining, by the one or more computing systems and for the selected one alternative shape match, a matching score between the estimated 3D room shape and the determined 3D room shape for the selected one alternative shape match in the placement for the selected one alternative shape match based on one or more matching criteria, wherein the one or more matching criteria include at least one of:
- determined distances in that placement between matching corners of the estimated 3D room shape and that determined 3D room shape; or
- determined distances in that placement between matching structural wall locations of the estimated 3D room shape and that determined 3D room shape; or
- determined distances in that placement between matching wireframe structural lines of the estimated 3D room shape and that determined 3D room shape; or
- determined distances in that placement between matching 3D bounding boxes for structural wall elements in the estimated 3D room shape and that determined 3D room shape; or
- determined distances in that placement between matching 3D bounding boxes for floor elements in the estimated 3D room shape and that determined 3D room shape.
13. The computer-implemented method of claim 12 wherein the determining of the matching score is further performed for each of the plurality of alternative shape matches and includes combining individual scores for multiple matching criteria, and wherein the selecting of one of the alternative shape matches for the estimated 3D room shape is further based at least in part on the selected one alternative shape match having a highest determined matching score.
14. The computer-implemented method of claim 12 further comprising updating, by the one or more computing systems, the placement for the selected one alternative shape match of the one determined 3D room shape relative to the estimated 3D room shape based on at least one of:
- using a least squares regression to reduce the determined distances between the matching corners of the estimated 3D room shape and the determined 3D room shape for the selected one alternative shape match; or
- using a least squares regression to reduce the determined distances between the matching structural wall locations of the estimated 3D room shape and the determined 3D room shape for the selected one alternative shape match; or
- using a least squares regression to reduce the determined distances between the matching wireframe structural lines of the estimated 3D room shape and the determined 3D room shape for the selected one alternative shape match; or
- using a least squares regression to reduce the determined distances between the matching 3D bounding boxes for structural wall elements in the estimated 3D room shape and the determined 3D room shape for the selected one alternative shape match; or
- using a least squares regression to reduce the determined distances between the matching 3D bounding boxes for floor elements in the estimated 3D room shape and the determined 3D room shape for the selected one alternative shape match; or
- performing a differentiable rendering optimization on differences between predicted pixel-level surface normal information for the estimated 3D room shape compared to other pixel-level surface normal values estimated for an existing image with a specified position in the identified room; or
- performing a differentiable rendering optimization on differences between predicted pixel-level surface segment information for the estimated 3D room shape compared to other pixel-level surface segment values estimated for an existing image with a specified position in the identified room, wherein the pixel-level surface segment information for the estimated 3D room shape is predicted based on at least one of segment surface normal information, or segment surface depth map information, or output of a convolutional neural network.
15. The computer-implemented method of claim 4 wherein the determining of the acquisition location of the target image further includes generating, by the one or more computing systems, the plurality of alternative shape matches for the estimated 3D room shape by, for each of one or more of the determined 3D room shapes, generating one or more of the plurality of alternative shape matches for that determined 3D room shape by, for each of the one or more alternative shape matches, performing at least one of:
- determining the placement of that determined 3D room shape relative to the estimated 3D room shape based at least in part on matching one or more first bounding boxes for one or more first objects on a wall of that determined 3D room shape to one or more second bounding boxes for one or more second objects on a wall for the estimated 3D room shape; or
- determining the placement of that determined 3D room shape relative to the estimated 3D room shape based at least in part on matching one or more third bounding boxes for one or more third objects on a floor of that determined 3D room shape to one or more fourth bounding boxes for one or more fourth objects on a floor for the estimated 3D room shape; or
- determining the placement of that determined 3D room shape relative to the estimated 3D room shape based at least in part on matching one or more first planar surfaces for that determined 3D room shape to one or more second planar surfaces for the estimated 3D room shape; or
- determining the placement of that determined 3D room shape relative to the estimated 3D room shape based at least in part on matching first 3D points for that determined 3D room shape to second 3D points for the estimated 3D room shape.
16. The computer-implemented method of claim 4 wherein the selecting of one of the alternative shape matches for the estimated 3D room shape based at least in part on its determined degree of match includes determining, by the one or more computing systems and for the selected one alternative shape match, a matching score based on one or more matching criteria, wherein the one or more matching criteria include at least one of:
- a match above a first defined matching threshold for multiple determined directions between the target image and an additional image associated with that determined 3D room shape for the selected one alternative shape, wherein one of the multiple determined directions is based on visual contents of the target image and the additional image, and wherein another of the multiple determined directions is based at least in part on acquisition metadata from acquiring the target image; or
- a match above a second defined matching threshold between a first estimated orientation for the target image based on visual contents of the target image, and a second estimated orientation for the target image based at least in part on acquisition metadata from acquiring the target image; or
- a match above a third defined matching threshold between a first estimated location at which the target image was acquired based on visual contents of the target image, and a second estimated location at which the target image was acquired based at least in part on acquisition metadata from acquiring the target image; or
- a match above a fourth defined matching threshold between a third estimated location at which the target image was acquired based on visual contents of the target image, and a fourth estimated location at which the target image was intended to be acquired.
17. The computer-implemented method of claim 4 wherein the target image is one of multiple target images having different acquisition locations in the building and each including 360 degrees of horizontal visual coverage around a vertical axis, and wherein the method further comprises:
- determining, by the one or more computing systems, the acquisition locations of each of the multiple target images on the floor plan, including performing generating of the estimated 3D room shape for each of the multiple target images without using any depth information from any depth-sensing sensors for distances from the acquisition location of that target image to surrounding surfaces; and
- updating, the determined acquisition locations of at least some of the multiple target images by performing a global optimization on the determined acquisition locations of the multiple target images in light of additional information about relationships between the multiple target images.
18. A system comprising:
- one or more hardware processors of one or more computing devices; and
- one or more memories with stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations including at least: determining, for a target image acquired in a building that has multiple rooms, an acquisition location of the target image in the building, including: obtaining, for each of one or more of the multiple rooms, a determined room shape of the room; generating, based at least in part on analyzing visual contents of the target image, an estimated room shape for the target image; selecting one of a plurality of alternative shape matches for the estimated room shape based at least in part on a degree of match determined for that selected one alternative shape match, wherein the alternative shape matches each includes a placement of one of the obtained determined room shapes of one of the multiple rooms relative to the estimated room shape and has a determined degree of match that is based at least in part on a fit of the estimated room shape to that placement of that one obtained determined room shape; locating, within the one room whose obtained determined room shape is used for the selected one alternative shape match, the acquisition location of the target image within that one room based on the placement for the selected one alternative shape match and by using information about the target image within the estimated room shape; and providing information about the determined acquisition location of the target image in the one room of the building whose obtained determined room shape is used for the selected one alternative shape match.
19. The system of claim 18 wherein the determining of the acquisition location of the target image in the building includes determining an orientation of the target image at the determined acquisition location that identifies a direction in at least two dimensions of at least a starting view from the determined acquisition location, and wherein the providing of the information includes presenting, on a client device, a display of at least some of the building that includes the identified room and that is overlaid with visual indications of the determined acquisition location and of the determined orientation.
20. The system of claim 18 wherein the obtaining of the determined room shape for each of the one or more rooms includes, for each of the one or more rooms:
- obtaining at least one additional panorama image acquired in that room; and
- generating the determined room shape for that room based at least in part on analyzing visual contents of the at least one additional panorama image acquired in that room.
21. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations including at least:
- receiving, by the one or more computing systems, a target image and multiple additional images that are acquired in a building having multiple rooms, wherein each of the target image and the multiple additional images is a panorama image having visual coverage of multiple surrounding walls;
- determining, by the one or more computing systems, an acquisition location of the target image in the building, including: obtaining, by the one or more computing systems, three-dimensional (“3D”) room shapes determined for the multiple additional images, wherein each determined 3D room shape identifies at least walls and corners visible in a surrounding room for an associated one of the multiple additional images; generating, by the one or more computing systems and based at least in part on analyzing visual contents of the target image, an estimated 3D room shape that includes at least walls and corners visible in the target image, and an indicated position within the estimated 3D room shape of the target image; selecting, by the one or more computing systems, one of a plurality of alternative shape matches for the estimated 3D room shape based at least in part on a degree of match determined for the selected one alternative shape match, wherein the alternative shape matches each includes a placement of one of the determined 3D room shapes relative to the estimated 3D room shape and has a determined degree of match that is based at least in part on a fit using its placement of the walls and corners of the estimated 3D room shape to the walls and corners of its one determined 3D room shape; locating, by the one or more computing systems, and within the surrounding room for the additional image associated with the one determined 3D room shape for the selected one alternative shape match, the indicated position within the estimated 3D room shape using the placement for the selected one alternative shape match, and identifying the acquisition location of the target image in the building based on the located indicated position within the surrounding room; and
- providing, by the one or more computing systems, information about the determined acquisition location of the target image within the surrounding room, to enable display of the determined acquisition location with information about the building.
22. The non-transitory computer-readable medium of claim 21 wherein the determining of the acquisition location of the target image in the building includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location that identifies a direction in at least two dimensions of at least a starting view from the determined acquisition location, and wherein the providing of the information includes presenting, by the one or more computing systems and on a client device, a display of at least some of the building that includes the surrounding room and that is overlaid with visual indications of the determined acquisition location and of the determined orientation.
23. The non-transitory computer-readable medium of claim 21 wherein the obtaining of the determined 3D room shapes includes, for each of one or more rooms in the building, generating the determined 3D room shape for that room by analyzing visual contents of at least one of the multiple additional images acquired in that room, wherein the determining of the acquisition location of the target image is performed dynamically at a time of acquiring the target image, and wherein the providing of the information includes presenting, by the one or more computing systems and at the time of acquiring the target image, a display of at least some of the building that includes the surrounding room.
24. The non-transitory computer-readable medium of claim 21 wherein the selecting of one of the alternative shape matches for the estimated 3D room shape based at least in part on its determined degree of match includes determining, by the one or more computing systems and for the selected one alternative shape match, a matching score between the estimated 3D room shape and the determined 3D room shape for the selected one alternative shape match in the placement for the selected one alternative shape match based on one or more matching criteria, wherein the one or more matching criteria include at least one of:
- distances in that placement between matching corners of the estimated 3D room shape and of that determined 3D room shape; or
- distances in that placement between matching structural wall locations of the estimated 3D room shape and of that determined 3D room shape; or
- distances in that placement between matching wireframe structural lines of the estimated 3D room shape and of that determined 3D room shape; or
- distances in that placement between matching 3D bounding boxes for structural wall elements in the estimated 3D room shape and in that determined 3D room shape; or
- distances in that placement between matching 3D bounding boxes for floor elements in the estimated 3D room shape and in that determined 3D room shape; or
- differences between visual contents of the target image and the one additional image associated with the surrounding room for the determined 3D room shape of the selected one alternative shape match; or
- differences between two measurements of a direction between the target image and that one additional image that includes a first measurement based on the visual contents of the target image and that one additional image and a second measurement based at least in part on acquisition metadata from acquiring the target image and that one additional image.
25. The non-transitory computer-readable medium of claim 21 wherein the selecting of one of the alternative shape matches for the estimated 3D room shape based at least in part on its determined degree of match includes determining, by the one or more computing systems and for the selected one alternative shape match, a matching score based on one or more matching criteria, wherein the one or more matching criteria include at least one of:
- a match above a first defined matching threshold between a first estimated orientation for the target image based on visual contents of the target image, and a second estimated orientation for the target image based at least in part on acquisition metadata from acquiring the target image; or
- a match above a second defined matching threshold between a first estimated location at which the target image was acquired based on visual contents of the target image, and a second estimated location at which the target image was acquired based at least in part on acquisition metadata from acquiring the target image; or
- a match above a third defined matching threshold between a third estimated location at which the target image was acquired based on visual contents of the target image, and a fourth estimated location at which the target image was intended to be acquired.
26. The non-transitory computer-readable medium of claim 21 wherein the selecting of one of the alternative shape matches for the estimated 3D room shape based at least in part on its determined degree of match includes determining, by the one or more computing systems and for the selected one alternative shape match, the degree of match based at least in part on comparing a first generated image embedding vector for the target image to a second generated image embedding vector for the one additional image associated with the surrounding room for the determined 3D room shape of the selected one alternative shape match, wherein the first generated image embedding vector encodes information about one or more features and/or attributes of the target image, and wherein the second generated image embedding vector encodes information about one or more features and/or attributes of that one additional image.
| 5140352 | August 18, 1992 | Moore et al. |
| 6031540 | February 29, 2000 | Golin et al. |
| 6141034 | October 31, 2000 | McCutchen |
| 6317166 | November 13, 2001 | McCutchen |
| 6320584 | November 20, 2001 | Golin et al. |
| 6323858 | November 27, 2001 | Gilbert et al. |
| 6337683 | January 8, 2002 | Gilbert et al. |
| 6654019 | November 25, 2003 | Gilbert et al. |
| 6683608 | January 27, 2004 | Golin et al. |
| 6690374 | February 10, 2004 | Park et al. |
| 6731305 | May 4, 2004 | Park et al. |
| 6738073 | May 18, 2004 | Park et al. |
| 7050085 | May 23, 2006 | Park et al. |
| 7129971 | October 31, 2006 | McCutchen |
| 7196722 | March 27, 2007 | White et al. |
| 7525567 | April 28, 2009 | McCutchen |
| 7620909 | November 17, 2009 | Park et al. |
| 7627235 | December 1, 2009 | McCutchen et al. |
| 7782319 | August 24, 2010 | Ghosh et al. |
| 7791638 | September 7, 2010 | McCutchen |
| 7909241 | March 22, 2011 | Stone et al. |
| 7973838 | July 5, 2011 | McCutchen |
| 8072455 | December 6, 2011 | Temesvari et al. |
| 8094182 | January 10, 2012 | Park et al. |
| RE43786 | November 6, 2012 | Cooper |
| 8463020 | June 11, 2013 | Schuckmann et al. |
| 8517256 | August 27, 2013 | Stone et al. |
| 8520060 | August 27, 2013 | Zomet et al. |
| 8523066 | September 3, 2013 | Stone et al. |
| 8523067 | September 3, 2013 | Stone et al. |
| 8528816 | September 10, 2013 | Stone et al. |
| 8540153 | September 24, 2013 | Stone et al. |
| 8594428 | November 26, 2013 | Aharoni et al. |
| 8654180 | February 18, 2014 | Zomet et al. |
| 8666815 | March 4, 2014 | Chau |
| 8699005 | April 15, 2014 | Likholyot |
| 8705892 | April 22, 2014 | Aguilera et al. |
| RE44924 | June 3, 2014 | Cooper et al. |
| 8854684 | October 7, 2014 | Zomet |
| 8861840 | October 14, 2014 | Bell et al. |
| 8861841 | October 14, 2014 | Bell et al. |
| 8879828 | November 4, 2014 | Bell et al. |
| 8953871 | February 10, 2015 | Zomet |
| 8989440 | March 24, 2015 | Klusza et al. |
| 8996336 | March 31, 2015 | Malka et al. |
| 9021947 | May 5, 2015 | Landa |
| 9026947 | May 5, 2015 | Lee et al. |
| 9035968 | May 19, 2015 | Zomet |
| 9041796 | May 26, 2015 | Malka et al. |
| 9071714 | June 30, 2015 | Zomet |
| 9129438 | September 8, 2015 | Aarts et al. |
| 9151608 | October 6, 2015 | Malka et al. |
| 9165410 | October 20, 2015 | Bell et al. |
| 9171405 | October 27, 2015 | Bell et al. |
| 9324190 | April 26, 2016 | Bell et al. |
| 9361717 | June 7, 2016 | Zomet |
| 9396586 | July 19, 2016 | Bell et al. |
| 9438759 | September 6, 2016 | Zomet |
| 9438775 | September 6, 2016 | Powers et al. |
| 9489775 | November 8, 2016 | Bell et al. |
| 9495783 | November 15, 2016 | Samarasekera et al. |
| 9576401 | February 21, 2017 | Zomet |
| 9619933 | April 11, 2017 | Spinella-Marno et al. |
| 9635252 | April 25, 2017 | Accardo et al. |
| 9641702 | May 2, 2017 | Bin-Nun et al. |
| 9760994 | September 12, 2017 | Bell et al. |
| 9786097 | October 10, 2017 | Bell et al. |
| 9787904 | October 10, 2017 | Birkler et al. |
| 9836885 | December 5, 2017 | Eraker et al. |
| 9852351 | December 26, 2017 | Aguilera Perez et al. |
| 9953111 | April 24, 2018 | Bell et al. |
| 9953430 | April 24, 2018 | Zakhor |
| 9990760 | June 5, 2018 | Aguilera Perez et al. |
| 9990767 | June 5, 2018 | Sheffield et al. |
| 10026224 | July 17, 2018 | Bell et al. |
| 10030979 | July 24, 2018 | Bjorke et al. |
| 10055876 | August 21, 2018 | Ford et al. |
| 10068344 | September 4, 2018 | Jovanovic et al. |
| 10083522 | September 25, 2018 | Jovanovic et al. |
| 10102639 | October 16, 2018 | Bell et al. |
| 10102673 | October 16, 2018 | Eraker et al. |
| 10120397 | November 6, 2018 | Zakhor et al. |
| 10122997 | November 6, 2018 | Sheffield et al. |
| 10127718 | November 13, 2018 | Zakhor et al. |
| 10127722 | November 13, 2018 | Shakib et al. |
| 10139985 | November 27, 2018 | Mildrew et al. |
| 10163261 | December 25, 2018 | Bell et al. |
| 10163271 | December 25, 2018 | Powers et al. |
| 10181215 | January 15, 2019 | Sedeffow |
| 10192115 | January 29, 2019 | Sheffield et al. |
| 10204185 | February 12, 2019 | Mrowca et al. |
| 10210285 | February 19, 2019 | Wong et al. |
| 10235797 | March 19, 2019 | Sheffield et al. |
| 10242400 | March 26, 2019 | Eraker et al. |
| 10339716 | July 2, 2019 | Powers et al. |
| 10366531 | July 30, 2019 | Sheffield |
| 10395435 | August 27, 2019 | Powers et al. |
| 10937247 | March 2, 2021 | Chuah |
| 20060256109 | November 16, 2006 | Acker et al. |
| 20080126022 | May 29, 2008 | Hoguet |
| 20100232709 | September 16, 2010 | Zhang et al. |
| 20120075414 | March 29, 2012 | Park et al. |
| 20120293613 | November 22, 2012 | Powers et al. |
| 20130050407 | February 28, 2013 | Brinda et al. |
| 20130342533 | December 26, 2013 | Bell et al. |
| 20140043436 | February 13, 2014 | Bell et al. |
| 20140044343 | February 13, 2014 | Bell et al. |
| 20140044344 | February 13, 2014 | Bell et al. |
| 20140125658 | May 8, 2014 | Bell et al. |
| 20140125767 | May 8, 2014 | Bell et al. |
| 20140125768 | May 8, 2014 | Bell et al. |
| 20140125769 | May 8, 2014 | Bell et al. |
| 20140125770 | May 8, 2014 | Bell et al. |
| 20140236482 | August 21, 2014 | Dorum et al. |
| 20140267631 | September 18, 2014 | Powers et al. |
| 20140307100 | October 16, 2014 | Myllykoski et al. |
| 20140320674 | October 30, 2014 | Kuang |
| 20150116691 | April 30, 2015 | Likholyot |
| 20150189165 | July 2, 2015 | Milosevski et al. |
| 20150262421 | September 17, 2015 | Bell et al. |
| 20150269785 | September 24, 2015 | Bell et al. |
| 20150302636 | October 22, 2015 | Arnoldus et al. |
| 20150310596 | October 29, 2015 | Sheridan et al. |
| 20150332464 | November 19, 2015 | O'Keefe et al. |
| 20160055268 | February 25, 2016 | Bell et al. |
| 20160134860 | May 12, 2016 | Jovanovic et al. |
| 20160140676 | May 19, 2016 | Fritze et al. |
| 20160217225 | July 28, 2016 | Bell et al. |
| 20160260250 | September 8, 2016 | Jovanovic et al. |
| 20160286119 | September 29, 2016 | Rondinelli |
| 20160300385 | October 13, 2016 | Bell et al. |
| 20170034430 | February 2, 2017 | Fu et al. |
| 20170067739 | March 9, 2017 | Siercks et al. |
| 20170194768 | July 6, 2017 | Powers et al. |
| 20170195654 | July 6, 2017 | Powers et al. |
| 20170263050 | September 14, 2017 | Ha et al. |
| 20170324941 | November 9, 2017 | Birkler |
| 20170330273 | November 16, 2017 | Holt et al. |
| 20170337737 | November 23, 2017 | Edwards et al. |
| 20180007340 | January 4, 2018 | Stachowski |
| 20180025536 | January 25, 2018 | Bell et al. |
| 20180075168 | March 15, 2018 | Tiwari et al. |
| 20180075652 | March 15, 2018 | Kim |
| 20180139431 | May 17, 2018 | Simek et al. |
| 20180143023 | May 24, 2018 | Bjorke et al. |
| 20180143756 | May 24, 2018 | Mildrew et al. |
| 20180144487 | May 24, 2018 | Bell et al. |
| 20180144535 | May 24, 2018 | Ford et al. |
| 20180144547 | May 24, 2018 | Shakib et al. |
| 20180144555 | May 24, 2018 | Ford et al. |
| 20180146121 | May 24, 2018 | Hensler et al. |
| 20180146193 | May 24, 2018 | Safreed et al. |
| 20180146212 | May 24, 2018 | Hensler et al. |
| 20180165871 | June 14, 2018 | Mrowca |
| 20180203955 | July 19, 2018 | Bell et al. |
| 20180241985 | August 23, 2018 | O'Keefe et al. |
| 20180293793 | October 11, 2018 | Bell et al. |
| 20180300936 | October 18, 2018 | Ford et al. |
| 20180306588 | October 25, 2018 | Bjorke et al. |
| 20180348854 | December 6, 2018 | Powers et al. |
| 20180365496 | December 20, 2018 | Hovden et al. |
| 20190012833 | January 10, 2019 | Eraker et al. |
| 20190026956 | January 24, 2019 | Gausebeck et al. |
| 20190026957 | January 24, 2019 | Gausebeck |
| 20190026958 | January 24, 2019 | Gausebeck et al. |
| 20190035165 | January 31, 2019 | Gausebeck |
| 20190041972 | February 7, 2019 | Bae |
| 20190050137 | February 14, 2019 | Mildrew et al. |
| 20190051050 | February 14, 2019 | Bell et al. |
| 20190051054 | February 14, 2019 | Jovanovic et al. |
| 20190087067 | March 21, 2019 | Hovden et al. |
| 20190122422 | April 25, 2019 | Sheffield et al. |
| 20190164335 | May 30, 2019 | Sheffield et al. |
| 20190180104 | June 13, 2019 | Sheffield et al. |
| 20190251645 | August 15, 2019 | Winans |
| 20190287164 | September 19, 2019 | Eraker et al. |
| 3058602 | April 2020 | CA |
| 107123144 | September 2017 | CN |
| 2413097 | February 2012 | EP |
| 2505961 | October 2012 | EP |
| 2506170 | October 2012 | EP |
| 101770648 | August 2017 | KR |
| 101930796 | December 2018 | KR |
| 2005091894 | October 2005 | WO |
| 2016154306 | September 2016 | WO |
| 2018204279 | November 2018 | WO |
| 2019083832 | May 2019 | WO |
| 2019104049 | May 2019 | WO |
| 2019118599 | June 2019 | WO |
- CubiCasa | From video to floor plan in under 5 minutes, retrieved on Mar. 26, 2019, from https://www.cubi.casa/, 6 pages.
- CubiCasa FAQ & Manual, retrieved on Mar. 26, 2019, from https://www.cubi.casa/faq/, 5 pages.
- Cupix Home, retrieved on Mar. 26, 2019, from https://www.cupix.com/, 1 page.
- Cupix—FAQ, retrieved on Mar. 26, 2019, from https://www.cupix.com/faq.html, 3 pages.
- Guide: 3D Virtual Tours, retrieved on Mar. 26, 2019, from https://goiguide.com/, 6 pages.
- Immoviewer.com | Automated Video Creation & Simple Affordable 3D 360 Tours, retrieved on Mar. 26, 2019, from https://www.immoviewer.com/, 5 pages.
- MagicPlan | #1 Floor Plan App, Construction & Surveying Samples, retrieved on Mar. 26, 2019, from https://www.magicplan.app/, 9 pages.
- EyeSpy360 Virtual Tours | Virtual Tour with any 360 camera, retrieved on Mar. 27, 2019, from https://www.eyespy360.com/en-us/, 15 pages.
- Indoor Reality, retrieved on Mar. 27, 2019, from https://www.indoorreality.com/, 9 pages.
- InsideMaps, retrieved on Mar. 27, 2019, from https://www.insidemaps.com/, 7 pages.
- Staging | Augmented & Virtual Reality Platform for Business, retrieved on Mar. 27, 2019, from https://www.istaging.com/en/, 7 pages.
- Metareal, retrieved on Mar. 27, 2019, from https://www.metareal.com/, 4 pages.
- PLNAR—The AR 3D Measuring / Modeling Platform, retrieved on Mar. 27, 2019, from https://www.plnar.co, 6 pages.
- YouVR Global, retrieved on Mar. 27, 2019, from https://global.youvr.io/, 9 pages.
- GeoCV, retrieved on Mar. 28, 2019, from https://geocv.com/, 4 pages.
- Biersdorfer, J.D., “How to Make a 3-D Model of Your Home Renovation Vision,” in the New York Times, Feb. 13, 2019, retrieved Mar. 28, 2019, 6 pages.
- Chen et al. “Rise of the indoor crowd: Reconstruction of building interior view via mobile crowdsourcing.” In: Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems. Nov. 4, 2015, 13 pages.
- Immersive 3D for the Real World, retrieved from https://matterport.com/, on Mar. 27, 2017, 5 pages.
- Leam About Our Complete 3D System, retrieved from https://matterport.com/how-it-works/, on Mar. 27, 2017, 6 pages.
- Surefield FAQ, retrieved from https://surefield.com/faq, on Mar. 27, 2017, 1 page.
- Why Surefield, retrieved from https://surefield.com/why-surefield, on Mar. 27, 2017, 7 pages.
- Schneider, V., “Create immersive photo experiences with Google Photo Sphere,” retrieved from http://geojournalism.org/2015/02/create-immersive-photo-experiences-with-google-photo-sphere/, on Mar. 27, 2017, 7 pages.
- Tango (platform), Wikipedia, retrieved from https://en.wikipedia.org/wiki/Tango_(platform), on Jun. 12, 2018, 5 pages.
- Zou et al. “LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Image” in arXiv:1803.08999, submitted Mar. 23, 2018, 9 pages.
- Lee et al. “RoomNet: End-to-End Room Layout Estimation” in arXiv:1703.00241v2, submitted Aug. 7, 2017, 10 pages.
- Time-of-flight camera, Wikipedia, retrieved from https://en.wikipedia.org/wiki/Time-of-flight_camera, on Aug. 30, 2018, 8 pages.
- Magicplan—Android Apps on Go . . . , retrieved from https://play.google.com/store/apps/details?id=com.sensopia.magicplan, on Feb. 21, 2018, 5 pages.
Type: Grant
Filed: Mar 15, 2021
Date of Patent: Oct 25, 2022
Assignee: Zillow, Inc. (Seattle, WA)
Inventors: Yuguang Li (Seattle, WA), Will Adrian Hutchcroft (Seattle, WA), Naji Khosravan (Seattle, WA), Ivaylo Boyadzhiev (Seattle, WA)
Primary Examiner: Jennifer Mehmood
Assistant Examiner: Andrew Shin
Application Number: 17/201,996
International Classification: G06T 7/73 (20170101); H04N 5/232 (20060101); G06T 11/00 (20060101); G06T 7/50 (20170101); G06K 9/62 (20220101); G06V 10/44 (20220101); G06V 10/75 (20220101); G06V 20/00 (20220101);
