IDENTIFYING OBJECTS USING A 3D SCANNING DEVICE, IMAGES, AND 3D MODELS

Abstract

This application is drawn to methods for identifying three-dimensional objects. Methods disclosed herein may include converting 3D model data to a set of simple surfaces, and generating searchable parameters from the image data and/or the simple surfaces. The searchable parameters may be stored in a database as metadata. A searcher may construct a search query by providing 3D model data from which search parameters and features may be extracted and used to formulate a query.

Description

I. BACKGROUND OF THE INVENTION

A. Field of Invention

Embodiments may generally relate to conducting similarity searches of three-dimensional models.

B. Description of the Related Art

While image capture technologies have existed for years, it is only relatively recently that the field of machine vision has developed to the point where computers are capable of decomposing images into elements. Furthermore the problem of matching complex shapes, such as body parts, to reference shapes is nontrivial and has been the subject of much research in recent years. Available methods are cumbersome and computation-intensive. Thus, there is a need in the art for methods having improved efficiency, accuracy, and/or resolution.

Some embodiments of the present invention may provide one or more benefits or advantages over the prior art.

II. SUMMARY OF THE INVENTION

Some embodiments may relate to a method for identifying three-dimensional objects, comprising: providing a set of three-dimensional model data of a subject; converting the set of three-dimensional model data to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface; generating searchable parameters from features of the set of continuous simple surfaces; storing the searchable parameters in a database as metadata of the three-dimensional model data; providing a search interface adapted to receive three-dimensional model data as input for a search query, the search interface being adapted to formulate a search query from the input; and matching the search query to one or more records of the database and retrieving said records.

According to some embodiments the set of three-dimensional model data comprises a point cloud or a mesh.

Embodiments may further comprise a step of extracting local geometric features directly from the set of three-dimensional model data of a subject without first converting the three-dimensional model data of a subject to a set of continuous simple surfaces.

According to some embodiments a user or an algorithm selects the features of the three-dimensional model data of a subject from which searchable parameters are generated, wherein the selected features are less than all of the features of the three-dimensional model data.

According to some embodiments the database is organized according to a pattern promoting efficient searching, the pattern being selected from one or more of organization according to feature size, feature type, or feature curvature.

According to some embodiments the database is organized according to a structure selected from one or more of relational, hierarchical, network, object-relational, object oriented, semi-structured, associative, Entity-Attribute-Value, or context model.

According to some embodiments the database contains simplified searchable parameters comprising one or more of a bag-of-words, or dimension reduction by Principal Component Analysis.

According to some embodiments the step of providing a search interface further comprises: converting the input to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface; generating a set of input features from the set of continuous simple surfaces; and formulating a query from the input features.

Embodiments may further comprise the step of extracting local geometric features directly from the input without first converting the input to a set of continuous simple surfaces.

According to some embodiments a user or an algorithm selects the features of the input for use in formulating a query, wherein the selected features are less than all of the features of the input.

Some embodiments may relate to a method for identifying three-dimensional objects, comprising: providing a set of three-dimensional model data of a subject; converting the set of three-dimensional model data to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface; generating searchable parameters from features of the a set of continuous simple surfaces; extracting local geometric features directly from the set of three-dimensional model data of a subject without first converting the three-dimensional model data of a subject to a set of continuous simple surfaces; storing the searchable parameters in a database as metadata of the three-dimensional model data; providing a search interface adapted to receive three-dimensional model data as input for a search query, the search interface being adapted to formulate a search query from the input; and matching the search query to one or more records of the database and retrieving said records.

Some embodiments may relate to a method for identifying three-dimensional objects, comprising: providing a set of three-dimensional model data of a subject; converting the set of three-dimensional model data to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface; generating searchable parameters from features of the a set of continuous simple surfaces; extracting local geometric features directly from the set of three-dimensional model data of a subject without first converting the three-dimensional model data of a subject to a set of continuous simple surfaces; storing the searchable parameters in a database as metadata of the three-dimensional model data; providing a search interface adapted to receive three-dimensional model data as input for a search query; converting the input to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface; generating a set of input features from the set of continuous simple surfaces; formulating a query from the input features; and matching the search query to one or more records of the database and retrieving said records.

Other benefits and advantages will become apparent to those skilled in the art to which it pertains upon reading and understanding of the following detailed specification.

III. BRIEF DESCRIPTION OF THE DRAWINGS

The invention may take physical form in certain parts and arrangement of parts, embodiments of which will be described in detail in this specification and illustrated in the accompanying drawings which form a part hereof and wherein:

FIG. 1 is a flow diagram of a process according to an embodiment of the invention;

FIG. 2 is an illustration of acquiring three-dimensional model data of a subject using a 3D scanning device;

FIG. 3 is an illustration of extracting feature data from an 3D model data for formulating a query; and

FIG. 4 is a conceptual drawing of a search interface for inputting a query and viewing search results.

IV. DETAILED DESCRIPTION OF THE INVENTION

Methods for identifying three-dimensional objects include the conversion of a set of three-dimensional model data to a set of continuous simple surfaces. Searchable parameters may be generated from features of the three-dimensional model data. The searchable parameters may be stored in a database as metadata of the three-dimensional model data. An interface may be provided to receive three-dimensional model data as an input for formulating a search query. The search query may be matched to records in the database which may be retrieved based on their similarity to the three-dimensional model data provided as input.

Referring now to the drawings wherein the showings are for purposes of illustrating embodiments of the invention only and not for purposes of limiting the same, FIG. 1 depicts a flow diagram 100 of an illustrative embodiment, wherein three-dimensional model data 110 is provided for the purpose of generating a three-dimensional model database for quick and efficient identification of three-dimensional objects. The three-dimensional model data may either be obtained from a 3D scanning device, from the device's on-board storage, or from a server where such data is pre-stored. In one embodiment, the three-dimensional model data stored in the database or used as an input for a search may be in the form of a point cloud or mesh of a three-dimensional subject. Alternatively, any and all other forms of three-dimensional data representation, now known or developed in the future, that are capable of being converted to point cloud or mesh form may be used.

The three-dimensional model data may further be converted to a set of continuous simple surfaces 112 by using a fitting method including but not limited to a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method. All of these methods are well understood in the art and their methodologies are incorporated herein by reference. Any simple surface including but not limited to a planar surface, cylindrical surface, spherical surface, sinusoidal surface, or a conic surface may be used to represent the three-dimensional data as the set of simple continuous surfaces.

Features of the three-dimensional model data may be analyzed in order to generate searchable parameters for the model data. The searchable parameters may be generated 114 based on the representation of 3D model data as features of the set of simple continuous surfaces. In another embodiment, local geometric features may be directly extracted from the set of three-dimensional model data of a subject without first converting the data to a set of continuous simple surfaces. This direct feature extraction may permit 3D representations other than point cloud or mesh data to be used for generating searchable parameters.

The searchable parameters may be generated based on manual feature selection by the user and/or automatically (including algorithmically). Either all or less than all features may be selected for generating searchable parameters based on specific objectives of database creation and specific use case. For instance, where the use case is a database of orthotic inserts for shoes, and the query is formulated from a 3D model of a patient's foot, the relevant searchable parameters may be related to describing the curvature of the foot arch, foot length, foot width, etc. Similarly, a user of an embodiment may select certain features or portions of a 3D model of a patient's foot from which search parameters will be formulated.

In one embodiment, searchable parameters may be stored in a database 116 as metadata. The metadata may describe the object mathematically based on its features as calculated from the set of simple surfaces and/or through direct extraction from the 3D model data. Furthermore, the metadata may be used for comparison of the inputted three-dimensional model data with the data stored on the database. The stored metadata includes information such as dimensions, surface features, creator/manufacturer details of the three-dimensional objects, etc. The database may also include local geometric features. The local geometric features extracted may include but are not limited to principal curvatures, average geodesic distances, heat kernel signatures, wave kernel signatures, shape contexts, spin images, features generated by Principal Component Analysis of the mesh graph, bounding box dimensions, mesh homology, and features generated by combining the aforementioned features or similar features in a manner known to a person skilled in the art. In an embodiment, the local geometric features are extracted after the conversion of the inputted three-dimensional model data into a set of continuous simple surfaces. In another embodiment, local geometric features are extracted from the input directly, without first converting the input into a set of continuous simple surfaces.

Organization of the database is an important aspect to ensure efficient searching. In order to facilitate faster searches. For example, and without limitation, the database may be organized based on one or more of feature size, feature type, or feature curvature, and may be arranged/ordered from the largest to the smallest or vice versa to improve search speed. Based on the use case scenario, the database can also be organized based one or more of relational, hierarchical, network, object-relational, object oriented, semi-structured, associative, entity-attribute value, or context model. A person skilled in the art will be able to determine the most efficient database structure based on the requirements. Sometimes the three-dimensional model data may have complex features, and efficient storing and searching would necessitate the use of simplified descriptors. In such scenarios, methods like bag-of-words, or dimension reduction by Principal Component Analysis may be used to reduce the complexity of the searchable parameters.

The objective of the method provided herein is to find one or more objects that match with the object provided by the user. And the process of search begins with the input of a search query 118. The search query is inputted 118 as 3D model data. The 3D model data may be generated through a 3D scanner. Any other methods of generating a 3D model data well known in the art may also be used. In one embodiment, the three-dimensional model data inputted may be in the form of a point cloud or mesh or any other form which may be capable of being converted to a point cloud or mesh.

Once, the 3D model data corresponding to a real world 3D object is generated, the search interface facilitates the generation of a set of input features and the formulation of a search query. The process of formulating a search query further includes the conversion of inputted three-dimensional model data to a set of continuous simple surfaces 112 by using a fitting method including but not limited to a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method. Any simple surface including but not limited to a planar surface, cylindrical surface, spherical surface, sinusoidal surface, or a conic surface may be used to represent the three-dimensional data as the set of simple continuous surfaces. In one embodiment, local geometric features may be extracted after the conversion of the inputted three-dimensional model data into a set of continuous simple surfaces. In an alternate embodiment, local geometric features are extracted from the input directly, without first converting the input into a set of continuous simple surfaces.

Some or all the elements extracted from the inputted three-dimensional model data may be used for formulating a search query. In another embodiment, a user or an algorithm may select the number of features to be used for formulating a search query, instead of inputting all the features; based on the need for identification and matching of only a portion of a three-dimensional model data set. Such an option is desirable as the reduction in the number of selected features for the search facilitates in the reduction of the computation time and help in faster retrieval of search results.

Once, the search query is formulated and inputted 118, the query is matched with database records 120. Based on the matching of the inputted search query to the database records 120, search results are retrieved 122. The retrieval of the search results 122 may be based on the matching of one or more features of the input with one or more records in the database.

In an example, FIGS. 2 through 5 illustrate a method for the selection of a close matching insole for an athlete's footwear involving the steps of 3D scanning the athlete's foot to obtain three-dimensional model data, converting the athlete's foot into continuous simple surfaces, generating a set of input features and formulating a search query, comparing it to an insole database to match and retrieve search results based on similarity to database records.

With particular reference to FIG. 2, this figure depicts an illustrative embodiment 200, wherein a three-dimensional scanner 210 is used to scan and obtain three-dimensional model data 216 of a real world subject 212 (a human foot). The three-dimensional model data 216 is obtained by scanning the subject 212 from various directions and orientations 214. While FIG. 2 suggests a particular orientation of the scanner and scan path, one skilled in the art will appreciate that this is merely for the convenience of illustration and that an actual scanner path may differ significantly so as to include features on the sole of the foot. The 3D scanner 210 may be any known or future developed 3D scanner including but not limited to mobile devices, smart phones or tablets configured to scan and obtain three-dimensional model data.

FIG. 3 depicts an illustrative embodiment 300, wherein the three-dimensional model data of the real world subject 212 (a human foot) of FIG. 2 is represented in the form of a mesh 216. This mesh representation may be further converted to a set of continuous simple surfaces (not shown). This conversion may be achieved by using a fitting method including but not limited to a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method. In another embodiment, the 3D scan can be represented in the form of a point cloud.

In one embodiment, a set of three features (312, 314, and 316) may be generated corresponding to the scanned three-dimensional model data 216 of a real world subject 212. Reference numbers 312, 314 and 316 represent relevant features of the scanned foot, i.e. the length, width and the arch of the right human foot respectively. Based on these features, a query may be formulated containing mathematical descriptions of one or more of the features that may be queried against insoles provided on the database (not shown). In another embodiment, a user or an algorithm may select a subset of the total number of features to be used for formulating a search query instead of formulating a query from all of the features. Such an option may help to reduce the computation time and help in faster retrieval of search results.

A database according to an embodiment of the invention may provide a repository of 3D model data, e.g. of all insoles in inventory, with simplified searchable parameters to mitigate the amount of computation necessary for matching which may arise out of the mathematical complexity of the surface being searched. Such simplified searchable parameters may be one or more of bag-of-words, or dimension reduction by Principal Component Analysis. Such methods are known in the art and one skilled in the art would be capable of applying these methods to the present complexity problem.

FIG. 4 depicts an illustrative embodiment 400 providing an input section 410 and an output section 412. The user can input three-dimensional model data 216 for formulating a query. The user may also view search results 412 based on the similarity of the query to records contained in the database.

It will be apparent to those skilled in the art that the above methods and apparatuses may be changed or modified without departing from the general scope of the invention. The invention is intended to include all such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.

Having thus described the invention, it is now claimed:

Claims

1. A method for identifying three-dimensional objects, comprising:

providing a set of three-dimensional model data of a subject;

converting the set of three-dimensional model data to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface;

generating searchable parameters from features of the set of continuous simple surfaces;

storing the searchable parameters in a database as metadata of the three-dimensional model data;

providing a search interface adapted to receive three-dimensional model data as input for a search query, the search interface being adapted to formulate a search query from the input; and

matching the search query to one or more records of the database and retrieving said records.

2. The method of claim 1, wherein the set of three-dimensional model data comprises a point cloud or a mesh.

3. The method of claim 1, further comprising the step of extracting local geometric features directly from the set of three-dimensional model data of a subject without first converting the three-dimensional model data of a subject to a set of continuous simple surfaces.

4. The method of claim 1, wherein a user or an algorithm selects the features of the three-dimensional model data of a subject from which searchable parameters are generated, wherein the selected features are less than all of the features of the three-dimensional model data.

5. The method of claim 1, wherein the database is organized according to a pattern promoting efficient searching, the pattern being selected from one or more of organization according to feature size, feature type, or feature curvature.

6. The method of claim 5, wherein the database is organized according to a structure selected from one or more of relational, hierarchical, network, object-relational, object oriented, semi-structured, associative, Entity-Attribute-Value, or context model.

7. The method of claim 1, wherein the database contains simplified searchable parameters comprising one or more of a bag-of-words, or dimension reduction by Principal Component Analysis.

8. The method of claim 1, wherein the step of providing a search interface further comprises:

converting the input to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface;

generating a set of input features from the set of continuous simple surfaces; and

formulating a query from the input features.

9. The method of claim 8, further comprising the step of extracting local geometric features directly from the input without first converting the input to a set of continuous simple surfaces.

10. The method of claim 8, wherein a user or an algorithm selects the features of the input for use in formulating a query, wherein the selected features are less than all of the features of the input.

11. A method for identifying three-dimensional objects, comprising:

providing a set of three-dimensional model data of a subject;

converting the set of three-dimensional model data to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface;

generating searchable parameters from features of the a set of continuous simple surfaces;

extracting local geometric features directly from the set of three-dimensional model data of a subject without first converting the three-dimensional model data of a subject to a set of continuous simple surfaces;

storing the searchable parameters in a database as metadata of the three-dimensional model data;

providing a search interface adapted to receive three-dimensional model data as input for a search query, the search interface being adapted to formulate a search query from the input; and

matching the search query to one or more records of the database and retrieving said records.

12. A method for identifying three-dimensional objects, comprising:

providing a set of three-dimensional model data of a subject;

converting the set of three-dimensional model data to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface;

generating searchable parameters from features of the a set of continuous simple surfaces;

extracting local geometric features directly from the set of three-dimensional model data of a subject without first converting the three-dimensional model data of a subject to a set of continuous simple surfaces;

storing the searchable parameters in a database as metadata of the three-dimensional model data;

providing a search interface adapted to receive three-dimensional model data as input for a search query;

converting the input to a set of continuous simple surfaces using a fitting method selected from one or more of a random sample consensus (RANSAC) method, an iterative closest point method, a least squares method, a Newtonian method, a quasi-Newtonian method, or an expectation-maximization method, wherein a simple surface comprises a planar surface, a cylindrical surface, a spherical surface, a sinusoidal surface, or a conic surface;

generating a set of input features from the set of continuous simple surfaces;

formulating a query from the input features; and

matching the search query to one or more records of the database and retrieving said records.

13. The method of claim 12, wherein the set of three-dimensional model data comprises a point cloud or a mesh.

14. The method of claim 13, wherein a user or an algorithm selects the features of the three-dimensional model data of a subject from which searchable parameters are generated, wherein the selected features are less than all of the features of the three-dimensional model data.

15. The method of claim 14, wherein the database is organized according to a pattern promoting efficient searching, the pattern being selected from one or more of organization according to feature size, feature type, or feature curvature.

16. The method of claim 15, wherein the database is organized according to a structure selected from one or more of relational, hierarchical, network, object-relational, object oriented, semi-structured, associative, Entity-Attribute-Value, or context model.

17. The method of claim 16, wherein the database contains simplified searchable parameters comprising one or more of a bag-of-words, or dimension reduction by Principal Component Analysis.