Patents by Inventor Nils Grimsmo
Nils Grimsmo has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).
-
Publication number: 20240135835Abstract: Systems and methods for augmented-reality tutoring can utilize optical character recognition, natural language processing, and/or augmented-reality rendering for providing real-time notifications for completing a determined task. The systems and methods can include utilizing one or more machine-learned models trained for quantitative reasoning and can include providing a plurality of different user interface elements at different times.Type: ApplicationFiled: October 18, 2022Publication date: April 25, 2024Inventors: Jessica Lee, David Trotter Oleson, Fabian Roth, Nils Grimsmo
-
Patent number: 11943181Abstract: A computer-implemented method is described. The method includes a computing system receiving an item of digital content from a user device. The computing system generates one or more labels that indicate attributes of the item of digital content. The computing system also generates one or more conversational replies to the item of digital content based on the one or more labels that indicate attributes of the item of digital content. The method also includes the computing system selecting a conversational reply from among the one or more conversational replies and providing the conversational reply for output to the user device.Type: GrantFiled: July 7, 2022Date of Patent: March 26, 2024Assignee: GOOGLE LLCInventors: Ibrahim Badr, Aayush Kumar, Goekhan Hasan Bakir, Nils Grimsmo, Bianca Madalina Buisman
-
Patent number: 11921988Abstract: Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.Type: GrantFiled: July 19, 2021Date of Patent: March 5, 2024Assignee: GOOGLE LLCInventors: Ibrahim Badr, Gokhan H. Bakir, Roland Peter Kehl, Nils Grimsmo
-
Publication number: 20230394816Abstract: Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.Type: ApplicationFiled: August 21, 2023Publication date: December 7, 2023Inventors: Ibrahim Badr, Nils Grimsmo, Gökhan Bakir
-
Patent number: 11734926Abstract: Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.Type: GrantFiled: November 10, 2020Date of Patent: August 22, 2023Assignee: GOOGLE LLCInventors: Ibrahim Badr, Nils Grimsmo, Gökhan Bakir
-
Patent number: 11688191Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.Type: GrantFiled: September 9, 2022Date of Patent: June 27, 2023Assignee: GOOGLE LLCInventors: Ibrahim Badr, Nils Grimsmo, Gokhan H. Bakir, Kamil Anikiej, Aayush Kumar, Viacheslav Kuznetsov
-
Publication number: 20230004597Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.Type: ApplicationFiled: September 9, 2022Publication date: January 5, 2023Inventors: Ibrahim Badr, Nils Grimsmo, Gokhan H. Bakir, Kamil Anikiej, Aayush Kumar, Viacheslav Kuznetsov
-
Patent number: 11514035Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining query refinements using search data. In one aspect, a method includes receiving a first query and a second query each comprising one or more n-grams for a user session, determining a first set of query refinements for the first query, determining a second set of query refinements from the first set of query refinements, each query refinement in the second set of query refinements including at least one n-gram that is similar to an n-gram from the first query and at least on n-gram that is similar to an n-gram from the second query, scoring each query refinement in the second set of query refinements, selecting a third query from a group consisting of the second set of query refinements and the second query, and providing the third query as input to a search operation.Type: GrantFiled: June 1, 2020Date of Patent: November 29, 2022Assignee: GOOGLE LLCInventors: Matthias Heiler, Behshad Behzadi, Evgeny A. Cherepanov, Nils Grimsmo, Aurelien Boffy, Alessandro Agostini, Karoly Csalogany, Fredrik Bergenlid, Marcin M. Nowak-Przygodzki
-
Publication number: 20220337539Abstract: A computer-implemented method is described. The method includes a computing system receiving an item of digital content from a user device. The computing system generates one or more labels that indicate attributes of the item of digital content. The computing system also generates one or more conversational replies to the item of digital content based on the one or more labels that indicate attributes of the item of digital content. The method also includes the computing system selecting a conversational reply from among the one or more conversational replies and providing the conversational reply for output to the user device.Type: ApplicationFiled: July 7, 2022Publication date: October 20, 2022Inventors: Ibrahim Badr, Aayush Kumar, Goekhan Hasan Bakir, Nils Grimsmo, Bianca Madalina Buisman
-
Patent number: 11442983Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.Type: GrantFiled: December 31, 2019Date of Patent: September 13, 2022Assignee: GOOGLE LLCInventors: Ibrahim Badr, Nils Grimsmo, Gokhan H. Bakir, Kamil Anikiej, Aayush Kumar, Viacheslav Kuznetsov
-
Patent number: 11405340Abstract: A computer-implemented method is described. The method includes a computing system receiving an item of digital content from a user device. The computing system generates one or more labels that indicate attributes of the item of digital content. The computing system also generates one or more conversational replies to the item of digital content based on the one or more labels that indicate attributes of the item of digital content. The method also includes the computing system selecting a conversational reply from among the one or more conversational replies and providing the conversational reply for output to the user device.Type: GrantFiled: September 14, 2020Date of Patent: August 2, 2022Assignee: GOOGLE LLCInventors: Ibrahim Badr, Aayush Kumar, Goekhan Hasan Bakir, Nils Grimsmo, Bianca Madalina Buisman
-
Publication number: 20210342047Abstract: Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.Type: ApplicationFiled: July 19, 2021Publication date: November 4, 2021Inventors: Ibrahim Badr, Gokhan H. Bakir, Roland Peter Kehl, Nils Grimsmo
-
Patent number: 11086493Abstract: Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.Type: GrantFiled: June 18, 2020Date of Patent: August 10, 2021Assignee: Google LLCInventors: Ibrahim Badr, Gokhan H. Bakir, Roland Peter Kehl, Nils Grimsmo
-
Publication number: 20210056310Abstract: Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.Type: ApplicationFiled: November 10, 2020Publication date: February 25, 2021Inventors: Ibrahim Badr, Nils Grimsmo, Gökhan Bakir
-
Patent number: 10922321Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving a query provided from a user device, and determining that the query implicitly references some entity, and in response: obtaining an approximate location and an approximate orientation of the user device, obtaining a set of entities including one or more entities, each entity in the set of entities being at least partially within a range that is based on the approximate location and that includes an extended portion based on the approximate orientation, selecting an entity from the set of entities, and providing a revised query based on the query and the entity, the revised query explicitly referencing the entity.Type: GrantFiled: January 7, 2019Date of Patent: February 16, 2021Assignee: Google LLCInventors: Nils Grimsmo, Behshad Behzadi, Fredrik Bergenlid
-
Publication number: 20200412673Abstract: A computer-implemented method is described. The method includes a computing system receiving an item of digital content from a user device. The computing system generates one or more labels that indicate attributes of the item of digital content. The computing system also generates one or more conversational replies to the item of digital content based on the one or more labels that indicate attributes of the item of digital content. The method also includes the computing system selecting a conversational reply from among the one or more conversational replies and providing the conversational reply for output to the user device.Type: ApplicationFiled: September 14, 2020Publication date: December 31, 2020Inventors: Ibrahim Badr, Aayush Kumar, Goekhan Hasan Bakir, Nils Grimsmo, Bianca Madalina Buisman
-
Patent number: 10867180Abstract: Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.Type: GrantFiled: March 21, 2019Date of Patent: December 15, 2020Assignee: GOOGLE LLCInventors: Ibrahim Badr, Nils Grimsmo, Gökhan Bakir
-
Publication number: 20200319765Abstract: Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.Type: ApplicationFiled: June 18, 2020Publication date: October 8, 2020Inventors: Ibrahim Badr, Gokhan H. Bakir, Roland Peter Kehl, Nils Grimsmo
-
Patent number: 10778619Abstract: A computer-implemented method is described. The method includes a computing system receiving an item of digital content from a user device. The computing system generates one or more labels that indicate attributes of the item of digital content. The computing system also generates one or more conversational replies to the item of digital content based on the one or more labels that indicate attributes of the item of digital content. The method also includes the computing system selecting a conversational reply from among the one or more conversational replies and providing the conversational reply for output to the user device.Type: GrantFiled: May 14, 2019Date of Patent: September 15, 2020Assignee: GOOGLE LLCInventors: Ibrahim Badr, Aayush Kumar, Goekhan Hasan Bakir, Nils Grimsmo, Bianca Madalina Buisman
-
Publication number: 20200250227Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.Type: ApplicationFiled: December 31, 2019Publication date: August 6, 2020Inventors: Ibrahim Badr, Nils Grimsmo, Gokhan H. Bakir, Kamil Anikiej, Aayush Kumar, Viacheslav Kuznetsov