Patents by Inventor Shumin Zhai

Shumin Zhai has filed for patents to protect the following inventions. This listing includes patent applications that are pending as well as patents that have already been granted by the United States Patent and Trademark Office (USPTO).

  • Publication number: 20240029728
    Abstract: Implementations described herein generally relate to generating a modification selectable element that may be provided for presentation to a user in a smart dictation session with an automated assistant. The modification selectable element may, when selected, cause a transcription, that includes textual data generated based on processing audio data that captures a spoken utterance and that is automatically arranged, to be modified. The transcription may be automatically arranged to include spacing, punctuation, capitalization, indentations, paragraph breaks, and/or other arrangement operations that are not specified by the user in providing the spoken utterance. Accordingly, a subsequent selection of the modification selectable element may cause these automatic arrangement operation(s), and/or the textual data locationally proximate to these automatic arrangement operation(s), to be modified.
    Type: Application
    Filed: September 2, 2022
    Publication date: January 25, 2024
    Inventors: Nicolo D'Ercole, Shumin Zhai, Swante Scholz, Mehek Sharma, Adrien Olczak, Akshay Kannan, Alvin Abdagic, Julia Proskurnia, Viesturs Zarins
  • Publication number: 20230360444
    Abstract: An example method includes initiating, by a computing device, a fingerprint sensing operation that is associated with a fingerprint sensor, outputting, by the computing device and over a defined period of time, a sequence of haptic signals with varying intensity, wherein each haptic signal in the sequence has an intensity that is different than a respective intensity of any haptic signal that was previously output in the sequence, obtaining, by the computing device and from the fingerprint sensor, fingerprint data associated with a fingerprint of the finger of the user, and responsive to determining that the finger of the user is still positioned at the fingerprint sensor upon completion of the fingerprint sensing operation, outputting, by the computing device, a discrete haptic signal indicating a successful completion of the fingerprint sensing operation.
    Type: Application
    Filed: September 22, 2020
    Publication date: November 9, 2023
    Inventors: Scott Jenson, Jianxun Wang, Debanjan Mukherjee, Shumin Zhai
  • Patent number: 11727212
    Abstract: A computing device outputs for display at a presence-sensitive display, a graphical keyboard comprising a plurality of keys, receives an indication of at least one gesture to select a group of keys of the plurality of keys, and determines at least one characteristic associated with the at least one gesture to select the group of keys of the plurality of keys. The computing device modifies a spatial model based at least in part on the at least one characteristic and determines a candidate word based at least in part on data provided by the spatial model and a language model, wherein the spatial model provides data based at least in part on the indication of the at least one gesture and wherein the language model provides data based at least in part on a lexicon. The computing device outputs for display at the presence-sensitive display, the candidate word.
    Type: Grant
    Filed: April 6, 2022
    Date of Patent: August 15, 2023
    Assignee: Google LLC
    Inventors: Shumin Zhai, Ciprian Ioan Chelba
  • Patent number: 11698699
    Abstract: A method includes detecting a touch input by a user at a position on a virtual keyboard. The method further includes applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heatmap of prior touch inputs by the user. The method further includes applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys. The method further includes selecting a particular key of the one or more first and second candidate keys as indicated by the touch input.
    Type: Grant
    Filed: May 3, 2022
    Date of Patent: July 11, 2023
    Assignee: Google LLC
    Inventors: Xu Liu, Shumin Zhai, Adrian Pittman, Billy Dou
  • Patent number: 11573698
    Abstract: In some examples, a computing device includes at least one processor; and at least one module, operable by the at least one processor to: output, for display at an output device, a graphical keyboard; receive an indication of a gesture detected at a location of a presence-sensitive input device, wherein the location of the presence-sensitive input device corresponds to a location of the output device that outputs the graphical keyboard; determine, based on at least one spatial feature of the gesture that is processed by the computing device using a neural network, at least one character string, wherein the at least one spatial feature indicates at least one physical property of the gesture; and output, for display at the output device, based at least in part on the processing of the at least one spatial feature of the gesture using the neural network, the at least one character string.
    Type: Grant
    Filed: September 8, 2021
    Date of Patent: February 7, 2023
    Assignee: Google LLC
    Inventors: Shumin Zhai, Thomas Breuel, Ouais Alsharif, Yu Ouyang, Francoise Beaufays, Johan Schalkwyk
  • Publication number: 20220261135
    Abstract: A method includes detecting a touch input by a user at a position on a virtual keyboard. The method further includes applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heatmap of prior touch inputs by the user. The method further includes applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys. The method further includes selecting a particular key of the one or more first and second candidate keys as indicated by the touch input.
    Type: Application
    Filed: May 3, 2022
    Publication date: August 18, 2022
    Applicant: Google LLC
    Inventors: Xu LIU, Shumin ZHAI, Adrian PITTMAN, Billy DOU
  • Publication number: 20220229988
    Abstract: A computing device outputs for display at a presence-sensitive display, a graphical keyboard comprising a plurality of keys, receives an indication of at least one gesture to select a group of keys of the plurality of keys, and determines at least one characteristic associated with the at least one gesture to select the group of keys of the plurality of keys. The computing device modifies a spatial model based at least in part on the at least one characteristic and determines a candidate word based at least in part on data provided by the spatial model and a language model, wherein the spatial model provides data based at least in part on the indication of the at least one gesture and wherein the language model provides data based at least in part on a lexicon. The computing device outputs for display at the presence-sensitive display, the candidate word.
    Type: Application
    Filed: April 6, 2022
    Publication date: July 21, 2022
    Applicant: Google LLC
    Inventors: Shumin Zhai, Ciprian Ioan Chelba
  • Patent number: 11379663
    Abstract: A computing device outputs a keyboard for display, receives an indication of a first gesture to select a first sequence of one or more keys, determines a set of candidate strings based in part on the first sequence of keys, and outputs for display at least one of the set of candidate strings. The computing device receives an indication of a second gesture to select a second sequence of one or more keys, and determines that characters associated with the second sequence of keys are included in a first candidate word based at least in part on the set of candidate strings, or are included in a second candidate word not based on the first sequence of keys. The computing device modifies the set of candidate strings based at least in part on the determination and outputs for display at least one of the modified candidate strings.
    Type: Grant
    Filed: March 4, 2021
    Date of Patent: July 5, 2022
    Assignee: Google LLC
    Inventors: Yu Ouyang, Shumin Zhai, Xiaojun Bi
  • Patent number: 11347352
    Abstract: A method includes detecting a touch input by a user at a position on a virtual keyboard. The method further includes applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heatmap of prior touch inputs by the user. The method further includes applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys. The method further includes selecting a particular key of the one or more first and second candidate keys as indicated by the touch input.
    Type: Grant
    Filed: October 25, 2020
    Date of Patent: May 31, 2022
    Assignee: Google LLC
    Inventors: Xu Liu, Shumin Zhai, Adrian Pittman, Billy Dou
  • Patent number: 11334717
    Abstract: A computing device outputs for display at a presence-sensitive display, a graphical keyboard comprising a plurality of keys, receives an indication of at least one gesture to select a group of keys of the plurality of keys, and determines at least one characteristic associated with the at least one gesture to select the group of keys of the plurality of keys. The computing device modifies a spatial model based at least in part on the at least one characteristic and determines a candidate word based at least in part on data provided by the spatial model and a language model, wherein the spatial model provides data based at least in part on the indication of the at least one gesture and wherein the language model provides data based at least in part on a lexicon. The computing device outputs for display at the presence-sensitive display, the candidate word.
    Type: Grant
    Filed: November 14, 2019
    Date of Patent: May 17, 2022
    Assignee: Google LLC
    Inventors: Shumin Zhai, Ciprian Ioan Chelba
  • Publication number: 20220129137
    Abstract: A method includes detecting a touch input by a user at a position on a virtual keyboard. The method further includes applying a spatial model to map the touch input to one or more first candidate keys of the virtual keyboard, wherein the spatial model maps the touch input to the one or more first candidate keys based on a heatmap of prior touch inputs by the user. The method further includes applying a language model to map the touch input to one or more second candidate keys of the virtual keyboard, wherein the language model maps the touch input to the one or more second candidate keys based on comparing the position of the touch input to one or more predicted next keys. The method further includes selecting a particular key of the one or more first and second candidate keys as indicated by the touch input.
    Type: Application
    Filed: October 25, 2020
    Publication date: April 28, 2022
    Applicant: Google LLC
    Inventors: Xu LIU, Shumin ZHAI, Adrian PITTMAN, Billy DOU
  • Patent number: 11237660
    Abstract: A method includes a processor of an electronic device receiving first input signals from a first sensor in response to user contact at a first edge of the device and second input signals from a second sensor in response to user contact at a second edge of the electronic device. The first and second sensors are covered by a housing of the device. The processor determines an external context of the device based on analysis of the first input signals and the second input signals. The determined external context indicates at least a position of the device relative to a user or an orientation of the device relative to a user. Responsive to determining the external context, the electronic device executes a particular user input action.
    Type: Grant
    Filed: April 27, 2020
    Date of Patent: February 1, 2022
    Assignee: Google LLC
    Inventors: Tyler Reed Kugler, Alexey Polyudov, Seungyon Lee, Yun-Ling Lee, Philip Quinn, Kishore Sundara-Rajan, Shumin Zhai, Debanjan Mukherjee, James B. Miller, Isaac William Reynolds
  • Publication number: 20210405868
    Abstract: In some examples, a computing device includes at least one processor; and at least one module, operable by the at least one processor to: output, for display at an output device, a graphical keyboard; receive an indication of a gesture detected at a location of a presence-sensitive input device, wherein the location of the presence-sensitive input device corresponds to a location of the output device that outputs the graphical keyboard; determine, based on at least one spatial feature of the gesture that is processed by the computing device using a neural network, at least one character string, wherein the at least one spatial feature indicates at least one physical property of the gesture; and output, for display at the output device, based at least in part on the processing of the at least one spatial feature of the gesture using the neural network, the at least one character string.
    Type: Application
    Filed: September 8, 2021
    Publication date: December 30, 2021
    Applicant: Google LLC
    Inventors: Shumin Zhai, Thomas Breuel, Ouais Alsharif, Yu Ouyang, Francoise Beaufays, Johan Schalkwyk
  • Patent number: 11150804
    Abstract: In some examples, a computing device includes at least one processor; and at least one module, operable by the at least one processor to: output, for display at an output device, a graphical keyboard; receive an indication of a gesture detected at a location of a presence-sensitive input device, wherein the location of the presence-sensitive input device corresponds to a location of the output device that outputs the graphical keyboard; determine, based on at least one spatial feature of the gesture that is processed by the computing device using a neural network, at least one character string, wherein the at least one spatial feature indicates at least one physical property of the gesture; and output, for display at the output device, based at least in part on the processing of the at least one spatial feature of the gesture using the neural network, the at least one character string.
    Type: Grant
    Filed: April 30, 2020
    Date of Patent: October 19, 2021
    Assignee: Google LLC
    Inventors: Shumin Zhai, Thomas Breuel, Ouais Alsharif, Yu Ouyang, Francoise Beaufays, Johan Schalkwyk
  • Publication number: 20210192135
    Abstract: A computing device outputs a keyboard for display, receives an indication of a first gesture to select a first sequence of one or more keys, determines a set of candidate strings based in part on the first sequence of keys, and outputs for display at least one of the set of candidate strings. The computing device receives an indication of a second gesture to select a second sequence of one or more keys, and determines that characters associated with the second sequence of keys are included in a first candidate word based at least in part on the set of candidate strings, or are included in a second candidate word not based on the first sequence of keys. The computing device modifies the set of candidate strings based at least in part on the determination and outputs for display at least one of the modified candidate strings.
    Type: Application
    Filed: March 4, 2021
    Publication date: June 24, 2021
    Applicant: Google LLC
    Inventors: Yu Ouyang, Shumin Zhai, Xiaojun Bi
  • Patent number: 10977440
    Abstract: A computing device outputs a keyboard for display, receives an indication of a first gesture to select a first sequence of one or more keys, determines a set of candidate strings based in part on the first sequence of keys, and outputs for display at least one of the set of candidate strings. The computing device receives an indication of a second gesture to select a second sequence of one or more keys, and determines that characters associated with the second sequence of keys are included in a first candidate word based at least in part on the set of candidate strings, or are included in a second candidate word not based on the first sequence of keys. The computing device modifies the set of candidate strings based at least in part on the determination and outputs for display at least one of the modified candidate strings.
    Type: Grant
    Filed: July 12, 2017
    Date of Patent: April 13, 2021
    Assignee: Google LLC
    Inventors: Yu Ouyang, Shumin Zhai, Xiaojun Bi
  • Patent number: 10846602
    Abstract: A computing device is described that receives first input, at an initial time, of a first textual character and a second input, at a subsequent time, of a second textual character. The computing device determines, based on the first and second textual characters, a first character sequence that does not include a space character between the first and second textual characters and a second character sequence that includes the space character between the first and second textual characters. The computing device determines a first score associated with the first character sequence and a second score associated with the second character sequence. The computing device adjusts, based on a duration of time between the initial and subsequent times, the second score to determine a third score, and responsive to determining that the third score exceeds the first score, the computing device outputs the second character sequence.
    Type: Grant
    Filed: August 8, 2019
    Date of Patent: November 24, 2020
    Assignee: Google LLC
    Inventors: Thomas Deselaers, Daniel Martin Keysers, Abraham Murray, Shumin Zhai
  • Publication number: 20200257447
    Abstract: In some examples, a computing device includes at least one processor; and at least one module, operable by the at least one processor to: output, for display at an output device, a graphical keyboard; receive an indication of a gesture detected at a location of a presence-sensitive input device, wherein the location of the presence-sensitive input device corresponds to a location of the output device that outputs the graphical keyboard; determine, based on at least one spatial feature of the gesture that is processed by the computing device using a neural network, at least one character string, wherein the at least one spatial feature indicates at least one physical property of the gesture; and output, for display at the output device, based at least in part on the processing of the at least one spatial feature of the gesture using the neural network, the at least one character string.
    Type: Application
    Filed: April 30, 2020
    Publication date: August 13, 2020
    Applicant: Google LLC
    Inventors: Shumin Zhai, Thomas Breuel, Ouais Alsharif, Yu Ouyang, Francoise Beaufays, Johan Schalkwyk
  • Publication number: 20200257391
    Abstract: A method includes a processor of an electronic device receiving first input signals from a first sensor in response to user contact at a first edge of the device and second input signals from a second sensor in response to user contact at a second edge of the electronic device. The first and second sensors are covered by a housing of the device. The processor determines an external context of the device based on analysis of the first input signals and the second input signals. The determined external context indicates at least a position of the device relative to a user or an orientation of the device relative to a user. Responsive to determining the external context, the electronic device executes a particular user input action.
    Type: Application
    Filed: April 27, 2020
    Publication date: August 13, 2020
    Inventors: Tyler Reed Kugler, Alexey Polyudov, Seungyon Lee, Yun-Ling Lee, Philip Quinn, Kishore Sundara-Rajan, Shumin Zhai, Debanjan Mukherjee, James B. Miller, Isaac William Reynolds
  • Patent number: 10671281
    Abstract: In some examples, a computing device includes at least one processor; and at least one module, operable by the at least one processor to: output, for display at an output device, a graphical keyboard; receive an indication of a gesture detected at a location of a presence-sensitive input device, wherein the location of the presence-sensitive input device corresponds to a location of the output device that outputs the graphical keyboard; determine, based on at least one spatial feature of the gesture that is processed by the computing device using a neural network, at least one character string, wherein the at least one spatial feature indicates at least one physical property of the gesture; and output, for display at the output device, based at least in part on the processing of the at least one spatial feature of the gesture using the neural network, the at least one character string.
    Type: Grant
    Filed: January 30, 2019
    Date of Patent: June 2, 2020
    Assignee: Google LLC
    Inventors: Shumin Zhai, Thomas Breuel, Ouais Alsharif, Yu Ouyang, Francoise Beaufays, Johan Schalkwyk