Methods and systems for detecting and processing speech signals

- Google

Provided are methods, systems, and apparatuses for detecting, processing, and responding to audio signals, including speech signals, within a designated area or space. A platform for multiple media devices connected via a network is configured to process speech, such as voice commands, detected at the media devices, and respond to the detected speech by causing the media devices to simultaneously perform one or more requested actions. The platform is capable of scoring the quality of a speech request, handling speech requests from multiple end points of the platform using a centralized processing approach, a de-centralized processing approach, or a combination thereof, and also manipulating partial processing of speech requests from multiple end points into a coherent whole when necessary.

Skip to: Description  ·  Claims  ·  References Cited  · Patent History  ·  Patent History
Description
CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 15/052,426, entitled “Method and System for Detecting and Processing Speech Signals,” filed Feb. 24, 2016, of which is hereby incorporated by reference in its entirety.

BACKGROUND

Media data (e.g., audio/video content) is sometimes shared between multiple modules on a network. To get the most out of such media sharing arrangements, it is desirous to have a platform that is capable of processing such media data from the multiple modules simultaneously.

SUMMARY

This Summary introduces a selection of concepts in a simplified form in order to provide a basic understanding of some aspects of the present disclosure. This Summary is not an extensive overview of the disclosure, and is not intended to identify key or critical elements of the disclosure or to delineate the scope of the disclosure. This Summary merely presents some of the concepts of the disclosure as a prelude to the Detailed Description provided below.

The present disclosure generally relates to methods and systems for processing audio signals. More specifically, aspects of the present disclosure relate to detecting and processing speech signals from multiple end points simultaneously.

One embodiment of the present disclosure relates to a method comprising: detecting, at one or more data modules in a group of data modules in communication with one another over a network, an activation command; computing, for each of the one or more data modules, a score for the detected activation command; receiving audio data from each detecting data module having a computed score above a threshold; sending a request to a server in communication with the group of data modules over the network, wherein the request includes the audio data received from each of the detecting data modules having a computed score above the threshold; receiving from the server, in response to the sent request, audio data associated with a requested action; and communicating the requested action to each of the data modules in the group of data modules.

In another embodiment, the method further comprises: combining the audio data received from each of the detecting data modules having a computed score above the threshold; and generating the request to the server based on the combined audio data.

In another embodiment, the method further comprises, in response to detecting the activation command, muting a loudspeaker of each data module in the group.

In yet another embodiment, the method further comprises activating a microphone of each detecting data module having a computed score above the threshold.

In still another embodiment, the method further comprises causing each of the data modules in the group to playout an audible confirmation of the requested action communicated to each of the data modules.

Another embodiment of the present disclosure relates to a system comprising a group of data modules in communication with one another over a network, where each of the data modules is configured to: in response to detecting an activation command, compute a score for the detected activation command; determine whether the computed score for the activation command is higher than a threshold number of computed scores for the activation command received from other detecting data modules in the group; in response to determining that the computed score for the activation command is higher than the threshold number of computed scores received from the other detecting data modules, send audio data recorded by the data module to a server in communication with the group of data modules over the network; receive from the server, in response to the sent audio data, a requested action; determine a confidence level for the requested action received from the server; and perform the requested action based on a determination that the confidence level determined by the data module is higher than confidence levels determined by a threshold number of other data modules that received the requested action from the server.

In another embodiment, each of the data modules in the system is configured to, in response to computing the score for the detected activation command, send the computed score to each of the other data modules in the group.

In another embodiment, each of the data modules in the system is configured to receive, from other detecting data modules in the group, scores for the activation command computed by the other detecting data modules.

In another embodiment, each of the data modules in the system is configured to broadcast the determined confidence level to the other data modules in the group that received the requested action from the server.

In another embodiment, each of the data modules in the system is configured to: compare the confidence level determined by the data module to confidence levels broadcasted by the other data modules in the group that received the requested action from the server; and determine, based on the comparison, that the confidence level determined by the data module is higher than the confidence levels determined by the threshold number of other data modules that received the requested action from the server.

In yet another embodiment, each of the data modules in the system is configured to, in response to determining that the confidence level determined by the data module is higher than the confidence levels determined by the threshold number of other data modules, playout an audible confirmation of the request action received from the server.

In still another embodiment, each of the data modules in the system is configured to compute a score for the detected activation command based on one or more of the following: a power of a signal received at the data module for the activation command; a determined location of a source of the activation command relative to the data module; and whether the detected activation command corresponds to a previously stored activation command.

In one or more other embodiments, the methods and systems described herein may optionally include one or more of the following additional features: the computed score for the activation command detected at a data module is based on one or more of a power of a signal received at the data module for the activation command, a determined location of a source of the activation command relative to the data module, and whether the detected activation command corresponds to a previously stored activation command; the audio data received from each detecting data module having a computed score above the threshold includes speech data captured and recorded by the data module; the speech data captured and recorded by the data module is associated with a speech command generated by a user; the speech data captured by each data module with an activated microphone is associated with a portion of a speech command generated by a user; the audio data recorded by the data module includes speech data recorded by the data module; the speech data recorded by the data module is associated with a speech command generated by a user; the confidence level for the requested action is determined based on an audio quality measurement for the audio data recorded by the data module and sent to the server, and/or the requested action received from the server is based on audio data recorded by a plurality of the other detecting data modules having computed scores higher than the threshold number of computed scores.

It should be noted that embodiments of some or all of the processor and memory systems disclosed herein may also be configured to perform some or all of the method embodiments disclosed above. In addition, embodiments of some or all of the methods disclosed above may also be represented as instructions embodied on transitory or non-transitory processor-readable storage media such as optical or magnetic memory or represented as a propagated signal provided to a processor or data processing device via a communication network such as an Internet or telephone connection.

Further scope of applicability of the methods and systems of the present disclosure will become apparent from the Detailed Description given below. However, it should be understood that the Detailed Description and specific examples, while indicating embodiments of the methods and systems, are given by way of illustration only, since various changes and modifications within the spirit and scope of the concepts disclosed herein will become apparent to those skilled in the art from this Detailed Description.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, features, and characteristics of the present disclosure will become more apparent to those skilled in the art from a study of the following Detailed Description in conjunction with the appended claims and drawings, all of which form a part of this specification. In the drawings:

FIG. 1 is a block diagram illustrating an example content management system and surrounding network environment according to one or more embodiments described herein.

FIG. 2 is a flowchart illustrating an example method for detecting, processing, and responding to speech signals from multiple end points according to one or more embodiments described herein.

FIG. 3 is a block diagram illustrating example components and data flows for detecting a speech command in a multi-device content management system according to one or more embodiments described herein.

FIG. 4 is a block diagram illustrating example components and data flows for assessing quality of a detected speech command in a multi-device content management system according to one or more embodiments described herein.

FIG. 5 is a block diagram illustrating example components and data flows for activating a device based on a detected speech command in a multi-device content management system according to one or more embodiments described herein.

FIG. 6 is a block diagram illustrating example components and data flows for processing a detected speech command in a multi-device content management system according to one or more embodiments described herein.

FIG. 7 is a block diagram illustrating example components and data flows for responding to a speech command in a multi-device content management system according to one or more embodiments described herein.

FIG. 8 is a block diagram illustrating an example computing device arranged for detecting and processing speech signals from multiple end points simultaneously according to one or more embodiments described herein.

The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of what is claimed in the present disclosure.

In the drawings, the same reference numerals and any acronyms identify elements or acts with the same or similar structure or functionality for ease of understanding and convenience. The drawings will be described in detail in the course of the following Detailed Description.

DETAILED DESCRIPTION

Various examples and embodiments of the methods and systems of the present disclosure will now be described. The following description provides specific details for a thorough understanding and enabling description of these examples. One skilled in the relevant art will understand, however, that one or more embodiments described herein may be practiced without many of these details. Likewise, one skilled in the relevant art will also understand that one or more embodiments of the present disclosure can include other features not described in detail herein. Additionally, some well-known structures or functions may not be shown or described in detail below, so as to avoid unnecessarily obscuring the relevant description.

Embodiments of the present disclosure relate to methods, systems, and apparatuses for detecting, processing, and responding to audio (e.g., speech) within an area or space (e.g., a room). For example, in accordance with at least one embodiment, a platform for multiple media devices connected via a network may be configured to process speech (e.g., voice commands) detected at the media devices, and respond to the detected speech by causing the media devices to simultaneously perform one or more requested actions.

As will be described in greater detail below, the methods and systems of the present disclosure use a distributive approach for handling voice commands by considering input from multiple end points of the platform. Such end points may be, for example, independent data modules (e.g., media and/or audio devices such as, for example, loudspeakers) connected to one another via a wired or wireless network (e.g., Wi-Fi, Ethernet, etc.).

The methods and systems described herein utilize a flexible architecture in which each data module (e.g., loudspeaker) plays a unique role (e.g., has particular responsibilities, privileges, and/or capabilities) in detecting, processing, and responding to speech commands (e.g., generated by a user). The flexibility of the architecture is partly based on the ability of the data modules to dynamically switch between different roles (e.g., operating roles) while the system is in active operation.

Among numerous other advantages, features, and functionalities that will be described in greater detail herein, the methods and systems of the present disclosure are capable of scoring the quality of a speech request (e.g., voice command, speech command, etc.), handling speech requests from multiple end points using a centralized processing approach, a de-centralized processing approach, or a combination thereof, and also manipulating partial processing of speech requests from multiple end points into a coherent whole when necessary.

For example, in a scenario involving multiple data modules (e.g., loudspeakers), where each data module has a set of microphones (e.g., microphone array), each data module may compute (e.g., determine) a score for audio data (e.g., speech command, activation command, etc.) it records. In the following description, the score computed by a data module may be referred to as a “Hot Word” score for the data module. The computed Hot Word scores may then be used by the system to evaluate which of the data modules received the best signal. In accordance with one or more embodiments, the Hot Word score computed by each of the data modules may be based on, for example, one or more of the following:

(i) Power of the signal. For example, the power of the signal received at the data module for the speech command may be compared to the power of the signal received prior to the speech command.

(ii) Score of a Hot Word recognizer/detector module (which, for example, might be based on or utilize neural network concepts). For example, in accordance with at least one embodiment, the audio data received or recorded at a given data module may be fed to a Hot Word detector, which may be configured to determine whether the audio data corresponds to a known (e.g., stored) Hot Word. For example, the Hot Word detector may utilize a neural network (NN) or a deep neural network (DNN), which takes features of the input audio data and determines (e.g., identifies, assesses, evaluates, etc.) whether there are any occurrences of a Hot Word. If a Hot Word is found to be present in the audio data, then the detector may, for example, set a flag. In accordance with at least one embodiment, the Hot Word detector may be configured to generate a score for any detection of a Hot Word that is made by the detector. The score may, for example, reflect a confidence of the NN or DNN with regard to the detection. For example, the higher the score, the more confident the network is that a Hot Word is present in the audio data. In accordance with one or more embodiments, the output of the DNN may be a likelihood (e.g., probability) of the Hot Word being present in the audio data recorded at the data module. The determined likelihood may be compared to a threshold (e.g., a likelihood threshold, which may be predetermined and/or dynamically adaptable or adjustable based on, for example, network conditions, scores calculated for other nearby data modules, some combination thereof, and the like), and if the determined likelihood is at or above the threshold then a flag may be set to indicate the detection of a Hot Word. The threshold may be set so as to achieve or maintain, for example, a target false-detection versus miss-detection rate. As will be described in greater detail herein, if a Hot Word detection confidence is higher for a particular one of the data modules, it intuitively follows that the module in question will likely have a higher chance of correctly recognizing the command query that follows the detected Hot Word.

(iii) Location of the user relative to the data module. For example, by using a localizer (which, for example, may be part of a beamformer, or may be a standalone module) the angle of the sound source may be obtained. In another example, the angles provided by different data modules may be triangulated to estimate the position of the user (this is based on the assumption that the positions of the data modules are known).

(iv) Additional processing performed on the audio (e.g., combining all microphone array outputs using a beamformer, applying noise suppression/cancellation, gain control, echo suppression/cancellation, etc.)

In accordance with one or more embodiments, the system of the present disclosure may be configured to handle speech requests from multiple end points (e.g., data modules) using a centralized processing approach, a de-centralized processing approach, or an approach based on a combination thereof. For example, in accordance with at least one embodiment, audio data (e.g., speech data) may be collected from all relevant sources (e.g., end points) in the system and the collected audio data sent to one centralized processor (e.g., which may be one of the data modules in a group of data modules, as will be further described below). The centralized processor may determine (e.g., identify, select, etc.), based on scores associated with the audio data received from each of the sources, one or more of the sources that recorded the highest quality audio data (e.g., the processor may determine the sources that have scores higher than the scores associated with a threshold number of other sources). The centralized processor may send the audio data received from the sources having the highest scores to a server (e.g., a server external to the system of data modules) for further processing. The centralized processor may then receive a response from the server and take appropriate action in accordance with the response.

In accordance with at least one other embodiment, each data module in a group of data modules may determine its own Hot Word score and broadcast its score to the other data modules in the group. If a data module in the group determines, based on the broadcasted scores, that the data module has one of the best (e.g., highest quality) signals, then the data module may send/upload its recorded audio data (e.g., speech data relating to a command from the user) to the server (e.g., the Voice Search Back-End, further details of which will be provided below). Upon receiving a response from the server, the data module may then broadcast its confidence level of the response and wait for similar broadcasts from other data modules in the group. If the data module determines that it has one of the highest confidence levels for the response, the data module may act on the response accordingly.

For example, in accordance with at least one embodiment, when a data module detects a Hot Word, the data module generates a score for the detected Hot Word, broadcasts the score to the other data modules in the group (e.g., an Ethernet broadcast), and waits for some period of time (which may be a predetermined period of time, a period of time based on a setting that may or may not be adjustable, or the like) to receive similar broadcasts from other modules. After the designated period of time has passed, the data module has access to the scores generated by all of the other data modules in the group that have also detected the Hot Word. As such, the data module (as well as each of the other detecting data modules in the group) can then determine (e.g., rank) how well it scored with respect to the other detecting data modules. For example, if the data module determines that it has one of the top (e.g., two, three, etc.) scores for the Hot Word, the data module can decide to take action.

The system may also be capable of performing partial processing of speech commands by utilizing portions of audio data received from multiple data modules. For example, in accordance with one or more embodiments, the system may capture each part of a sentence spoken by the user from the “best” loudspeaker for that particular part. Such partial processing may be applicable, for example, when a user speaks a command while moving around within a room. A per-segment-score may be created for each data module and each word processed independently. It should be noted that because the clocks of the data modules in a given group are synchronized, the system is able to compare signal-to-noise ratio (SNR) values between speech segments.

In an example application of the methods and systems of the present disclosure, users are given the ability to play audio content available from an audio source (e.g., audio content stored on a user device, audio content associated with a URL and accessible through the user device, etc.) to any combination of audio devices that share a common wireless or wired network. For example, in the context of a multi-room house, a system of speakers may be located in each room (e.g., living room, dining room, bedroom, etc.) of the house, and the speakers forming a system for a given room may be at various locations throughout the room. In accordance with one or more embodiments described herein, audio will be played out synchronously across all of the audio devices selected by the user. It should be understood, however, that the methods and systems described herein may be applicable to any system that requires time synchronization of any data type between different modules on a network, and thus the scope of the present disclosure is not in any way limited by the example application described above.

FIG. 1 is an example content management system 100 in which one or more embodiments described herein may be implemented. Data Source 110 (e.g., a content source such as an audio source (e.g., an online streaming music or video service, a particular URL, etc.)) may be connected to Data Module 115 over a Network 105 (e.g., any kind of network including, for example, Ethernet, wireless LAN, cellular network, etc.). Content (e.g., audio, video, data, mixed media, etc.) obtained from Data Source 110 may be played out by Data Module 115 and/or transported by Data Module 115 to one or more of Data Modules 120a-120n (where “n” is an arbitrary number) over Network 125 (e.g., a wireless LAN or Ethernet). Similarly, content obtained at Data Modules 120a-120n may be played out by Data Modules 120a-120n and/or transported over Network 135 to corresponding Data Modules 130a-130m, Data Modules 140a-140p, or some combination thereof (where “m” and “p” are both arbitrary numbers). It should also be noted that Networks 125 and 135 may be the same or different networks (e.g., different WLANs within a house, one wireless network and the other a wired network, etc.).

A Control Client 150 may be in communication with Data Module 115 over Network 105. In accordance with at least one embodiment, Control Client 150 may act as a data source (e.g., Data Source 110) by mirroring local data from the Control Client to Data Module 115.

In accordance with one or more embodiments, the data modules (e.g., Data Module 115, Data Modules 120a-120n, and Data Modules 130a-130m) in the content management system 100 may be divided into groups of data modules. Each group of data modules may be divided into one or more systems, which, in turn, may include one or more individual data modules. In accordance with at least one embodiment, group and system configurations may be set by the user.

Data modules within a group may operate in accordance with different roles. For example, data modules within a group may be divided into Player Modules, Follower Modules, and Renderer Modules (sometimes referred to herein simply as “Players,” “Followers,” and “Renderers,” respectively). Example features and functionalities of the Players, Followers, and Renderers will be described in greater detail below. In accordance with at least one embodiment, the methods and systems of the present disclosure allow for multiple configurations and Player/Follower/Renderer combinations, and further allow such configurations and/or combinations to be modified on-the-fly (e.g., adaptable or adjustable by the user and/or system while the system is in operation). As is further described below, the resulting configuration (Player/Follower/Renderer) is determined based on the grouping, audio source/type, network conditions, etc.

The Player acts as “master” or a “leader” of a group of data modules (e.g., Data Module 115 may be the Player in the example group comprising Data Module 115, Data Modules 120a-120n, and Data Modules 130a-130m in the example content management system 100 shown in FIG. 1). For example, the Player may fetch (e.g., retrieve, obtain, etc.) the data (e.g., audio) from the source (e.g., Data Source 110) and forward the data out to the other data modules (e.g., loudspeakers) in the group. The source of the data obtained by the Player may be, for example, an online audio/video streaming service or website, a portable user device (e.g., cellular telephone, smartphone, personal digital assistant, tablet computer, laptop computer, smart television, etc.), a storage device containing memory for storing audio/video data (e.g., a standalone hard drive), and the like. The Player may also be configured to packetize the data obtained from the source and send raw or coded data packets over the network to the data modules in the group. In accordance with at least one embodiment, the Player may be configured to determine whether to send raw or coded data packets to the other data modules in the group based on available bandwidth of the network and/or the capabilities of each particular data module (e.g., each Follower or Renderer's capabilities). For example, it may be the case that one or more devices (e.g., loudspeakers) in the system are not capable of decoding all codecs. Also, in a scenario involving degraded (e.g., limited) network conditions (e.g., low bandwidth), it may be difficult for the Player to send raw data to the other data modules in the group. In such a scenario, the Player may instead send coded data to the other modules. In other instances, the system may be configured to re-encode the data following the initial decoding by the Player. However, it should also be understood that the data originally received by the Player is not necessarily coded in all instances (and thus there may not be an initial decoding performed by the Player).

In addition to the example features and functionalities of the Player described above, in accordance with one or more embodiments of the present disclosure, the Player may also act as a centralized processor in detecting, processing, and responding to speech commands (e.g., generated by a user). For example, as will be described in greater detail below with respect to the example arrangements illustrated in FIGS. 3-7, the Player (or “Group Leader Module”) may be configured to receive (e.g., retrieve, collect, or otherwise obtain) “Hot Word” command scores from each of the other data modules in the group, determine the data modules with the highest scores, activate or cause to activate the microphones on the data modules with the highest scores, receive audio data containing a speech command of the user from the data modules with the activated microphones, and combine the received audio data into a request that is sent to an external server for processing (e.g., interpretation). In addition, in response to sending the audio data containing the user's speech command, the Player may receive from the server a response containing a requested action corresponding to the speech command, which the Player may then fan out (e.g., distribute) to the other data modules in the group so that the requested action is performed. In accordance with one or more embodiments described herein, the response received at the Player from the server may also include audio data corresponding to the requested action, which the Player may also fan out to the other data modules in the group. Such audio data may, for example, be played out by each of the data modules in the group as an audible confirmation to the user that the user's command was received and is being acted on.

It should also be understood that a Player may also be a Follower and/or a Renderer, depending on the particulars of the group configuration.

The Follower is the head of a local system of data modules (e.g., Data Modules 120a-120n may be Followers in different systems of data modules made up of certain Data Modules 130a-130m in the example content management system 100 shown in FIG. 1). The Followers may receive data from a Player and fan (e.g., forward) out the data to the connected Renderers in their respective systems. In accordance with one or more embodiments, the Follower may receive over the network raw or coded data packets from the Player and send the data packets to the Renderers in the system. The Follower may send the packets to the Renderers in the same format as the packets are received from the Player, or the Follower may parse the packets and perform various operations (e.g., transcoding, audio processing, etc.) on the received data before re-packeting the data for sending to the connected Renderers. It should be noted that a Follower may also be a Renderer.

In accordance with at least one embodiment of the present disclosure, the Renderer is the endpoint of the data pipeline in the content management system (e.g., Data Modules 130a-130m in the example content management system 100 shown in FIG. 1). For example, the Renderer may be configured to playout the data received from the Follower that heads its respective system. The Renderer may perform additional local processing (e.g., fade in/fade out in the context of audio) on the data received from the Follower prior to playing out the data.

As described above, one or more of the data modules in the content management system may be in communication with and/or receive control commands from a control client connected to the network (e.g., Control Client 150 may be in communication with Data Module 115 over Network 105 in the example content management system 100 shown in FIG. 1). The control client is not a physical data module (e.g., not a physical loudspeaker), but instead may be a device (e.g., cellular telephone, smartphone, personal digital assistant, tablet computer, laptop computer, smart television, etc.) that can control and send messages to the Player. For example, the control client may be used to relay various control messages (e.g., play, pause, stop, volume updates, etc.) to the Player. In accordance with one or more embodiments, the control client may also act as a data source (e.g., Data Source 110) for the content management system, for example, by mirroring local data from the control client to the Player. The control client may use the same communication protocol as the data modules in the content management system.

It should be understood that the platform, architecture, and system of the present disclosure are extremely dynamic. For example, a user of the system and/or the system itself may modify the unique roles of the data modules, the specific data modules targeted for playout, the grouping of data modules, the designation of an “active” group of data modules, or some combination thereof while the system is in active operation.

In accordance with one or more embodiments of the present disclosure, the selection of a group leader (e.g., a Player Module) may be performed using a system in which each data module advertises its capabilities to a common system service, which then determines roles for each of the modules, including the election of the group leader, based on the advertised capabilities. For example, the leader selection process may be based on a unique score computed (e.g., by the common system service) for each of the data modules (e.g., loudspeakers). In accordance with at least one embodiment, this score may be computed based on one or more of the following non-limiting parameters: (i) CPU capabilities; (ii) codec availability (e.g., a select or limited number of codecs may be implemented in particular data modules); and (iii) bandwidth/latency.

FIG. 2 illustrates an example process 200 for detecting, processing, and responding to speech signals (e.g., speech commands) from multiple end points. In accordance with one or more embodiments described herein, one or more of blocks 205-240 in the example process 200 may be performed by one or more of the components in the example content management system shown in FIG. 1, and described in detail above. For example, one or more of Control Client 150, Data Module 115, Data Modules 120a-120n, and Data Modules 130a-130m in the example content management system 100 may be configured to perform one or more of the operations associated with blocks 205-240 in the example process 200 for detecting, processing, and responding to speech commands, further details of which are provided below.

It should also be noted that, in accordance with one or more embodiments, the example process 200 for detecting, processing, and responding to speech commands may be performed without one or more of blocks 205-240, and/or performed with one or more of blocks 205-240 being combined together.

At block 205, a Hot Word command (which may sometimes be referred to herein as an “activation command,” “initialization command,” or the like) may be generated (e.g., by a user) during audio playback by data modules in a group of data modules (e.g., a group of data modules comprising Data Module 115, Data Modules 120a-120n, and Data Modules 130a-130m in the example content management system 100 shown in FIG. 1).

At block 210, the data modules in the group that detect the generated Hot Word command (e.g., which may or may not be all of the data modules in the group) may determine (e.g., compute, calculate, etc.) a score for the detected command (a “Hot Word” score). For example, in accordance with at least one embodiment, the Hot Word score that may be determined by each of the data modules may be based on, for example, one or more of the following non-exhaustive and non-limiting factors: (i) power of the signal (e.g., the power of the signal received at the data module for the speech command may be compared to the power of the signal received prior to the speech command); (ii) score of a Hot Word recognizer/detector module (the details of which are described above); (iii) location of the user relative to the data module. For example, by using the localizer of a beamformer, the angle of the sound source may be obtained. In another example, the angles provided by different data modules may be triangulated to estimate the position of the user (this is based on the assumption that the positions of the data modules are known); and (iv) additional processing performed on the audio (e.g., combining all microphone array outputs using a beamformer, applying noise suppression/cancellation, gain control, echo suppression/cancellation, etc.).

At block 215, each of the data modules in the group may send its computed “Hot Word” score to a group leader data module (e.g., a Player Module, as described above). In accordance with one or more embodiments of the present disclosure, the group leader data module may act as a centralized processor of sorts in that the group leader collects (e.g., receives) the computed Hot Word scores from the other data modules in the group.

At block 220, the group leader data module may pause or mute audio playback by the other data modules in the group and determine (e.g., identify), based on the computed Hot Word scores received from the data modules at block 215, those data modules having the highest computed Hot Word scores for the Hot Word command generated at block 205. For example, the group leader data module may utilize the received Hot Word scores (at block 215) to rank or order the data modules in the group according to their corresponding scores. The group leader data module may then determine the data modules that have one of the top (e.g., two, three, etc.) scores for the Hot Word command generated at block 205. In another example, the group leader data module may determine the data modules that have Hot Word scores higher than the scores of some threshold number of the detecting data modules.

At block 225, the group leader data module may activate microphone(s) at the data module(s) in the group determined to have the highest computed scores for the Hot Word command.

At block 230, the data modules with activated microphones (from block 225) may record a generated command/request (e.g., a command/request generated by the user) and send audio data containing the recorded command/request to the group leader data module.

At block 235, the group leader data module may generate a request based on the audio data containing the recorded command/request received from the data modules with activated microphones (at block 230), and send the generated request to an external server for processing (e.g., interpretation). For example, the group leader data module may generate the request sent to the external server by combining the audio data received from the data modules. In addition, in accordance with one or more embodiments, the external server may be a back-end server (e.g., Voice Search Back-End 660 or 760 as shown in the example component and data flows in FIGS. 6 and 7, respectively) that receives the request from the group leader and is configured to interpret the combined audio data (e.g., the audio data containing the recorded command/request from, for example, the user).

At block 240, the group leader data module may receive from the external (e.g., back-end) server a response to the request sent by the group leader data module (e.g., at block 235). The group leader data module may process the received response and take appropriate control action based on the response, and/or the group leader module may distribute (e.g., fan out, transmit, etc.) the response to the other data modules in the group so that the requested action is performed. For example, in accordance with at least one embodiment, the response received at the group leader data module at block 240 may contain a requested action corresponding to the generated command/request (e.g., speech command) recorded by the data modules with activated microphones (at block 230). In another example, the response received at the group leader data module from the server (at block 240) may also include audio data corresponding to the requested action, which the group leader data module may also fan out to the other data modules in the group. Such audio data may, for example, be played out by each of the data modules in the group as an audible confirmation to the user that the user's command was received and is being acted on.

It should be noted that, in accordance with one or more embodiments of the present disclosure, one or more of the operations associated with blocks 205-240 in the example process 200 for detecting, processing, and responding to speech commands may optionally be modified and/or supplemented without loss of any of the functionalities or features described above. For example, each data module in the group of data modules may determine (e.g., calculate, compute, etc.) its own Hot Word score and broadcast its score to the other data modules in the group. If a data module in the group determines, based on the broadcasted scores, that the data module has one of the best (e.g., highest quality) signals, then the data module may send/upload its recorded audio data (e.g., speech data relating to a command from the user) to the external server for processing/interpretation (e.g., to Voice Search Back-End 660 or 760 as shown in the example component and data flows in FIGS. 6 and 7, respectively, further details of which are provided below). Upon receiving a response from the server, the data module may then broadcast its confidence level of the response and wait for similar broadcasts from other data modules in the group. If the data module determines that it has one of the highest confidence levels for the response, the data module may act on the response accordingly (e.g., perform a requested action contained in the response received from the server).

For example, in accordance with at least one embodiment, when a data module detects a Hot Word, the data module may generate a score for the detected Hot Word, broadcast the score to the other data modules in the group (e.g., an Ethernet broadcast), and wait for some period of time (which may be, for example, a predetermined period of time, a period of time based on a setting that may or may not be adjustable, or the like) to receive similar broadcasts from other data modules. After the designated period of time has passed, the data module has access to the scores generated by the other data modules in the group that have also detected the Hot Word. As such, the data module (as well as each of the other detecting data modules in the group) can then determine (e.g., rank) how well it scored with respect to the other detecting data modules. For example, if the data module determines that it has one of the top (e.g., two, three, etc.) scores for the Hot Word, the data module can decide to take action (e.g., send/upload its recorded audio data (e.g., speech data relating to a command from the user) to the external server for processing/interpretation).

It should also be noted that the system of the present disclosure may also be capable of performing partial processing of speech commands by utilizing portions of audio data received from multiple data modules. For example, in accordance with one or more embodiments, the system may capture each part of a sentence spoken by the user from the “best” loudspeaker for that particular part. Such partial processing may be applicable, for example, when a user speaks a command while moving around within a room. A per-segment-score may be created for each data module and each word processed independently. It should be noted that because the clocks of the data modules in a given group are synchronized, the system is able to compare signal-to-noise ratio (SNR) values between speech segments.

FIGS. 3-7 illustrate example components and data flows for various operations that may be performed by the content management system 100 shown in FIG. 1 (and described in detail above). In accordance with one or more embodiments of the present disclosure, a Player data module (or “Group Leader Module”) may also act as a centralized processor in detecting, processing, and responding to speech commands (e.g., generated by a user). For example, the Player (e.g., 315, 415, 515, 615, and 715 in the example arrangements shown in FIGS. 3-7, respectively) may be configured to receive “Hot Word” command scores from each of the other data modules in the group (e.g., data modules 320a-320n, 420a-420n, 520a-520n, 620a-620n, and 720a-720n in the example arrangements shown in FIGS. 3-7, respectively), determine the data modules with the highest scores, activate or cause to activate the microphones on the data modules with the highest scores, receive audio data containing a speech command of the user (e.g., 370, 470, 570, 670, and 770) from the data modules with the activated microphones, and combine the received audio data into a request that is sent to an external server (e.g., 660 and 760 in FIGS. 6 and 7, respectively) for processing (e.g., interpretation).

In addition, in response to sending the audio data containing the user's speech command, the Player (e.g., 715 in FIG. 7) may receive from the server (760) a response containing a requested action corresponding to the speech command, which the Player may then fan out (e.g., distribute) to the other data modules in the group (720a-720n) so that the requested action is performed. In accordance with one or more embodiments described herein, the response received at the Player from the server may also include audio data corresponding to the requested action, which the Player may also fan out to the other data modules in the group. Such audio data may, for example, be played out by each of the data modules in the group as an audible confirmation to the user that the user's command was received and is being acted on.

FIG. 8 is a high-level block diagram of an exemplary computer (800) that is arranged for detecting, processing, and responding to speech commands in a multi-device content management system in accordance with one or more embodiments described herein. In a very basic configuration (801), the computing device (800) typically includes one or more processors (810) and system memory (820). A memory bus (830) can be used for communicating between the processor (810) and the system memory (820).

Depending on the desired configuration, the processor (810) can be of any type including but not limited to a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor (810) can include one more levels of caching, such as a level one cache (811) and a level two cache (812), a processor core (813), and registers (814). The processor core (813) can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof. A memory controller (815) can also be used with the processor (810), or in some implementations the memory controller (815) can be an internal part of the processor (810).

Depending on the desired configuration, the system memory (820) can be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. System memory (820) typically includes an operating system (821), one or more applications (822), and program data (824). The application (822) may include a system for detecting and processing speech commands (823). In accordance with at least one embodiment of the present disclosure, the system for detecting and processing speech commands (823) is further designed to perform partial processing of speech commands by utilizing portions of audio data received from multiple data modules in a content management system (e.g., Data Module 115, Data Modules 120a-120n, and/or Data Modules 130a-130m in the example content management system 100 shown in FIG. 1 and described in detail above).

Program Data (824) may include storing instructions that, when executed by the one or more processing devices, implement a system (823) and method for detecting and processing speech commands using multiple data modules operating on a network. Additionally, in accordance with at least one embodiment, program data (824) may include network, Hot Words, and module data (825), which may relate to various statistics routinely collected from the local network on which the system (823) is operating, certain voice/speech commands that activate scoring an processing operations, as well as one or more characteristics of data modules included in a group of modules. In accordance with at least some embodiments, the application (822) can be arranged to operate with program data (824) on an operating system (821).

The computing device (800) can have additional features or functionality, and additional interfaces to facilitate communications between the basic configuration (801) and any required devices and interfaces.

System memory (820) is an example of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 800. Any such computer storage media can be part of the device (800).

The computing device (800) can be implemented as a portion of a small-form factor portable (or mobile) electronic device such as a cell phone, a smartphone, a personal data assistant (PDA), a personal media player device, a tablet computer (tablet), a wireless web-watch device, a personal headset device, an application-specific device, or a hybrid device that include any of the above functions. The computing device (800) can also be implemented as a personal computer including both laptop computer and non-laptop computer configurations.

The foregoing detailed description has set forth various embodiments of the devices and/or processes via the use of block diagrams, flowcharts, and/or examples. Insofar as such block diagrams, flowcharts, and/or examples contain one or more functions and/or operations, it will be understood by those within the art that each function and/or operation within such block diagrams, flowcharts, or examples can be implemented, individually and/or collectively, by a wide range of hardware, software, firmware, or virtually any combination thereof. In accordance with at least one embodiment, several portions of the subject matter described herein may be implemented via Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signal processors (DSPs), or other integrated formats. However, those skilled in the art will recognize that some aspects of the embodiments disclosed herein, in whole or in part, can be equivalently implemented in integrated circuits, as one or more computer programs running on one or more computers, as one or more programs running on one or more processors, as firmware, or as virtually any combination thereof, and that designing the circuitry and/or writing the code for the software and or firmware would be well within the skill of one of skill in the art in light of this disclosure.

In addition, those skilled in the art will appreciate that the mechanisms of the subject matter described herein are capable of being distributed as a program product in a variety of forms, and that an illustrative embodiment of the subject matter described herein applies regardless of the particular type of non-transitory signal bearing medium used to actually carry out the distribution. Examples of a non-transitory signal bearing medium include, but are not limited to, the following: a recordable type medium such as a floppy disk, a hard disk drive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape, a computer memory, etc.; and a transmission type medium such as a digital and/or an analog communication medium (e.g., a fiber optic cable, a waveguide, a wired communications link, a wireless communication link, etc.).

With respect to the use of substantially any plural and/or singular terms herein, those having skill in the art can translate from the plural to the singular and/or from the singular to the plural as is appropriate to the context and/or application. The various singular/plural permutations may be expressly set forth herein for sake of clarity.

It should also be noted that in situations in which the systems and methods described herein may collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features associated with the systems and/or methods collect user information (e.g., information about a user's preferences). In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be treated so that no personally identifiable information can be determined for the user. Thus, the user may have control over how information is collected about the user and used by a server.

Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method comprising:

receiving, at a centralized processing device, a corresponding hotword confidence score from each of multiple media devices in communication with the centralized processing device via a network, each hotword confidence score indicating a likelihood that audio data corresponding to a first utterance of a user received by the corresponding media device includes a particular, predefined hotword;
determining, by the centralized processing device, that two or more of the received hotword confidence scores satisfy a hotword score threshold;
for each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold, receiving, at the centralized processing device, second audio data from the corresponding media device, the second audio data recorded by the corresponding media device and including a user speech command; and
generating, by the centralized processing device, a request associated with the user speech command based on the second audio data received from each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold.

2. The computer-implemented method of claim 1, further comprising, in response to receiving the corresponding hotword confidence score from each of the two or more media devices in communication with the centralized processing device, instructing, by the centralized processing device, each of the two or more media devices to mute a corresponding loudspeaker.

3. The computer-implemented method of claim 1 further comprising, prior to receiving the second audio data from the corresponding media device, instructing, by the centralized processing device, the corresponding media device to activate a corresponding microphone.

4. The computer-implemented method of claim 1, wherein each of the multiple media devices are configured to:

record the first audio data corresponding to the first user utterance;
detect the particular, predefined hotword in the first audio data using a corresponding hotword data module;
compute the corresponding hotword confidence score indicating the likelihood that the first audio data recorded by the corresponding media device includes the particular, predefined hotword; and
transmit the corresponding hotword confidence score over the network to the centralized processing device.

5. The computer-implemented method of claim 1, further comprising:

transmitting the request associated with the user speech command from the centralized processing device to an external server;
receiving, at the centralized processing device, an audio response associated with the user speech command from the external server; and
transmitting the audio response to at least one of the multiple media devices, the audio response when received by the at least one media device causing the at least one media device to play the audio response over a corresponding loudspeaker associated with the at least one media device.

6. The computer-implemented method of claim 1, wherein generating the request associated with the user speech command based on the second audio data comprises combining the second audio data received from each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold to generate the request associated with the user speech command.

7. The computer-implemented method of claim 1, further comprising, for each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold, determining, by the centralized computing device, a word transcription and a word segment score for each word in the second audio data received from the corresponding media device.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving a corresponding hotword confidence score from each of multiple media devices in communication with the one or more computers via a network, each hotword confidence score indicating a likelihood that audio data corresponding to a first utterance of a user received by the corresponding media device includes a particular, predefined hotword; determining that two or more of the received hotword confidence scores satisfy a hotword score threshold; for each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold, receiving second audio data from the corresponding media device, the second audio data recorded by the corresponding media device and including a user speech command; and generating a request associated with the user speech command based on the second audio data received from each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold.

9. The system of claim 8, wherein the operations further comprise in response to receiving the corresponding hotword confidence score from each of the two or more media devices, instructing each of the two or more media devices to mute a corresponding loudspeaker.

10. The system of claim 8, wherein the operations further comprise, prior to receiving the second audio data from the corresponding media device, instructing the corresponding media device to activate a corresponding microphone.

11. The system of claim 8, wherein each of the multiple media devices are configured to:

record the first audio data corresponding to the first user utterance;
detect the particular, predefined hotword in the first audio data using a corresponding hotword data module;
compute the corresponding hotword confidence score indicating the likelihood that the first audio data recorded by the corresponding media device includes the particular, predefined hotword; and
transmit the corresponding hotword confidence score over the network to the one or more computers.

12. The system of claim 8, wherein the operations further comprise:

transmitting the request associated with the user speech command to an external server;
receiving an audio response associated with the user speech command from the external server; and
transmitting the audio response to at least one of the multiple media devices, the audio response when received by the at least one media device causing the at least one media device to play the audio response over a corresponding loudspeaker associated with the at least one media device.

13. The system of claim 8, wherein generating the request associated with the user speech command based on the second audio data comprises combining the second audio data received from each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold to generate the request associated with the user speech command.

14. The system of claim 8, wherein the operations further comprise, for each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold, determining a word transcription and a word segment score for each word in the second audio data received from the corresponding media device.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

receiving a corresponding hotword confidence score from each of multiple media devices in communication with the one or more computers via a network, each hotword confidence score indicating a likelihood that audio data corresponding to a first utterance of a user received by the corresponding media device includes a particular, predefined hotword;
determining that two or more of the received hotword confidence scores satisfy a hotword score threshold;
for each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold, receiving second audio data from the corresponding media device, the second audio data recorded by the corresponding media device and including a user speech command; and
generating a request associated with the user speech command based on the second audio data received from each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold.

16. The computer-readable medium of claim 15, wherein the operations further comprise, in response to receiving the corresponding hotword confidence score from each of the two or more media devices, instructing each of the two or more media devices to mute a corresponding loudspeaker.

17. The computer-readable medium of claim 15, wherein the operations further comprise, prior to receiving the second audio data from the corresponding media device, instructing the corresponding media device to activate a corresponding microphone.

18. The computer-readable medium of claim 15, wherein each of the multiple media devices are configured to:

record the first audio data corresponding to the first user utterance;
detect the particular, predefined hotword in the first audio data using a corresponding hotword data module;
compute the corresponding hotword confidence score indicating the likelihood that the first audio data recorded by the corresponding media device includes the particular, predefined hotword; and
transmit the corresponding hotword confidence score over the network to the one or more computers.

19. The computer-readable medium of claim 15, wherein the operations further comprise:

transmitting the request associated with the user speech command to an external server;
receiving an audio response associated with the user speech command from the external server; and
transmitting the audio response to at least one of the multiple media devices, the audio response when received by the at least one media device causing the at least one media device to play the audio response over a corresponding loudspeaker associated with the at least one media device.

20. The computer-readable medium of claim 15, wherein generating the request associated with the user speech command based on the second audio data comprises combining the second audio data received from each of the two or more media devices having hotword confidence scores that satisfy the hotword score threshold to generate the request associated with the user speech command.

Referenced Cited
U.S. Patent Documents
4363102 December 7, 1982 Holmgren
5659665 August 19, 1997 Whelpley, Jr.
5897616 April 27, 1999 Kanevsky
5983186 November 9, 1999 Miyazawa et al.
6141644 October 31, 2000 Kuhn
6567775 May 20, 2003 Maali
6671672 December 30, 2003 Heck
6718308 April 6, 2004 Nolting
6744860 June 1, 2004 Schrage
6826159 November 30, 2004 Shaffer
6931375 August 16, 2005 Bossemeyer
6973426 December 6, 2005 Schier
7016833 March 21, 2006 Gable
7222072 May 22, 2007 Chang
7571014 August 4, 2009 Lambourne et al.
7640160 December 29, 2009 Di Cristo
7668718 February 23, 2010 Kahn
7720012 May 18, 2010 Borah
7904297 March 8, 2011 Mirkovic et al.
8099288 January 17, 2012 Zhang
8194624 June 5, 2012 Park et al.
8200488 June 12, 2012 Kemp et al.
8209174 June 26, 2012 Al-Telmissani
8214447 July 3, 2012 Deslippe et al.
8340975 December 25, 2012 Rosenberger
8588949 November 19, 2013 Lambourne et al.
8612211 December 17, 2013 Shires
8670985 March 11, 2014 Lindahl et al.
8713119 April 29, 2014 Lindahl
8717949 May 6, 2014 Crinon
8719009 May 6, 2014 Baldwin et al.
8719018 May 6, 2014 Dinerstein
8768687 July 1, 2014 Quasthoff et al.
8775191 July 8, 2014 Sharifi et al.
8805890 August 12, 2014 Zhang et al.
8838457 September 16, 2014 Cerra et al.
8938394 January 20, 2015 Faaborg et al.
8996372 March 31, 2015 Secker-Walker
9129602 September 8, 2015 Shepard
9142218 September 22, 2015 Schroeter
9171541 October 27, 2015 Kennewick
9292487 March 22, 2016 Weber
9420227 August 16, 2016 Shires
20020049596 April 25, 2002 Burchard et al.
20020072905 June 13, 2002 White
20020123890 September 5, 2002 Kopp
20020193991 December 19, 2002 Bennett et al.
20030200090 October 23, 2003 Kawazoe
20030231746 December 18, 2003 Hunter
20040101112 May 27, 2004 Kuo
20050165607 July 28, 2005 Di Fabbrizio et al.
20060074656 April 6, 2006 Mathias
20060085188 April 20, 2006 Goodwin et al.
20060184370 August 17, 2006 Kwak et al.
20070100620 May 3, 2007 Tavares
20070133437 June 14, 2007 Wengrovitz
20070198262 August 23, 2007 Mindlin
20070798262 August 2007 Mindlin
20080252595 October 16, 2008 Boillot
20090258333 October 15, 2009 Yu
20090292541 November 26, 2009 Daya
20100070276 March 18, 2010 Wasserblat
20100110834 May 6, 2010 Kim
20110026722 February 3, 2011 Jing
20110054892 March 3, 2011 Jung
20110060587 March 10, 2011 Phillips et al.
20110066429 March 17, 2011 Shperling
20110112827 May 12, 2011 Kennewick
20110112921 May 12, 2011 Kennewick
20110184730 July 28, 2011 LeBeau et al.
20110304648 December 15, 2011 Kim et al.
20120084087 April 5, 2012 Yang
20120232896 September 13, 2012 Taleb
20120265528 October 18, 2012 Gruber et al.
20120316471 December 13, 2012 Rahman
20130060571 March 7, 2013 Soemo et al.
20130110521 May 2, 2013 Hwang
20130124207 May 16, 2013 Sarin et al.
20130132086 May 23, 2013 Xu et al.
20130183944 July 18, 2013 Mozer
20140012573 January 9, 2014 Hung
20140012578 January 9, 2014 Morioka
20140088961 March 27, 2014 Woodward
20140222430 August 7, 2014 Rao
20140257821 September 11, 2014 Adams
20140278383 September 18, 2014 Fan
20140278435 September 18, 2014 Ganong, III et al.
20150061969 March 5, 2015 Chi
20150106091 April 16, 2015 Wetjen
20150154953 June 4, 2015 Bapat
20150262577 September 17, 2015 Nomura
20160049152 February 18, 2016 Kennewick
20160104483 April 14, 2016 Foerster et al.
20160260431 September 8, 2016 Newendorp
Foreign Patent Documents
1998/040875 September 1998 WO
2014/008194 January 2014 WO
2015/025330 February 2015 WO
Other references
  • International Search Report and Written Opinion in International Application No. PCT/US2015/052860, dated Dec. 8, 2015, 12 pages.
  • Office Action issued in U.S. Appl. No. 14/675,932 dated Jun. 12, 2015, 9 pages.
  • Office Action issued in U.S. Appl. No. 14/675,932 dated Oct. 1, 2015, 19 pages.
  • Notice of Allowance issued in U.S. Appl. No. 14/675,932 dated Jan. 25, 2016, 12 pages.
  • Office Action issued in U.S. Appl. No. 15/088,477 dated May 25, 2016, 12 pages.
  • Notice of Allowance issued in U.S. Appl. No. 15/088,477 dated Sep. 9, 2016, 17 pages.
  • Office Action in Korean Application No. 10-2016-7021778, dated Dec. 13, 2016, 4 pages.
  • Extended European Search Report issued in Application No. 16195834.3-1910, dated Nov. 23, 2016, 9 pages.
  • International Preliminary Report on Patentability in International application No. PCT/US2015/030569, dated Jan. 24, 2017, 8 pages.
  • Office Action issued in U.S. Appl. No. 15/346,914, dated Feb. 3, 2017, 22 pages.
  • International Search Report and Written Opinion in International Application No. PCT/US2015/052870, dated Dec. 4, 2015, 11 pages.
  • Office Action in U.S. Appl. No. 15/190,739, dated Aug. 12, 2016, 9 pages.
  • Office Action in U.S. Appl. No. 15/190,739, dated Feb. 24, 2017, 13 pages.
  • Office Action in U.S. Appl. No. 14/659,861, dated Dec. 30, 2015, 10 pages.
  • Notice of Allowance in U.S. Appl. No. 14/659,861, dated Jun. 8, 2016, 8 pages.
  • Office Action in U.S. Appl. No. 15/052,426, dated Apr. 19, 2017, 8 pages.
Patent History
Patent number: 10163442
Type: Grant
Filed: May 17, 2017
Date of Patent: Dec 25, 2018
Patent Publication Number: 20170249943
Assignee: Google LLC (Mountain View, CA)
Inventors: Jay Pierre Civelli (Sunnyvale, CA), Mikhal Shemer (Tel-Aviv), Turaj Zakizadeh Shabestary (Mountain View, CA), David Tapuska (Waterloo)
Primary Examiner: Vijay B Chawan
Application Number: 15/597,249
Classifications
Current U.S. Class: Speech Controlled System (704/275)
International Classification: G10L 21/00 (20130101); G10L 15/30 (20130101); G10L 15/22 (20060101); G10L 15/32 (20130101); G10L 15/02 (20060101); G10L 15/08 (20060101);