Abstract: In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

Abstract: Apparatus and methods for leveraging machine learning and artificial intelligence to generate a response to an utterance expressed by a user during an interaction between an interactive response system and the user is provided. The methods may include a natural language processor processing the utterance to output an utterance intent. The methods may also include a signal extractor processing the utterance, the utterance intent and previous utterance data to output utterance signals. The methods may additionally include an utterance sentiment classifier using a hierarchy of rules to extract, from a database, a label, the extracting being based on the utterance signals. The methods may further include a sequential neural network classifier using a trained algorithm to process the label and a sequence of historical labels to output a sentiment score. The methods may further include, based on the utterance intent, the label and the score, to output a response.

Abstract: A method includes adding, by a first computing device, a first audio watermark to first speech data corresponding to playback of a first utterance including a hotword used to invoke an attention of a second computing device. The method includes outputting, by the first computing device, the playback of the first utterance corresponding to the watermarked first speech data. The second computing device is configured to receive the watermarked first speech data and determine to cease processing of the watermarked first speech data.

Abstract: A multi-tag concurrent identification method and a system for a query tree based on feature groups are provided in this disclosure. In the disclosure, a whole data string space is divided into a plurality of disjoint subsets according to features of data strings returned by tags, where each of the subsets contains several different data strings, each of the data strings in the each of the subsets is regarded as a complete tag ID or a partial ID, and the each of the subsets corresponds to a unique query prefix, a length of the prefix is fixed and does not dynamically increase with an actual location of a collision, and when multiple data strings from a same subset return at a same time, a reader is capable of identifying them at a same time in a slot.

Abstract: In one aspect, a networked microphone device is configured to (i) receive sound data, (ii) determine, via the wake-word engine, that a first portion of the sound data is representative of a wake word, (iii) determine that a second networked microphone device was added to a media playback system, (iv) transmit the first portion of the sound data to a second networked microphone device, (v) begin determining a command to be performed by the first networked microphone device, (vi) receive an indication of whether the first portion of the sound data is representative of the wake word, and (vii) output a response indicative of whether the first portion of the sound data is representative of the wake word.

Abstract: An electronic apparatus including a memory configured to store first voice recognition information related to a first language and second voice recognition information related to a second language, and a processor to obtain a first text corresponding to a user voice that is received on the basis of first voice recognition information, based on an entity name being included in the user voice according to the obtained first text, identify a segment in the user voice in which the entity name is included. The processor is to obtain a second text corresponding to the identified segment of the user voice on the basis of the second voice recognition information, and obtain control information corresponding to the user voice on the basis of the first text and the second text.

Abstract: Techniques for performing multi-stage entity resolution (ER) processing are described. A system may determine a portion of a user input corresponding to an entity name, and may request an entity provider component to perform a search to determine one or more entities corresponding to the entity name. The preliminary search results may be sent to a skill selection component for processing, while the entity provider component performs a complete search to determine entities corresponding to the entity name. A selected skill component may request the complete search results to perform its processing, including determining an output responsive to the user input.

Abstract: Techniques are described herein for detecting and/or enrolling (or commissioning) new “hot commands” that are useable to cause an automated assistant to perform responsive action(s) without having to be first explicitly invoked. In various implementations, an automated assistant may be transitioned from a limited listening state into a full speech recognition state in response to a trigger event. While in the full speech recognition state, the automated assistant may receive and perform speech recognition processing on a spoken command from a user to generate a textual command. The textual command may be determined to satisfy a frequency threshold in a corpus of textual commands Consequently, data indicative of the textual command may be enrolled as a hot command. Subsequent utterance of another textual command that is semantically consistent with the textual command may trigger performance of a responsive action by the automated assistant, without requiring explicit invocation.

Abstract: Aspects of the disclosure relate to using an apparatus for flagging and removing real time workflows that produce sub-optimal results. Such an apparatus may include an utterance sentiment classifier. The apparatus stores a hierarchy of rules. Each of the rules is associated with one or more rule signals. In response to receiving the one or more utterance signals, the classifier iterates through the hierarchy of rules in sequential order to identify a first rule for which the one or more utterance signals are a superset of the rule's one or more rule signals. In response to receiving the one or more alternate utterance signals from the signal extractor, the classifier may iterate through the hierarchy of rules in sequential order to identify the first rule in the hierarchy for which the one or more alternate utterance signals are a superset of the first rule's one or more rule signals.

Abstract: A device performs a method for using image data to aid voice recognition. The method includes the device capturing image data of a vicinity of the device and adjusting, based on the image data, a set of parameters for voice recognition performed by the device. The set of parameters for the device performing voice recognition include, but are not limited to: a trigger threshold of a trigger for voice recognition; a set of beamforming parameters; a database for voice recognition; and/or an algorithm for voice recognition. The algorithm may include using noise suppression or using acoustic beamforming.

Abstract: The technology described in this document can be embodied in a computer-implemented method that includes receiving, at a processing system, a first signal including an output of a speaker device and an additional audio signal. The method also includes determining, by the processing system, based at least in part on a model trained to identify the output of the speaker device, that the additional audio signal corresponds to an utterance of a user. The method further includes initiating a reduction in an audio output level of the speaker device based on determining that the additional audio signal corresponds to the utterance of the user.

Abstract: Disclosed is an electronic apparatus which may receive a user voice of a user to be registered as a start command for activating a speech recognition function of the electronic apparatus; identify whether the user voice is misrecognized as a reference audio, wherein the reference audio includes a voice uttered by a user with high frequency from among a plurality of voices previously uttered by the user; and based on the user voice being not misrecognized as the reference audio, register the user voice as the start command.

Abstract: In some aspects, a server device may identify one or more services of a cloud infrastructure via a management layer. The server device may determine service information and configuration information for the one or more services. The server device may generate an environment model based at least in part on the service information and the configuration information, the environment model providing information on relationship between one or more components of the cloud infrastructure. The server device may determine one or more threats to the one or more services based at least in part on analyzing the environment model and accessing a threat information database. The server device may generate a threat model that lists the one or more threats to the one or more services. The server device may generate one or more recommendations for the cloud infrastructure based at least on the threat model.

Abstract: Implementations can receive, at a computing device, audio data corresponding to a spoken utterance of a user, process the audio data to generate, for one or more parts of the spoken utterance, a plurality of speech hypotheses, select a given one of the speech hypotheses, cause the given one of the speech hypotheses to be incorporated as a portion of a transcription associated with the software application, and store the plurality of speech hypotheses. In some implementations, the plurality of speech hypotheses can be loaded at an additional computing device when the transcription is accessed at the additional computing device. In additional or alternative implementations, the plurality of speech hypotheses can be loaded into memory of the computing device when the software application is reactivated and/or when a subsequent dialog session associated with the transcription is initiated.

Abstract: Aspects of the disclosure relate to receiving a stateless application programming interface (“API”) request. The API request may store an utterance, previous utterance data and a sequence of labels, each label in the sequence of labels being associated with a previous utterance expressed by a user during an interaction. The previous utterance data may, in certain embodiments, be limited to a pre-determined number of utterances occurring prior to the utterance. Embodiments process the utterance, using a natural language processor in electronic communication with the first processor, to output an utterance intent, a semantic meaning of the utterance and an utterance parameter. The utterance parameter may include words in the utterance and be associated with the intent. The natural language processor may append the utterance intent, the semantic meaning of the utterance and the utterance parameter to the API request. A signal extractor processor may append the plurality of utterance signals to the API request.

Abstract: Apparatus and methods for leveraging machine learning and artificial intelligence to assess a sentiment of an utterance expressed by a user during an interaction between an interactive response system and the user is provided. The methods may include a natural language processor processing the utterance to output an utterance intent. The methods may also include a signal extractor processing the utterance, the utterance intent and previous utterance data to output utterance signals. The methods may additionally include an utterance sentiment classifier using a hierarchy of rules to extract, from a database, a label, the extracting being based on the utterance signals. The methods may further include a sequential neural network classifier using a trained algorithm to process the label and a sequence of historical labels to output a sentiment score.

Abstract: Disclosed is an operating method of an electronic device, including receiving a first user model from another electronic device of a user registered in the electronic device, through a communication circuit of the electronic device, refining a user model of the electronic device based on the first user model, and identifying the user based on a first voice input of the user by using the refined user model, wherein the user model is trained by the electronic device based on a second user model of at least one another user other than the user before the user is registered, and wherein the first user model is trained by the another electronic device based on a second voice input of the user obtained by the another electronic device of the user.

Abstract: A method, performed by a hub device, of storing a voice assistant model for controlling a device includes receiving information about a first device connected to the hub device, requesting a voice assistant server to update a device determination model stored in the hub device; receiving the updated device determination model from the voice assistant server and storing the received updated device determination model; requesting, from the voice assistant server, information about a function determination model corresponding to the first device; receiving the information about the function determination model corresponding to the first device from the voice assistant server, and determining whether to store the function determination model in the hub device based on the received information; and based on the function determination model being determined to be stored in the hub device, storing, in the hub device, the function determination model corresponding to the first device.

Abstract: Apparatus and methods for leveraging machine learning and artificial intelligence to assess a sentiment of an utterance expressed by a user during an interaction between an interactive response system and the user is provided. The methods may include a natural language processor processing the utterance to output an utterance intent. The methods may also include a signal extractor processing the utterance, the utterance intent and previous utterance data to output utterance signals. The methods may additionally include an utterance sentiment classifier using a hierarchy of rules to extract, from a database, a label, the extracting being based on the utterance signals. The methods may further include a sequential neural network classifier using a trained algorithm to process the label and a sequence of historical labels to output a sentiment score.

Abstract: Phrase alternative data structures are generated from the lattice output of an audio input to Automatic Speech Recognition (ASR) system. A user interface is supported for users to view phrase alternatives to selected portions of an audio transcript of the audio input, search the transcript based on query phrases, or edit the transcript based on phrase alternatives.

Abstract: Interface and mode selection for digital action execution is provided. For example, a system loads a script library embedded in an electronic resource. The system determines a historic level of engagement between a client computing device and one or more digital assistants. The system selects, based on a first property of the client computing device and the historic level of engagement, a type of digital interface. The system generates, based on the type of digital interface, a digital interface with the call-to-action. The system determines, responsive to an instruction to execute the call-to-action, a mode of execution. The system selects a digital assistant and a second client device to execute the call-to-action. The system transmits the call-to-action to the second client device for execution.

Abstract: A voice-based peer-to-peer communication system may be used to detect voice commands from users to provide a wireless communication voice connection that allows the users to directly communicate with each other. The system may include a first computing device of a first user communicatively coupled to a second computing device of a second user over the wireless connection. The system may process the detected voice command having a phrase, contact name, and voice message. The phrase may include a wake, answer, or stop phrase. The contact name may be utilized to determine whether that contact name matches an entry within a predetermined contact list of the first user, where the matched contact name may be associated with the second user. Finally, the system may generate audio data based on the processed voice command that is then transmitted to the second computing device of the second user over the wireless connection.

Abstract: A computer-implemented input-method editor process includes receiving a request from a user for an application-independent input method editor having written and spoken input capabilities, identifying that the user is about to provide spoken input to the application-independent input method editor, and receiving a spoken input from the user. The spoken input corresponds to input to an application and is converted to text that represents the spoken input. The text is provided as input to the application.

Abstract: Systems and methods dynamically extracting n-grams for automated vocabulary updates. Text is received. An n-gram extracted from the text is matched to a canonical n-gram from a vocabulary to identify a tag for the text. An n-gram weight is computed for the n-gram extracted from the text. The n-gram weight may be computed by adjusting a term frequency of the n-gram. A relevancy score is computed for the tag using the n-gram weight and using an n-gram frequency of the canonical n-gram. The relevancy score is computed by dividing the n-gram weight by a value proportional to the n-gram frequency of the canonical n-gram. The relevancy score of the n-gram is presented.

Abstract: A method includes receiving, from an entity, a request to organize a survey on a topic, based on the request, organizing a survey of a plurality of people, recording a video of the survey, obtaining a transcription of the video and linking the transcription of the video in time to the video to yield a processed video. The method can further include presenting, on a user interface to the entity based on the processed video, the video and the transcription of the video, wherein each word in the transcription of the video is selectable by the entity, receiving a selection of text by the entity from the transcription of the video and, based on the selection of the text, presenting a portion of the video at a time that is associated with when a participant in the video spoke the text. The user can also select a “clip to next speaker” option to generate a clip.

Abstract: This disclosure describes, in part, context-driven device arbitration techniques to select a speech interface device from multiple speech interface devices to provide a response to a command included in a speech utterance of a user. In some examples, the context-driven arbitration techniques may include executing multiple pipeline instances to analyze audio signals and device metadata received from each of the multiple speech interface devices which detected the speech utterance. A remote speech processing service may execute the multiple pipeline instances and analyze the audio signals and/or metadata, at various stages of the pipeline instances, to determine which speech interface device is to respond to the speech utterance.

Abstract: Computer-implemented method and system for receiving and processing one or more moment-associating elements. For example, the computer-implemented method includes receiving the one or more moment-associating elements, transforming the one or more moment-associating elements into one or more pieces of moment-associating information, and transmitting at least one piece of the one or more pieces of moment-associating information.

Abstract: In cases in which a confidence score of an inferred intent label is a predetermined threshold or less, an intent inference section searches for whether or not wording pertaining to a location, such as “on the door”, is present in a question. In cases in which a word relating to a location is present, the intent inference section consults individual function identification data associated with OM item codes in order to find intent labels including individual functions relevant to the location (such as “door”). In cases in which an intent label including an individual function relevant to the “door” is found, an OMA interaction control section consults QA data to find and acquire associated response information based on the found intent label and the OM item code, and notifies a HMI interaction control section of such response information.

Abstract: An acoustic system and method is disclosed for providing spatial and temporal classification of a range of different types of sound producing targets in a geographical area. The system includes an optical signal transmitter arrangement for repeatedly transmitting, at multiple instants, interrogating optical signals into each of one or more optical fibres distributed across the geographical area and forming at least part of an installed fibre-optic communications network. An optical signal detector arrangement receives, during an observation period following each of the multiple instants, returning optical signals scattered in a distributed manner over distance along the one or more of optical fibres.

Abstract: Methods, systems, and media for providing direct and hybrid data acquisition approaches are provided.

Abstract: A medication confirmation method and apparatus. The method of an embodiment of the invention includes the steps of capturing one or more video sequences of a user administering medication via a medication administration apparatus, storing the captured one or more video sequences, and analyzing at least one of the stored video sequences to confirm that the user has properly administered the medication, and has properly positioned the medication administration apparatus.

Abstract: Devices, systems and processes for a dynamic microphone system that enhances the passenger experience in autonomous vehicles are described. One example method for enhancing a passenger experiences includes generating, using an artificial intelligence algorithm, a plurality of filters based on a plurality of stored waveforms previously recorded by each of one or more passengers and a plurality of recordings of one or more noise sources, capturing voice commands from at least one of the one or more passengers inside the autonomous vehicle, generating voice commands with reduced distortion based on processing the voice commands using the plurality of filters, and instructing, based on the voice commands with reduced distortion, the autonomous vehicle to perform one or more actions.

Abstract: A message playing method includes: receiving a first message, and asking in a voice manner, whether to play the first message; if a first voice of a user does not match a keyword of a positive reply, continuing to detect a voice of the user; if a second voice of the user detected, matches the keyword of the positive reply, playing the first message in the voice manner, and recording a quantity of times of using a text corresponding to the first voice; and when the quantity of times of using the text that corresponds to the first voice and that is recorded is greater than a first threshold, adding the text to the keyword of the positive reply.

Abstract: A vehicle includes a rear door, at least one sensor component proximate to the rear door, an alert system, a memory in which program instructions are stored, and a processor operably connected to the alert system, the at least one sensor component, and the memory. The processor executes the program instructions to detect an object within a detection zone of the at least one sensor component relative to the rear door. The processor further activates the alert system in response to the object being within a predefined protection zone of the detection zone when the rear door is not moving.

Abstract: A vehicle system includes: at least one of: (a) a particulate matter sensor configured to measure an amount of particulate within a passenger cabin of a vehicle; and (b) a volatile organic compounds (VOC) sensor configured to measure an amount of VOCs within the passenger cabin of the vehicle; and a control module configured to, based on at least one of the amount of particulate and the amount of VOCs within the passenger cabin of the vehicle, diagnose at least one of: a characteristic of the vehicle; presence of a chemical in the passenger cabin of the vehicle; occurrence of an event within the passenger cabin of the vehicle; and a remaining life of engine oil of the vehicle.

Abstract: A computer-implemented technique is described herein for expediting a user's interaction with a digital assistant. In one implementation, the technique involves receiving a system prompt generated by a digital assistant in response to an input command provided by a user via an input device. The technique then generates a predicted response based on linguistic content of the system prompt, together with contextual features pertaining to a circumstance in which the system prompt was issued. The predicted response corresponds to a prediction of how the user will respond to the system prompt. The technique then selects one or more dialogue actions from a plurality of dialogue actions, based on a confidence value associated with the predicted response. The technique expedites the user's interaction with the digital assistant by reducing the number of system prompts that the user is asked to respond to.

Abstract: Techniques related to a method and system of audio false keyphrase rejection using speaker recognition are described herein. Such techniques use speaker recognition of a computer originated voice to omit actions triggered when a keyphrase is present in captured audio and omitted when speech of the captured audio was spoken by the computer originated voice.

Abstract: Systems, devices, and methods are discussed relating to plasma sources using load current switch timing of zero volt switching resonant topology.

Abstract: An audio spotting system configured for various operating modes including a regular mode and sensitivity mode is described. An example cascade audio spotting system may include a high-power subsystem including a high-power trigger and a transfer module. This high-power trigger includes one or more detection models used to detect whether a target sound activity is included in the one or more audio streams. The one or more detection models are associated with a first set of hyperparameters when the cascade audio spotting system is in a regular mode, and the one or more detection models are associated with a second set of hyperparameters when the cascade audio spotting system is in a sensitivity mode. The transfer module provides at least one of one or more processed audio streams for further processing in response to the high-power trigger detecting the target sound activity in the one or more audio streams.

Abstract: Features are disclosed for training and using a word embedding model configured to receive textual and context data associated with an utterance of a user. A word embedding model can be trained with text data and context data to account for context associated with the text data. The word embedding model can receive an input vector including text data and one or more sets of context data associated with the text data and perform word embedding based on the input vector. In some embodiments, the input vector can include an automatic speech recognition (“ASR”) confidence score generated by an ASR model and one or more labels generated by an NLU model. In some embodiments, the input vector can include user information associated with the user.

Abstract: Described are techniques for linking generating a skill-stored user profile, and linking same with a natural language processing (NLP) system-stored user profile. In at least some examples, a user may provide a natural language input to a NLP system. The NLP system may determine a skill is to process to perform an action responsive to the natural language input. To perform the action, the skill may require the user have a user profile stored by the skill, but the user may not have such a user profile. However, the NLP system may store a user profile for the user. The NLP system may determine the user profile stored thereby and may send, with user permission, information in the user profile to the skill. The skill may use the received information to generate and store a user profile for the user. Thereafter, the skill may provide the NLP system with a user profile identifier that the skill may use to identify the user's profile stored thereby.

Abstract: Systems and methods are described for handling interruptions during a digital assistant session between a user and a digital assistant by detecting if an interruption event is to occur during the digital assistant session. In response to detecting that the interruption event is to occur, options to address the interruption are provided.

Abstract: Methods, devices, and systems for processing audio information are disclosed. An exemplary method includes receiving an audio stream. The audio stream may be monitored by a low power integrated circuit. The audio stream may be digitized by the low power integrated circuit. The digitized audio stream may be stored in a memory, wherein storing the digitized audio stream comprises replacing a prior digitized audio stream stored in the memory with the digitized audio stream. The low power integrated circuit may analyze the stored digitized audio stream for recognition of a keyword. The low power integrated circuit may induce a processor to enter an increased power usage state upon recognition of the keyword within the stored digitized audio stream. The stored digitized audio stream may be transmitted to a server for processing. A response received from the server based on the processed audio stream may be rendered.

Abstract: Embodiments described herein provide for a computer that detects one or more keywords of interest using acoustic features, to detect or query commonalities across multiple fraud calls. Embodiments described herein may implement unsupervised keyword spotting (UKWS) or unsupervised word discovery (UWD) in order to identify commonalities across a set of calls, where both UKWS and UWD employ Gaussian Mixture Models (GMM) and one or more dynamic time-warping algorithms. A user may indicate a training exemplar or occurrence of call-specific information, referred to herein as “a named entity,” such as a person's name, an account number, account balance, or order number. The computer may perform a redaction process that computationally nullifies the import of the named entity in the modeling processes described herein.

Abstract: Generally discussed herein are devices, systems, and methods for on-device detection of a wake word. A device can include a memory including model parameters that define a custom wake word detection model, the wake word detection model including a recurrent neural network transducer (RNNT) and a lookup table (LUT), the LUT indicating a hidden vector to be provided in response to a phoneme of a user-specified wake word, a microphone to capture audio, and processing circuitry to receive the audio from the microphone, determine, using the wake word detection model, whether the audio includes an utterance of the user-specified wake word, and wake up a personal assistant after determining the audio includes the utterance of the user-specified wake word.

Abstract: Methods, systems, apparatus, including computer programs encoded on a computer storage medium, for a user device to learn offline voice actions. In one aspect, the method includes actions of detecting, by the user device, an utterance at a first time when the user device is connected to a server by a network, providing, by the user device, the utterance to the server using the network, receiving, by the user device and from the server, an update to the grammar of the user device, detecting, by the user device, a subsequent utterance of the utterance at a second time when the user device is not connected to the server by a network, and in response to detecting, by the user device, the subsequent utterance of the utterance at the second time, identifying, by the user device, an operation to perform based on (i) the subsequent utterance, and (ii) the updated grammar.

Abstract: Methods, apparatus, systems, and computer-readable media are provided for transferring dialog sessions between devices using deep links. The dialog sessions can correspond to interactions, mediated by an automated assistant, between a user and a third party application. During the dialog session, a user can request that the dialog session be transferred to a different device, for example, to interact with the third party application through a different modality. In response, the automated assistant and/or the third party application can generate a link that can be transferred to the transferee device to allow the transferee device to seamlessly take over the dialog session. In this way, computational resources and electrical power can be preserved by not requiring a recipient device to re-process natural language inputs previously provided during the dialog session.

Abstract: A system for synthesizing real IVR scenarios, automatically reproducing test scenarios, and providing accurate reports on those scenarios. An IVR tester uses a telephony device (website, mobile app, VOIP, etc.) and an automated IVR test case generator to place test calls to an IVR system. The automated IVR test case generator intercepts and collects the phone call's audio stream, dual-tone multi-frequency signals, and available metadata (e.g., response times, post-speech silence timeout, etc.). Once the call is complete, the automated IVR test case generator pre-populates a webpage or other user-interface technology and a database with the audio stream, dual-tone multi-frequency signals, and available metadata from the test scenario phone call. The IVR tester may then save that test scenario and continue to the next scenario. The saved scenario may now be used automatically by the IVR test case generator for regression testing or duplicated and edited for use in other systems.

Abstract: An information processing apparatus is connected to a voice processing server that analyzes text data transmitted from a voice input/output apparatus that converts an instruction by an utterance of a user to the text data and outputs the text data, and outputs an instruction obtained by analysis and utterance language information indicating a language of the utterance, and the information processing apparatus includes: a communicator that communicates with the voice processing server; a destination searcher that determines on the basis of the utterance language information whether to include a space character in a target of the search, and searches for a name indicated in a search character string from a destination list on the basis of a result of the determination; and a hardware processor that performs control to transmit a search result of a destination by the destination searcher to the voice processing server via the communicator.

Abstract: A donation processing system receives a text message from a donor indicating that they would like to make a donation to a non-profit organization. In response to the text message, a response message indicating the particulars of the donation including the donation amount and the non-profit organization is sent to the user. The donor reviews the information and, if the information is correct, a confirmation message is sent from the donor in response to the response message. The confirmation message confirms the donation. The donation is then made to the non-profit organization in response to the confirmation message.