Abstract: Various arrangements are detailed herein related to managing video recording based on spoken commands. A system receives a video stream from a video camera and analyzes a field of view in the received video stream to determine a location for one or more identified or potential users. The system can beamform audio from microphones of a home assistant device based on the location of the one or more identified or potential users. The system adjusts an audio output based on the location of the one or more identified or potential users, receives a spoken command from the one or more identified or potential users, and outputs a response to the spoken command.

Abstract: A display device according to one embodiment of the present invention comprises an input unit, a storage unit, a display unit for displaying a menu for providing at least one function, and a control unit, which generates a user pattern, including function information corresponding to an execution command and information about the time at which the execution command was received, when receiving the execution command from a user, controls the storage unit so as to store the generated user pattern, reconfigures the menu on the basis of the user pattern, and controls the display unit so as to display the reconfigured menu.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing assistance for customer service agents are disclosed. In one aspect, a method includes the actions of receiving, by a computing device, customer interaction data that reflects an interaction between a first user and a second user. The actions further include receiving, by the computing device, a customer summary file that reflects characteristics of the first user. The actions further include, based on the customer interaction data and the customer summary file, determining, by the computing device, instructions for the second user to continue interacting with the first user during the interaction between the first user and the second user. The actions further include, based on determining the instructions, providing, for output to the second user, the instructions for the second user to continue interacting with the first user.

Abstract: The present disclosure generally relates to using voice interaction to access call functionality of a companion device. In an example process, a user utterance is received. Based on the user utterance and contextual information, the process causes a server to determine a user intent corresponding to the user utterance. The contextual information is based on a signal received from the companion device. In accordance with the user intent corresponding to an actionable intent of answering the incoming call, a command is received. Based on the command, instructions are provided to the companion device, which cause the companion device to answer the incoming call and provide audio data of the answered incoming call. Audio is outputted according to the audio data of the answered incoming call.

Abstract: The present disclosure provides systems and methods that generating health diagnostic information from an audio recording. A computing system can include a machine-learned health model comprising that includes a sound model trained to receive data descriptive of a patient audio recording and output sound description data. The computing system can include a diagnostic model trained to receive the sound description data and output a diagnostic score. The computing system can include at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed, cause the processor to perform operations. The operations can include obtaining the patient audio recording; inputting data descriptive of the patient audio recording into the sound model; receiving, as an output of the sound model, the sound description data; inputting the sound description data into the diagnostic model; and receiving, as an output of the diagnostic model, the diagnostic score.

Abstract: Systems and processes for providing a virtual assistant service are provided. In accordance with one or more examples, a method includes receiving, from an accessory device communicatively coupled to the first electronic device, a representation of a speech input representing a user request. The method further includes detecting a second electronic device and transmitting, from the first electronic device, a representation of the user request and data associated with the detected second electronic device to a third electronic device. The method further includes receiving, from the third electronic device, a determination of whether a task is to be performed by the second electronic device in accordance with the user request; and in accordance with a determination that a task is to be performed by the second electronic device, requesting the second electronic device to performed the task in accordance with the user request.

Abstract: Systems and methods for optimizing network microphone devices using noise classification are disclosed herein. In one example, individual microphones of a network microphone device (NMD) detect sound. The sound data is analyzed to detect a trigger event such as a wake word. Metadata associated with the sound data is captured in a lookback buffer of the NMD. After detecting the trigger event, the metadata is analyzed to classify noise in the sound data. Based on the classified noise, at least one performance parameter of the NMD is modified.

Abstract: Systems and methods for identifying audio events in one or more audio streams include the use of a cascade audio spotting system (such as a cascade keyword spotting system (KWS)) to reduce power consumption while maintaining a desired performance. An example cascade audio spotting system may include a first module and a high-power subsystem. The first module is to receive an audio stream from one or more audio streams, process the audio stream to detect a first target sound activity in the audio stream, and provide a first signal in response to detecting the first target sound activity in the audio stream. The high-power subsystem is to (in response to the first signal being provided by the first module) receive the one or more audio streams and process the one or more audio streams to detect a second target sound activity in the one or more audio streams.

Abstract: Recommending an automated assistant action for inclusion in an existing automated assistant routine of a user, where the existing automated assistant routine includes a plurality of preexisting automated assistant actions. If the user confirms the recommendation through affirmative user interface input, the automated assistant action can be automatically added to the existing automated assistant routine. Thereafter, when the automated assistant routine is initialized, the preexisting automated assistant actions of the routine will be performed, as well as the automated assistant action that was automatically added to the routine in response to affirmative user interface input received in response to the recommendation.

Abstract: The present disclosure is directed techniques for executing a task or service using a virtual agent. A method includes: executing, using a virtual agent, one or more tiers of a plurality of tiers of machine learning analysis to identify a desired action to be performed based on a user command, the user command being received from an external computing device; responsive to the one or more tiers of the plurality of tiers of machine learning analysis identifying a plurality of actions associated with the user command, determining a series of inquiries to present via the external computing device, wherein each inquiry of the series of inquiries is selected based on a number of actions associated with each inquiry, and wherein each subsequent inquiry in the series of inquires is based on a user response to a preceding inquiry; identifying, based on responses to the series of inquiries, the desired action to be performed; and executing the desired action to be performed.

Abstract: An occupancy tracking device configured to receive a plurality of sound samples over a predetermined time period. The device is further configured to compute an audio signature for each sound sample. The device is further configured to populate entries in the voice data log for the sound samples, to identify one or more clusters based on an audio signature that is associated with the populated entries, and to determine a number of clusters that are identified. The device is further configured to determine a predicted occupancy level based on the number of clusters that are identified and to control a Heating, Ventilation, and Air Conditioning (HVAC) system based on the predicted occupancy level.

Abstract: Systems and methods for identifying content corresponding to a language are provided. Language spoken by a first user based on verbal input received from the first user is automatically determined with voice recognition circuitry. A database of content sources is cross-referenced to identify a content source associated with a language field value that corresponds to the determined language spoken by the first user. The language field in the database identifies the language that the associated content source transmits content to a plurality of users. A representation of the identified content source is generated for display to the first user.

Abstract: In a first aspect of the invention, it is claimed a computer-implemented method for assisting the movement of a visually impaired user by means of a wearable device 1, comprising the following steps: S1—Acquiring data from the environment of the visually impaired user S2—Fusing the acquired data, creating, repeatedly updating of a Live Map S3—Determining, repeatedly updating and storing, of at least one navigation path together with associated navigation guiding instructions for the visually impaired user to navigate from the current position of the visually impaired user to a point of interest, repeatedly selecting one preferred navigation path from the at least one navigation path, and repeatedly sending to the visually impaired user the preferred navigation path, together with associated navigation guiding instructions.

Abstract: Systems and methods are disclosed configured to detect impairment issues, and via an interlock device, inhibit operation of an item of equipment when impairment is detected. The interlock device may comprise a solid state relay, an electromechanical relay, and/or a solenoid. The interlock device may perform power isolation and/or may use a mechanism, such as a rotating cam or gear, to immobilize a control and/or other components. Based on detected impairment, a determination is made as to whether the interlock is to be activated or deactivated.

Abstract: Some natural language command processing systems may handle some commands on a user device rather than sending input to another system for processing. Such a system may include an arbitration component for arbitrating between device and/or system processing. The arbitration component may execute in the system and render a device-specific decision as to whether the device will be able to process the input and/or execute the command, based on information known to the system about the device's capabilities. If the arbitration component predicts that the device will not be able to execute the command, the system may execute the command without waiting for a signal from the device. If the arbitration component predicts that the device will be able to execute the command, the system may halt processing to prevent duplicate execution.

Abstract: Systems and methods are presented herein for generating a new language understanding model, based on a user request. A user may input a root language and a locale into an application for generating a student language model. The application may generate the student language model and may identify a teacher language model related to the student language model. The application may compare data from the identified teacher language model to the student language model. The application may determine a subset of data from the teacher language model is not contained in the student language model. If the application determines at least a subset of data from the teacher language model is not in the student language model, the application may add at least the subset of data from the teacher language model to the student language model.

Abstract: A computing system for enabling a user to control a legacy application of an enterprise using voice commands includes a processor and a memory storing instructions that, when executed by the one or more processors, cause the computing system to receive a user utterance; generate an output by analyzing the utterance using a speech-to-text application programming interface; and perform an action with respect to an element of the legacy application. A computer-implemented method includes receiving a user utterance; generating an output by analyzing the utterance using a speech-to-text application programming interface; and performing an action with respect to an element of the legacy application. A non-transitory computer readable medium includes program instructions that when executed, cause a computer to receive a user utterance; generate an output by analyzing the utterance using a speech-to-text application programming interface; and perform an action with respect to an element of the legacy application.

Abstract: A system including at least one interface configured to receive data from and transmit data to a first computing device of a plurality of computing devices involved in a virtual online meeting through an external application, and a processor communicatively coupled to the at least one interface. The processor is configured to receive, via the at least one interface, a request from the first computing device of the plurality of computing devices via a voice command of the request to change a display of a shared content of the virtual online meeting on the first computing device, and output instructions to a virtual assistant that is communicatively coupled to the first computing device to change the display of the shared content on the first computing device.

Abstract: This relates to an intelligent automated assistant in a video communication environment. An example includes, during a video communication session between at least two devices, receiving a voice input at one device, generating and transmitting to a server a textual representation of the voice input, receiving from the server a shared transcription including both the textual representation of the voice input and one or more additional textual representations generated by another device, and determining and presenting one or more candidate tasks based on the shared transcription.

Abstract: An audio file associated with a user voice query may be received at a user device. The audio file may be compared to a plurality of references, such as cache entries, corresponding to a plurality of other voice queries. Based on a determination that the voice query corresponds to one of the references, an operation associated with the voice query may be executed. An indication may be received that the operation was not an intended operation associated with the voice query. Based on receiving this indication, the incorrectly identified operation, associated reference, e.g., voice query, may be disabled for the user or the device. However, the cache entry may remain enabled for one or more of a plurality of other devices.

Abstract: The present disclosure relates to automated chatbot generation for different domains from available human-to-human chat logs. The systems and methods may be configured to cluster user utterances as well as agent utterances from the human chat logs. A data miner mines intents and entities from the user utterance clustering and mines actions from agent utterances. The intents, entities and actions mined are used to generate a set of stories or flows which are further used by a machine learning engine to train the chatbot. The stories or flows are also generated automatically by mapping the intents with the actions.

Abstract: A method of providing an assistant service, performed by an electronic device, includes: determining content identification information for identifying content displayed on the electronic device; determining a user context for identifying a use situation of a user of the electronic device by using the determined content identification information; generating an utterance list based on the determined content identification information and the determined user context; and, in response to an occurrence of a predefined utterance providing event, outputting the generated utterance list.

Abstract: Disclosed are methods and systems for voice phishing monitoring. For instance, a method includes receiving voice data of an incoming call to a communication device from an application associated with a user account and executing on the device, identifying an entity and interaction allegedly associated with the incoming call from the voice data, determining first fraud indicator data based on a number of the incoming call and second fraud indicator data based on a correspondence of user account interaction data to the entity and/or interaction, and providing the voice data to a trained machine learning system to receive third fraud indicator data based on content and/or a voice characteristic identified from the voice data. The method may further include determining a status for the incoming call of fraudulent or confirmed based on the first, second, and third fraud indicator data, and generating a notification indicating the status for display.

Abstract: The technology disclosed herein enables controlled access to portions of a communication session recording. In a particular embodiment, a method includes accessing a recording of user communications exchanged between participants over a communication session. The method further includes determining that a first participant subset of the participants is participating over the communication session during a first time and a second participant subset of the participants is participating over the communication session during a second time. A first user included the first participant subset is not included in the second participant subset. Upon receiving a request for the first user to access the recording, providing, to a first endpoint of the first user, a first portion of the recording corresponding to the first time and preventing access to a second portion of the recording corresponding to the second time.

Abstract: Systems and methods presented herein generally include multi-wake phrase detection executed on a single device utilizing multiple voice assistants. Systems and methods presented herein can further include continuously running a Voice Activity Detection (VAD) process which detects presence of human speech. The multi-wake phrase detection can activate when the VAD process detects human speech. Once activated, the multi-wake phrase detection can determine which (if any) of the wake phrases of the multiple voice assistants might be in the detected speech. Operation of the multi-wake phrase detection can have a low miss-rate. In some examples, operation of the multi-wake phrase detection can be granular to accomplish the low miss-rates at low power with a tolerance for false positives on wake phrase detection.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a prediction of an audio signal. One of the methods includes receiving a request to generate an audio signal; obtaining a semantic representation of the audio signal; generating, using one or more generative neural networks and conditioned on at least the semantic representation, an acoustic representation of the audio signal; and processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal.

Abstract: A method for providing privacy protection in preparation for a phone call between a vehicle occupant of a vehicle and a remote conversation partner is provided. The method includes detecting, by an occupant detection system of the vehicle, additional listeners inside the vehicle. The method further includes receiving a call request with a telecommunication system in the vehicle for a phone call between the vehicle occupant and the remote conversation partner, the phone call being an outgoing call or an incoming call. The method further includes sending a notification to the remote conversation partner with the telecommunication system prior to establishment of the requested phone call informing the remote conversation partner about the presence of detected additional listeners.

Abstract: Examples of systems and methods for a wearable system to automatically select or filter available user interface interactions or virtual objects are disclosed. The wearable system can select a group of virtual objects for user interaction based on contextual information associated with the user, the user's environment, physical or virtual objects in the user's environment, or the user's physiological or psychological state.

Abstract: A method of training a language model for rare-word speech recognition includes obtaining a set of training text samples, and obtaining a set of training utterances used for training a speech recognition model. Each training utterance in the plurality of training utterances includes audio data corresponding to an utterance and a corresponding transcription of the utterance. The method also includes applying rare word filtering on the set of training text samples to identify a subset of rare-word training text samples that include words that do not appear in the transcriptions from the set of training utterances or appear in the transcriptions from the set of training utterances less than a threshold number of times. The method further includes training the external language model on the transcriptions from the set of training utterances and the identified subset of rare-word training text samples.

Abstract: A gesture recognition and control device recognizes a mobile terminal and ascertains a current graphical user interface generated by a display device of the mobile terminal. The gesture recognition and control device provides an output signal describing, as display content, the graphical user interface generated by the display device of the mobile terminal, and transmits the output signal to an output apparatus that can be worn on the head for outputting the display content in a predefined output region in the interior of the motor vehicle as part of augmented reality or virtual reality output by the output apparatus. During the process of outputting the display content, the gesture recognition and control device recognizes a spatial gesture of the user, generates a remote control signal for triggering an operating function of the mobile terminal, and transmits the remote control signal to a control device of the recognized mobile terminal.

Abstract: Provided is a voice assistance system with proactive routines that couples a remote server and respective user voice interactive devices to deliver a complete experience to the end user of the device. The voice assistance system can also provide a platform to connect remote users to customize commands (e.g., proactive or reactive) to improve day-to-day operation across groups of devices and/or users. For example, routines can be built for an organization and made available to voice interactive devices distributed by or on behalf of the organization. Administrators can select features for the device and pre-configure voice assistance devices with groups of pre-selected routines to deliver a device that is ready to go out of the box. Updates to a device or routines can be based on such groups. Additionally, primary routine instances can be linked to groups where changes to the primary instance are propagated to any linked users.

Abstract: An augmented reality (AR) content generation method includes: acquiring, with a camera of an AR device, one or more images of a component of a medical imaging or medical therapy device; receiving, from a microphone of the AR device, a triggering audio segment; generating one or more query data structures from both the one or more images and the triggering audio segment; retrieving AR instructional content related to the medical imaging or medical therapy device matching the generated one or more query data structures from a database; and outputting the AR instructional content one or more of (i) displayed superimposed on video displayed by the AR device and/or (ii) displayed on a head mounted display of the AR device and/or (iii) as audio content via a loudspeaker of the AR device.

Abstract: The present disclosure relates to chatbot systems, and more particularly, to techniques for identifying an intent for an utterance based on semantic framing. For an input utterance, a semantic frame is generated. The semantic frame includes semantically relevant grammatical relations and corresponding words identified in the utterance. The semantically relevant grammatical relations define context and relationships of words in the utterance. The semantic frame is used to identify an intent for the utterance, based on an intent model. The intent model maps features to corresponding words for a given intent. The semantic frame is compared to a plurality of intent models, and a best-matching intent model is used to identify the intent for the utterance.

Abstract: A method for streaming action fulfillment receives audio data corresponding to an utterance where the utterance includes a query to perform an action that requires performance of a sequence of sub-actions in order to fulfill the action. While receiving the audio data, but before receiving an end of speech condition, the method processes the audio data to generate intermediate automated speech recognition (ASR) results, performs partial query interpretation on the intermediate ASR results to determine whether the intermediate ASR results identify an application type needed to perform the action and, when the intermediate ASR results identify a particular application type, performs a first sub-action in the sequence of sub-actions by launching a first application to execute on the user device where the first application is associated with the particular application type. The method, in response to receiving an end of speech condition, fulfills performance of the action.

Abstract: Examples relate to on-device non-semantic representation fine-tuning for speech classification. A computing system may obtain audio data having a speech portion and train a neural network to learn a non-semantic speech representation based on the speech portion of the audio data. The computing system may evaluate performance of the non-semantic speech representation based on a set of benchmark tasks corresponding to a speech domain and perform a fine-tuning process on the non-semantic speech representation based on one or more downstream tasks. The computing system may further generate a model based on the non-semantic representation and provide the model to a mobile computing device. The model is configured to operate locally on the mobile computing device.

Abstract: Systems and methods for document editing are provided. One aspect of the systems and methods includes obtaining a document and a natural language edit request. Another aspect of the systems and methods includes generating a structured edit command using a machine learning model based on the document and the natural language edit request. Yet another aspect of the systems and methods includes generating a modified document based on the document and the structured edit command, where the modified document includes a revision of the document that incorporates the natural language edit request.

Abstract: Various embodiments of the present disclosure relate generally to providing services to users via communication channels. More specifically, various embodiments of the present disclosure relate to systems and methods for modifying, updating, and/or changing communication channel interactions based on the tracking or listening for events within other communication channels.

Abstract: Apparatuses, methods, and systems for implicitly annotating textual data in conversational messaging are disclosed. One method includes receiving, by a server, a first input text message from a user (customer), displaying, by the server, the first input text message to an agent (merchant), displaying, by the server, a configurable menu of responses to the first input text message to the agent (merchant), receiving, by the server, a selection of one of the configurable menu of responses from the agent (merchant), facilitating, by the server, sending of the selected of one of the configurable menu of responses to the user (customer); and associating and recording the selected one of the configurable menu of responses with the first input text message.

Abstract: A computing system receives an instruction to initiate audio presentation of electronic communications for a recipient, and outputs an audio presentation responsive to the instruction. The audio presentation includes an initial portion that includes a presentation road map, and a subsequent portion that includes audible output of text content of a plurality of unreviewed electronic communications for the recipient. The presentation road map identifies an estimated duration of time to present the subsequent portion of the audio presentation.

Abstract: An input assistance system includes a terminal device including a display screen, an acquisition unit, a recognition unit, an input item display unit, a recognition result display unit, and a reception unit. The acquisition unit acquires utterance voice data of a user. The recognition unit performs voice recognition of the utterance voice data to generate text data. The input item display unit displays a plurality of input items including the input item associated with the text data. The recognition result display unit displays the text data. The reception unit accepts an operation of selecting the input item associated with the text data displayed by the recognition result display unit from the plurality of input items displayed by the input item display unit. The reception unit accepts the operation of selecting the input item associated with the text data when the plurality of input items and the text data are displayed.

Abstract: An information processing device presents first information indicating that voice input for the voice operation is possible and second information representing a domain of utterance in which voice operation is possible in response to an occurrence of a predetermined state transition, and performs voice recognition for voice input by a user.

Abstract: Systems and methods are disclosed herein for building contextual transcripts. A computing system may receive a textual transcript of a meeting that contains a variety of statements made by various attendees of the meeting, select the first statement made during the meeting, and determine which meeting attendee made the statement. A machine learning model corresponding to the particular attendee that has been trained using previously received statements by the particular attendee may be used on the utterance to determine the tone of the utterance. That tone may be recorded within the transcript and this process may be repeated for each utterance to build a contextual transcript.

Abstract: Methods and systems are provided herein for playing back indexed conversations based on the presence of other people. When a user asks a query, the system monitors the area, determines the other users in the area, and searches its database for a conversation that addresses the query in consideration of the other users present in the area. The system filters the indexed conversations to find conversations that included all the users present and determines the best matching conversation based on the words of the query as well as the keywords from the conversation. Once the system has determined the best match conversation, the system plays back the conversation to the user.

Abstract: A device may include a processor, a receiver, and a transmitter. The transmitter may be configured to transmit an audible signal, an inaudible signal, or both. The inaudible signal may be associated with a content identifier of the audible signal. The transmitter may be configured to transmit the audible signal, the inaudible signal, or both, to a first electronic device, a second electronic device, or both. The receiver may be configured to receive a first message that includes a first input and a second message that includes a second input. The processor may be configured to determine whether the first input matches the second input. The transmitter may be further configured to transmit the first message to the first service on a condition that the first input and the second input are determined to match.

Abstract: Disclosed are a display method and a display apparatus for operation prompt information of an input control. The display method includes: during a user's interaction with the display apparatus, obtaining an operation instruction from a user; in response to the operation instruction, determining a target interaction mode for the user's interaction with the display apparatus; in response to a start instruction of an input control generated by invoking of the input control in the target interaction mode, obtaining target operation prompt information for the input control; and generating the input box on the user interface and display the target operation prompt information in the input box.

Abstract: Provided is a quick-responsive voice control technique even in use in a planetarium. A control device of a projector of a planetarium includes: a storage unit that stores a plurality of commands for controlling the projector, flags indicating whether or not the respective commands can be executed, and keywords associated with the respective commands; a voice acquisition unit that acquires voice data; a control unit that controls the control device; and a communication unit that communicates with the projector.

Abstract: Methods and systems are disclosed for improving dialog management for task-oriented dialog systems. The disclosed dialog builder leverages machine teaching processing to improve development of dialog managers. In this way, the dialog builder combines the strengths of both rule-based and machine-learned approaches to allow dialog authors to: (1) import a dialog graph developed using popular dialog composers, (2) convert the dialog graph to text-based training dialogs, (3) continuously improve the trained dialogs based on log dialogs, and (4) generate a corrected dialog for retraining the machine learning.

Abstract: Apparatus transmits an identifier for association with a virtual area by an administering network service, generates output data from human perceptible stimulus in a physical space, transmits the output data in connection with the virtual area, receives input data associated with the virtual area, and generates human perceptible stimulus in the physical space from the input data. A persistent association is created between the apparatus and a virtual area. A respective presence is established in the virtual area for a communicant operating a client network node connected to the virtual area. A respective connection between each active pair of complementary sources and sinks of the client network node and the apparatus are administered in association with the virtual area. A client network node displays a graphical user interface, establishes the administered connections, and presents interaction controls associated with the object for interacting with communicants in the physical space.

Abstract: The disclosure discloses a method for generating a conversation, an electronic device, and a storage medium. The detailed implementation includes: obtaining a current conversation and historical conversations of the current conversation; selecting multiple reference historical conversations from the historical conversations and adding the multiple reference historical conversations to a temporary conversation set; and generating reply information of the current conversation based on the current conversation and the temporary conversation set.

Abstract: An apparatus for maintaining an event listing using voice control, the apparatus includes a sound capturing device configured to capture acoustic data and a computing device connected to the sound device configured to receive the acoustic data, identify a voice input based on the acoustic data using a voice recognition module, wherein the voice recognition module is configured to identify a target entity and identify event activation data, obtain entity data associated with the target entity containing historical event data, generate a voice-activated command using the voice input via a command interpretation module, wherein the command interpretation module is configured to determine a maintenance operation for an event related to the target entity as a function of event activation data and the historical event data, maintain an event listing using the voice-activated command by executing the at least a maintenance operation, and display the event listing using a user interface.