Abstract: Provided are methods and systems for suggesting an enhanced multimodal interaction. The method for suggesting at least one modality of interaction, includes: identifying, by an electronic device, initiation of an interaction by a user with a first device using a first modality; detecting, by the electronic device, an intent of the user and a state of the user based on the identified initiated interaction; determining, by the electronic device, at least one of a second modality and at least one second device, to continue the initiated interaction, based on the detected intent of the user and the detected state of the user; and providing, by the electronic device, a suggestion to the user to continue the interaction with the first device using the determined second modality, by indicating the second modality on the first device or the at least one second device.

Abstract: In some examples, a software agent executing on a server receives a communication comprising a first utterance from a customer and predicts, using an intent classifier, a first intent of the first utterance. Based on determining that the first intent is order-related, the software agent predicts, using a dish classifier, a cart delta vector based at least in part on the first utterance and modifies a cart associated with the customer based on the cart delta vector. The software agent predicts, using a dialog model, a first dialog response based at least in part on the first utterance and provides the first dialog response to the customer using a text-to-speech converter.

Abstract: A computer system may connect to various customer-facing devices and manage or automate the order process between a retail store and the customer. The computer system may perform the dialogue and receive an order for items from the retail store and may perform quality control monitoring of the dialogue between customers and employees taking orders. The ordering system may utilize the ordered items in combination with various contextual cues to determine a customer identity which may then be linked to past orders and/or various order preferences. Based on the determined customer identity, the system may provide recommendations of additional order items or order alterations to the customer before personally identifying information has been collected from the customer. The determination of the customer identity and the determination of recommendations may be performed by machine learning algorithms that were trained on customer data and the retail store products.

Abstract: Embodiments herein disclose a method for an enhanced interaction with a voice assistant using user accessories data by a first electronic device. The method includes detecting an utterance from a first user, where the utterance comprises a data item indicative of an identity of at least one second user. Further, the method includes determining at least one of position information and direction information of at least one wearable electronic device connected to the first electronic device of the first user. Further, the method includes determining the identity of the at least one second user indicated in the utterance of the first user based on at least one of the position and the direction of the at least one wearable electronic device connected to the first electronic device.

Abstract: Provided are methods and systems for suggesting an enhanced multimodal interaction. The method for suggesting at least one modality of interaction, includes: identifying, by an electronic device, initiation of an interaction by a user with a first device using a first modality; detecting, by the electronic device, an intent of the user and a state of the user based on the identified initiated interaction; determining, by the electronic device, at least one of a second modality and at least one second device, to continue the initiated interaction, based on the detected intent of the user and the detected state of the user; and providing, by the electronic device, a suggestion to the user to continue the interaction with the first device using the determined second modality, by indicating the second modality on the first device or the at least one second device.

Abstract: In some aspects, a computing device receives a scan of a code displayed on an order post located near a restaurant, determines that the code is associated with the restaurant, and automatically opens a software application and navigates the software application to an ordering page associated with the restaurant. The computing device initiates receiving, via the software application, input associated with an order, sends the input to a machine learning based software agent executing on a server, receives a predicted response to the input, provides the predicted response as audio output and/or displays the predicted response on the touchscreen display device. After the order is complete, the computing device sends order data associated with the order to the restaurant. After receiving an indication from the restaurant that the order is ready, the computing device indicates that the order is ready to be picked up.

Abstract: In some examples, a server may receive an utterance from a customer. The utterance may be included in a conversation between the artificial intelligence engine and the customer. The server may convert the utterance to text and determine a customer intent based on the text and a user history. The server may determine a user model of the customer based on the text and the customer intent. The server may update a conversation state associated with the conversation based on the customer intent and the user model. The server may determine a user state based on the user model and the conversation state. The server may select, using a reinforcement learning based module, a particular action from a set of actions, the particular action including a response and provide the response to the customer.

Abstract: In some aspects, a computing device receives a scan of a code displayed on an order post located near a restaurant, determines that the code is associated with the restaurant, and automatically opens a software application and navigates the software application to an ordering page associated with the restaurant. The computing device initiates receiving, via the software application, input associated with an order, sends the input to a machine learning based software agent executing on a server, receives a predicted response to the input, provides the predicted response as audio output and/or displays the predicted response on the touchscreen display device. After the order is complete, the computing device sends order data associated with the order to the restaurant. After receiving an indication from the restaurant that the order is ready, the computing device indicates that the order is ready to be picked up.

Abstract: A computer system may connect to various customer-facing devices and manage or automate the order process between a retail store and the customer. The computer system may perform the dialogue and receive an order for items from the retail store and may perform quality control monitoring of the dialogue between customers and employees taking orders. The ordering system may utilize the ordered items in combination with various contextual cues to determine a customer identity which may then be linked to past orders and/or various order preferences. Based on the determined customer identity, the system may provide recommendations of additional order items or order alterations to the customer before personally identifying information has been collected from the customer. The determination of the customer identity and the determination of recommendations may be performed by machine learning algorithms that were trained on customer data and the retail store products.

Abstract: In some examples, a server may receive, by a menu manager executing on the server, a menu having a particular format from a point-of-sale (POS) terminal and parse the menu to create a parsed menu. The menu is parsed using POS data indicating a formatting of the menu. The server automatically converts the parsed menu into multiple menu items, stores the multiple menu items in a menu item database, and creates a mapping database that includes pricing data, pronunciation data, and voice tags associated with individual menu items of the multiple menu items. The server provides multiple software agents access to the mapping database and instructs individual software agents of the one or more software agents to initiate a conversation with a customer to receive a voice-based order. The individual software agents comprise an instance of an artificial intelligence engine.

Abstract: In some examples, a software agent executing on a server receives a communication comprising a first utterance from a customer and predicts, using an intent classifier, a first intent of the first utterance. Based on determining that the first intent is order-related, the software agent predicts, using a dish classifier, a cart delta vector based at least in part on the first utterance and modifies a cart associated with the customer based on the cart delta vector. The software agent predicts, using a dialog model, a first dialog response based at least in part on the first utterance and provides the first dialog response to the customer using a text-to-speech converter.

Abstract: In some examples, a server may receive an utterance from a customer. The utterance may be included in a conversation between the artificial intelligence engine and the customer. The server may convert the utterance to text and determine a customer intent based on the text and a user history. The server may determine a user model of the customer based on the text and the customer intent. The server may update a conversation state associated with the conversation based on the customer intent and the user model. The server may determine a user state based on the user model and the conversation state. The server may select, using a reinforcement learning based module, a particular action from a set of actions, the particular action including a response and provide the response to the customer.

Abstract: In some examples, a software agent executing on a server an utterance from a customer. The software agent converts the utterance to text. The software agent creates an audio representation of the text and performs a comparison of the audio representation and the utterance. The software agent creates edited text based on the comparison. For example, the software agent may determine, based on the comparison, audio differences between the audio representation and the utterance, create a sequence of edit actions based on the audio differences, and apply the sequence of edit actions to the text to create the edited text. The software agent outputs the edited text as a dialog response to the utterance.

Abstract: In some examples, a software agent executing on a server receives a communication comprising a first utterance from a customer and predicts, using an intent classifier, a first intent of the first utterance. Based on determining that the first intent is order-related, the software agent predicts, using a dish classifier, a cart delta vector based at least in part on the first utterance and modifies a cart associated with the customer based on the cart delta vector. The software agent predicts, using a dialog model, a first dialog response based at least in part on the first utterance and provides the first dialog response to the customer using a text-to-speech converter.

Abstract: In some aspects, an order post detects, using one or more sensors, a presence of a customer, determines an identity of the customer, retrieves previous orders of the customer, indicates at least one item in the previous orders, receives an order comprising input that includes an utterance of the customer, modifies the utterance to create a modified utterance, sends the modified utterance to a software agent comprising a natural language processor and one or more classifiers, receives a predicted response to the modified utterance from the software agent, plays back the predicted response via the speaker, determines that the order is complete, receives payment information for the order from the customer, sends order data associated with the order to a restaurant, receives an indication from the restaurant that the order is ready for pickup, and instructs the customer to pick up the order.

Abstract: A computer system may connect to various customer-facing devices and manage or automate the order process between a retail store and the customer. The computer system may perform the dialogue and receive an order for items from the retail store and may perform quality control monitoring of the dialogue between customers and employees taking orders. The ordering system may utilize the ordered items in combination with various contextual cues to determine a customer identity which may then be linked to past orders and/or various order preferences. Based on the determined customer identity, the system may provide recommendations of additional order items or order alterations to the customer before personally identifying information has been collected from the customer. The determination of the customer identity and the determination of recommendations may be performed by machine learning algorithms that were trained on customer data and the retail store products.