Abstract: The present teaching relates to method, system, medium, and implementation for activating an animatronic device. Information about a user is obtained for whom an animatronic device is to be configured to carry out a dialogue with the user. The animatronic device includes a head portion and a body portion and the head portion is configured based on one of a plurality of selectable head portions. One or more preferences of the user are identified from the obtained information and used to select, from the plurality of selectable head portions, a first selected head portion. A configuration of the head portion of the animatronic device is then configured based on the first selected head portion for carrying out the dialogue.

Abstract: The present teaching relates to method, system, medium, and implementations for managing a user machine dialogue. A request is received by a server from a device for a response to be directed to a user engaged in a dialogue with the device. The request includes information related to a current state of the dialogue. The response is determined based on a dialogue tree and the information related to the current state of the dialogue. A sub-dialogue tree, which corresponds to a portion of the dialogue tree, is then created based on the response and the dialogue tree and is then used to generate a local dialogue manager for the device. The response, the sub-dialogue tree, and the local dialogue manager are then sent to the device, wherein the local dialogue manager, once deployed on the device, is capable of driving the dialogue with the user based on the sub-dialogue tree on the device.

Abstract: The present teaching relates to method, system, medium, and implementations for enabling communication with a user. Information representing surrounding of a user engaged in an on-going dialogue is received via the communication platform, wherein the information includes a current response from the user in the on-going dialogue and is acquired from a current scene in which the user is present and captures characteristics of the user and the current scene. Relevant features are extracted from the information. A state of the user is estimated based on the relevant features and a dialogue context surrounding the current scene is determined based on the relevant features. A feedback directed to the current response of the user is generated based on the state of the user and the dialogue context.

Abstract: The present teaching relates to method, system, medium, and implementations for managing a user machine dialogue. Sensor data is received at a device, including an utterance representing a speech of a user engaged in a dialogue with the device. The speech of the user is determined based on the utterance and a response to the user is searched by a local dialogue manager residing on the device against a sub-dialogue tree stored on the device. The response, if identified from the sub-dialogue tree, is rendered to the user in response to the speech. A request is sent to a server for the response, if the response is not available in the sub-dialogue tree.

Abstract: The present teaching relates to method, system, medium, and implementations for configuring an animatronic device. Information about a user is obtained for whom an animatronic device is to be configured to carry out a dialogue with the user and is used to select, from a plurality of selectable programs, a program related to a topic to be covered in the dialogue, where the program is to be used by the animatronic device to drive the dialogue with the user. The animatronic device is then configured based on the program for carrying out the dialogue with the user.

Abstract: The present teaching relates to method, system, medium, and implementations for understanding a three dimensional (3D) scene. Image data acquired by a camera at different time instances with respect to the 3D scene are received wherein the 3D scene includes a user or one or more objects. The face of the user is detected and tracked at different time instances. With respect to some of the time instances, a 2D user profile representing a region in the image data occupied by the user is generated based on a corresponding face detected and a corresponding 3D space in the 3D scene is estimated based on calibration parameters associated with the camera. Such estimated 3D space occupied by the user in the 3D scene is used to dynamically update a 3D space occupancy record of the 3D scene.

Abstract: The present teaching relates to method, system, medium, and implementations for enabling communication with a user. Information representing surrounding of a user to be engaged in a new dialogue is received via the communication platform, wherein the information is acquired from a scene in which the user is present and captures characteristics of the user and the scene. Relevant features are extracted from the information. A state of the user is estimated based on the relevant features, and a dialogue context surrounding the scene is determined based on the relevant features. A topic for the new dialogue is determined based on the user, and a feedback is generated to initiate the new dialogue with the user based on the topic, the state of the user, and the dialogue context.

Abstract: The present teaching relates to method, system, medium, and implementations for detecting a source of speech sound in a dialogue. A visual signal acquired from a dialogue scene is first received, where the visual signal captures a person present in the dialogue scene. A human lip associated with the person is detected from the visual signal and tracked to detect whether lip movement is observed. If lip movement is detected, a first candidate source of sound is generated corresponding to an area in the dialogue scene where the lip movement occurred.

Abstract: The present teaching relates to method, system, and medium for cross network communications. Information related to an application running on a user device is first received, which includes a state of the application and sensor data obtained with respect to a user interacting with the application on the user device. A request is sent to an application server for an instruction of a state transition of the application. A light weight model (LWM) for an object involved in the state transition is received and is personalized based on at least one of the sensor data and one or more preferences related to the user to generate a personalized model (PM) for the object, which is then sent to the user device.

Abstract: The present teaching relates to method, system, medium, and implementations for user machine dialogue. An instruction is received by an agent device for rendering a communication directed to a user involved in a dialogue in a dialogue scene and is used to render the communication. A first representation of a mindset of the agent is updated accordingly after the rendering. Input data are received in one or more media types capturing a response from the user and information surrounding the dialogue scene and a second representation of a mindset of the user is updated based on the response from the user and the information surrounding of the dialogue scene. A next communication to the user is then determined based on the first representation of the mindset of the agent and the second representation of the mindset of the user.

Abstract: The present teaching relates to method, system, medium, and implementations for identifying object of interest. Image data acquired by a camera with respect to a scene are received. One or more users are detected, during a period of time, from the image data who are present at the scene. Three dimensional (3D) gazing rays of the one or more users during the period of time are estimated. One or more intersections of such 3D gazing rays are identified and are used to determine at least one object of interest of the one or more users.

Abstract: The present teaching relates to method, system, medium, and implementations for speech recognition. An audio signal is received that represents a speech of a user engaged in a dialogue. A visual signal is received that captures the user uttering the speech. A first speech recognition result is obtained by performing audio based speech recognition based on the audio signal. Based on the visual signal, lip movement of the user is detected and a second speech recognition result is obtained by performing lip reading based speech recognition. The first and the second speech recognition results are then integrated to generate an integrated speech recognition result.

Abstract: The present teaching relates to method, system, medium, and implementations for user machine dialogue. Historic dialogue data related to past dialogues are accessed and used to learn, via machine learning, expected utilities. During a dialogue involving a user and a machine agent, a representation of a shared mindset between the user and the agent is obtained to characterize the current state of the dialogue, which is then used to update the expected utilities. Continuous expected utility functions are then generated based on the updated expected utilities, wherein the continuous expected utility functions are to be used in determining how to conduct a dialogue with a user.

Abstract: The present teaching relates to method, system, medium, and implementations for configuring an animatronic device. Information about a user is obtained for whom an animatronic device is to be configured to carry out a dialogue with the user. One or more preferences of the user are identified from the obtained information and are used to select, from the plurality of selectable profiles, a first selected profile, which specifies parameters to be used to control a manner by which the animatronic device is to communicate with the user. The animatronic device is then configured based on the first selected profile for carrying out the dialogue in the manner specified.

Abstract: The present teaching relates to method, system, medium, and implementations for configuring an animatronic device. Information is received about performance of a user exhibited in a dialogue between the user and an animatronic device which conducts the dialogue with the user in accordance with a configuration. The effectiveness of the configuration with respect to the user is assessed based on the information about the performance of the user and is used for machine learning at least one model, which is then used to adjust the configuration to generate an updated configuration for the animatronic device to use to continue the dialogue.

Abstract: The present teaching relates to method, system, medium, and implementations for rendering a moving object. An object data package related to a moving object appearing in a monitored scene with respect to a first time instance is first received and features characterizing the moving object at the first time instance are extracted from the package, that are estimated at a monitoring rate and include a current position of the object and a current motion vector at the first time instance. Information associated with a previously rendered object at a previously rendered position at a previous time instance is retrieved and a next rendering position of the object is determined based on the current position, the current motion vector, and a rendering rate lower than the monitoring rate. The object is rendered at the next rendering position based on a motion vector and the information associated with the previously rendered object.