Abstract: Methods and arrangements for identifying burden comprehension in multimedia content. A contemplated method includes: accepting multimedia input; detecting components of the multimedia input; determining a comprehension burden score of each of the detected components; and thereupon calculating a total comprehension burden score for the multimedia input. Other variants and embodiments are broadly contemplated herein.

Abstract: Methods and arrangements for identifying burden comprehension in multimedia content. A contemplated method includes: accepting multimedia input; detecting components of the multimedia input; determining a comprehension burden score of each of the detected components; and thereupon calculating a total comprehension burden score for the multimedia input. Other variants and embodiments are broadly contemplated herein.

Abstract: A manual annotation system of multi-modal characteristics in multimedia files. There is provided an arrangement for selection an observation modality of video with audio, video without audio, audio with video, or audio without video, to be used to annotate multimedia content. While annotating video or audio features is isolation results in less confidence in the identification of features, observing both audio and video simultaneously and annotating that observation results in a higher confidence level.

Abstract: An arrangement for yielding enhanced audio features towards the provision of enhanced audio-visual features for speech recognition. Input is provided in the form of noisy audio-visual features and noisy audio features related to the noisy audio-visual features.

Abstract: Methods and apparatus for enabling a Pervasive Authentication Domain. A Pervasive Authentication Domain allows many registered Pervasive Devices to obtain authentication credentials from a single Personal Authentication Gateway and to use these credentials on behalf of users to enable additional capabilities for the devices. It provides an arrangement for a user to store credentials in one device (the Personal Authentication Gateway), and then make use of those credentials from many authorized Pervasive Devices without re-entering the credentials. It provides a convenient way for a user to share credentials among many devices, particularly when it is not convenient to enter credentials as in a smart wristwatch environment. It further provides an arrangement for disabling access to credentials to devices that appear to be far from the Personal Authentication Gateway as measured by metrics such as communications signal strengths.

Abstract: An arrangement for yielding enhanced audio features towards the provision of enhanced audio-visual features for speech recognition. Input is provided in the form of noisy audio-visual features and noisy audio features related to the noisy audio-visual features.

Abstract: The use of multiple complementary classes of measure to assess face and speech consistency in video. In an exemplary embodiment, both synchrony measures and plausibility measures are employed.

Abstract: A manual annotation system of multi-modal characteristics in multimedia files. There is provided an arrangement for selection an observation modality of video with audio, video without audio, audio with video, or audio without video, to be used to annotate multimedia content. While annotating video or audio features in isolation results in less confidence in the identification of the features, observing both audio and video simultaneously and annotating that observation results in a higher confidence level.

Abstract: A system for activating a microphone based on visual speech cues, in accordance with the invention, includes a feature tracker coupled to an image acquisition device. The feature tracker tracks features in an image of a user. A region of interest extractor is coupled to the feature tracker. The region of interest extractor extracts a region of interest from the image of the user. A visual speech activity detector is coupled to the region of interest extractor and measures changes in the region of interest to determine if a visual speech cue has been generated by the user. A microphone is turned on by the visual speech activity detector when a visual speech cue has been determined by the visual speech activity detector. Methods for activating a microphone based on visual speech cues are also included.

Abstract: An arrangement for yielding enhanced audio features towards the provision of enhanced audio-visual features for speech recognition. Input is provided in the form of noisy audio-visual features and noisy audio features related to the noisy audio-visual features.

Abstract: The use of multiple complementary classes of measure to assess face and speech consistency in video. In an exemplary embodiment, both synchrony measures and plausibility measures are employed.

Abstract: In accordance with the invention, a method and system for accessing a dialog system employing a plurality of different clients, includes providing a first client device for accessing a conversational system and presenting a command to the conversational system by converting the command to a form understandable to the conversational system. The command is interpreted by employing a mediator, a dialog manager and a multi-modal history to determine the intent of the command based on a context of the command. A second client device is determined based on a predetermined device preference stored in the conversational system. An application is abstracted to perform the command, and the results of the performance of the command are set to the second client device.

Abstract: A method of speech driven lip synthesis which applies viseme based training models to units of visual speech. The audio data is grouped into a smaller number of visually distinct visemes rather than the larger number of phonemes. These visemes then form the basis for a Hidden Markov Model (HMM) state sequence or the output nodes of a neural network. During the training phase, audio and visual features are extracted from input speech, which is then aligned according to the apparent viseme sequence with the corresponding audio features being used to calculate the HMM state output probabilities or the output of the neutral network. During the synthesis phase, the acoustic input is aligned with the most likely viseme HMM sequence (in the case of an HMM based model) or with the nodes of the network (in the case of a neural network based system), which is then used for animation.

Abstract: A method for noise-robust speech processing with cochlea filters within a computer system is disclosed. This invention provides a method for producing feature vectors from a segment of speech, that is more robust to variations in the environment due to additive noise. A first output is produced by convolving a speech signal input with spatially dependent impulse responses that resemble cochlea filters. The temporal transient and the spatial transient of the first output is then enhanced by taking a time derivative and a spatial derivative, respectively, of the first output to produce a second output. Next, all the negative values of the second output are replaced with zeros. A feature vector is then obtained from each frame of the second output by a multiple resolution extraction.