Abstract: A method for processing multi-modal input includes receiving multiple signal inputs, each signal input having a corresponding input mode. Each signal input is processed in a series of mode-specific processing stages. Each successive mode-specific stage is associated with a successively longer scale of analysis of the signal input. A fused output is generated based on the output of a series of fused processing stages. Each successive fused processing stage is associated with a successively longer scale of analysis of the signal input. Multiple fused processing stages receive inputs from corresponding mode-specific processing stages, so that the fused output depends on the multiple of signal inputs.

Abstract: Data augmentation is used for speech emotion recognition tasks where certain emotional labels, e.g., sadness, are significantly underrepresented in a training dataset. This is typical for data collected in real-life applications. We propose conditioned data augmentation using Generative Adversarial Networks (GANs), in order to generate samples for underrepresented emotions. We propose a conditional GAN architecture to generate synthetic spectrograms for the minority class. For comparison purposes, we implement a series of signal-based data augmentation methods. Results on the speech emotion recognition task show that the proposed data augmentation method significantly improves classification performance as compared to traditional speech data augmentation methods.

Abstract: A method for processing multi-modal input includes receiving multiple signal inputs, each signal input having a corresponding input mode. Each signal input is processed in a series of mode-specific processing stages. Each successive mode-specific stage is associated with a successively longer scale of analysis of the signal input. A fused output is generated based on the output of a series of fused processing stages. Each successive fused processing stage is associated with a successively longer scale of analysis of the signal input. Multiple fused processing stages receive inputs from corresponding mode-specific processing stages, so that the fused output depends on the multiple of signal inputs.

Abstract: Behavioral profiling and shaping is used in a “closed-loop” in that an interaction with at least one human is monitored and based on inferred characteristics of the interaction with that human (e.g., their behavioral profile) the interaction is guided. In one exemplary embodiment, the interaction is between two humans, for example, a “customer” and an “agent” and the interaction is monitored and the agent is guided according to the inferred behavioral profile of the customer (or optionally of the agent themselves).

Abstract: A method and apparatus for performing automatic speech recognition (ASR) in a distributed ASR system for use over a wireless channel takes advantage of probabilistic information concerning the likelihood that a given, portion of the data has been accurately decoded to a particular value. The probability of error in each feature in a transmitted feature set is employed to improve speech recognition performance under adverse channel conditions. Bit error probabilities for each of the bits which are used to encode a given ASR feature are used to compute the confidence level that the system may have in the decoded value of that feature. Features that have been corrupted with high probability are advantageously either not used or are weighted less in the acoustic distance computation performed by the speech recognizer.

Abstract: A system for, and method of, representing and resolving ambiguity in natural language text and a spoken dialogue system incorporating the system for representing and resolving ambiguity or the method. In one embodiment, the system for representing and resolving ambiguity includes: (1) a context tracker that places the natural language text in context to yield candidate attribute-value (AV) pairs and (2) a candidate scorer, associated with the context tracker, that adjusts a confidence associated with each candidate AV pair based on system intent.

Abstract: A system for, and method of, measuring a degree of independence of semantic classes in separate domains. In one embodiment, the system includes: (1) a cross-domain distance calculator that estimates a similarity between n-gram contexts for the semantic classes in each of the separate domains to determine domain-dependent relative entropies associated with the semantic classes and (2) a distance summer, associated with the cross-domain distance calculator, that adds the domain-dependent distances over a domain vocabulary to yield the degree of independence of the semantic classes.

Abstract: An unsupervised, discriminative, sentence level, HMM adaptation based on speech-silence classification is presented. Silence and speech regions are determined either using a speech end-pointer or the segmentation obtained from the recognizer in a first pass. The discriminative training procedure using a GPD or any other discriminative training algorithm, employed in conjunction with the HMM-based recognizer, is then used to increase the discrimination between silence and speech.

Abstract: Frequency warping approaches to speaker normalization have been proposed and evaluated on various speech recognition tasks. In all cases, frequency warping was found to significantly improve recognition performance by reducing the mismatch between test utterances presented to the recognizer and the speaker independent HMM model. This invention relates to a procedure which compensates utterances by simultaneously scaling the frequency axis and reshaping the spectral energy contour. This procedure is shown to reduce the error rate in a telephone based connected digit recognition task by 30%.

Abstract: An improved speech recognition system, in which transformation process parameters are generated in response to selected characteristics derived from speech inputs obtained from both carbon and linear microphones. The transformation process parameters are utilized in conjunction with selected digitized speech models to improve the speech recognition process based on the carbon microphone property of suppressing speech spectral energy for low energy invoiced sounds, and also for low energy regions of the spectrum between formant peaks for voices sounds.