Abstract: A method includes generating a base model by training with a first dataset of data pairs and generating an adapted model by training the base model on a second dataset of data pairs. The method also includes determining a contrastive score for each data pair of a third dataset of data pairs using the base model and the adapted model. The contrastive score is indicative of a probability of quality of the respective data pair. The method also includes training a target model using the data pairs of the third dataset and the contrastive scores.

Abstract: A method includes generating a base model by training with a first dataset of data pairs and generating an adapted model by training the base model on a second dataset of data pairs. The method also includes determining a contrastive score for each data pair of a third dataset of data pairs using the base model and the adapted model. The contrastive score is indicative of a probability of quality of the respective data pair. The method also includes training a target model using the data pairs of the third dataset and the contrastive scores.

Abstract: A method includes generating a base model by training with a first dataset of data pairs and generating an adapted model by training the base model on a second dataset of data pairs. The method also includes determining a contrastive score for each data pair of a third dataset of data pairs using the base model and the adapted model. The contrastive score is indicative of a probability of quality of the respective data pair. The method also includes training a target model using the data pairs of the third dataset and the contrastive scores.

Abstract: This invention provides a language processing apparatus analyzing a dependency structure of an input sentence, including: a segment unit model parameter estimating part estimating a segment unit model parameter as a parameter of a probability model to analyze the dependency structure by using a feature in a unit of segment; a sentence unit model parameter estimating part estimating a sentence unit model parameter as a parameter of a probability model to analyze the dependency structure by using a feature in a unit of sentence; a dynamic sample generating part generating a sample of the dependency structure of the input sentence from the probability model characterized by the sentence unit model parameter and the segment unit model parameter; and a dependency structure determining part determining a suitable dependency structure from the sample of the dependency structure of the input sentence generated by the dynamic sample generating part.

Abstract: The morphological analysis apparatus according to the present invention, comprises a spelling recovery unit that recovers the spellings of words, a morphological analysis candidate generation unit that segments a word sequence composed of words, the spellings of which have been recovered, into morphemes, appends POS tags to the morphemes and generates a single morphological analysis candidate or a plurality of morphological analysis candidates, a generation probability calculation unit that calculates a generation probability for each morphological analysis candidate having been generated based upon the product of the probability of the pre-spelling recovery word being converted to the post-spelling recovery word and the probability of a morpheme sequence and a POS sequence being generated from the post-spelling recovery word sequence and a solution search unit that selects through a search the most likely candidate as a solution from all the morphological analysis candidates for which the generation probabil

Abstract: A morphological analyzer divides a received text into known words and unknown words, divides the unknown words into their constituent characters, analyzes known words on a word-by-word basis, and analyzes unknown words on a character-by-character basis to select a hypothesis as to the morphological structure of the received text. Although unknown words are divided into their constituent characters for analytic purposes, they are reassembled into words in the final result, in which any unknown words are preferably tagged as being unknown. This method of analysis can process arbitrary unknown words without requiring extensive computation, and with no loss of accuracy in the processing of known words.

Abstract: An input text is analyzed into morphemes by using a prescribed morphological analysis procedure to generate word strings with part-of-speech tags, including form information for parts of speech having forms, as hypotheses. The probabilities of occurrence of each hypothesis in a corpus of text are calculated by use of two or more part-of-speech n-gram models, at least one of which takes the forms of the parts of speech into consideration. Lexicalized models and class models may also be used. The models are weighted and the probabilities are combined according to the weights to obtain a single probability for each hypothesis. The hypothesis with the highest probability is selected as the solution to the morphological analysis. By combining multiple models, this method can resolve ambiguity with a higher degree of accuracy than methods that use only a single model.