Abstract: A method and system for language identification are provided. The system includes a feature set of a plurality of character strings of varying length with associated information. The associated information includes one or more significance scores for a character string for one or more of a plurality of languages. Means are provided for detecting character strings from the feature set within a token from an input text. The system uses a finite-state device and the associated information is provided as glosses at the final nodes of the finite-state device for each character string. The associated information can also include significance scores based on linguistic rules.

Abstract: A method and system for language identification are provided. The system includes a feature set of a plurality of character strings of varying length with associated information. The associated information includes one or more significance scores for a character string for one or more of a plurality of languages. Means are provided for detecting character strings from the feature set within a token from an input text. The system uses a finite-state device and the associated information is provided as glosses at the final nodes of the finite-state device for each character string. The associated information can also include significance scores based on linguistic rules.

Abstract: A method and arrangement for handling case and other orthographic variations in linguistic databases by explicit representation comprising: explicit storage of all orthographic and case variations of words in the dictionary, and use of extended cut and paste codes to control dictionary size explosion and to make the restoration of the lemma more efficient. This provides the advantage of allowing very efficient handling of case and orthographic variants while performing a dictionary lookup.