APPARATUS FOR CLASSIFYING OR DISAMBIGUATING DATA
A computing system has a data storage device (4, 5, 6) for storing a database consisting of a classified vocabulary of terms. A processor (1) of the apparatus is arranged to associate each term with one of a number of different categories of data and to associate all terms falling within the same category with a common code identifying a collocation of terms that exemplify that category so that terms in different categories are associated with different codes and can be disambiguated. The processor (1) is arranged to write, directly or indirectly, a classified vocabulary consisting of the terms together with the associated code onto a computer-readable storage medium (RDD2) or to supply an electrical signal via, for example a MODEM (10) or a LAN/WAN (11). The database may be used in classification of documents, spelling checking of documents and refining of keyword search results.
Latest CRYSTAL REFERENCE SYSTEMS LIMITED Patents:
This application is a Continuation of and is based upon and claims the benefit of priority under 35 U.S.C. §120 for U.S. Ser. No. 11/949,571, filed Dec. 3, 2007 which is a Continuation of U.S. Ser. No. 10/990,534, filed Nov. 18, 2004, which is a Continuation of U.S. Ser. No. 09/412,754, filed Oct. 5, 1999 and claims the benefit of priority under 35 U.S.C. §119 from British Patent Application No. 9821787.0, filed Oct. 6, 1998, the entire contents of each which are incorporated herein by reference.
This invention relates to apparatus for classifying or processing data. In particular this invention is concerned with apparatus for enabling use, storage, disambiguating or manipulating of an item of data in accordance with a category, for example a subject matter area, within which that item of data is determined to fall.
Classification schemes are used to enable items of data in a particular category to be retrieved either from a physical location or electronically. Various different specific classification schemes exist. Thus, for example, the Dewey Decimal, Universal Decimal and Library of Congress classification schemes are all used to classify library material to enable librarians and other people using a library to identify the location of books and other publications by title, by author or by subject matter. In addition, international standard industry codes exist to classify commercial products and the Whittacker system classifies living organisms. Each of these existing classification schemes is thus particular to a certain type of subject matter and, moreover, requires that each individual item of data such as a book or publication be manually classified to enable its subsequent retrieval using the classification scheme.
Since such manual classification is a time-consuming and costly activity, several attempts have been made to devise a means of automatically classifying documents, primarily by comparing words in the document with words known to occur frequently in particular subject areas. Such an approach is described in WO97/10557. Where the words in the document include sufficient of the frequently-occurring subject words, the document is determined to be about that subject. A drawback to this approach is that when a large number of subject areas are involved, the speed of comparison may be slow. It is also the case that, since this approach is based on word frequencies, a document which contains unusual words may be classified incorrectly.
The Internet provides, via the world wide web, access to a large amount of data. A number of search engines are available via the world wide web to enable retrieval of documents containing text on a specific topic. To retrieve documents relating to a specific topic, a keyword (which may consist of one or more terms) is entered and the search engine then searches for documents available electronically via the world wide web and containing that keyword. The results of the search are then collated and the titles displayed to the user who can then access the individual documents. However, such search engines are extremely inefficient frequently returning very large numbers of ‘hits’ or documents which are not directly related to the search because, in many cases, it is not possible to identify precisely the field of enquiry simply by means of a keyword. For example, if the keyword is ‘depression’, documents relating to each of the meteorological, economic and medical meanings of the term ‘depression’ will be retrieved. Some search engines seek to improve results by offering additional keywords for selection by the user in order to expand the search term. Such keywords are generally based on frequency counts and may therefore exclude the required subject area if this is less common.
It is an aim of the present invention to provide an apparatus for classifying terms in a manner which can be universal and which enables more efficient and accurate identification and extraction of terms relating to a specific desired topic or subject matter area, so enabling disambiguation.
In one aspect, the present invention provides apparatus for storing data on a computer readable storage medium having means for associating all terms falling within a common category with a common code identifying a collocation associated with that category and means for directly or indirectly writing each term together with the associated code onto a computer readable storage medium. The writing means may be arranged also to write the collocation for the associated code onto the computer readable storage medium. The writing means may be replaced or supplemented by signal generating means for generating a signal carrying each term together with the associated code and optionally also the associated collocation.
The categories may comprise different subject matter areas which are desirably sufficient to encompass all data currently available in the world. Typically, the subject matter areas may be the universe, the earth, the environment, natural history, humanity, recreation, society, the mind, human history and human geography. Each of these subject matter areas may be divided into smaller subject matter areas which may themselves in turn be divided into even smaller subject matter areas. Desirably, each category comprises a combination of a subject matter area and a species or genus with each item of data being allocated to only one species or genus. Typically, there may be five species or genus which may consist of, for example, people, places, organisations, products and terminology with the latter genus including general concepts within the subject matter area. The classification of terms into both subject matter areas and genera enables efficient and accurate retrieval of terms in a context specific manner and enables a distinction to be made between the use of the same term as the name of the person, the name of a place and the name of an organisation, for example.
In one aspect, the present invention provides apparatus for storing data on a computer readable storage medium, comprising:
means for storing terms;
means for associating each term with one of a number of different subject matter areas;
means for associating each term with one of a number of different species areas such that each item of data is associated with one or more subject matter areas but only with one species area; and
means for directly or indirectly writing each term onto a computer readable storage medium in association with a code or codes identifying the associated subject matter and species areas.
The writing means may be replaced or supplemented by means for generating a signal carrying the same data as is written onto the computer readable storage medium.
In one aspect, the present invention provides apparatus for processing data by determining which of a number of collocations each associated with a specific different category is relevant to a received term.
In one aspect, the present invention provides apparatus for checking the spelling of terms in a text which comprises means for determining a category relevant to the text and means for highlighting or otherwise identifying to a user terms which may have been incorrectly used. Such apparatus may desirably comprise: means storing a vocabulary and means for comparing the terms used in the text with the terms in the vocabulary to identify any terms in the text not present in the vocabulary; means for determining, when unknown terms are identified in the text, likely possible alternative terms in the vocabulary that have the same category and means for advising a user of the possible alternative term or terms. Such apparatus may be used as part of a word processing arrangement to check the spelling of terms or words in a text document. Such apparatus may also be used to check, where the spelling is correct, that none of the terms used in the text being checked are inappropriate for the determined category of the document.
In one aspect, the present invention provides apparatus for classifying a text which comprises means for comparing terms used in the text with the terms used in a classified vocabulary in which classified terms are associated with categories and means for allocating a classification code to the text in accordance with the results of the comparison. The text to be classified may be supplied in a computer readable form or may be optically scanned and then converted into a computer readable form using known optical character recognition software. Such apparatus enables text to be classified automatically without the need for a person skilled in the subject matter area of the text or in document classification to study the text to determine the subject matter area to which the text relates.
In one aspect, the present invention provides apparatus for refining the results of a subject matter search carried out by a search engine using a keyword, for example an Internet search engine, the apparatus comprising:
means for accessing a plurality of collocations, each collocation being associated with a respective different one of a number of categories;
means for determining whether the keyword falls in one or more of the different categories and, if the keyword used falls within a number of different categories, advising a user of these different categories; user operable selection means for selecting one of the determined categories;
means for comparing the terms used in each text located by the search with the terms in the collocation associated with the selected category; and
means for filtering the search results in accordance with the number of terms the search result texts have in common with the collocation associated with the selected category.
The present invention also provides a computer usable storage medium carrying processor implementable instructions for causing operation of apparatus according to any of the aspects referred to above.
The present invention also provides a computer readable storage medium or signal carrying the results of operation of apparatus in accordance with any one of the aspects referred to above.
Embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which:
For ease of understanding definitions of several of the terms or phrases used herein will now be given.
As used herein the phrase “item of data” means an entry in the classified vocabulary that includes a term, its description and at least one of a corresponding category identification and a classification code.
As used herein the word “term” means a term which may consist of one or more words (including made up words, proper nouns, etc.) or abbreviations and which may have one or more different meanings but which, for a given meaning, conveys a single concept. It will be understood that a single term may have more than one meaning. Thus, for example, the term “depression” has a number of meanings including a meteorological, a medical and an economic meaning.
As used herein “classification scheme” means the set of subject matter areas or domains and associated genera used to classify terms.
As used herein “category” means a specific combination of the subject area and genus in which a term is classified.
As used herein “classification code” means the code allocated to a term and which identifies the category within which the term falls.
As used herein “category identification” means a code unique to a classification code and a particular collocation.
As used herein “classified vocabulary” means a set of terms classified in accordance with the classification scheme.
As used herein “classification data set” means a set of items each consisting of a collocation, a characterisation or description of that collocation and at least one of the corresponding category identification and classification code.
As used herein “collocation” means a collection of terms (not necessarily organised in any specific order) that exemplify a category of data and which would frequently be found in documents that should fall within that category.
As used herein “keyword” means a search term (which may be made up of one or more words and/or abbreviations) entered by a user.
As shown in
As shown in
In the classification scheme, each of these major subject matter areas is divided into subsidiary subject matter areas or subsidiary domains.
Each (major or subsidiary) subject area or domain has five species areas or genera 23 which are, in this example, people, locations, products, organisations and terminology. The genus ‘product’ includes the names by which anything may be sold which will include, in addition to trade names and trade marks, song and book titles, for example. The genus ‘terminology’ includes general concepts in the related subject matter area or domain. Any one item of data can belong only to one genus although it may belong to more than one (major or subsidiary) subject matter area or domain. Thus, each meaning of a term in the classified vocabulary will be allocated to a specific category in the classification scheme with the specific category being defined by its allocated major and subsidiary subject matter areas or domains and its allocated genus. This facilitates differentiation between use of the same word as a common noun, a person's name and the name of an organisation because the database treats the three different meanings of the same word as being different terms because they are allocated to different ones of the five genera.
To facilitate understanding of the database structure, specific examples will be given below.
Thus, a term which relates to space exploration will be classified in the subsidiary subject matter area or domain (SPA) within the major subject matter area or domain (UN). Each classified term within the subsidiary subject matter area (SPA) will then be allocated to one of the five genera. Thus, for example, terms consisting of the names of astronauts, cosmonauts and mission control personnel will be allocated to the genus ‘people’ and so to a category defined by the combination of the subject matter and the genus with, in this example, a classification code: UN SPA SPAP, where the latter four letter term indicates the genus, that is people (P), in the subsidiary domain SPA. In contrast, terms consisting of the names of space exploration organisations will be allocated to the genus ‘organisations’ and will have a category or classification code: UN SPA SPAORG where the last three letters of the final part of the code indicates that the genus is the organisation genus.
To take another example, one of the subsidiary subject matter areas of the major subject matter area or domain ‘the earth’ is climate (CLI) and the field of meteorology is classified at: EA CLI. Terms consisting of the names of meteorologists are classified in category: earth-climate-people (classification code EA CL CLIP) while the term “the UK meteorological office” is classified in the category: earth-climate-organisations (classification code EA CLI CLIORG). The term “UK meteorological office” may also be classified in: human geography; Europe; UK; organisations (classification code HG EU UKIORG) to enable it to be identified as a UK organisation independently of its existence within the field of meteorology.
It will, of course, be appreciated that the above subsidiary subject matter areas are examples only and that the person skilled in the art may adopt or add different subject matter divisions. Generally, however, the ten major subject matter areas or domain will be those given above. Similarly, the five particular genus selected are exemplary and it is possible that alternative genera may be used. What is, however, important is that all terms are classified in accordance with the classification scheme with each classified term being allocated to one or more specific subject matter areas (which may be a subsidiary subject matter area within a major or other subsidiary subject matter area) but only to one specific one of the available genera so as to enable disambiguation between different meanings of the same word, phrase or abbreviation.
As illustrated schematically by
Each entry in the classified vocabulary may also include a field 35 for containing part of speech (for example noun, verb, adjective, adverb) information to assist in document classification and fields 36 and 37 for containing inflected forms and abbreviations and derivatives so that the classified vocabulary need contain only an entry for the root term and does not require separate entries for inflections, derivatives and abbreviations.
Two examples of vocabulary entries are shown below. These entries omit, in the interests of clarity, inflections and abbreviations or derivatives.
EXAMPLE 1
Each different category (that is each specific combination of subject matter subsidiary domain and genus) is associated with a unique classification scheme data set item CL in the classification scheme data set.
Each classification scheme data set item CL includes the corresponding classification code and collocation for the category and a characterisation which gives a brief description of the category.
As noted above, the collocation consists of terms that exemplify the category and which would frequently be found in documents that should fall within the category. For example, a collocation will include terms which may be used to describe the function, appearance or relationship with other objects of the classified terms in the associated category or any other terms (for example ‘buy’, ‘sell’ in relation to cars) which may generally be used in the same context as the classified terms. For example, where the item of data is the term ‘depression’ in the economic sense, then terms which may be included in the corresponding collocation include: economy, employment, low, poor, poverty, market, social, failure, money, jobs etc.
It should, of course, be understood that the classification scheme data set items CL are in no way the same as the set of sub-headings which will generally be found in a standard library classification under each subject matter heading. Such sub-headings are analogous to the subsidiary subject matter domains mentioned above in that they define subject matter areas or specific topics which fall within the main headings. Such sub-headings do not relate to terms which may be used in discussing or describing items of data falling within the category or heading.
The collocations for the categories recognised within the classification system are determined using a mixture of encyclopaedic and lexicographical criteria. They are not just subject lexicons in the usual sense; for example, as a test case, a collocation lexicon for the category of meat within nutrition would include terms for various kinds of meat foodstuffs (lamb, pork, beef, poultry, etc) but also general words to do with the category (eat, cook, joint, fat, grilled, etc).
The collocations do not just identify domain A compared with domain B (e.g. meteorology vs literature), but levels of sub-domain within a domain (e.g. literature vs novel vs types of novel). The terms within the collocations are derived from three main sources:
1) Encyclopaedic Sources Including:
-
- i) relevant headwords and words within entries belonging to a particular domain, as displayed in encyclopedias such as The Cambridge Encyclopedia, and associated publications; and
- ii) relevant headwords taken from specialist sources outside of the above, for example place-names for a particular country from atlases, environmental terms from the indexes of various specialised works on the environment.
2) Lexicographic Sources Including:
-
- i) relevant headwords taken from dictionaries such as the Chambers Dictionary; and
- ii) relevant headwords taken from conceptually and alphabetically organised thesauri.
3) Other Sources Such as:
-
- i) relevant words found in a set of records after searching a particular subject matter domain on the Internet;
- ii) relevant words taken from a frequency listing of words in a set of Internet records; and
- iii) human input from a person collating the collocations using the above information.
The terms providing a collocation may be grouped within the collocation, according to their relevance to the category.
Where a classified vocabulary entry 30 gives, as shown in
The attached Appendix A lists examples of items classified vocabulary entries and the associated classification scheme data set items.
Section 1 of Appendix A lists two entries in the classified vocabulary both relating to the word ‘bayonet’. The first example given in Appendix A is for the term ‘bayonet’ when used in the term of a light bulb fitting while the second entry is for the term ‘bayonet’ when used in the context of a camera lens fitting. As can be seen from Appendix A, these two meanings of the term ‘bayonet’ have different category IDs with the category ID for the light bulb fitting being 00010 and the category ID for the camera lens fitting being 0020 in this example.
Section 2 of Appendix A shows the classification scheme data set items identified by the category numbers 00010 and 00020. As can be seen from Appendix A, each classification scheme data set item is headed by its category ID followed by the classification code defined by the code for the main domain followed by the code for each subsidiary domain with these in turn being followed by the collocation only a part of which is shown in Appendix A for each of the two classification scheme data set items.
Terms to be classified using the apparatus shown in
Individual terms may be manually classified by the user using the input device. Thus, the processor 1 will first cause the display 7 to display the table shown in
Once the term to be classified has been entered into the cell 31a, the user then enters in the cell 32a a description in the form of a word or phrase describing the general nature or subject matter area of the term. For example, where the term is ‘depression’ in the economic sense as mentioned above, then the description entered by the user may be ‘economics’.
Once the user has entered the description, the processor 1 prompts the user to enter a definition of the specific term into cell 33a. Where the term is ‘depression’ then the user may enter: ‘a period of low business and industrial activity accompanied by a rise in unemployment’ or some other similar short description.
The category ID may be determined manually by the user referring to a hard copy list of the classification codes or may be determined using the computer. Thus, for example, the processor may first request the user to select one of the ten major subject matter areas or domains and then, once the major subject matter area or domain has been selected, request the user to select one of the available subsidiary domains and, once the subsidiary domain has been selected, a subsidiary domain of that domain if it exists, and so on. Once the subject area subsidiary domain has been determined, the processor may then request the user to select the required genus. Once the user has done this, then the processor 1 determines the classification code and category ID from a classification code key stored in memory (for example in the ROM 3 or on the hard disk 4). Once the category ID has been determined and entered in the cell 34a, the processor 1 may prompt the user to enter, in turn, data indicating the part of speech in cell 35a, details of inflected forms in cell 36a and details of abbreviations and derivatives in cell 37a. Where the processor 1 has access to a dictionary, for example, where an electronic dictionary is stored on the hard disc drive 4 or on a removable disc inserted into one of the drives RD1 and RD2 or an electronic dictionary is accessible via the LAN/WAN interface 11 or over the Internet then the processor 1 may be programmed to determine inflections, abbreviations and derivatives automatically from electronically available dictionary sources. Once the data has been entered in cell 37a, then the processor 1 may request the user to confirm that the entry is correct and, once this has been done, the processor will store the classified term in the classified vocabulary so that the category ID determined by the user links the classified term to the appropriate item in the classification scheme data set.
Once all the desired terms have been classified, the classified vocabulary consisting of the classified terms each with their description, definition and category ID may be written onto a removable disk of the removable disk disk drive 5 or 6 or supplied as a signal to, via a network or the Internet, for example, another computing system. It will be appreciated that although the classified vocabulary may change or need to be updated fairly frequently, updating or changing of the classification scheme data set may be required less frequently. Accordingly, because the classification scheme data set would generally constitute a relatively large amount of data which requires infrequent modification, the classification scheme data set may be stored separately from the classified vocabulary, for example on a separate CD ROM. It will, of course, be appreciated that the computer apparatus shown in
For convenience, the classified vocabulary and classification scheme data set may both be written by the processor onto a removable disk which may be, for example, a writable CD (compact disc) or both be supplied as a signal to another computing system. Where the classified vocabulary is specific to one or more of the subject matter areas 21 shown in
The database described above comprising the classified vocabulary and the classification scheme consisting of the classification scheme data set has many applications. For example, once the processor 1 has access to the classified vocabulary and the classification scheme data set, text documents can be classified automatically using the apparatus shown in
In order for the computer apparatus to classify a text document it must, of course, be in computer readable form. Where the text document is supplied as an electrical signal via the LAN/WAN 11, the MODEM 10 or via a removable disk inserted into one of the removable disk disk drives 5 and 6, this will already be the case. Where the document to be classified is not in an electronic form, then the scanner 12 and conventional optical character recognition software may be used to convert the text document into a form readable by the computer. As another possibility, the text may be entered verbally if the computing system has speech recognition software.
Whichever way the text document is provided to the computing system, it is first stored on the hard disk 4. The processor 1 then reads the document at step S1, matches the terms used in the text document being classified against the classified vocabulary at step S2, identifies (at step S3) the classification codes of the terms found in both the classified vocabulary and the text document by using the classified vocabulary and classification scheme data set (see
Weighting of the classification codes may be carried out according to a number of different parameters and the criteria for assigning a classification code with confidence will vary from application to application. However, one way of weighting the classification codes which works well in practice is to assign each term in the text document a total weighting of one and to divide that total weighting by the number of classification codes which may relate to that term so that where a term has a number of different senses (such as the term “depression”, for example) the processor 1 will identify the classification code for each sense and will assign each classification code a weighting of 1/n where n is the number of classification codes identified for the term. Another approach is for the processor 1 to assign a weighting only to terms associated with the single classification codes, however this does not give good results in practice. Another alternative approach is for the processor 1 to process the text document sentence by sentence, determine a weighted classification code for each sentence and then to combine the sentence classification codes. Provided the processor 1 has access to some elementary grammatical rules (for example stored on the hard disc drive), this approach enables the processor 1 to take advantage of the part of speech information in the classified vocabulary to assist in differentiating between different senses of the same word. Generally extremely frequent words such as “a”, “the”, “but”, “and”, “can”, “it” etc. will be ignored in step S2.
The description above with reference to
The automatic classification software may also provide a user with a mechanism for overriding or modifying an automatic allocated classification code. For example, the instructions supplied to the processor may cause a user to be alerted via the display 7 if the processor 1 has been unable to allocate a classification code or codes to the text document, so allowing the user to classify such documents manually.
Initially, when a user wishes to search for documents relating to a particular topic, the user activates one of the search engines available on the world wide web causing a user interface similar to that shown in
Once the user interface has been displayed, the user is prompted to enter the required search keyword in box 41 and then to instigate the search by, for example, positioning the cursor using the mouse or other pointing device over the phrase ‘Search Now!’ and then clicking.
Once the user has initiated the search, the search engine carries out the search in conventional manner. However, when the search engine returns the results of the search, the processor 1 intercepts and stores these before displaying them to the user and reads the search keyword input by the user (step S6 in
The processor 1 then checks the classified vocabulary of the database for matches to the keyword used to initiate the search (step S7). Where matches in different categories (which may or may not be genus specific) are identified, the processor 1 reads the description from the classified vocabulary for each term and displays it to the user with a request for the user to select the category required (step S8).
The user interface prompts the user to enter the desired domain in box 42 in
If the user enters the required domain in
The processor then determines at step S13 which of the search results documents have at least a predetermined number of matches with the collocation terms and then displays to the user at step S14 only those search results documents having at least the predetermined number of collocation terms. If the domain ‘other’ is selected, the processor lists those documents not containing (or containing the least number of) terms used in the collocations associated with the other three domains. The processor may order the search results in accordance with the number of matches with the collocation terms of the selected domain and may list all of the search results in an order determined by the number of matches with the selected collocation with the highest number of matches being listed first or may display a given number of the search results for example the first ten or twenty search results to the user.
By using the collocations, the processor 1 can disambiguate different meanings of the same word and the search results produced by the search engine can be refined so as to select only those documents which use terms relevant to or which would be used in discussing or describing the keyword in the subject matter area or domain selected by the user. Thus, the search results relating to the use of the term ‘AA’ in subject matter areas different from the one selected by the user can be filtered out so that, for example, if the user selects the domain: ‘AA:HEALTH’, he will be provided with only the documents relating to Alcoholics Anonymous and not documents relating to the Automobile Association or anti-aircraft weapons.
A further application of the database will now be described with reference to
Commonly used software applications such as word processors, databases and spreadsheets need to be able to validate words. However, current spelling checkers are extremely limited in their application. For example, most current spelling checkers cannot identify place names, product names, company names and the names of people, particularly surnames, where these words are not also common nouns.
The spelling checkers of such word processors, database and spreadsheets may, however, be modified using the apparatus described above and the classification scheme data set to enable far more accurate verification of text.
In this example, the dictionary of a conventional spelling checker is replaced by the database described above. When instructed to verify the text, the processor first reads the document at step S20, compares the terms used in the document with the classified vocabulary of the database at step S21, identifies at step S22 any terms not in the vocabulary then matches at step S23 the document terms against the terms in classified vocabulary so as to determine at step S24 the domain having the most matches so as to determine the subject matter area and so the classification code of the document. This is carried out in a similar manner to the automatic document classification discussed above with reference to
Once the subject matter area of the document has been determined, the processor 1 at step S25 checks for terms in classified vocabulary which have the same classification code as that allocated to the document and are closest in spelling to the unknown term and displays these to the user at step S26. This enables the selection of the possible alternatives for the unknown word or term to be specifically directed toward the subject matter of the document being checked so that inappropriate alternatives are not presented.
The above description suggests that a single general database consisting of the classified vocabulary and the accompanying classification scheme data set will be provided. This need, however, not be the case. Rather, the contents of the database provided may be specific to the requirements of the user with, for example, a particular user perhaps only being provided with the classified vocabulary for a specific subject matter area or areas and the associated classification scheme data set item or items. Additionally, the general database or a specific such database may be supplemented by additional classified terms specific to a particular user's requirements. Thus, individual lists of specialist classified terms may be prepared and supplied together with related items of the classification scheme data set. Examples of such specialist classified vocabulary lists are, for example, lists of pharmaceutical compound names and chemical names for the pharmaceutical industry, specialist lists of persons involved in a specific field, for example a list of all recognised chemists in a particular field or all recognised scientists such as, for example, people like Einstein, Oppenheimer, Newton etc.
Such classified lists may provide a key to standardised data and therefore greatly improve retrieval of data from a database. At present, some companies may have their own internal standards or authority files to ensure that employees are using the same terminology but with the growing use of the Internet and intranets there is a fast growing need for standard data than can be used for all organisations around the world. Classified lists provide a powerful way of establishing standard specialist vocabularies. Such specialist vocabulary classified lists may be used, for example, to supplement word processing spell checkers such as those described above with reference to
The apparatus described above may also be used to index documents. Thus, for example, where specialist classified lists are provided, then documents in the field of the specialist classified list may be indexed in accordance with that list. For example, the processor 1 may index documents in the field of chemistry in accordance with the names of recognised chemists appearing in those documents by comparing the terms used in the documents with specialist classified lists accessible to the processor 1 and then indexing each document under each term in the specialist classified list identified in the document. This would enable, for example, a researcher to identify all papers published by a specific person identified in the classified list or to extract all documents referring to each of a number of persons identified in the classified list.
As noted above, because the database is classified both as to subject matter and as to genus, it enables the processor 1 to validate words including proper nouns which are stored in the classified vocabulary, to differentiate between semantic items, for example the use of the word ‘wood’ as a surname or as a material, to identify the use of common terms as also being names of products, to provide via the classified lists variants on forms or spellings of names such as Vienna/Wien and to provide, again via the classified lists, lists of specialist terms for example all chemical compounds, all mathematicians, all units of currency as required by the end user. Moreover, because the classification scheme is modular, an end user may be supplied with only a part of the classified vocabulary specific to his particular needs with the associated classification scheme data set items without having to make any modifications to the classified vocabulary. Furthermore, the subject matter areas or domains can easily be refined by the addition of deeper and deeper levels of subsidiary domains without disturbing the overall structure of the database.
The classified vocabulary or items of data may be provided in different languages. Different classification scheme data sets will however be required for different languages because there is not always a direct correlation in meaning. The apparatus described above may be used to assist in translation of documents. In order to achieve this, the apparatus is given access to two different language versions of the database and to an electronically stored conventional dictionary providing translations of the source language into the required final language. In order to assist in the translation of the document, the apparatus first determines, in a manner similar to that described above with reference to
As noted above as used herein, the term ‘collocation’ means a collection or list of terms which exemplify the domain or category with which the collocation is associated. However, the collocations may be ranked so that the terms within each collocation are arranged in order of significance. For example, the terms used in the collocation may be split into a number of groups of terms with the groups of terms being ordered in accordance with their significance to the domain with which they are associated. This would enable, where necessary or desired, limited numbers of the groups of terms to be used by the computing system. Limiting the number of terms in the collocation which are actually used in practice to those of most significance in relation to the subject matter area should facilitate more rapid carrying out by the computing system of the processors described above, for example, searching, classification or spell checking, with only a slight degradation in accuracy.
The classification scheme discussed above with reference to
Although in the arrangements described above, each specific category is associated with a particular classification scheme data set item and thus with a specific collocation, items of data of different genus but falling within the same subject matter area or domain may share a collocation because frequently the same terms will be used in relation to items of data falling within different genus in the same subject matter area.
In the arrangement described above with reference to
In the above examples, the classified vocabulary consists of classified terms. Conceivably, however, the classified vocabulary may be images, music or other sounds or non-textual matter. Of course, manual classification will be necessary if the items of data are not accompanied by related text.
It will be appreciated that the processor implementable instructions for causing the processor 1 to carry out any of the operations described above may be supplied via a storage medium insertable into a removable disk disk drive as discussed above. Alternatively, or additionally, the computer or processor implementable instructions can be supplied as a signal by, for example, downloading the code over a network which may be an intranet or the Internet. An aspect of the present invention thus provides a storage medium storing processor implementable instructions for controlling the processor to carry out one or more of the processes described above. Another aspect of the present invention provides an electrical signal carrying processor implementable instructions for controlling the processor to carry out one or more of the methods described above.
As noted above, the database for use by the apparatus may be supplied on a storage medium insertable into one of the removable disk disk drives or may be accessed remotely as a signal downloaded over a network such as the Internet or an intranet. Also, the classification scheme data set may be supplied separately from the classified vocabulary or items of data. The present invention thus also provides a storage medium storing a classified vocabulary or items of data and/or the classification scheme data set or items therefrom as discussed above. The present invention also provides an electrical signal carrying a classified vocabulary and/or the or some of the items from the classification scheme data set as discussed above.
In one aspect, the present invention provides apparatus for storing data on a computer readable storage medium, comprising:
means for storing items of data;
means for associating each item of data with one of a number of different categories of data;
means for associating all items of data falling within the same category with a common code identifying a collocation of terms that exemplify that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation of terms being specific to the associated category; and
means for directly or indirectly writing each item of data together with the associated code onto a computer readable storage medium.
In one aspect, the present invention provides apparatus for storing data on a computer readable storage medium, comprising:
means for storing items of data;
means for storing a plurality of different collocations of terms with the terms in each different collocation being terms that exemplify a specific different one of a plurality of categories of data;
means for associating each item of data with one of said number of different categories of data;
means for associating all items of data falling within the same category with a common code identifying which one of said collocations contains terms that exemplify items of data in that category so that items of data in different categories are associated with different codes identifying different ones of said collocations of terms; and
means for directly or indirectly storing the plurality of collocations and each item of data together with its associated code onto a computer readable storage medium.
In one aspect, the present invention provides apparatus for storing data on a computer readable storage medium, comprising:
means for storing items of data;
means for associating each item of data with one of a number of different species of data and one of a number of different subject matter areas such that the associated species and subject matter area define a category for that item of data;
means for associating all items of data falling within the same category with a common code identifying a collocation of terms that exemplify that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation of terms being specific to the associated category; and
means for directly or indirectly writing each item of data together with the associated code onto a computer readable storage medium.
In one aspect, the present invention provides apparatus for storing data on a computer readable storage medium, comprising:
means for storing items of data;
means for storing a plurality of different collocations of terms with the terms in each different collocation being terms that exemplify items of data falling within a specific different combination of one of a number of different species of data and one of a number of different subject matter areas such that the associated species and subject matter area define a category for that item of data;
means for associating each item of data with a category;
means for associating all items of data falling within the same category with a common code identifying which one of said collocations contains terms exemplifying items of data in that category so that items of data in different categories are associated with different codes identifying different ones of said collocations of terms; and
means for directly or indirectly storing the plurality of collocations and each item of data together with its associated code onto a computer readable storage medium.
In one aspect, the present invention provides apparatus for processing computer usable data, comprising:
means for storing items of data;
means for associating each item of data with one of a number of different categories of data;
means for associating all items of data falling within the same category with a common code identifying a collocation of terms usable in relation to items of data in that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation of terms being specific to the associated category; and
means for generating a signal carrying each item of data together with its associated code for supply to a computer readable storage medium.
In one aspect, the present invention provides apparatus for processing computer usable data, comprising:
means for storing items of data;
means for storing a plurality of different collocations of terms with the terms in each different collocation exemplifying items of data falling within a specific different one of a plurality of categories of data;
means for associating each item of data with one of said number of different categories of data;
means for associating all items of data falling within the same category with a common code identifying which one of said collocations contains terms exemplifying items of data in that category so that items of data in different categories are associated with different codes identifying different ones of said collocations of terms; and
means for generating a signal carrying each item of data together with its associated code for supply to a computer readable storage medium.
In one aspect, the present invention provides apparatus for processing computer usable data, comprising:
means for storing items of data;
means for associating each item of data with one of a number of different species of data and one of a number of different subject matter areas such that the associated species and subject matter area define a category for that item of data;
means for associating all items of data falling within the same category with a common code identifying a collocation of terms usable in relation to items of data in that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation of terms being specific to the associated category; and
means for generating a signal carrying each item of data together with its associated code for supply to a computer readable storage medium.
In one aspect, the present invention provides apparatus for storing data on a computer readable storage medium, comprising:
means for storing items of data;
means for storing a plurality of different collocations of terms with the terms in each different collocation exemplifying items of data falling within a specific different combination of one of a number of different species of data and one of a number of different subject matter areas such that the associated species and subject matter area define a category for that item of data;
means for associating each item of data with a category;
means for associating all items of data falling within the same category with a common code identifying which one of said collocations contains terms usable in relation to items of data in that category so that items of data in different categories are associated with different codes identifying different ones of said collocations of terms; and
means for generating a signal carrying each item of data together with its associated code for supply to a computer readable storage medium.
In one aspect, the present invention provides a computer usable medium having computer readable instructions stored therein for causing the computer:
to associate each of a plurality of items with one of number of different categories;
to associate all the items of data falling within the same category with a common code identifying a collocation of terms exemplifying items of data in that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation of terms being specific to the associated category; and
to generate a signal carrying each item of data together with its associated code for supply to a computer readable storage medium.
In one aspect, the present invention provides a computer usable medium having computer readable instructions stored therein for causing the computer:
to associate each of a plurality of items of data with one of a number of different species of data and one of a number of different subject matter areas such that the associated species and subject matter area define a category for that item of data;
to associate all items of data falling within the same category with a common code identifying a collocation exemplifying items of data in that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation of terms being specific to the associated category; and
to generate a signal carrying each item of data together with its associated code for supply to a compute readable storage medium.
In one aspect, the present invention provides a computer usable medium having computer readable instructions stored therein for causing the computer:
to associate each of a plurality of items of data with one of a number of different categories of data;
to associate all items of data falling within the same category with a common code identifying a collocation of terms exemplifying items of data in that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation being specific to the associated category; and
directly or indirectly to write each item of data together with the associated code onto a computer readable storage medium.
In one aspect, the present invention provides a computer usable medium having computer readable instructions stored therein for causing the computer:
to associate each of a plurality of items of data with one of a number of different species of data and one of a number of different subject matter areas such that the associated species and subject matter area define a category for that item of data;
to associate all items of data falling within the same category with a common code identifying a collocation of terms exemplifying that category so that items of data in different categories are associated with different codes identifying different collocations of terms with each collocation of terms being specific to the associated category; and
directly or indirectly to write each item of data together with the associated code onto a computer readable storage medium.
In one aspect, the present invention provides apparatus for processing data comprising:
means for accessing from storage means a plurality of collocations of terms with each collocation being associated with a different category of data and containing terms exemplifying that category;
means for receiving items of data;
means for determining a collocation which is relevant to a received item of data; and
means for processing the received item of data using terms from that collocation.
In one aspect, the present invention provides apparatus for checking the spelling of terms in a text, comprising:
means for receiving the text to be checked;
means for accessing first storage means storing a plurality of different collocations of terms with the terms in each collocation being usable in relation to a particular different category;
means for accessing second storage means storing a vocabulary with each term in the vocabulary being associated with a respective code identifying a specific one of said different collocations and a specific category for each different context or meaning of the term;
means for comparing the terms used in the text with the terms in the vocabulary to identify any terms in the text not present in the vocabulary;
means for, when unknown terms not present in the vocabulary are identified, comparing the rest of the terms in the text with the terms in the collocations to determine the collocation which has terms most closely matching the terms of the text to determining the category to which the text should be allocated;
means for determining any term in the vocabulary associated with the determined category for which the unknown term may be a misspelling; and
means for advising a user of the determined term(s).
In one aspect, the present invention provides apparatus for classifying a text into one of a number of different subject matter categories, comprising:
means for receiving the text to be classified;
means for accessing storage means storing a plurality of different collocations of terms with the terms in each collocation being usable in relation to a particular subject matter category and each collocation being associated with a classification code identifying the particular subject matter category to which the collocation is relevant;
means for comparing terms used in the text with the terms in the collocations;
means for determining which of the collocations has the most terms in common with the text being classified; and
means for allocating to the text the classification code associated with the determined collocation.
In one aspect, the present invention provides apparatus for refining the results of a subject matter search carried out by a search engine using a keyword, comprising:
means for accessing first storage means storing a plurality of different collocations of terms with the terms in each collocation exemplifying a particular different subject matter category;
means for accessing second storage means storing a vocabulary with each term in the vocabulary being associated with a respective code identifying a specific one of said different collocations and a specific category for each different context or meaning of the term;
means for receiving the results of the subject matter search;
means for comparing the keyword used to carry out the search with the term in the vocabulary to determine each category with which the keyword is associated;
means for advising a user of the different categories with which the keyword is associated;
user operable selection means for selecting one of the categories with which the keyword is associated;
means for comparing the terms used in text in each of the search results with the collocation of terms of the selected category; and
means for advising the user of the search results for which the text has greater than a predetermined number of terms in common with the collocation for the selected category.
In one aspect, the present invention provides apparatus for checking the usage of terms in a text, comprising:
means for receiving the text to be checked;
means for accessing first storage means storing a classified vocabulary in which the terms are allocated to categories;
means for comparing terms in the text with the terms in the classified vocabulary to determine a category for the text; and
means for identifying any terms not in the classified vocabulary; and
means for advising the user of any term in the classified vocabulary similar to an unidentified term and having the determined category.
Other modifications will be apparent to those skilled in the art.
Appendix A: Data Samples
1. Classified Vocabulary
- TERM bayonet
- DESCRIPTION technology
- DEFINITION type of fitting for a light bulb in which prongs on its side fit into slots to hold it in place
- CAT ID 00010
- TERM bayonet
- DESCRIPTION Photography
- DEFINITION type of fitting for a camera lens in which prongs on its side fit into slots to hold it in
- CAT ID 00020
2. Classification Scheme
- CAT ID=00010
- DOMAIN MI SUBDOMAIN TEC SUBDOMAIN POW SUBDOMAIN POWGEN COLLOCATIONS; A; AF; AGR; CAD; Calor gas; EP; P; acceptor; accident; accumulator; acoustic coupler; actuator; adapter; adaptor; advanced gas-cooled reactor; afterdamp; alternating current; alternator; ambisonics; ammeter; amp; amplification; amplifier; analogue-to-digital converter; anode; anthracite; antinuclear; armature; audio; audiometer; bank; barrel; battery; bayonet; bell; bezel; binaural; biological shield; bipolar; bipolarity; blackout; bleep; blip; bloop; blow-out; blow; boiler; booster; bore; borehole; bowser; brakeman; brakesman; brazier; breadboard; break; breed; breeder reactor; bridge; briquet; briquette; bromine; brush; bulb; bunker; burn-up; butane; button cell battery; button cell; buzzer; bypass; cable; cage; candle; capacitor; capstan; ceramic stratus; chemical laser; codec; coder/decoder; cut-out; cut; damp; damper; deck; derrick; diaphragm; diesel; diffuser; disc; discharge; dross; earth; electro; element; envelope; excitant; exciter; excitor; fantail; feedback; feeder; fender; fidelity; filament; filter; fireman; flasher; flashlight; flip side; flip-flop; fuel; fuse; gain; gap; gas; gate; geyser; kieselguhr; oiler; outage; paraffin . . .
- <CAT ID=00020><BRANCH><DOM>MI<SUBDOM>TEC<SUBDOM&g-t;OPT<SUBDOM>OPTGEN</BRANCH><COLLS>; Betacam; Betamax; Brownie; Calotype; Overcoat; PAL; aberration; achromat; achromatic; adaptive optics; aliasing; amplifier; anaglyph; anamorphic lens; aperture synthesis; aperture; apochromat; aspect ratio; atomic force microscope; autofocus; automatic exposure; autotype; b/w; back projection; bath; bayonet; bellows; bifocal; binocular; black and white; blimp; blow-up; blue-backing shot; box camera; bromide paper; bromine; bromoil; bull's-eye; camcorder; camera lucida; camera obscura; camera; carbro; color cinematography; color negative; colorization; colour cinematography; colour negative; conforming; coronagraph; couplers; daguerreotype; develop; developer; diaphragm; dolly; emulsion; exposure; film; filter; fix; fixer; flash; flashlight; flood; fog; frame; freezeframe; gauge; ghost; meniscus; microdot; mil; monitor; mount; negative; nosepiece; objective; ocular; opaque; pan . . . .
Claims
1. A computer processing apparatus for classifying a document, comprising:
- means for accessing a database structure providing a plurality of different subject matter categories, the database containing a classified vocabulary consisting of terms in all of the different subject matter categories with each term being classified in accordance with the subject matter category structure of the database;
- means for receiving in computer-readable form a text documents to be classified;
- processor means operable to compare terms appearing in the text document with the terms in the classified vocabulary and to determine from the comparison the category for the document; and
- means for supplying a signal carrying data representing the text document and data associating the text document with the determined category.
Type: Application
Filed: Dec 11, 2009
Publication Date: Apr 15, 2010
Applicant: CRYSTAL REFERENCE SYSTEMS LIMITED (HOLYHEAD)
Inventors: JUDI VERNAU (CLAPHAM), DAVID CRYSTAL (HOLYHEAD)
Application Number: 12/636,482
International Classification: G06F 17/30 (20060101);
