Abstract: A distributed indexing system spreads out the load on an index of stored data in a data storage system. Rather than maintain a single index, the distributed indexing system maintains an index in each media agent of a federated data storage system and a master index that points to the index in each media agent. In some embodiments, the distributed indexing system includes an index server (or group of servers) that handles indexing requests and forwards the requests to the appropriate distributed systems. Thus, the distributed indexing system, among other things, increases the availability and fault tolerance of a data storage index.

Abstract: The present invention extends to methods, systems, and computer program products for deriving document similarity indices. Embodiments of the invention include scalable and efficient mechanisms for deriving and updating a document similarity index for a plurality of documents. The number of maintained similarities can be controlled to conserve CPU and storage resources.

Abstract: When an index is partitioned and managed in a plurality of indexes in a search system which deals with large amounts of data, the number of documents registered in each partitioned index may be varied, and inefficient data transfer may occur at the time of index reconfiguration when search nodes are added or deleted. In the present invention, mapping information is used to map search node IDs to slot regions obtained by equally partitioning a circular hash value space in which a start point and an end point of the hash value space are assumed to be identical, so that each document is mapped to each partitioned index.

Abstract: Systems and methods for mapping and propagating a source dataset to a plurality of target tables in a target database are described herein. Embodiments include a physical database design with a template-based loader, and a method to propagate changes in the source dataset to a target database. Other embodiments include a database physical design with a plurality of small fact and summary tables. Still other embodiments include a method of extracting and loading data automatically into a target database, while simultaneously maintaining current summary tables.

Abstract: A computer-implemented method, system, and article of manufacture for determining a set of correlated data among heterogeneous computer applications. The method includes providing a computer system having software modules, receiving statistics on data relating to a first event and a second event, generating a confidence score for an attribute set, where the attribute set includes an attribute from the first event data and an attribute from the second event data, and selecting the attribute set as a set of correlated data if the confidence score is within a threshold value.

Abstract: An apparatus and method are described for the discovery of semantics, relationships and mappings between data in different software applications, databases, files, reports, messages, or systems. In one aspect, semantics and relationships and mappings are identified between a first and a second data source. A binding condition is discovered between portions of data in the first and the second data source. The binding condition is used to discover correlations between portions of data in the first and the second data source. The binding condition and the correlations are used to discover a transformation function between portions of data in the first and the second data source.

Abstract: A system and method for indexing and searching a free format data source is provided. Indexing the free format data source includes normalizing data from the free format data source and creating a master index based on the normalized data. The master index may be updated through synchronization with the free format data source when a change in the data occurs. A search query may be executed against the master index and/or one or more child indexes to search for records. The search query may be normalized and transformed before being executed. The master index may be replicated into the one or more child indexes for scaling and load balancing purposes. Searches may be performed by search agents in parallel at the master index and/or the child indexes.

Abstract: Embodiments disclosed herein provide systems and methods that can bridge the gap between business operations and data management operations. An interface layer provides customizable interfaces for interactions with a master entity index (MEI) system to accomplish association, storage, management or processing of data records based on user-defined business service operations. The interface layer allows a user to define business services, operations, and logical procedures according to a desired data model such that the inputs and outputs for each operation of the service correspond to the particular needs of the user and allow different terminology to be employed by the user.

Abstract: A mechanism is provided for efficiently searching a database for data associated with a key value that is unidentifiable in a set of nodes of the database. A request is received to retrieve data from the database, where the request specifies a searched-for key value that is unidentifiable in the set of nodes of the database. The data associated with the searched-for key value is identified in a lowest level node of the database between a current leftmost node and a current rightmost node. A key index is updated in at least one node of the database with the key index value associated with the data and identified from the lowest level node thereby making the searched-for key value identifiable within the at least one node. The data associated with the searched-for key value is then returned to a user.

Abstract: Updating an index is disclosed. An indication of one or more changes to be made to data is received. The one or more changes are written to a storage medium prior to updating one or more file name index entries. One or more file name index entries are updated prior to writing a new index tree root. A new index tree root is written.

Abstract: A file system driver stored in a computational device receives a request to provide access to a snapshot of a rewritable file system at a selected point in time, wherein a plurality of tokens corresponding to a plurality of data units are stored in the rewritable file system, and wherein an ordering relationship is maintained among the plurality of tokens. The file system driver determines a token corresponding to the selected point in time based on the ordering relationship. A virtual subdirectory name is generated corresponding to the determined token. A virtual subdirectory is created with the generated virtual subdirectory name, wherein the virtual subdirectory is a subdirectory of a root directory. The snapshot is accessed via the subdirectory.

Abstract: Provided are techniques for selecting row identifiers from an initial index structure storing rows of randomized indexes. The row identifiers are randomized. Groups are formed with the randomized row identifiers so that each group has a predetermined number of row identifiers. At least one group is selected from the groups. Indexes are retrieved from the initial index structure that correspond to the row identifiers in the selected at least one group. The retrieved indexes are encoded by adding product information to form new identifiers.

Abstract: A method and apparatus for proxying search requests for a storage system and maintaining a central index for performing the search requests is described herein. An index manager on the storage system may initially produce the central index by examining each file in a file system and update the central index thereafter by examining only those files that have changed since the central index was initially produced or last updated. The index manager may receive a changed file list from a differencing layer configured for comparing snapshots of the file system at different time points to produce changed file lists. A search proxy module may receive search requests in a search protocol and proxy the search requests to a search engine by converting the search requests to another search protocol compatible with the search engine. The search engine may then use the central index for performing the search request.

Abstract: A method for effectively utilizing database servers. A middle tier system, interfacing between clients and a database system, receives requests to create, delete or update a record from a client. An index is used by the middle tier system to determine if a record previously exists, and if so, its current record location. In this manner, the middle tier system keeps track of the locations of the records stored in the databases. By keeping track of the data load, the middle tier system identifies the most available resource to store the records. When a record is to be created, deleted or updated, the middle tier system rebalances the load capability of the database servers by distributing the data across the databases in a randomized manner based on the server load. In this manner, changes to the server load are more effectively managed.

Abstract: Provides is an apparatus for question answering based on answer trustworthiness including: an answer indexer that indexes documents of which document trustworthiness satisfying a threshold value among documents included in a document collection and stores it in a knowledge Bases; an answer candidate extractor that extracts answer candidate documents for a user's question from the knowledge Bases; an answer source trustworthiness measurement unit; an answer extraction strategy trustworthiness measurement unit; and a trustworthiness integrator that generates an answer candidate trustworthiness list by ranking the answer candidate documents on the basis of the document trustworthiness, the source trustworthiness, and the extraction strategy trustworthiness of the answer candidate documents.

Abstract: A server provides the user's desired data without the server knowing the contents or keywords of data by using a method of searching the desired data without decrypting the encrypted data, such that the privacy for the important data of the user can be secured. Also, the present invention shortens the operation time when searching the encrypted data, such that it can prevent the degradation in efficiency due to excess operation involved in the previous existing methods based on the pairing operation.

Abstract: The present application is directed to an indexing system. In one example, the indexing system includes one or more processors, one or more electronic memories that provide random access memory, one or more mass storage devices that provide persistent data storage, and one or more indexing routines, executed by the one or more processors, that create and manage an index data structure comprising nodes that include key-values/reference pairs, the index data structure additionally including a sibling reference for each index-data-structure node stored in the one or more electronic memories, the one or more indexing routines removing, from the index data structure, the sibling references for index-data-structure nodes when transferring the index-data-structure nodes from the one or more electronic memories to the one or more mass storage devices.

Abstract: According to some embodiments, a system and method for a parallel join of relational data tables may be provided by calculating, by a plurality of concurrently executing execution threads, hash values for join columns of a first input table and a second input table; storing the calculated hash values in a set of disjoint thread-local hash maps for each of the first input table and the second input table; merging the set of thread-local hash maps of the first input table, by a second plurality of execution threads operating concurrently, to produce a set of merged hash maps; comparing each entry of the merged hash maps to each entry of the set of thread-local hash maps for the second input table to determine whether there is a match, according to a join type; and generating an output table including matches as determined by the comparing.

Abstract: A system and method for efficiently reducing latency of accessing an index for a data segment stored on a server. A server both removes duplicate data and prevents duplicate data from being stored in a shared data storage. The file server is coupled to an index storage subsystem holding fingerprint and pointer value pairs corresponding to a data segment stored in the shared data storage. The pairs are stored in a predetermined order. The file server utilizes an ordered binary search tree to identify a particular block of multiple blocks within the index storage subsystem corresponding to a received memory access request. The index storage subsystem determines whether an entry corresponding to the memory access request is located within the identified block. Based on at least this determination, the file server processes the memory access request accordingly. In one embodiment, the index storage subsystem is a solid-state disk (SSD).

Abstract: A computer-implemented contact management method includes creating a contact record in a contact management system, generating a unique serial number corresponding to the contact record, conveying the serial number to a recipient, entering, by the recipient, the serial number into an internet-enabled computer application, requesting, by the application, the record from the contact management system corresponding to the serial number, and receiving data associated with the contact record. The received contact data may be automatically updated through an update function. A contact management system includes a data storage and a contact management server. A plurality of interne-enabled computer applications associated to the same user account may receive the same information using a synchronization function without further user input.

Abstract: Method and apparatus for indexing subblocks in a reduced-redundancy storage system. Each subblock is hashed to an K-bit key and an entry for the subblock added to an index data structure comprising of a tree of hash tables. In a further aspect, by replacing the top of the tree with an array, the data structure can achieve O(1) access time for random keys while still providing relatively smooth growth.

Abstract: A current key is received at a current arrival time at a computer. An index in an array corresponding to the current key is determined using a hash function. A previous key and a previous arrival time are retrieved from the array at the index. The array is transformed by replacing the previous key and the previous arrival time with the current key and the current arrival time in the array at the index. The previous key and the previous arrival time are inserted into a nearest eligible sequential index in the array.

Abstract: The inter-system communications techniques include a system, and/or a method. In some embodiments of these techniques, the method includes receiving data from a first system. The method further includes storing the received data in an in-memory database. The method further includes determining a second system to inform of the received data using a system assignment table. The system assignment table includes a source system identifier, a target system identifier, a first field, and a first field value. The method further includes informing the second system of the received data when the first field value matches a value associated with the first field of the received data. This Abstract is provided for the sole purpose of complying with the Abstract requirement rules. This Abstract is submitted with the explicit understanding that it will not be used to interpret or to limit the scope or the meaning of the claims.

Abstract: In embodiments of the disclosed technology, indexes, such as inverted indexes, are updated only as necessary to guarantee answer precision within predefined thresholds which are determined with little cost in comparison to the updates of the indexes themselves. With the present technology, a batch of daily updates can be processed in a matter of minutes, rather than a few hours for rebuilding an index, and a query may be answered with assurances that the results are accurate or within a threshold of accuracy.

Abstract: A distributed indexing system spreads out the load on an index of stored data in a data storage system. Rather than maintain a single index, the distributed indexing system maintains an index in each media agent of a federated data storage system and a master index that points to the index in each media agent. In some embodiments, the distributed indexing system includes an index server (or group of servers) that handles indexing requests and forwards the requests to the appropriate distributed systems. Thus, the distributed indexing system, among other things, increases the availability and fault tolerance of a data storage index.

Abstract: Techniques for extracting information from a database are provided. A database such as the Web is searched for occurrences of tuples of information. The occurrences of the tuples of information that were found in the database are analyzed to identify a pattern in which the tuples of information were stored. Additional tuples of information can then be extracted from the database utilizing the pattern. This process can be repeated with the additional tuples of information, if desired.

Abstract: Methods and systems provide a tool for prioritizing the ordering of outstanding indexing work in order to bring a particular portion of an indexing source up to date quickly and to reduce the likelihood of inconsistencies between an index-backed view and a direct view of a source. In accordance with the described embodiments, indexing of items can be prioritized based upon a user's view or metadata contained within a query. Further, in at least some embodiments, the tool can decide the order to index items based upon multiple prioritization requests.

Abstract: Disclosed herein is a method and system for hardware-accelerating the generation of metadata for a data stream using a coprocessor. Using these techniques, data can be richly indexed, classified, and clustered at high speeds. Reconfigurable logic such a field programmable gate arrays (FPGAs) can be used by the coprocessor for this hardware acceleration. Techniques such as exact matching, approximate matching, and regular expression pattern matching can be employed by the coprocessor to generate desired metadata for the data stream.

Abstract: Disclosed are methods and apparatus for organizing data in a database in a set-oriented manner. Data is organized by linking data in the form of key-value pairs stored in storage media of the database to corresponding key-value pair identifiers. A set having a corresponding set identifier is then associated with one or more of the key-value pair identifiers where the set includes the stored key-value pairs linked to the key-value pair identifiers.

Abstract: In general, one aspect of the subject matter described in this specification can be embodied in a system that includes a network-based information server system; and a computer operable to interact with a user interface device and operable to perform operations including: receiving from the network-based information server system status information corresponding to software applications, the status information including black-list information and phoenix-list information; updating a local cache, retained by the computer, with the status information received from the network-based information server system; generating an identifier for a software application, using a digital certificate and a hash value, when the software application is launched on the computer; checking the identifier for the software application against the status information; and notifying a user of the computer when a match is found during the checking.

Abstract: The present disclosure provides systems and methods for accessing secure and certified electronic messages using a combination of biometric security, a separate and secure network and email infrastructure, email management processes, and the addition of text, audio and visual format options to sending emails messages.

Abstract: A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Abstract: A current key is received at a current arrival time at a computer. An index in an array corresponding to the current key is determined using a hash function. A previous key and a previous arrival time are retrieved from the array at the index. The array is transformed by replacing the previous key and the previous arrival time with the current key and the current arrival time in the array at the index. The previous key and the previous arrival time are inserted into a nearest eligible sequential index in the array.

Abstract: Disclosed is a system in which an index registration unit registers an index, which will be used for search processing, as a partitioned index which is partitioned on a time series basis, and a search means reads indexes older than a specified point in time, which is used as a search base point, to perform search processing, thereby searching for information based on a point in time in the past.

Abstract: Methods and systems provide a tool for prioritizing the ordering of outstanding indexing work in order to bring a particular portion of an indexing source up to date quickly and to reduce the likelihood of inconsistencies between an index-backed view and a direct view of a source. In accordance with the described embodiments, indexing of items can be prioritized based upon a user's view or metadata contained within a query. Further, in at least some embodiments, the tool can decide the order to index items based upon multiple prioritization requests.

Abstract: A query that identifies an input data source is received. The input data source is partitioned into a plurality of partitions. Each of the partitions includes a set of data elements with an associated set of indices for indicating an ordering of the data elements. A query type for a query operator in the received query is identified. It is determined whether a reordering of data elements will be performed based on the identified query type. The data elements in at least one of the partitions are reordered when it is determined based on the identified query type that reordering will be performed.

Abstract: A process for reading entries in a directory is initiated. A first index is maintained to indicate how far the read has progressed in the directory. If, during execution of the process, the directory is partitioned into subdirectories, then a second index is maintained for each of the subdirectories to indicate how far the read has progressed in each of the subdirectories. A third index that indicates how far the read has progressed in the partitioned directory is also maintained.

Abstract: In a particular embodiment, a method includes storing, at a staging table of a database, a data update generated based on a transaction performed with respect to a data table that is associated with one or more indexes. Each index of the one or more indexes is maintained externally to the database. The method further includes maintaining the data update at the staging table at least until index update data is received at the database. The index update data represents a modification, based on the data update, to a particular index of the one or more the indexes.

Abstract: A system and method for maintaining version information. An identifier (“ID”) that identifies a collection of associated files is obtained. An index is generated that specifies the contents of the collection of associated files. The ID may be saved along with the index in a target version file to convey version information about the collection of associated files. Subsequently, the index may be extracted from the target version file to compare with a corresponding index extracted from a reference version file. The result of the comparison may be used to determine whether the contents of the collection of associated files match a reference.

Abstract: Overflow access records (OARs) are managed in a database system. An OAR is created in response to receiving an update command for a data record and to the updated data record generated by the update command not fitting onto the page in the table where the data record was stored. The OAR that is created includes an index counter that indicates a number of indexes associated with the table. When an OAR is accessed in response to a query command, an identifier of the accessed OAR is replaced in the index by an identifier of a data record pointed to by the OAR, and the index counter in the accessed OAR is changed by a predefined amount. When the index counter reaches a predefined value, the accessed OAR is removed from the table.

Abstract: A data storage apparatus of the present invention includes a data collector that collects time-series data and a sampler that calculates, for each piece of the data, a plurality of change indices indicating change in each piece of the data and determines whether or not the piece of data is to be sampled.

Abstract: Apparatus, methods and articles of manufacture for file integrity and baseline maintenance on a computer system or systems are shown. A file integrity check is made of files on a system by hashing the files and comparing the hash to a stored hash of the files. If the comparison is false, the system requests a copy of the file from a server, which may be unknown or untrusted. The system then compares the received file to the stored hash. If the comparison is true, the copy of the file will be installed on the system. If false, the system repeats the request.

Abstract: A method and system for performing at least one transaction on at least one column of a table in a database system is disclosed. The method includes determining whether a first transaction of the at least one transaction fulfills all of at least one criteria. The method further includes ensuring that the at least one column is in a shared numeric lock if the first transaction fulfills all of the at least one criteria, wherein the shared numeric lock is a lock that allows at least a second transaction fulfilling all of the at least one criteria to access the same column concurrently with the first transaction without releasing the shared numeric lock between transactions, and wherein the shared numeric lock is released only after all of the transactions fulfilling all of the criteria have completed processing. The method further includes processing the first transaction.

Abstract: Managing database indexes includes creating a main index and creating at least one service index that is configured for recording a change to a node to be updated in the main index. Managing database indexes also includes detecting whether an operation that involves the main index and is performed on the database appears in the database, and maintaining the main index using at least one service index in response to the operation that involves the main index and is performed on the database, appearing in the database. The maintaining is performed based on changes to a node to be updated in the main index that are recorded in the at least one service node.

Abstract: Systems and methods for performing an updating process to an in-memory index are provided. Upon receiving notice of document modifications covered by an inverted index associated with a search engine, in the form of an update file, a representation of the modification is published onto various index serving machines. Each index serving machine receiving the update file determines if the modifications are applicable to the index serving machine. If an index serving machine determines that it contains mapping information corresponding to the modified documents, the index serving machine utilizes the update file and associated mapping information to update an in-memory index. In embodiments, the in-memory index is used to provide results to user queries in tandem with the inverted index. In some embodiments, an extra in-memory index is maintained that is revised with constantly incoming metadata updates and the existing in-memory index is periodically swapped with the revised in-memory index.

Abstract: Systems and methods for applying user behavior data to improve search query result ranking are provided. Upon receiving an update file indicating that recent, significant user behavior data is available for a document associated with an inverted index, the update file is published periodically and frequently to an index server. After filtering out the relevant update information from the update file, the index server extracts identifiers of the documents having the associated user behavior data. The update file and the identifier of the documents are utilized to update an in-memory index containing representations of metadata indicative of the user behavior. The in-memory index is continuously updated and utilized to serve search query results in response to user search queries. Search query results from the in-memory index are ranked using the user behavior data prior to serving. Thus, results associated with recent, significant user-behavior metadata receive prominent placement on the search results page.

Abstract: The invention is directed to a computer-implemented process for processing information related to entities and said entities being contained in a stream of information, the entities being stored in resource directories of a system, each resource directory containing entities and being annotated with a version number modified with the addition of at least one new entity.

Abstract: A system and method for maintaining version information. An identifier (“ID”) that identifies a collection of associated files is obtained. An index is generated that specifies the contents of the collection of associated files. The ID may be saved along with the index in a target version file to convey version information about the collection of associated files. Subsequently, the index may be extracted from the target version file to compare with a corresponding index extracted from a reference version file. The result of the comparison may be used to determine whether the contents of the collection of associated files match a reference.

Abstract: A method for managing data includes receiving individually accessible data units, each identified by a key value; storing a plurality of blocks of data, each of at least some of the blocks being generated by combining a plurality of the data units; and providing an index that includes an entry for each of the blocks. One or more of the entries enable location, based on a provided key value, of a block that includes data units corresponding to a range of key values that includes the provided key value.

Abstract: A database of hyperlinks, stored in a hyperlink store or distributed across multiple machines such as a scalable hyperlink store, may be incrementally updated. When data is added, instead of modifying an existing data store, a hierarchy of data stores is built. The data stores are merged together, such that a new store is a suffix on an old store. Additions and updates go into new stores, which are relatively small. Lookups consult new stores first. A background thread merges adjacent stores. For example, a batch of updates is collected and incorporated into a new store and then the store is sealed. Subsequent updates are added to yet another new store. Stores are merged occasionally to prevent the chain of stores from becoming too long. Once the batch has been integrated, the new stores are sealed and are used to answer subsequent queries.