Abstract: A memory stores therein, with respect to a database containing records each having a first data item and a second data item, an index that includes, in association with each candidate value that is used as the first data item, record specification information specifying two or more records with the candidate value and a statistical value obtained from values of the second data item registered in the two or more records. A processor receives a query including a search condition specifying a requested value of the first data item and a command requesting statistical processing of values of the second data item registered in records satisfying the search condition, retrieves the statistical value associated with a candidate value satisfying the search condition from the index, and outputs a processing result based on the retrieved statistical value for the query.

Abstract: An information processing apparatus includes a history acquisition section configured to acquire history data including a history indicating that a plurality of selection subjects have selected selection objects; a learning processing section configured to allow a choice model to learn a preference of each selection subject for a feature and an environmental dependence of selection of each selection object in each selection environment using the history data, where the choice model uses a feature value possessed by each selection object, the preference of each selection subject for the feature, and the environmental dependence indicative of ease of selection of each selection object in each of a plurality of selection environments to calculate a selectability with which each of the plurality of selection subjects selects each selection object; and an output section configured to output results of learning by the learning processing section.

Abstract: In accordance with an embodiment, the system enables access to a sharded database using a cache and a shard topology. A shard-aware client application connecting to a sharded database can use a connection pool (e.g., a Universal Connection Pool, UCP), to store or access connections to different shards or chunks of the sharded database within a shared pool. As new connections are created, a shard topology layer can be built at the database driver layer, which learns and caches shard key ranges to locations of shards. The shard topology layer enables subsequent connection requests from a client application to use a fast key path access to the appropriate shard or chunk.

Abstract: Embodiments of the invention utilize an improved LSM-tree-based key-value approach to strike the optimal balance between the costs of updates and lookups with any given main memory budget. The improved approach involves allocating memory to Bloom filters differently across different levels so as to minimize the sum of the false positive rates associated with the Bloom filters. In addition, the improved approach may predict the impact of the system design parameter(s) and/or environmental parameter(s) on the lookup performance. Subsequently, the improved approach may “self-tune” the system design parameter(s) and/or environment parameter(s) to maximize the throughput.

Abstract: Techniques related to flexible record definitions for semi-structured data in a relational database system are disclosed. The techniques include a machine readable medium, on which are stored instructions for accessing flexible record definitions for efficient database storage, comprising instructions that when executed cause a device to: receive a first request to create a first row in a database table, the first request indicating a first set of columns associated with the first row, determine that columns of the first set of columns are included in a second set of columns associated with the database table, select a first variant, from a set of variants, the first variant associated with a third set of columns, the third set of columns including at least the first set of columns, and write the first row to the database table based on the first variant and the third set of columns.

Abstract: The present disclosure provides systems and methods for extracting signal differences from sparse data sets. Data sets for comparison, including a control data set and one or more test data sets, may be normalized and separated into subsets or groupings via a MapReduce function. Normalization may account for large values present in both control and test data sets that would otherwise reduce the significance of smaller correlated values, creating false negatives. The MapReduce may provide identification and analysis of correlations between sets via related entities. Accordingly, via the systems and methods discussed herein, a computing device may extract statistically significant differences between data sets, without requiring extensive entity by entity comparison (or entity to every entity comparison, which, for a data set of millions of entities, may be too computationally expensive or take too long), reducing memory footprint and processor requirements.

Abstract: Methods and systems for converting a point cloud dataset into a hierarchical data structure for a lossless representation of an input file are provided herein. In various embodiments the method comprises recursing subtrees using a preordered traversal for a plurality of nodes, the preordered traversal being checking a parent node and then checking children nodes and stopping the recursing of the subtrees using the preordered traversal of the plurality of nodes when a node of the plurality of nodes is a below threshold node. The below threshold node being less than a configuration specified maximum number of data points, the below threshold node being a leaf of a tileset. Further comprising, aggregating the leaf of the tileset into a lossless representation of the collection of input files.

Abstract: A method of managing a storage system having one or more storage devices includes receiving a request to access, in a tiered data structure, a data object and one or more attributes of the data object and processing the access request. The request includes a first key value for the data object and respective second key values for the one or more attributes of the data object. Modified key values for the attributes of the data object are generated in accordance with at least a portion of the first key value and used to access tree entries for the data object attributes in one or more leaf nodes of the tiered data structure. A tree entry for the data object is also accessed from the same set of one or more leaf nodes.

Abstract: One embodiment provides a system that facilitates a single interface for file I/O and network communication. During operation, the system receives from a local application a first command to create content. The first command includes a content name and access control information that indicates how the content may be accessed. The content name is a hierarchically structured variable length identifier that includes contiguous name components ordered from a most general level to a most specific level. The system creates a first manifest which is a content object that indicates a set of content objects by a name and a digest. The first manifest name is the content name corresponding to the first command, and access to the manifest is based on the access control information included in the first command. The system transmits to the local application the first manifest and an indicator of success for creating the first manifest.

Abstract: Embodiments presented herein provide techniques for setting different methods of skipping duplicate values when executing a query statement in a relational database. A distance between a two distinct keys in an index, a current index key and a next distinct index key, are estimated. Based on the estimated distance, an appropriate duplicate-skipping method is determined. If the proximity between the distinct keys is relatively far apart (e.g., the keys reside in index pages that are at least an index page apart), then a “big skip” method is performed. Otherwise, if the proximity between the distinct keys is relatively near (e.g., the keys reside in the same index page), then a “little skip” method is performed.

Abstract: A system includes storage of data of a hierarchy, where each node of the hierarchy is represented by a row, and each row includes a level of its respective node, a pointer to a lower bound entry of an order index structure associated with the hierarchy, and a pointer to an upper bound entry of the order index structure associated with the hierarchy, reception of a pointer l, and determination of an entry e of the order index structure to which the received pointer l points.

Abstract: Embodiments presented herein provide techniques for setting different methods of skipping duplicate values when executing a query statement in a relational database. A distance between a two distinct keys in an index, a current index key and a next distinct index key, are estimated. Based on the estimated distance, an appropriate duplicate-skipping method is determined. If the proximity between the distinct keys is relatively far apart (e.g., the keys reside in index pages that are at least an index page apart), then a “big skip” method is performed. Otherwise, if the proximity between the distinct keys is relatively near (e.g., the keys reside in the same index page), then a “little skip” method is performed.

Abstract: A method of analyzing a program code of an electronic device includes configuring a tree by using a key string included in the program code and; in response to a command to find a specific key being received, performing a predetermined order traversal of the tree by using a string included in the specific key; in response to a node which matches a last string included in the specific key having a leaf node as a result of the predetermined order traversal, returning a value of the leaf node; and analyzing the program code by using the return value.

Abstract: The present invention discloses a look-up table creation method and query method. The creation method includes: receiving a look-up table create message sent by a controller, where the look-up table create message includes an identifier of a look-up table and a look-up mode of the look-up table; creating a look-up table according to the look-up table create message and setting a look-up mode of the look-up table; receiving an entry add message sent by the controller, where the entry add message includes a to-be-added forwarding action and an index parameter of the forwarding action; and adding, according to the entry add message, an entry corresponding to the entry add message to the look-up table.

Abstract: In a system for enabling a user to access a product or service, resources for accessing the product or service, such as apps on a user device, are explored in a specified order. A relatively lower priority resource is not explored until it is determined that all higher priority resources are unavailable. Within the content corresponding to an available resource, the user can be readily directed to the portion of the content that is associated with the particular product or service.

Abstract: A method and apparatus queries a table in a database where the table includes at least one column declared to be sparse. A binary large object may be used to store the sparse column data. The object includes a column-id and column-value pair for each non-null value. To answer a query with a constraint on a sparse column, the object is searched for one or more column ids to obtain the column values. Rows whose column values match a constraint are returned. In another embodiment, an internal table is used. Each tuple in the internal table has a column id and a value array indexed by an ordinal row number. To answer a query with a constraint on a sparse column, the column value in the internal table is found and matched against the constraint. If the match is successful, the index of the column value in the internal table is returned.

Abstract: A system for executing services in a distributed way, the execution of each service involving the processing of a plurality of tasks of predetermined types, each task type to be processed being represented by a corresponding type of tuples of data relating to the task type, wherein: the system includes a plurality of peer entities; each peer entity being adapted to store and execute a plurality of worker modules, each worker module being adapted to process at least one task type; each peer entity including tuple handling modules that, for a tuple of a type T, are adapted: a) to check if at least one of the plurality of the worker modules in the peer entity is adapted to process the task represented by the tuple, b) in the positive case of a), to take charge of the processing of the task represented by the tuple, and c) in the negative case of a), to execute a forwarding procedure for forwarding the tuple to another peer entity of the plurality of peer entities of the system.

Abstract: The information processing system includes a data management unit that manages a group of records having a plurality of attribute values in a data structure including an index key-value pair and a record key-value pair associated with each other. The data management unit is configured to generate the index key-value pair including a value including a classification reference value indicating a criterion for classifying given attribute values included in the group of records, and a key associated with the value, and also generate the record key-value pair including a key associated with the classification reference value in the value of the index key-value pair, and a value including information of the records having the given attribute values corresponding to the classification reference value.

Abstract: Techniques for indexing file paths of items in a content repository may include taking turns in querying each different item type or folder type in a round robin schedule to visit select nodes of the folder tree of that type to update and maintain the file path indexes. Item types or folder types may be associated with a count of instances or children of instances that are missing indexes. For each item type or folder type, a query may be performed for instances of the item type or folder type having children that are missing indexes, the instances or children of the instances returned may be associated with file path indexes, and the count of instances or children of instances may be adjusted based on the associating.

Abstract: A plurality of storage devices is organized into a physical volume called an aggregate, and the aggregate is organized into a global storage space, and a data block is resident on one of the storage devices of the plurality of storage devices. A plurality of virtual volumes is organized within the aggregate and the data block is allocated to a virtual volume. A physical volume block number (pvbn) is selected for the data block from a pvbn space of the aggregate, and virtual volume block number (vvbn) for the data block is selected from a vvbn space of the selected vvol. Both the selected pvbn and the selected vvbn are inserted in a parent block as block pointers to point to the allocated data block on the storage device.

Abstract: An information re-organization system includes a plurality of counters coordinated to meaning attributes, and a re-organization incentive notification unit that updates, in case the information stored in preset storage unit has been updated, value of a counter out of the multiple counters that has the meaning attribute associated with contents updated. The information re-organization system also includes an information re-organization processor that executes, in case the value of the counter section updated has met one of a number of predetermined conditions for information re-organization, a processing for information re-organization corresponding to the condition for information re-organization on the information stored in the preset storage unit.

Abstract: A method to allocate memory, in response to application requests, for a compact data structure having location data and a trailer section is provided. The trailer section of the compact data structure is checked to determine an offset for listings and indices representing the location data. Upon determining the offset, the listings and indices are loaded into memory and responses to the application requests are generated by utilizing the listings and indices stored in the memory.

Abstract: A set of index keys is included in an index search system that are associated with the scope of the search rather than the content of the documents that are the target of the search. These scope related index keys, or scope keys allows the scope of the search to be selected, reducing the number of documents that a search is required to sift through to obtain results. Furthermore, compound scopes are recognized and stored such that an index of complex search scopes is provided to eliminate rehashing of the searches based on these complex search scopes.

Abstract: A facility management system has a facility GIS database configured with electronic map data and facility management GIS data containing data for identifying a location and a form of a facility located along a railway or a road, a facility register database configured to manage a name of a rail or a road, a kilometer post and attribute information in association with one another, and a kilometer post information management section configured to manage coordinate data of the facility on the general map and the kilometer post in association with each other. When searching for attribute information from the facility register database, the kilometer post information management section is referred to obtain kilometer post information by using the coordinate data of the facility and the attribute information of the facility is searched from the facility register database by using the kilometer post information.

Abstract: A method, apparatus, and program product are provided for creating an Encoded Matrix Index for a column in a database table. An element of the column for all rows in the database table is compared to a corresponding reference value in a reference data structure, and in response to at least one value for the element of the column not matching the reference value, indicating a variation in a variation data structure and creating a value data structure. Queries executed using the Encoded Matrix Index include terms associated with a sub-column defined in a column of a database table. The variation data structure is accessed to determine whether any variation exists between rows belonging to a sub-column of the database table. If no variation exists, a value is accessed from the reference data structure; otherwise, a value for each row of the sub-column is accessed from a value data structure.

Abstract: Systems and methods for deleting non-key values from an index distributed over a plurality of computing devices maintains a non-key master list that includes the non-key values that are stored on each of the plurality of computing devices and receives a list of non-key values to delete. The systems and methods further intersect the list of non-key values to delete with the non-key master list, creating a first delete list for a first one of the plurality of computing devices that includes non-key values to be deleted that are stored on the first computing device. The systems and methods further transmit the first delete list to the first computing device and update the non-key master list based on the list of non-key values to delete.

Abstract: Inverted indexes for terms and for term separators are separately provided to minimize data redundancy. Search queries are parsed to identify terms and term separators, if any, and the corresponding inverted indexes are searched for responsive documents. Related apparatus, systems, techniques and articles are also described.

Abstract: Systems and methods for storing and retrieving data elements transmitted via data streams received from distributed devices connected via a network. The received data elements may be stored in block stores on the distributed devices. The stored data-elements may be allocated to data-blocks of a block-store that have assigned block-identifiers and further allocated to events of the data-blocks that have assigned token-names. Stream-schema of the received data-streams may comprise a list of token-names and an index-definition for each corresponding data-stream. Indices may be generated for the event-allocated data-elements. A query may be executed in order to retrieve data-elements of the received data-streams based on the indices.

Abstract: Methods, systems and computer program products are provided for predicting data. A name or title is obtained from a taste profile. There is an index into a data set based on the name or title, and a set of terms and corresponding term weights associated with the name or title are retrieved. A sparse vector is constructed based on the set of terms and term weights. The sparse vector is input to a training model including target data. The target data includes a subset of test data which has a correspondence to a predetermined target metric of data. A respective binary value and confidence level is output for each term, corresponding to an association between the term and the target metric.

Abstract: In general, techniques are described for an RDF (Resource Description Framework) database system which can scale to huge size for realistic data sets of practical interest. In some examples, a database system includes a Resource Description Framework (RDF) database that stores a plurality of data chunks to one or more storage drives, wherein each of the plurality of data chunks includes a plurality of triples of the RDF database. The database system also includes a working memory, a query interface that receives a query for the RDF database, a SPARQL engine that identifies a subset of the data chunks relevant to the query, and an index interface that includes one or more bulk loaders that load the subset of the data chunks to the working memory. The SPARQL engine executes the query only against triples included within the loaded subset of the data chunks to obtain a query result.

Abstract: A method, apparatus, and program product are provided for creating an Encoded Matrix Index for a column in a database table. An element of the column for all rows in the database table is compared to a corresponding reference value in a reference data structure, and in response to at least one value for the element of the column not matching the reference value, indicating a variation in a variation data structure and creating a value data structure. Queries executed using the Encoded Matrix Index include terms associated with a sub-column defined in a column of a database table. The variation data structure is accessed to determine whether any variation exists between rows belonging to a sub-column of the database table. If no variation exists, a value is accessed from the reference data structure; otherwise, a value for each row of the sub-column is accessed from a value data structure.

Abstract: The method and system of the present invention provides an improved technique for integrated asset management. Information is aggregated from a variety of sources into a centralized computerized database. Thereafter, asset transition events are scheduled. Information from the centralized computerized database is used in the performance of the asset transition events and information relating to the asset transition events is added to the centralized computerized database. Subsequent changes to the asset are also recorded into the centralized computerized database. As a result, a plethora of information is available within said database for the purpose of managing future asset transition events.

Abstract: A Searchable Symmetric Encryption (SSE) mechanism is described which allows efficient dynamic updating of encrypted index information. The encrypted index information includes pointer information that is encrypted using a malleable encryption scheme. The SSE mechanism updates the encrypted index information by modifying at least one instance of the pointer information without decrypting the pointer information, and thereby without revealing the nature of the changes being made. In one implementation, the SSE mechanism includes a main indexing structure and a deletion indexing structure. An updating operation involves patching applied to both the main indexing structure and deletion indexing structure.

Abstract: A method performs a database query in a relational database, the query being carried out by a database engine and being based on user-defined search criteria. The method includes retrieving a number N of properties of a record within a main database table, the number N being higher than zero, creating a search criteria option for each of the N properties, creating a search criteria table for every search criteria option, creating an index for every search criteria table, and performing the database query based on a user-defined combination of a plurality of the search criteria options.

Abstract: A transactionally consistent indexer is a tiered middleware framework component that updates a transactional index for a data blob according to a data transaction requested by an application. The transactionally consistent indexer determines index entries to be added or removed from a transaction index based on the application request. The transactionally consistent indexer further inserts each index entry to be added into the transaction index. With respect to each index entry to be removed, the transactionally consistent indexer uses a time stamp or version number of the index entry for ensuring optimistic concurrency during deletion. The transactionally consistent indexer then updates a data blob that is associated with each index entry to be added or each index entry to be removed based on the application request.

Abstract: Inverted indexes for terms and for term separators are separately provided to minimize data redundancy. Search queries are parsed to identify terms and term separators, if any, and the corresponding inverted indexes are searched for responsive documents. Related apparatus, systems, techniques and articles are also described.

Abstract: Provided is a method for quickly obtaining an intensity value at a desired m/z value in a compressed data obtained by run-length encoding of a mass analysis data. An index is created by pairing either the start position of a section where zero-intensity consecutively occurs two or more times in an array of an original spectrum data, or the start position of a sequence of data having significant intensity values in an array of the original spectrum data, with the corresponding position in an array of a compressed data. This index is stored separate from the compressed data. The creation of the index does not affect the array of the compressed data. Therefore, the data can be decompressed even by a data processing system that does not use the index. The index helps to quickly locate a compressed data corresponding to the desired m/z and obtain the necessary intensity value.

Abstract: An information re-organization system includes a plurality of counters coordinated to meaning attributes, and a re-organization incentive notification unit that updates, in case the information stored in preset storage unit has been updated, value of a counter out of the multiple counters that has the meaning attribute associated with contents updated. The information re-organization system also includes an information re-organization processor that executes, in case the value of the counter section updated has met one of a number of predetermined conditions for information re-organization, a processing for information re-organization corresponding to the condition for information re-organization on the information stored in the preset storage unit.

Abstract: The present invention relies on the two-dimensional information in documents and encodes two-dimensional structures into a one-dimensional synthetic language such that two-dimensional documents can be searched at text search speed. The system comprises: an indexing module, a retrieval module, an encoder, a quantization module, a retrieval engine and a control module coupled by a bus. Electronic documents are first indexed by the indexing module and stored as a synthetic text library. The retrieval module then converts an input image to synthetic text and searches for matches to the synthetic text in the synthetic text library. The matches can be in turn used to retrieve the corresponding electronic documents. In one or more embodiments, the present invention includes a method for comparing the synthetic text to documents that have been converted to synthetic text for a match.

Abstract: A disappearing index allows faster processing of a database query without the startup time of a prior art sparse index. The disappearing index starts as a full index but is trimmed of index values that point to a row in the table that is not selected by the query. Thus the traditional index disappears as it becomes a sparse index tailored to the query. The query is able to execute faster using the now sparse index since the target database table is not accessed for duplicate values in the base table of the query. The query optimizer may determine to use a disappearing index based on estimates of the number of duplicate values in the base table. When the query is complete, the created disappearing table may be discarded or used for other queries that match the selection of the query that created the disappearing index.

Abstract: An apparatus for generating indexes of data may include a processor and memory storing executable computer code causing the apparatus to at least perform operations including obtaining an order number responsive to receipt of a request from a device to index an item(s) of data. The computer program code may further cause the apparatus to map the order number to a key value and link the key value to the data and provide one or more index entries to a memory device to enable storage of the index entries. The index entries may include information corresponding to the key value and the data. The computer program code may further cause the apparatus to assign a new index row(s) including the data for inclusion in a set of index rows of a designated partition(s) to obtain a built index(es) of the data. Corresponding methods and computer program products are also provided.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying similar images. In some implementations, a method is provided that includes receiving a collection of images and data associated with each image in the collection of images; generating a sparse feature representation for each image in the collection of images; and training an image similarity function using image triplets sampled from the collection of images and corresponding sparse feature representations.

Abstract: The present application is directed to an indexing system. In one example, the indexing system includes one or more processors, one or more electronic memories that provide random access memory, one or more mass storage devices that provide persistent data storage, and one or more indexing routines, executed by the one or more processors, that create and manage an index data structure comprising nodes that include key-values/reference pairs, the index data structure additionally including a sibling reference for each index-data-structure node stored in the one or more electronic memories, the one or more indexing routines removing, from the index data structure, the sibling references for index-data-structure nodes when transferring the index-data-structure nodes from the one or more electronic memories to the one or more mass storage devices.

Abstract: Data processing apparatus comprising: a chunk store containing specimen data chunks, a manifest store containing a plurality of manifests, each of which represents at least a part of a data set and each of which comprises at least one reference to at least one of said specimen data chunks, a sparse chunk index containing information on only some specimen data chunks, the processor being operable to: process input data into input data chunks; identify manifests having at least one reference to one of said specimen data chunks that corresponds to one of said input data chunks and on which there is information contained in the sparse chunk index; and prioritize the identified manifests for subsequent operation.

Abstract: A method and a system for summarizing a concept are provided. A query corresponding to a concept is received from a user. A plurality of images and corresponding descriptive information may be collected based on the query. The plurality of images and the descriptive information may be processed to form feature vectors and processed descriptive information respectively. Further, one or more topics may be identified for the plurality of images. Each of the plurality of images may be assigned with one or more topic distribution values corresponding to the one or more topics. The one or more topics correspond to the processed descriptive information. A sparse set of images may be determined based on the feature vectors and the assigned topic distribution values, to summarize the concept. Also, a target summary may be built from the summarized concept, by regularizing one or more distribution constraints.

Abstract: A method to allocate memory, in response to application requests, for a compact data structure having location data and a trailer section is provided. The trailer section of the compact data structure is checked to determine an offset for listings and indices representing the location data. Upon determining the offset, the listings and indices are loaded into memory and responses to the application requests are generated by utilizing the listings and indices stored in the memory.

Abstract: A disappearing index allows faster processing of a database query without the startup time of a prior art sparse index. The disappearing index starts as a full index but is trimmed of index values that point to a row in the table that is not selected by the query. Thus the traditional index disappears as it becomes a sparse index tailored to the query. The query is able to execute faster using the now sparse index since the target database table is not accessed for duplicate values in the base table of the query. The query optimizer may determine to use a disappearing index based on estimates of the number of duplicate values in the base table. When the query is complete, the created disappearing table may be discarded or used for other queries that match the selection of the query that created the disappearing index.

Abstract: The present invention relies on the two-dimensional information in documents and encodes two-dimensional structures into a one-dimensional synthetic language such that two-dimensional documents can be searched at text search speed. The system comprises: an indexing module, a retrieval module, an encoder, a quantization module, a retrieval engine and a control module coupled by a bus. A number of electronic documents are first indexed by the indexing module and stored as a synthetic text library. The retrieval module then converts and input image to synthetic text and searches for matches to the synthetic text in the synthetic text library. The matches can be in turn used to retrieve the corresponding electronic documents. It should be noted that a plurality of matches and corresponding electronic documents may be retrieves ranked by order according the similarity of the synthetic text.

Abstract: Prefixes are registered on a first list as index elements for respective registration patterns. Each prefix is selected as the longest of different-length prefixes that are extractable from a registration pattern in accordance with an extraction rule. Suffixes, which are the remaining parts of the registration patterns excluding the respective prefixes, are registered on a second list. Using different-length prefixes that are extracted from a retrieval key in accordance with the extraction rule, a prefix retriever searches the first list to retrieve a registration pattern whose prefix matches any of the prefixes of the retrieval key. A suffix checker carries out a check on the suffix of the registration pattern retrieved by the prefix retriever, among the suffixes on the second list, as to whether the suffix of the registration pattern matches the suffix of the retrieval key.

Abstract: Systems, methods, and other embodiments associated with logically expanding a document and determining the relevance of the logically expanded document to a query are described. One method embodiment includes searching an index to locate a document identifier for a document in which a query term appears. The method includes determining whether the index entry includes an expansion identifier, and, if so, producing a logically expanded document. The logically expanded document may include both a document associated with the document identifier and a document associated with the expansion identifier. The method may then determine a relevance value of the logically expanded document with respect to the query and may provide a signal corresponding to the relevance value.