Abstract: Described are systems and methods for storing a variable number of instances of data objects (e.g., 1, 2, 3, or up to N?1 instances, where N is the number of instances of the data object included in primary data) in secondary storage across a data storage network. In some examples, a system for storing a variable number of instances of data objects includes, one or more computing devices storing a set of data objects and multiple storage devices distinct from the one or more computing devices. Each of the multiple storage devices is configured to store at least a single instance of a data object. The system also includes a database configured to store information associated with the data objects. This information includes substantially unique identifiers for the data objects and, for each of the data objects, a number of instances of the data object stored on the multiple storage devices.

Abstract: A method and system are provided for capturing and applying changes to a data structure made by one or more processing nodes. The method includes providing a data structure including one or more elements (200-205), maintaining a linked record of changes (200a-200c, 201a-201c, 205a-205b) to each element (200-205) made by a processing node. A new value of an element (200-205) is linked to the previous value. A record (250) of the sequence of changes to the elements (200-205) is also maintained. In one embodiment, the linked record of changes (200a-200c, 201a-201c, 205a-205b) is a push down stack. The data structure is defined including indirections (210-215) pointing to most recent values of an element (200-205), which in turn points to the previous value of the element (200-205), forming a linked record to the initial value of the element.

Abstract: A method and system for data recovery in a parallel database system in which data stored in the database has been corrupted by a transaction or operation. Transaction log files corresponding to each node of the parallel database are scanned to determine an operation that caused the corruption of the stored data. Information corresponding to the corrupted data before the operation causing the corruption is then generated based on the scanned transaction log. The operation responsible for corrupting the data is then rolled back, and the corrupted data is replaced with the original data stored in the database prior to the corruption.

Abstract: A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Abstract: A disclosure range determination method includes accepting a disclosure range of data and metadata of the data, extracting attribute information of a user that belongs to the disclosure range of the data accepted in the accepting based on association information associating a disclosure range of data with attribute information of a user that belongs to the disclosure range, converting the metadata accepted in the accepting into a keyword based on association information associating a conversion rule that converts the metadata into significant information with a keyword corresponding to the conversion rule, and determining, by a computer, whether or not a user that belongs to the disclosure range accepted in the accepting qualifies for the disclosure range accepted in the accepting, based on whether or not the keyword converted in the converting is included in the attribute information extracted in the extracting.

Abstract: According to one embodiment of the present invention, a system selects member objects from among data objects stored in one or more data sources by evaluating criteria against information about the data objects in an index, wherein the information about the data objects in the index is derived from the data sources. The system creates a set of data objects comprising a collection of references to the selected member objects, and associates the set with a set-identifier. The system receives a request to apply an action to a subset of the data objects stored in the one or more data sources, wherein the request specifies the subset by including the set-identifier, and applies the action to the set of data objects associated with the set-identifier within the request. Embodiments of the present invention further include methods and computer program-products for managing data in substantially the same manners described above.

Abstract: A system can provide unique user metrics associated with a node in a hierarchy and the node's decedents in real time. This can occur by merging sketches of the node and its decedents to form a merged sketch, and retrieving the unique user metrics from the merged sketch. The sketches and the merged sketch may each be a separate data form, such as a hash. The hierarchy may include an online content taxonomy and the sketches of each node may include unique user metrics associated with the node, such as user interaction metrics associated with a category of content of the node. The system may build sketches for the nodes of a hierarchy, associate each sketch with its ancestor sketches, invert the associations, tie corresponding descendant sketches of an ancestor sketch based on the inversions, and merge the descendant sketches with their ancestor sketches to generate the merged sketches.

Abstract: An information management system according certain aspects may be configured to generate a snapshot of data relating to a plurality of applications. The system may include first and second data agents associated with first and second applications, respectively. The system may also include a snapshot manager configured to: in response to receiving notifications from the first and second data agents that the first and second applications are in consistent states: obtain a snapshot of the logical volume including data generated by the first and second applications; generate mapping information between the first application and a portion of the snapshot relating to the first application based at least in part on metadata obtained by the first data agent; and generate mapping information between the second application and a portion of the snapshot relating to the second application based at least in part on metadata obtained by the second data agent.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for storing semi-structured data. One of the methods includes maintaining a plurality of schemas; receiving a first semi-structured data item; determining that the first semi-structured data item does not match any of the schemas in the plurality of schemas; and in response to determining that the first semi-structured data item does not match any of the schemas in the plurality of schemas: generating a new schema, encoding the first semi-structured data item in the first data format to generate the first new encoded data item in accordance with the new schema, storing the first new encoded data item in the data item repository, and associating the first new encoded data item with the new schema.

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a search engine to provide an entity some indication of topics in which a user may have an interest. The methods, systems, and apparatus include actions of receiving information at a search engine from a classifier indicating that a user is likely interested in a set of topics based on information about the user provided by the search engine to the classifier. Additional actions may include selecting a subset of the topics and generating a search results page that includes one or more references for one or more resources that are responsive to a search query. At least one reference of the one or more references may include information based on the received information that indicates that the user is likely interested in the subset of topics.

Abstract: A method reads and writes data from a database table. Each row in the table has a primary key and multiple non-key columns. Each non-key column has one or more column values, and each column value has an associated timestamp that identifies when the column value was stored. The timestamps associated with the column values in each non-key column provide a unique order for the column values. A read transaction is initiated to read from a first non-key column of a first row. A write transaction is in progress that is updating a second non-key column of the first row, where the second non-key column is distinct from the first non-key column. The write transaction holds a lock on the second non-key column of the first row. The method concurrently reads the data from the first non-key column and writes a new column value to the second non-key column.

Abstract: A data set of spatial data having a plurality of dimensions and including linestrings can be processing by decomposing each linestring of the plurality of linestrings into a plurality of line segments. Each coordinate dimension appears in at least one line segment of the plurality of line segments can be listed in one of a plurality of dimensional dictionaries that each correspond to a dimension of the plurality of dimensions. A linestring of the plurality of linestrings can be represented as a set of the line segments using the plurality of dimensional dictionaries.

Abstract: Virtualization sprawl can lead to virtual machines with no designated periodic backup. If the data associated with these unprotected virtual machines is not backed up, it cannot be restored if needed, leading to system failures. A data storage system identifies and protects the unprotected virtual machines. For instance, the system compares a list of virtual machines with a list of computing devices having a designated backup policy in the data storage system and determines which of the virtual machines are unprotected. The system further automatically categorizes the unprotected virtual machines, identifies those unprotected virtual machines that remain uncategorized, and applies a default backup policy to the uncategorized and unprotected virtual machines to provide protected virtual machines.

Abstract: According to one embodiment of the present invention, a system with a plurality of memories, generates a first filter associated with a first memory, and generates one or more second filters each associated with a corresponding memory. Each second filter is generated with a size at least partially accommodated within the storage capacity of the associated memory. The system determines absence of items from a set by selectively probing the generated filters in an ascending order of data access speed to the associated memories and aborts further probing for an item in response to a probe indicating that the item is absent from the set. Embodiments of the present invention further include a method and computer program product for testing set membership in substantially the same manners described above.

Abstract: According to one embodiment of the present invention, a system with a plurality of memories, generates a first filter associated with a first memory, and generates one or more second filters each associated with a corresponding memory. Each second filter is generated with a size at least partially accommodated within the storage capacity of the associated memory. The system determines absence of items from a set by selectively probing the generated filters in an ascending order of data access speed to the associated memories and aborts further probing for an item in response to a probe indicating that the item is absent from the set. Embodiments of the present invention further include a method and computer program product for testing set membership in substantially the same manners described above.

Abstract: Querying a data store can include receiving a query defined by a number of phrases. Querying a data store can include identifying within the data store service component metrics, service component logs, and service component events associated with the query. Querying a data store can include displaying the identified service component metrics, service component logs, and service component events.

Abstract: A computer implemented method includes the steps of establishing a plurality of storage control sessions, wherein each storage control session associates at least one primary storage controller and at least one secondary storage controller with a predetermined path through a data transport network that interconnects the primary and secondary storage controllers; identifying storage control sessions that have data available to be transferred from primary storage controllers to secondary storage controllers; accumulating available data for each session and determining transfer time through the network for each data element to estimate prospective bandwidth utilization; and determining, for a predetermined time interval, whether transferring the available data would fully occupy available bandwidth of the data transport network for the predetermined time interval.

Abstract: The technique described herein provides a way to summarize data and can also minimize unnecessary accesses to a data store. In one embodiment, the technique creates and stores an adaptive range filter that contains a compact summary of the contents of an index for a data store in the form of a trie data structure. Before accessing the index of the data store in response to a query, the technique checks the filter to determine whether the data store does not contain any keys for a specific range of data. If the adaptive range filter indicates that the index contains no keys satisfying the query predicate, the index of the data store and the data itself is not accessed. The adaptive range filter of the technique supports both range predicates and equality predicates. It is adaptive to changes in data and queries by learning the query and data distribution.

Abstract: A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Abstract: A process control method that automatically and comprehensively manages the creation, storage and use of electronic records during the operation of a controlled process and for governing and documenting the users that operate the computer systems that are involved in the creation, storage and use of the electronic records. The method controls a process, such as a chemical process, by receiving measurement information relating to the process, applying rules to the received measurement information to determine a desired response, and sending control information to at least one device associated with the process, so as to modify the process. The method maintains the rules, measurements and control activity in transaction logs or audit trails related thereto in a secure database.