Abstract: An efficient method to apply an erasure encoding and decoding scheme across dispersed data stores that receive constant updates. A data store is a persistent memory for storing a data block. Such data stores include, without limitation, a group of disks, a group of disk arrays, or the like. An encoding process applies a sequencing method to assign a sequence number to each data and checksum block as they are modified and updated onto their data stores. The method preferably uses the sequence number to identify data set consistency. The sequencing method allows for self-healing of each individual data store, and it maintains data consistency and correctness within a data block and among a group of data blocks. The inventive technique can be applied on many forms of distributed persistent data stores to provide failure resiliency and to maintain data consistency and correctness.

Abstract: A data management system or “DMS” provides an automated, continuous, real-time data protection service to one or more data sources associated with a set of application host servers. To facilitate the service, a host driver embedded in an application server captures real-time data transactions. When a data protection command for a given data source is forwarded to a host driver, an event processor enters into an initial upload state. During this state, the event processor gathers a list of data items to be protected and creates a data list. Then, the event processor moves the data to a DMS core to create initial baseline data. The upload is a stream of application-aware data chunks that are attached to upload events. A resynchronization state is entered when there is a suspicion that the state of the data in the host is out-of-sync with the state of the most current data in the DMS. During upload or upward resynchronization, the application does not have to be shut down.

Abstract: The present invention provides a distributed clustering method to allow multiple active instances of consistency management processes that apply the same encoding scheme to be cooperative and function collectively. The techniques described herein facilitate an efficient method to apply an erasure encoding and decoding scheme across dispersed data stores that receive constant updates. The technique can be applied on many forms of distributed persistent data stores to provide failure resiliency and to maintain data consistency and correctness.

Abstract: A data management system or “DMS” provides a wide range of data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. To facilitate the data service, a host driver embedded in an application server connects an application and its data to a cluster. The host driver provides a method and apparatus for capturing real-time data modifications and application state notifications and, in response, generating data transactions in the form of an event journal that is provided to the data management system. The driver functions to translate traditional file/database/block I/O into a continuous, application-aware, output data stream.

Abstract: A data management system or “DMS” provides a wide range of data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. To facilitate the data service, a host driver embedded in an application server connects an application and its data to a cluster. The host driver provides a method and apparatus for capturing real-time data modifications and application state notifications and, in response, generating data transactions in the form of an event journal that is provided to the data management system. The driver functions to translate traditional file/database/block I/O into a continuous, application-aware, output data stream.

Abstract: A data management system or “DMS” provides an automated, continuous, real-time, substantially no downtime data protection service to one or more data sources associated with a set of application host servers. To facilitate the data protection service, a host driver embedded in an application server captures real-time data transactions, preferably in the form of an event journal that is provided to other DMS components. The driver functions to translate traditional file/database/block I/O and the like into a continuous, application-aware, output data stream. The host driver includes an event processor. When a data protection command for a given data source is forwarded to a host driver, the event processor enters into an initial upload state. During this state, the event processor gathers a list of data items of the data source to be protected and creates a data list. Then, the event processor moves the data (as an upload, preferably one data element at a time) to a DMS core to create initial baseline data.

Abstract: A data management system (DMS) includes a continuous real-time object store that captures all real-time activities, with associated object metadata information. The DMS is capable of reintroducing any point-in-time view of data ranging from a granular object to an entire file system. A set of algorithms (for creation of a file or directory, modification of a file or directory, deletion of a file or directory, and relocation/renaming of a file or directory) are used to generate and maintain a file system history in the DMS and to ensure that a latest version of a directory always refers to a latest version of its children until the directory changed. Any point-in-time recovery is implemented using the file system history in one of various ways to provide strong individual file integrity, exact point-in-time crash consistency, and/or recovery of last version of all files in the file system.

Abstract: A data management system or “DMS” provides an automated, continuous, real-time, substantially no downtime data protection service to one or more data sources associated with a set of application host servers. To facilitate the data protection service, a host driver embedded in an application server captures real-time data transactions, preferably in the form of an event journal that is provided to other DMS components. The driver functions to translate traditional file/database/block I/O and the like into a continuous, application-aware, output data stream. The host driver includes an event processor that provides the data protection service, preferably by implementing a finite state machine (FSM). In particular, the data protection is provided to a given data source in the host server by taking advantage of the continuous, real-time data that the host driver is capturing and providing to other DMS components.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: A data management system or “DMS” provides an automated, continuous, real-time, substantially no downtime data protection service to one or more data sources associated with a set of application host servers. To facilitate the data protection service, a host driver embedded in an application server captures real-time data transactions, preferably in the form of an event journal that is provided to other DMS components. The driver functions to translate traditional file/database/block I/O and the like into a continuous, application-aware, output data stream. The host driver includes an event processor that provides the data protection service, preferably by implementing a finite state machine (FSM). In particular, the data protection is provided to a given data source in the host server by taking advantage of the continuous, real-time data that the host driver is capturing and providing to other DMS components.

Abstract: A scalable content delivery network (SCDN) employs a parallel download mechanism to ensure that a demanded file is present at a station in time for user consumption. This mechanism is used in solving the content caching and storage problem for applications such as video-on-demand, which is commonly perceived as a tough problem in the industry. In the network, files are divided into smaller units called tracks according to the nature of data contained in each of them. Tracks are further divided into smaller equally sized units called block files. This division builds the foundation for parallel download. A sequence server provides a lock-free mechanism for multiple threads or processes to access data atomically. The sequence server allows clients to gain sequential access to data, or to find out whether the sequence has been violated so that they can retry their operation or take corrective action. Advantages of the invention include the ability to handle distribution of large files and process sequencing.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: A method for initializing a new node in a network. The network has multiple nodes arranged in a virtual tree format. The new node is a node of the tree, and each node of the tree has a set of attributes and a set of rolled up attributes to identify each node. A query is automatically sent to the nodes to determine what contents to download. The contents are then stored as block files in the nodes. The query contains the set of attributes and rolled up attributes for the new node. The query receives replies from a subset of the nodes that have the contents needed for the new node. Each reply identifies what subset of the block files is available and the performance characteristics of that replying node. Then the desired contents from the subset of the block files from nodes that are least congested is downloaded.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.

Abstract: Large payload files are selectively partitioned in blocks and the blocks distributed to a plurality of distribution stations at the edge of the network qualified to have the data. Each qualified station decides how much and what portion of the content to save locally, based on information such as network location and environment, usage, popularity, and other distribution criteria defined by the content provider. Different pieces of a large payload file may be available from different nodes, however, when a user requests access to the large payload file, for example, through an application server, a virtual file control system creates an illusion that the entire file is present at the connected node. However, since only selective portions of the large payload file may actually be resident at that node's storage at the time of request, a cluster of distribution servers at the distribution station may download the non-resident portions of the file as the application server is servicing the user.