Abstract: A system and method for automatically scaling consumer servers in a data processing system. To build an automatic scaling system, the present disclosure allows consumers to obtain additional information, e.g., the number of events that await to be read from an aggregator when receiving an event from the aggregator. This additionally obtained number provides a direct gauge for the data processing system to determine when the consumers are over-provisioned, i.e., when the number of events left to be read is close to zero, as well as when the consumers are under-provisioned, e.g., when the number of events left to be read continues to increase. As a result, the consumers can be automatically scaled to handle the dynamic data processing demand while providing optimal resource allocation.

Abstract: A system and method for automatically scaling consumer servers in a data processing system. To build an automatic scaling system, the present disclosure allows consumers to obtain additional information, e.g., the number of events that await to be read from an aggregator when receiving an event from the aggregator. This additionally obtained number provides a direct gauge for the data processing system to determine when the consumers are over-provisioned, i.e., when the number of events left to be read is close to zero, as well as when the consumers are under-provisioned, e.g., when the number of events left to be read continues to increase. As a result, the consumers can be automatically scaled to handle the dynamic data processing demand while providing optimal resource allocation.

Abstract: Aspects of the subject matter described herein relate to an offload provider. In aspects, an offload provider may provide a token that represents data. The offload provider may be expected to ensure that the data the token represents will not change while the token is valid. The offload provider may take actions to ensure the immutability of the data. The actions may be taken, for example, in conjunction with receiving an offload read and/or offload write, and/or in conjunction with receiving another write that, if allowed to proceed, would otherwise change the data represented by the token.

Abstract: The storage of data sets in a storage set (e.g., data sets written to hard disk drives comprising a RAID array) may diminish the performance of the storage set through non-sequential writes, particularly if the storage devices promptly write data sets that are followed by sequentially following data sets. Additionally, storage sets may exhibit inconsistencies due to non-atomic writes of data sets and verifiers (e.g., checksums) and an intervening failure, such as an occurrence of the RAID write hole. Instead, data sets and verifiers may first be written to a stored on the nonvolatile media of a storage device before being committed to the storage set. Such writes may be sequentially written to the journal, irrespective of the locations of the data sets in the storage set; and recovery of a failure may simply involve re-committing the consistent records in the journal to correct incomplete writes to the storage set.

Abstract: Aspects of the subject matter described herein relate to virtualization and offload reads and writes. In aspects, an offload read allows a requestor to obtain a token that represents data while an offload write allows the requestor to request that the data (or a part thereof) represented by a token be logically written. Offload reads and writes may be used to perform various actions for virtual environments.

Abstract: The representation of storage devices on computers (e.g., as logical volumes) may be complicated by the pooling of multiple storage devices in order to apply redundancy plans such as mirroring and checksumming. Presented herein is a storage device driver configured to operate as a storage device interface generating representations of the storage regions of the storage devices; to claim those regions as a storage controller; and to expose pooled storage regions as logical disks. Additionally, the storage device driver may support the inclusion of storage devices in a cluster, comprising nodes that may be appointed as managers of the storage pool configuration; as managers of the storage devices; as owners having exclusive read/write access to the storage pool or cluster resources; and as cluster resource writers having exclusive write access to a cluster resource. The nodes of the cluster may interoperate to share the storage devices while avoiding write conflicts.

Abstract: Aspects of the subject matter described herein relate to a zero token. In aspects, a zero token may be used to represent one or more zeroes in an offload write command. A component receiving an offload write command with the zero token is free to write the zeroes in a variety of ways including, for example, changing data structures the component uses to represent the zeroes, issuing another command that writes zeroes, writing physical zeroes, and so forth. A component receiving an offload write command with the zero token does not need to verify that the zero token was obtained from a successful offload read or that the zero token is still valid. In response to an offload read request, a component may provide the zero token to represent all or a portion of the data associated with the offload read request.

Abstract: Aspects of the subject matter described herein relate to an offload provider. In aspects, an offload provider may provide a token that represents data. The offload provider may be expected to ensure that the data the token represents will not change while the token is valid. The offload provider may take actions to ensure the immutability of the data. The actions may be taken, for example, in conjunction with receiving an offload read and/or offload write, and/or in conjunction with receiving another write that, if allowed to proceed, would otherwise change the data represented by the token.

Abstract: The representation of storage devices on computers (e.g., as logical volumes) may be complicated by the pooling of multiple storage devices in order to apply redundancy plans such as mirroring and checksumming. Presented herein is a storage device driver configured to operate as a storage device interface generating representations of the storage regions of the storage devices; to claim those regions as a storage controller; and to expose pooled storage regions as logical disks. Additionally, the storage device driver may support the inclusion of storage devices in a cluster, comprising nodes that may be appointed as managers of the storage pool configuration; as managers of the storage devices; as owners having exclusive read/write access to the storage pool or cluster resources; and as cluster resource writers having excusive write access to a cluster resource. The nodes of the cluster may interoperate to share the storage devices while avoiding write conflicts.

Abstract: The storage of data sets in a storage set (e.g., data sets written to hard disk drives comprising a RAID array) may diminish the performance of the storage set through non-sequential writes, particularly if the storage devices promptly write data sets that are followed by sequentially following data sets. Additionally, storage sets may exhibit inconsistencies due to non-atomic writes of data sets and verifiers (e.g., checksums) and an intervening failure, such as an occurrence of the RAID write hole. Instead, data sets and verifiers may first be written to a stored on the nonvolatile media of a storage device before being committed to the storage set. Such writes may be sequentially written to the journal, irrespective of the locations of the data sets in the storage set; and recovery of a failure may simply involve re-committing the consistent records in the journal to correct incomplete writes to the storage set.

Abstract: Described are embodiments which allow token-based file operations. The client may request a special offload file operation that is formatted according to a file access protocol. The file operation may be an offload read operation or an offload write operation. In an offload read operation, the client requests that data be logically read from a stored file, or a portion thereof. In response, the file server provides a response that includes a token that represents the logically read data. In some embodiments, the file server may return a response with a token that represents less than all of the requested data if for some reason it cannot provide a token that represents all of the data. The token can then be used by the client in a subsequent offload write operation. In embodiments, the tokens represent immutable data that can be safely and securely used across servers and clients.

Abstract: In one embodiment, a host application may manage a data set maintained at a storage device using a token. A processor 220 of a host computer executing a host application may obtain a token representing a data set. The processor 220 may read a data set result based on the data set into a memory local to the host application. The data set result may be a data set copy, a data set digest, or a data set transformation.

Abstract: Aspects of the subject matter described herein relate to virtualization and offload reads and writes. In aspects, an offload read allows a requestor to obtain a token that represents data while an offload write allows the requestor to request that the data (or a part thereof) represented by a token be logically written. Offload reads and writes may be used to perform various actions for virtual environments.

Abstract: Aspects of the subject matter described herein relate to offload reads and writes. In aspects, a requestor that seeks to transfer data sends a request for a representation of the data. In response, the requestor receives one or more tokens that represent the data. The requestor may then provide one or more of these tokens to a component with a request to write data represented by the one or more tokens. In some exemplary applications, the component may use the one or more tokens to identify the data and may then read the data or logically write the data without additional interaction with the requestor. Tokens may be invalidated by request or based on other factors.

Abstract: A file system which is enabled to use a Metadata Partition instead of VAT for write-once discs, is provided by a pseudo-overwrite method. On applying this invention to a drive apparatus which supports pseudo-overwrite media, the file system distinguishes data to overwrite from data to append. When the data is newly written to a logical sector (S1305), the drive apparatus writes the data to a physical sector to which the logical sector corresponds (S1311). When the logical sector is overwritten the data is written to another unrecorded physical sector in the volume space (1321), and remapping information that specifies the original address, and the remapping address are stored in the remapping table (S1322).

Abstract: Method and system for establishing and maintaining quotas. An auto quota is defined and applied to a directory. Input and output is monitored to detect a successful operation that involves a subdirectory of the directory. A determination is made as to whether to apply a quota associated with the auto quota to the subdirectory. If the determination is that the quota is to be applied to the subdirectory, it is automatically applied.

Abstract: In order to provide a more efficient persistent storage device, one or more long-term storage media are included along with a non-volatile memory. In one embodiment, one portion of the non-volatile memory is used as a write buffer and a read cache for writes and reads to the long-term storage media. Interfaces are provided for controlling the use of the non-volatile memory as a write buffer and a read cache. Additionally, a portion of the non-volatile memory is used to provide a direct mapping for specified sectors of the long-term storage media. Descriptive data regarding the persistent storage device is stored in another portion of the non-volatile memory.

Abstract: An operating system copies data from memory pages into a paging file on disk, in order to free up space in the memory. A mechanism is disclosed that causes the data to be encrypted as it is copied into the paging file, thereby protecting the paged data from unauthorized (or otherwise undesired) observation. The data that is stored in the paging file is encrypted with a session key, that is generated shortly after the machine on which the paging file exists is started. The session key, which is used both for encryption and decryption of the paging file data, is stored in volatile memory, so that the key is not persisted across boots of the machine. Since the key is not persisted across boots, old paging file data that was stored prior to the most recent boot cannot be recovered in clear text, thereby protecting the data from observation.

Abstract: Late-binding/dynamic pathname resolution can be performed by a variable identifier operating in user or kernel mode, and a pathname engine and a data structure operating in kernel mode. The variable identifier identifies a variable in a pathname. The pathname engine evaluates the variable by converting the variable into a character string such that a path to an object file is established. The pathname engine determines a scope of potential character string values for the variable by mapping the variable to corresponding values stored in the data structure. Each variable has a well-defined scope of values because the only values available are the values selected for storage in the data structure. The mechanism for establishing values is extensible such that variable/value pairs can be implemented in the data structure at any time.