Abstract: A partitioning policy, comprising an indication of an initial mapping of data records of a stream to a plurality of partitions, is selected to distribute data records of a data stream among a plurality of nodes of a stream management service. Data ingestion nodes and storage nodes are configured according to the initial mapping. In response to a determination that a triggering criterion for dynamically repartitioning the data stream has been met, a modified mapping is generated, and a different set of ingestion and storage nodes are configured. For at least some time during which arriving data records are stored in accordance with the modified mapping, data records stored at the first set of storage nodes in accordance with the initial mapping are retained.

Abstract: Systems and methods are presented for processing and auditing data records using a stream based data processing system. Data output by data center computers may be collected and used to generate data records that include values for metrics related to computer resource consumption. These data records may be inserted into a stream which can include auditors and various other processors. The auditors may determine whether any of the data records include discrepancies. A gating processor can determine which processors, if any, to provide data records that include discrepancies. Further, an amendment processor can be used to resolve discrepancies detected by the auditors. In addition, a billing processor can be used to generate bills that identify the discrepancies and include information relating to the cause and actions taken in response to the discrepancies detected in the data records.

Abstract: A replication chain comprising one or more replication nodes of a multi-tenant stream management system is assigned to store data records of a partition of a particular data stream. A data record of the partition is received at a selected replication node of the replication chain. In a sequential order, a respective replica of the data record is stored at each replication node of the chain. An acknowledgement of a successful storage of the data record is provided after the replications are completed.

Abstract: Methods, systems, and computer-readable media for generating message envelopes for heterogeneous events are disclosed. A plurality of events are received. The events are associated with computing resources in a multi-tenant environment. The events comprise an event formatted using a content schema and a different event formatted using a different content schema. A message is generated comprising a message envelope and the event. The message envelope comprises a plurality of field names and one or more field values formatted using an envelope schema. A different message is generated comprising a different message envelope and the different event. The different message envelope comprises the field names and one or more different field values formatted using the envelope schema. The message is filtered based at least in part on the message envelope, and the different message is filtered based at least in part on the different message envelope.

Abstract: A local de-duplication table for at least a particular partition of a data stream is instantiated at a particular ingestion node of a multi-tenant stream management service. A submission request indicating a data record of the partition is received at the ingestion node. In response to a determination that (a) the submission request was received within a de-duplication time window corresponding to the particular partition, and (b) the local de-duplication table does not indicate that the data record is a duplicate, a write operation to store the data record at one or more storage locations of the stream management system is initiated.

Abstract: Virtual resource provisioning may be enhanced by coloring virtual resource instances and/or underlying implementation resources. Particular resource colors may be associated with particular treatments during allocation of implementation resources to virtual resources. There may be different types of colors corresponding to different types of allocation treatment. Exclusory colors may be utilized to reduce clustering of virtual resources with respect to implementation resources. Assignment of exclusory colors to virtual resources can help strike a balance between lower costs through efficient implementation resource utilization and higher fault tolerance through spreading across an available implementation resource pool. Inclusive colors may be utilized to require and/or prefer allocation of virtual resources to implementation resources painted with the inclusive color. Proximity colors may be utilized to enhance a computational performance of a set of virtual resources.

Abstract: A configuration request comprising a security option selected for a particular data stream is received. Nodes of a plurality of functional categories, such as a data ingestion category and a data retrieval category are to be configured for the stream. The security option indicates a security profile of a resource to be used for nodes of at least one functional category. In accordance with the configuration request, a node of a first functional category is configured at a resource with a first security profile, and configuration of a node of a second functional category is initiated at a different resource with a different security profile.

Abstract: Methods and apparatus for vibration cancellation are disclosed. Vibration data from one or more vibration detectors associated with a storage device is collected. The vibration data represents vibrations experienced by the storage device. In response to the vibration data from the one or more vibration detectors, one or more movements for respective ones of one or more counter-vibration actuators to at least partially cancel of the vibrations experienced by the storage device is calculated. The one or more counter-vibration actuators perform the one or more movements.

Abstract: Methods and apparatus for vibration cancellation are disclosed. Vibration data from one or more vibration detectors associated with a storage device is collected. The vibration data represents vibrations experienced by the storage device. In response to the vibration data from the one or more vibration detectors, one or more movements for respective ones of one or more counter-vibration actuators to at least partially cancel of the vibrations experienced by the storage device is calculated. The one or more counter-vibration actuators perform the one or more movements.

Abstract: A control node of a multi-tenant stream management service receives a request to initialize a data stream to be comprised of a plurality of data records. The control node determines, based on a partitioning policy, parameters to be used to configure subsystems for ingestion, storage and retrieval of the records. The control node identifies resources to be used for a node of retrieval subsystem The retrieval node is configured to implement programmatic record retrieval interfaces, including respective interfaces to implement non-sequential and sequential access patterns. The control node configures the retrieval node using the selected resources.

Abstract: A control node of a multi-tenant stream processing service receives a request indicating an operation to be performed on data records of a particular data stream. Based on a stream partitioning policy, the control node determines an initial number of worker nodes to be used. The control node configures a worker node to perform the operation on received records. In response to a determination that the worker node is in an unhealthy state, the control node configures a replacement worker node.

Abstract: A programmatic interface is implemented, enabling a client of a stream management service to select a data ingestion policy for a data stream. A client request selecting an at-least-once ingestion policy is received. In accordance with the at-least-once policy, a client may transmit an indication of a data record one or more times to the service until a positive acknowledgement is received. In response to receiving a plurality of transmissions indicating a particular data record, respective positive acknowledgements are sent to the client. Based on a persistence policy selected for the stream, copies of the data record are stored at one or more storage locations in response to one particular transmission of the plurality of transmissions.

Abstract: A configuration request comprising a security option selected for a particular data stream is received. Nodes of a plurality of functional categories, such as a data ingestion category and a data retrieval category are to be configured for the stream. The security option indicates a security profile of a resource to be used for nodes of at least one functional category. In accordance with the configuration request, a node of a first functional category is configured at a resource with a first security profile, and configuration of a node of a second functional category is initiated at a different resource with a different security profile.

Abstract: A partitioning policy, comprising an indication of an initial mapping of data records of a stream to a plurality of partitions, is selected to distribute data records of a data stream among a plurality of nodes of a stream management service. Data ingestion nodes and storage nodes are configured according to the initial mapping. In response to a determination that a triggering criterion for dynamically repartitioning the data stream has been met, a modified mapping is generated, and a different set of ingestion and storage nodes are configured. For at least some time during which arriving data records are stored in accordance with the modified mapping, data records stored at the first set of storage nodes in accordance with the initial mapping are retained.

Abstract: Methods and apparatus for vibration cancellation are disclosed. Vibration data from one or more vibration detectors associated with a storage device is collected. The vibration data represents vibrations experienced by the storage device. In response to the vibration data from the one or more vibration detectors, one or more movements for respective ones of one or more counter-vibration actuators to at least partially cancel of the vibrations experienced by the storage device is calculated. The one or more counter-vibration actuators perform the one or more movements.

Abstract: Briefly, aspects of the subject matter described herein relate to job submission. In aspects, a client (which itself may be a scheduler) sends a request to a scheduler to execute a job. The client may indicate which extensions to a base job submission protocol the client supports. The scheduler may implement a base case protocol and may also implement extensions to the base case. The client and scheduler may communicate information and requests about a job based on extensions both support, if any. The scheduler maintains state information about executing jobs that may include substate information without affecting interoperability with clients that do not recognize substates. A job may be in multiple substates at the same time.

Abstract: Fault-recovery information for transactional operations between computer systems and the outcome of those operations is logged at the originating point of the units of work, such as source-computer systems, which may include client computers in a client/server system. By using these distributed logs of fault-recovery information, a target-computer system, which may include a server computer, can reconstruct the target-computer system's previous interaction with various source-computer systems. Source-computer systems may use a proxy server for logging fault-recovery information to persistent storage. A target-computer system may digitally sign the fault-recovery information it sends to a source-computer system thereby making the fault-recovery information's authenticity verifiable. Similarly, the target-computer system may encrypt the fault-recovery information before sending it to the source-computer system to preserve the privacy of the fault-recovery information.

Abstract: Fault-recovery information for transactional operations between computer systems and the outcome of those operations is logged at the originating point of the units of work, such as source-computer systems, which may include client computers in a client/server system. By using these distributed logs of fault-recovery information, a target-computer system, which may include a server computer, can reconstruct the target-computer system's previous interaction with various source-computer systems. Source-computer systems may use a proxy server for logging fault-recovery information to persistent storage. A target-computer system may digitally sign the fault-recovery information it sends to a source-computer system thereby making the fault-recovery information's authenticity verifiable. Similarly, the target-computer system may encrypt the fault-recovery information before sending it to the source-computer system to preserve the privacy of the fault-recovery information.