Abstract: A journaling file system may implement remote durable logging. Updates to a file system may be received, and log records describing the updates may be stored in a locally-accessible file system change log. The update may then be acknowledged as committed. The log records may then be sent to be stored in a network-based data store in a remote version of the file system change log. Once it may be determined that the log records are stored in the remote version, storage space for the log records in the local file system change log may be reclaimed. Various types of restoration and duplication techniques may be implemented based on the remote version of the change log to restore a file system at an originating device or to duplicate the file system at a different device.

Abstract: Methods, systems, and computer-readable media for execution of replicated tasks using redundant resources are disclosed. Replicas of a task are generated. Computing resources are selected from at least one pool of computing resources of a provider network. The provider network includes a plurality of pools of computing resources that vary in a characteristic, and the computing resources are selected based (at least in part) on the characteristic. Concurrent execution of the replicas of the task is initiated using the selected computing resources. Input data for the concurrent execution does not vary from one of the replicas to another of the replicas, and at least a portion of the replicas produce individual results for the input data. Based (at least in part) on a policy, an individual result of one or more of the replicas is selected as a final result of the task.

Abstract: Methods, systems, and computer-readable media for job scheduling based on job execution history are disclosed. A request is received to schedule a workload comprising a plurality of jobs. A resource allocation score for the workload is determined. The resource allocation score represents (at least in part) an estimated likelihood of successful execution of the workload. A first portion of the workload is scheduled for execution, and a remaining portion (if any) of the workload is delayed. A quantity of jobs in the first portion of the workload is determined based (at least in part) on the resource allocation score. Execution of the first portion of the workload is initiated.

Abstract: Methods, systems, and computer-readable media for job execution with managed compute environments are disclosed. A specification of a managed compute environment comprises one or more constraints associated with computing resources in the managed compute environment. A queue or other data structure that is associated with the managed compute environment is monitored. The data structure is configured to store jobs. Data indicative of a job is detected in the data structure. One or more computing resources are reserved for the job from a pool of available computing resources. The one or more computing resources are selected for the job based at least in part on the one or more constraints associated with computing resources in the managed compute environment. Execution of the job using the one or more computing resources is initiated.

Abstract: A scheduler of a batch job management service determines that a set of resources a client is insufficient to execute one or more jobs. The scheduler prepares a multi-dimensional statistical representation of resource requirements of the jobs, and transmits it to a resource controller. The resource controller uses the multi-dimensional representation and resource usage state information to make resource allocation change decisions.

Abstract: Methods, systems, and computer-readable media for job execution with managed compute environments are disclosed. A specification of a managed compute environment comprises one or more constraints associated with computing resources in the managed compute environment. A queue or other data structure that is associated with the managed compute environment is monitored. The data structure is configured to store jobs. Data indicative of a job is detected in the data structure. One or more computing resources are reserved for the job from a pool of available computing resources. The one or more computing resources are selected for the job based at least in part on the one or more constraints associated with computing resources in the managed compute environment. Execution of the job using the one or more computing resources is initiated.

Abstract: Methods, systems, and computer-readable media for job execution with scheduled reserved compute instances are disclosed. One or more queues are mapped to a compute environment. The queue(s) are configured to store data indicative of jobs. The compute environment is associated with one or more scheduled reserved compute instances, and the one or more scheduled reserved compute instances are reserved for use in the compute environment for a window of time. The queue(s) are mapped to the compute environment prior to the window of time opening. During the window, at least one of the scheduled reserved compute instances is added to the compute environment. The scheduled reserved compute instance(s) are provisioned from a pool of available compute instances. During the window, execution is initiated of one or more jobs from the queue(s) on the scheduled reserved compute instance(s) in the compute environment.

Abstract: A scheduler of a batch job management service determines that a set of resources a client is insufficient to execute one or more jobs. The scheduler prepares a multi-dimensional statistical representation of resource requirements of the jobs, and transmits it to a resource controller. The resource controller uses the multi-dimensional representation and resource usage state information to make resource allocation change decisions.

Abstract: Methods, systems, and computer-readable media for job scheduling based on job execution history are disclosed. A request is received to schedule a workload comprising a plurality of jobs. A resource allocation score for the workload is determined. The resource allocation score represents (at least in part) an estimated likelihood of successful execution of the workload. A first portion of the workload is scheduled for execution, and a remaining portion (if any) of the workload is delayed. A quantity of jobs in the first portion of the workload is determined based (at least in part) on the resource allocation score. Execution of the first portion of the workload is initiated.

Abstract: A scheduler of a batch job management service determines that a set of resources a client is insufficient to execute one or more jobs. The scheduler prepares a multi-dimensional statistical representation of resource requirements of the jobs, and transmits it to a resource controller. The resource controller uses the multi-dimensional representation and resource usage state information to make resource allocation change decisions.

Abstract: Methods, systems, and computer-readable media for job scheduling based on job execution history are disclosed. A request is received to schedule a workload comprising a plurality of jobs. A resource allocation score for the workload is determined. The resource allocation score represents (at least in part) an estimated likelihood of successful execution of the workload. A first portion of the workload is scheduled for execution, and a remaining portion (if any) of the workload is delayed. A quantity of jobs in the first portion of the workload is determined based (at least in part) on the resource allocation score. Execution of the first portion of the workload is initiated.

Abstract: A journaling file system may implement remote durable logging. Updates to a file system may be received, and log records describing the updates may be stored in a locally-accessible file system change log. The update may then be acknowledged as committed. The log records may then be sent to be stored in a network-based data store in a remote version of the file system change log. Once it may be determined that the log records are stored in the remote version, storage space for the log records in the local file system change log may be reclaimed. Various types of restoration and duplication techniques may be implemented based on the remote version of the change log to restore a file system at an originating device or to duplicate the file system at a different device.

Abstract: Methods, systems, and computer-readable media for task-level optimization of compute environments are disclosed. Execution is initiated of one or more tasks using a plurality of computing resources provisioned from a multi-tenant provider network. At least some of the computing resources vary in configuration. One or more metrics are determined that are associated with the execution of the one or more tasks. A configuration of the computing resources is selected based at least in part on the one or more metrics. A modified job definition associated with the one or more tasks is generated. The modified job definition indicates the selected configuration.

Abstract: Technology is provided for data set scoring. In one example, a method includes analyzing first and second characteristics of a data set. The first and second characteristics represent a quality of data values in the data set. At least the first characteristic is independent of the data values in the data set. The method further includes assigning a score to the data set based on the first and second characteristics. The data set may be ranked against a plurality of other data sets based on the score. The score of the data set may be provided together with a scoring scale to enable a determination of the quality of the data values based on the score.

Abstract: A journaling file system may implement remote durable logging. Updates to a file system may be received, and log records describing the updates may be stored in a locally-accessible file system change log. The update may then be acknowledged as committed. The log records may then be sent to be stored in a network-based data store in a remote version of the file system change log. Once it may be determined that the log records are stored in the remote version, storage space for the log records in the local file system change log may be reclaimed. Various types of restoration and duplication techniques may be implemented based on the remote version of the change log to restore a file system at an originating device or to duplicate the file system at a different device.

Abstract: A scheduler of a batch job management service determines that a set of resources a client is insufficient to execute one or more jobs. The scheduler prepares a multi-dimensional statistical representation of resource requirements of the jobs, and transmits it to a resource controller. The resource controller uses the multi-dimensional representation and resource usage state information to make resource allocation change decisions.