Abstract: Methods, systems, and computer-readable media for a client-side filesystem for a remote repository are disclosed. One or more files of a repository are sent from a storage service to a client device. The file(s) are obtained by the client using a credential sent by a repository manager. Local copies of the file(s) are accessible via a local filesystem mounted at the client device. One or more new files associated with the repository are generated at the client device. Using the credential, the one or more new files are obtained at the storage service from the client device. The one or more new files are added to the repository.

Abstract: A customer of a computing resource service provider submits a request to a service in a first data region to obtain all customer data stored across all data regions managed by the computing resource service provider. Accordingly, the service may refer to a chatter table that includes a listing of locations wherein customer data may be stored to determine which data regions comprise customer data. Based on the determination, the service may transmit one or more requests to other services within the determined data regions to obtain the customer data while ignoring any data regions that do not comprise customer data. Upon receiving the customer data, the service may aggregate this data and provide the aggregated data to the customer to fulfill the customer request.

Abstract: A request to add tags (e.g., labels, key-value pairs, or metadata) to resources can be digitally signed by the entity making the request, such that the source can be verified and an authorization determination made for each tag. For a request involving multiple services (or entities) that can each add tags, any tag added by a service can be included in the request and digitally signed by that service. Each service processing the request can also digitally sign the request before forwarding, so that each service signs a version of the request, which includes elements signed by other services earlier in the request chain. When the request is received to a tagging service, the service ensures that every tag was digitally signed by the appropriate authorized entity or service, and validates the signatures to ensure that no data was modified or omitted, before adding the tags to the designated resource(s).

Abstract: In a network-based services platform, customer resources may be automatically associated with a resource creator. Upon receipt of a request to create a computing resource, the computing resource may be created and thereafter executed within the network-based services platform. In response to the request, an entity associated with the request may be identified. In response to the identification of the entity, the computing resource may be automatically associated with the identified entity specifying the entity as the creator of the computing resource. In some embodiments, a cost of operating the resource over a period of time may be calculated, and the cost allocated to the identified entity based at least in part on the creator-to-resource association that was automatically generated upon creation of the resource. In this manner, a usage allocation report may be provided to the customer that shows the cost associated with the entity.

Abstract: An embodiment includes selecting a job to upload to a remote system, uploading a portion of data for the job to the remote system, and receiving notice that the portion of data for the job has been staged at the remote system. The embodiment also includes, based on the portion of data being staged, instructing the remote system to begin performing operations for the job on the portion of the data, where after the remote system begins to perform the operations for the job on the portion of data prior to all data for the job being uploaded. From an alternate perspective an embodiment includes receiving a description of a job to be performed on data, receiving a portion of the data, receiving an instruction to begin performing computations on the portion of data, and beginning to perform identified computations on the portion of data before receiving all the data.

Abstract: A user of a tag manager utilizes a tag manager to request creation of one or more zonal metadata tags. When the metadata tags are created, the tag manager updates a tag manager database to include information concerning the created metadata tags. Subsequently, the tag manager attempts to update a changes table within a discovery service database to include this information, the changes table usable by a discovery service to support regional and global tag queries. If the update of the changes table is initially unsuccessful, the tag manager may utilize a replicator to synchronize the tag manager database and the changes table when the discovery service database is once again available.

Abstract: The preemption of running jobs by other running or queued jobs in a system that has processing resources. The system has running jobs, and queued jobs that are awaiting processing by the system. In a scheduling operation, preemptor jobs are identified, the preemptor jobs being jobs that are candidates for preempting one or more of the running jobs. The preemptor jobs include queued jobs, as well as running jobs that are capable of using more processing resource of the system. One of the other running jobs is preempted to free processing resources for the running job that was identified as a preemptor job. Accordingly, not only may queued jobs preempt running jobs, but currently running jobs may preempt other currently running jobs.

Abstract: Collecting usage data in a cluster computing environment. A method includes at a tracker service receiving a request from an at least partially cloud based deployment for an interval for the deployment to report usage information in usage reports. The usage information includes information defining how software in a deployment is used. In response to the request from the deployment, the method further includes, the tracker service providing an interval to the deployment. The method further includes at the tracker service, receiving usage reports from the deployment according to the provided interval.

Abstract: A scheduler for a compute cluster that allocates computing resources to jobs to achieve a balanced distribution. The balanced distribution maximizes the number of executing jobs to provide fast response times for all jobs by, to the extent possible, assigning a designated minimum for each job. If necessary to achieve this minimum distribution, resources in excess of a minimum previously allocated to a job may be de-allocated, if those resources can be used to meet the minimum requirements of other jobs. Resources above those used to meet the minimum requirements of executing jobs are allocated based on a computed desired allocation, which may be developed based on respective job priorities. To meet the desired allocation, resources may be de-allocated from jobs having more than their desired allocation and re-allocated to jobs having less than their desired allocation of resources.

Abstract: An embodiment includes selecting a job to upload to a remote system, uploading a portion of data for the job to the remote system, and receiving notice that the portion of data for the job has been staged at the remote system. The embodiment also includes, based on the portion of data being staged, instructing the remote system to begin performing operations for the job on the portion of the data, where after the remote system begins to perform the operations for the job on the portion of data prior to all data for the job being uploaded. From an alternate perspective an embodiment includes receiving a description of a job to be performed on data, receiving a portion of the data, receiving an instruction to begin performing computations on the portion of data, and beginning to perform identified computations on the portion of data before receiving all the data.

Abstract: Moving data to a remote system and performing computations at the remote system. A method includes receiving from one or more users a plurality jobs. The one or more jobs specify both data to be operated on and the operations to be performed on the data at the remote system. At a data service, one of the jobs to upload to the remote system in priority over other jobs in the plurality of jobs is selected. The method further includes beginning to upload the prioritized job including staging data from the job. At the remote system, once a portion of the data has been received, a service in the remote system begins to perform the specified operation on the data for the prioritized job. Beginning to perform the specified operation is performed prior to all of the data from the prioritized job being uploaded to the remote system.

Abstract: Embodiments are directed to managing and verifying licenses in a cluster computer system environment. In an embodiment, a license management application running on a computer system cluster manager receives a job that has multiple job tasks as well as portions of job information. The license management application determines from the job information how many licenses and computer nodes are to be assigned to the job. The license management application checks out the determined number of licenses from a license distributing application on behalf of the received job. The license management application indicates to a scheduler of the computer system cluster manager that one job task is to be run per checked out license.

Abstract: Moving data to a remote system and performing computations at the remote system. A method includes receiving from one or more users a plurality jobs. The one or more jobs specify both data to be operated on and the operations to be performed on the data at the remote system. At a data service, one of the jobs to upload to the remote system in priority over other jobs in the plurality of jobs is selected. The method further includes beginning to upload the prioritized job including staging data from the job. At the remote system, once a portion of the data has been received, a service in the remote system begins to perform the specified operation on the data for the prioritized job. Beginning to perform the specified operation is performed prior to all of the data from the prioritized job being uploaded to the remote system.

Abstract: The scheduling of a processing job to be performed by at least a portion of a cluster of processing resources distributed across multiple machines. The processing job is associated with a certain entity having a certain amount of guaranteed processing resources on the cluster. If there are enough processing resources to perform the job, then the job may be initiated. On the other hand, if there are not enough processing resources to perform the job, and there are yet some remaining processing resources that are guaranteed to the certain entity, then one or more other jobs that are associated with other entities are at least partially preempted until there are enough processing resources to perform the processing job.

Abstract: Collecting usage data in a cluster computing environment. A method includes at a tracker service receiving a request from an at least partially cloud based deployment for an interval for the deployment to report usage information in usage reports. The usage information includes information defining how software in a deployment is used. In response to the request from the deployment, the method further includes, the tracker service providing an interval to the deployment. The method further includes at the tracker service, receiving usage reports from the deployment according to the provided interval.

Abstract: The preemption of running jobs by other running or queued jobs in a system that has processing resources. The system has running jobs, and queued jobs that are awaiting processing by the system. In a scheduling operation, preemptor jobs are identified, the preemptor jobs being jobs that are candidates for preempting one or more of the running jobs. The preemptor jobs include queued jobs, as well as running jobs that are capable of using more processing resource of the system. One of the other running jobs is preempted to free processing resources for the running job that was identified as a preemptor job. Accordingly, not only may queued jobs preempt running jobs, but currently running jobs may preempt other currently running jobs.

Abstract: Embodiments are directed to managing and verifying licenses in a cluster computer system environment. In an embodiment, a license management application running on a computer system cluster manager receives a job that has multiple job tasks as well as portions of job information. The license management application determines from the job information how many licenses and computer nodes are to be assigned to the job. The license management application checks out the determined number of licenses from a license distributing application on behalf of the received job. The license management application indicates to a scheduler of the computer system cluster manager that one job task is to be run per checked out license.

Abstract: The scheduling of a processing job to be performed by at least a portion of a cluster of processing resources distributed across multiple machines. The processing job is associated with a certain entity having a certain amount of guaranteed processing resources on the cluster. If there are enough processing resources to perform the job, then the job may be initiated. On the other hand, if there are not enough processing resources to perform the job, and there are yet some remaining processing resources that are guaranteed to the certain entity, then one or more other jobs that are associated with other entities are at least partially preempted until there are enough processing resources to perform the processing job.

Abstract: A command to perform a task can be received and the task can be started. A command to cancel the task can also be received. The task can be provided with a warning signal and a predetermined grace period of time before cancelling the task, which can allow the task to prepare for cancellation, such as by shutting down cleanly. If the task has not shut down within the grace period, then the task can be cancelled after the grace period expires.

Abstract: A scheduler for a compute cluster that allocates computing resources to jobs to achieve a balanced distribution. The balanced distribution maximizes the number of executing jobs to provide fast response times for all jobs by, to the extent possible, assigning a designated minimum for each job. If necessary to achieve this minimum distribution, resources in excess of a minimum previously allocated to a job may be de-allocated, if those resources can be used to meet the minimum requirements of other jobs. Resources above those used to meet the minimum requirements of executing jobs are allocated based on a computed desired allocation, which may be developed based on respective job priorities. To meet the desired allocation, resources may be de-allocated from jobs having more than their desired allocation and re-allocated to jobs having less than their desired allocation of resources.