Abstract: An execution environment in a computer system allows user code to be executed using multiple execution runtimes. The execution environment translates the user code into a runtime agnostic representation, selects an execution runtime for executing the runtime agnostic representation, and invokes a scheduler for the selected execution runtime. The scheduler dispatches tasks from the runtime agnostic representation for execution by the computer system using concurrency mechanisms in the selected execution runtime.

Abstract: An execution environment in a computer system provides dynamic data and compute resources elasticity for user code to improve execution efficiency. The execution environment translates the user code into a runtime agnostic representation with a set of tasks. For each task, the execution environment determines a level of concurrency for executing the task based on the size of the set of input data for the task, the amount of compute resources available at the time of invocation of the task, and any context-sensitive heuristics provided by the user code.

Abstract: An execution environment in a computer system supports a declarative programming model where user code is written with a query syntax in a native programming language to express inherent parallelism in terms of data flow. The execution environment translates queries in the user code into a runtime agnostic representation and dynamically selects an execution runtime for executing the runtime agnostic representation.

Abstract: In various embodiments, methods and systems for implementing multiple transaction logs in a distributed storage system are provided. A log stream component detects performance metrics of a plurality of log streams. The performance metrics are associated with requests from partitions in the distributed storage system. A transaction component receives a request to execute a transaction using a log stream. The request is received from a partition of the distributed storage system. The performance metrics of the plurality of log streams can be referenced, where the performance metrics indicate a performance capacity of a selected log stream to process the request. A log stream for executing the transaction is determined based on the performance capacity. The log stream selected can also factor request attributes of the request. The transaction component communicates the request to be executed, using the log stream to perform the transaction.

Abstract: Embodiments are directed to dynamically allocating processing resources among a plurality of resource schedulers. A resource manager dynamically allocates resources to a first resource scheduler. The resource manager is configured to dynamically allocate resources among a plurality of resource schedulers, and each scheduler is configured to manage various processing resources. The resource manager determines that at least one of the processing resources dynamically allocated to the first resource scheduler is idle. The resource manager determines that at least one other resource scheduler needs additional processing resources and, based on the determination, loans the determined idle processing resource of the first resource scheduler to a second resource scheduler.

Abstract: Functionality is described for providing a compiled program that can be executed in a parallel and a distributed manner by any selected runtime environment. The functionality includes a compiler module for producing the compiled program based on a dataflow representation of a program (i.e., a dataflow-expressed program). The dataflow-expressed program, in turn, includes a plurality of tasks that are connected together in a manner specified by a graph (such as a directed acyclic graph). The compiler module also involves performing static type-checking on the dataflow-expressed program to identify the presence of any mismatch errors in the dataflow-expressed program. By virtue of this approach, the above-described functionality can identify any errors in constructing the graph prior to its instantiation and execution in a runtime environment.

Abstract: An execution environment in a computer system supports a declarative programming model where user code is written with a query syntax in a native programming language to express inherent parallelism in terms of data flow. The execution environment translates queries in the user code into a runtime agnostic representation and dynamically selects an execution runtime for executing the runtime agnostic representation.

Abstract: An execution environment in a computer system allows user code to be executed using multiple execution runtimes. The execution environment translates the user code into a runtime agnostic representation, selects an execution runtime for executing the runtime agnostic representation, and invokes a scheduler for the selected execution runtime. The scheduler dispatches tasks from the runtime agnostic representation for execution by the computer system using concurrency mechanisms in the selected execution runtime.

Abstract: An execution environment in a computer system provides dynamic data and compute resources elasticity for user code to improve execution efficiency. The execution environment translates the user code into a runtime agnostic representation with a set of tasks. For each task, the execution environment determines a level of concurrency for executing the task based on the size of the set of input data for the task, the amount of compute resources available at the time of invocation of the task, and any context-sensitive heuristics provided by the user code.

Abstract: Embodiments are directed to dynamically allocating processing resources among a plurality of resource schedulers. A resource manager dynamically allocates resources to a first resource scheduler. The resource manager is configured to dynamically allocate resources among a plurality of resource schedulers, and each scheduler is configured to manage various processing resources. The resource manager determines that at least one of the processing resources dynamically allocated to the first resource scheduler is idle. The resource manager determines that at least one other resource scheduler needs additional processing resources and, based on the determination, loans the determined idle processing resource of the first resource scheduler to a second resource scheduler.

Abstract: A system that facilitates the storage of data using a write barrier. The system interfaces to a hardware component that stores data, and includes a write barrier component that dynamically employs instructions compatible with the hardware component to ensure data integrity during storage of the data. The write barrier component is independent of at least an operating system and an application and can operate in a least one of a user mode and a kernel mode. The write barrier component includes at least one of software instructions, routines, and methods, the selection of one or more of which is based on hardware data extracted from the hardware component. A selection component interrogates the hardware component for hardware data to facilitate selection of one or more instructions most suitable for interfacing to the hardware component. A coalescing component combines cache synchronization requests into a single set of instructions, which set is processed to flush a disk cache in one process.

Abstract: A system that facilitates the storage of data using a write barrier. The system interfaces to a hardware component that stores data, and includes a write barrier component that dynamically employs instructions compatible with the hardware component to ensure data integrity during storage of the data. The write barrier component is independent of at least an operating system and an application and can operate in a least one of a user mode and a kernel mode. The write barrier component includes at least one of software instructions, routines, and methods, the selection of one or more of which is based on hardware data extracted from the hardware component. A selection component interrogates the hardware component for hardware data to facilitate selection of one or more instructions most suitable for interfacing to the hardware component. A coalescing component combines cache synchronization requests into a single set of instructions, which set is processed to flush a disk cache in one process.