Abstract: The insertion and processing of programming language code (e.g., C#) in SQL statements, and the dynamic compiling of the code to detect errors prior to statement execution. The SQL statement with arbitrary programming language code can then be executed concurrently as a query on a computer cluster. External libraries can be imported and the associated functionality leveraged from within a SQL statement. The programming language compiler performs checking at build time, rather than retuning an error during the statement execution against the cluster, a potentially costly proposition in both computing time and human resources.

Abstract: Core commands and aggregations of such commands are provided to programmers to enable them to generate programs that can be parallel-processed without requiring the programmer to be aware of parallel-processing techniques. The core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the programmer to focus on higher-level concepts. The core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. The output of each core command can implement an interface that can enable underlying mechanisms to stitch together multiple core commands in a cohesive manner to perform more complex actions.

Abstract: Scripting core commands and aggregations of such commands are provided to script authors to enable them to generate scripts that can be parallel-processed without requiring the author to be aware of parallel-processing techniques. The scripting core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the script author to focus on higher-level concepts. The scripting core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. For added flexibility, one or more scripting core commands can utilize functions written in a different programming language and referenced appropriately in code blocks.

Abstract: Structured queries, such as those written using the Structured Query Language (SQL) can be an efficient mechanism for expressing the information sought to be obtained from a collection of data. Mechanisms can enable the use of structured queries to express data processing that can be performed in parallel in order to obtain the attendant efficiencies of such processing. The structured queries, whether standing alone or integrated into another programming context, can be translated into an intermediate form that can be compatible with, or equivalent to, the programming context into which they were integrated or some other high-level programming language. The intermediate, translated form can use core commands that abstract mechanisms that can be executed in parallel. The core commands include commands for applying a function in parallel and distributing and joining data in parallel, and also include aggregations of core commands directed to commonly performed functions.

Abstract: Efficient processing of time-bound messages is described herein. In an implementation, messages are classified either time-bounded messages or non-time-bounded messages. The time-bounded messages are then processed separately from the non-time-bound messages. Examples of time-bounded messages can include spam e-mails or other types or classes of e-mails subject to a retention policy, such that the are retained only for some set period of time. Metadata relating to the time-bound messages are compiled and stored separately from the messages, thereby enabling optimization of the metadata storage and processing apart from the messages. The time-bounded messages are accumulated before they are processed in bulk, thereby reducing the number if I/O operations consumed by the time-bound messages, and reducing the amount and cost of resources supporting the process of the time-bounded messages.

Abstract: The insertion and processing of programming language code (e.g., C#) in SQL statements, and the dynamic compiling of the code to detect errors prior to statement execution. The SQL statement with arbitrary programming language code can then be executed concurrently as a query on a computer cluster. External libraries can be imported and the associated functionality leveraged from within a SQL statement. The programming language compiler performs checking at build time, rather than retuning an error during the statement execution against the cluster, a potentially costly proposition in both computing time and human resources.

Abstract: A graphical application development tool for developing parallel computation applications. The tool facilitates insertion of computational elements by a drag-and-drop operation onto a canvas area for creating a computational graph. The graphical application tool reduces the barriers to the development of parallel computation applications by entry-level developers, for example, by allowing these users to write applications by using a graphical tool, thereby avoiding complexities of having to write well-formed code and learning a new language. The tool includes built-in functionality that allows the developer to write arbitrary code (e.g., C#) to perform various functions on massive amounts of data.

Abstract: Ambiguous stack references in low level software instructions can be resolved when raising a representation of low level software instructions, such as native code. The number of parameters passed to a function can be determined by heuristics, including direct call and indirect call heuristics. Indirect call heuristics can include dataflow-to-target-function, dataflow-to-target-type, import name, and cleanup instructions heuristics. A control flow heuristic can use an alternating direction propagation of seed points to determine the stack height at basic block boundaries. Interpolation can be used within the basic blocks as needed. The raised representation can be used for static analysis of the low level software instructions.

Abstract: Described are systems and techniques for losslessly restarting subsystems in a distributed file system. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. In particular, a map assisted state transfer may include receiving one or more state updates, marshaling one or more active data-structures into a marshaled shadow, applying the received state updates to the marshaled shadow and re-instantiating the active data-structures by unmarshaling the marshaled shadow.

Abstract: Structured queries, such as those written using the Structured Query Language (SQL) can be an efficient mechanism for expressing the information sought to be obtained from a collection of data. Mechanisms can enable the use of structured queries to express data processing that can be performed in parallel in order to obtain the attendant efficiencies of such processing. The structured queries, whether standing alone or integrated into another programming context, can be translated into an intermediate form that can be compatible with, or equivalent to, the programming context into which they were integrated or some other high-level programming language. The intermediate, translated form can use core commands that abstract mechanisms that can be executed in parallel. The core commands include commands for applying a function in parallel and distributing and joining data in parallel, and also include aggregations of core commands directed to commonly performed functions.

Abstract: Scripting core commands and aggregations of such commands are provided to script authors to enable them to generate scripts that can be parallel-processed without requiring the author to be aware of parallel-processing techniques. The scripting core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the script author to focus on higher-level concepts. The scripting core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. For added flexibility, one or more scripting core commands can utilize functions written in a different programming language and referenced appropriately in code blocks.

Abstract: Core commands and aggregations of such commands are provided to programmers to enable them to generate programs that can be parallel-processed without requiring the programmer to be aware of parallel-processing techniques. The core commands and aggregations abstract mechanisms that can be executed in parallel, enabling the programmer to focus on higher-level concepts. The core commands provided include commands for applying a function in parallel and distributing and joining data in parallel. The output of each core command can implement an interface that can enable underlying mechanisms to stitch together multiple core commands in a cohesive manner to perform more complex actions.

Abstract: An architecture and implementation for losslessly restarting subsystems in a distributed file system is described. By partitioning functionality and logging appropriately across the kernel and user-level boundaries on a client, the user-level subsystem may be made losslessly restartable. Practical mechanisms for supporting state-based recovery in replicated state machines and like replica are described. In particular, each client daemon may include an operations log and an applied log sequence number. Each client driver may include a potentially different operations log. Each client daemon may be configured to request logged operations associated with log sequence numbers in one or more ranges specified by a specification that includes the applied log sequence number. The requested logged operations may reside in the operations log maintained by a client driver.

Abstract: An index of a search engine includes two portions: a long-term portion that is optimized for lookup performance and is stored in bulk storage, for example, non-volatile memory, and a short-term portion that is easily updatable and is stored solely or primarily in random access memory (RAM). Both portions of the index are searchable. The vast majority of documents in the location space are indexed in the long-term portion in a format optimized for lookup, while new documents are immediately searchable in the easily updatable short-term portion, which has a different format. The long-term portion is updated with indexing information of the short-term portion.

Abstract: Each component binary in a heterogeneous program is translated from a platform-specific instruction set into a set of intermediate representation (IR) instructions that are platform-neutral. The IR instructions are grouped into IR code blocks, the IR code blocks into IR procedures, and the IR procedures into IR components to create an intermediate representation hierarchy for the program. An application program interface is provided that permits user access to the IR hierarchy for instrumentation, optimization, navigation, and manipulation of the IR hierarchy. The transformed IR hierarchy is then translated into platform-specific instructions and output as a modified binary. The user can designate a different platform for the output translation of a code block than the platform for which the code block was originally written. Prologue and epilog code is added to contiguous blocks that are translated into different architectures.

Abstract: Each component binary in a heterogeneous program is translated from a platform-specific instruction set into a set of intermediate representation (IR) instructions that are platform-neutral. The IR instructions are grouped into IR code blocks, the IR code blocks into IR procedures, and the IR procedures into IR components to create an intermediate representation hierarchy for the program. An application program interface is provided that permits user access to the IR hierarchy for instrumentation, optimization, navigation, and manipulation of the IR hierarchy. The transformed IR hierarchy is then translated into platform-specific instructions and output as a modified binary. The user can designate a different platform for the output translation of a code block than the platform for which the code block was originally written. Prologue and epilog code is added to contiguous blocks that are translated into different architectures.

Abstract: Pre-defined performance tools enable instrumentation and optimization of a heterogeneous program by modifying a platform-neutral intermediate representation (IR) of the program. The intermediate representation is hierarchy of base elements that correspond to instructions, code blocks, procedures and components within the program. Each base element exports an application program interface that provides for navigation, querying and modification of the corresponding element in the hierarchy. An application program interface for the entire intermediate representation includes the application program interfaces for the individual elements along with additional functionality. A user can request instrumentation and optimization of any level of the intermediate representation hierarchy and the performance tools perform the required modifications to the intermediate representation.