Abstract: A computer-implemented method for processing input data in a mapreduce framework includes: receiving, in the mapreduce framework, a data processing request for input data; initiating, based on the data processing request, a map operation on the input data by multiple mappers in the mapreduce framework, each of the mappers using an aggregator to partially aggregate the input data into one or more intermediate key/value pairs; initiating a reduce operation on the intermediate key/value pairs by multiple reducers in the mapreduce framework, wherein, without sorting the intermediate key/value pairs, those of the intermediate key/value pairs with a common key are handled by a same one of the reducers, each of the reducers using the aggregator to aggregate the intermediate key/value pairs into one or more output values; and providing the output values in response to the data processing request.

Abstract: Provided are a computer program product, system, and method for allocating physical nodes for processes in an execution plan. An execution plan is generated indicating a plurality of processes. A resource requirement is generated indicating requested physical nodes and an assignment of the processes to execute on the requested physical nodes. A determination is made from the resource requirement of a resource allocation of physical nodes for the requested physical nodes and the processes. The execution plan is updated to generate an updated execution plan indicating the physical nodes on which the processes will execute according to the received resource allocation.

Abstract: Techniques described herein are generally related to retrieval of data in computer systems having multi-level caches. The multi-level cache may include at least a first cache and a second cache. The first cache may be configured to receive a request for a cache line. The request may be associated with an instruction executing on a tile of the computer system. A suppression status of the instruction may be determined by a first cache controller to determine whether look-up of the first cache is suppressed based upon the determined suppression status. The request for the cache line may be forwarded to the second cache by the first cache controller after the look-up of the first cache is suppressed.

Abstract: Technologies are generally described for methods and systems to assign threads in a multi-core processor. In an example, a method to assign threads in a multi-core processor may include determining data relating to memory controllers fetching data in response to cache misses experienced by a first core and a second core. Threads may be assigned to cores based on the number of cache misses processed by respective memory controllers. Methods may further include determining that a thread is latency-bound or bandwidth-bound. Threads may be assigned to cores based on the determination of the thread as latency-bound or bandwidth-bound. In response to the assignment of the threads to the cores, data for the thread may be stored in the assigned cores.

Abstract: A method for reducing a compile time of a source program includes receiving, by a compiler hosted on a computer, the source program. The compiler may have a compile time that depends non-linearly on a size of a function in the source program. The method involves identifying a source function in the source program and splitting the source function in to two or more target functions having sizes smaller than a size of the source function. The method further includes compiling the source program with the two or more target functions having sizes smaller than a size of the source function replacing the source function in the source program.

Abstract: Shuffle optimization in map-reduce processing. The method includes: obtaining intermediate results from a plurality of mappers for an application on a computing device; combining the intermediate results from the plurality of mappers; and partitioning the combined intermediate results into intermediate results for respective reducers for the application based on respective keys of the combined intermediate results.

Abstract: An electronic apparatus and a method of controlling the same are described. A method of controlling an electronic apparatus having a main processor and an image processor includes: booting the main processor and the image processor according to a power input signal of the photographing apparatus; when the image processor is completely booted, performing a photographing preparation operation; photographing an object, performed by the image processor, in response to a shutter release signal; and generating photographing data of the object, performed by the image processor.

Abstract: A digital data processing system that is designed to facilitate use of UML activity diagrams.

Abstract: A method for processing computer program code to enable different parts of the computer program code to be executed by different processing elements of a plurality of communicating processing elements. The method comprises identifying at least one first part of the computer program code, which is to be executed by a particular one of said processing elements. The method further comprises identifying at least one further part of the computer code which is related to the at least one first part of the computer code. The at least one first part of the computer program code and the at least one further part of the computer program code are caused to be executed by the particular one of said processing elements.

Abstract: Data flow disruptions over a series of data processing operators can be detected by a computer system that generates a profile for data flow at an operator. The profile can include data input, processing, and output wait times. Using the profile, the system can detect potential flow disruptions. If the potential disruption satisfies a rule, it is considered a data flow disruption and a recommendation associated with the satisfied rule is identified. The recommendation and the operator identity is displayed.

Abstract: Embodiments for a method of interfacing with a remote application programming interface (API) by defining an execution plan using an interface definition language and a dependency configuration file to generate a constrained directed graph of hierarchically dependent functions of the API, and executing the execution plan using an executing engine that traverses the graph to call the API functions in a defined order and convert data output by a parent function call to input required by a child function call until a terminal vertex is reached that results in directing the resulting in an action such as data to persist and/or affecting the state of a system.

Abstract: Aspects of the disclosure relate to learning the most useful backup tasks when processing a job in a distributed computing application. For example, training data may be generated by running a job and running backup tasks for some of the tasks for that job. List of features may be generated for each backup task at different times. A representative list of features may be labeled according to the backup tasks usefulness to the job. The lists and usefulness labels may be used to generate a model of usefulness. The model may then be used to evaluate the usefulness of the tasks for the next job. This information may then be used to schedule backup tasks for execution. In addition, lists of features for these backup tasks may be generated and then used to update the model in a continuous learning loop.

Abstract: A computing system for handling barrier commands includes a memory, an interface, and a processor. The memory is configured to store a pre-barrier spreading range that identifies a target computing system associated with a barrier command. The interface is coupled to the memory and is configured to send a pre-barrier computing probe to the target computing system identified in the pre-barrier spreading range and receive a barrier completion notification messages from the target computing system. The pre-barrier computing probe is configured to instruct the target computing system to monitor a status of a transaction that needs to be executed for the barrier command to be completed. The processor is coupled to the interface and is configured to determine a status of the barrier command based on the received barrier completion notification messages.

Abstract: Data is received that includes at least a portion of a program. Thereafter, entry point locations and execution-relevant metadata of the program are identified and retrieved. Regions of code within the program are then identified using static disassembly and based on the identified entry point locations and metadata. In addition, entry points are determined for each of a plurality of functions. Thereafter, a set of possible call sequences are generated for each function based on the identified regions of code and the determined entry points for each of the plurality of functions. Related apparatus, systems, techniques and articles are also described.

Abstract: A method and system for operating a data center. The method includes, in response to a map task distributed by a job tracker, a map task tracker executes the map task to generate a map output including version information. The map task tracker stores the generated map outputs. The map task tracker informs the job tracker of related information of the map output. In response to a reduce task distributed by the job tracker, the reduce task tracker acquires the map outputs for key names including given version information from the map task trackers, wherein the acquired map outputs include the map outputs with the given version information and historical map outputs with the version information prior to the given version information. The reduce task tracker executes the reduce task on the acquired map outputs.

Abstract: A map-reduce compatible distributed file system that consists of successive component layers that each provide the basis on which the next layer is built provides transactional read-write-update semantics with file chunk replication and huge file-create rates. A primitive storage layer (storage pools) knits together raw block stores and provides a storage mechanism for containers and transaction logs. Storage pools are manipulated by individual file servers. Containers provide the fundamental basis for data replication, relocation, and transactional updates. A container location database allows containers to be found among all file servers, as well as defining precedence among replicas of containers to organize transactional updates of container contents. Volumes facilitate control of data placement, creation of snapshots and mirrors, and retention of a variety of control and policy information.

Abstract: Aspects of this disclosure relate to a method of compiling high-level software instructions to generate low-level software instructions. In an example, the method includes identifying, with a computing device, a set of high-level (HL) control flow (CF) instructions having one or more associated texture load instructions, wherein the set of HL CF instructions comprises one or more branches. The method also includes converting, with the computing device, the identified set of HL CF instructions to low-level (LL) instructions having a predicate structure. The method also includes outputting the converted (LL) instructions having the predicate structure.

Abstract: A non-transitory computer-readable storage medium may comprise a set of instructions for creating a subsystem process independent from a main system process for data computation. The set of instructions may direct at least one processor to perform acts of: in a main system process, receiving a user input; in the main system process, sending the user input to a subsystem process, the subsystem process is independent from and in parallel with the main system process; in the main system process, generating a first computation result according to the user input; in the subsystem process, generating a second computation result in parallel with the main system process according to the user input; in the main system process, receiving the second computation result from the subsystem process; and in the main system process, combining the first computation result and the second computation result.

Abstract: Data processing in a hybrid computing environment that includes a host computer, a plurality of accelerators, the host computer and the accelerators adapted to one another for data communications by a system level message passing module, the host computer having local memory shared remotely with the accelerators, the accelerators having local memory for the plurality of accelerators shared remotely with the host computer, where data processing according to embodiments of the present invention includes performing, by the plurality of accelerators, a local reduction operation with the local shared memory for the accelerators; writing remotely, by one of the plurality of accelerators to the shared memory local to the host computer, a result of the local reduction operation; and reading, by the host computer from shared memory local to the host computer, the result of the local reduction operation.

Abstract: The technology provides a hybrid local/remote hosted MapReduce framework and infrastructure comprising systems and methods for improving setup, configuration, controllability, debuggability, and integration of a compute job and systems and methods for increasing programmer productivity. The system applies an interpreted programming language for the programmer's custom Map and Reduce algorithms, such that those algorithms can execute identically on both the hosted service as well as locally (e.g., on the programmer's local computing system or device) for development and debugging purposes. Furthermore, the system delivers this service—a hosted MapReduce infrastructure—in a simple and transparent web service.

Abstract: A system and method for issuing a processor instruction to multiple processing sections arranged in an out-of-order processing pipeline architecture. The multiple processing sections include a first execution unit with a pipeline length and a second execution unit operating upon data produced by the first execution unit. An instruction issue unit accepts a complex instruction that is cracked into respective micro-ops for the first execution unit and the second execution unit. The instruction issue unit issues the first micro-op to the first execution unit to produce intermediate data. The instruction issue unit then delays for a time period corresponding to the processing pipeline length of the first execution unit. After the delay, a second micro-op is issued to the second execution unit.

Abstract: A data flow controller for reconfigurable computers. The novel data flow controller includes a first circuit for selecting one of a plurality of operating conditions and a second circuit for determining if the selected condition is met and outputting a control signal accordingly. In an illustrative embodiment, the operating conditions include: when all enabled data available signals are asserted and all enabled space available signals are asserted; when any enabled data available signal is asserted and all enabled space available signals are asserted; when all enabled data available signals are asserted and any enabled space available signal is asserted; and when any enabled data available signal is asserted and any enabled space available signal is asserted. By allowing a configurable element to operate under different possible conditions, data flow signals can also then be used to control what operation the element performs, in addition to controlling when.

Abstract: A method and apparatus for detecting signals. According to an embodiment, energy waves are received at a first receiver system and a second receiver system. The first receiver system generates first information using the energy waves received at the first receiver system. The first receiver system receives second information generated using the energy waves received at the second receiver system from the second receiver system. The first receiver system identifies desired information about a repetitive portion of a signal carried in the energy waves received at the first receiver system using the first information and the second information.

Abstract: A large-scale data processing system and method including a plurality of processes, wherein a master process assigns input data blocks to respective map processes and partitions of intermediate data are assigned to respective reduce processes. In each of the plurality of map processes an application-independent map program retrieves a sequence of input data blocks assigned thereto by the master process and applies an application-specific map function to each input data block in the sequence to produce the intermediate data and stores the intermediate data in high speed memory of the interconnected processors. Each of the plurality of reduce processes receives a respective partition of the intermediate data from the high speed memory of the interconnected processors while the map processes continue to process input data blocks an application-specific reduce function is applied to the respective partition of the intermediate data to produce output values.

Abstract: Some embodiments of a system and a method to tune a computing system based on a profile have been presented. A profile as used herein broadly refers to a file containing various parameters of a computing system, such as kernel parameters (e.g., buffer size, network setup, etc.), usable to configure the computing system. For instance, a set of profiles are stored in a computer-readable storage device in a computing system, such as a server, a personal computer, a laptop computer, etc. A processing device miming on the computing system may receive a user selection of one of the set of profiles. In response to the user selection, the processing device may load the selected profile onto the computing system in order to tune the computing system according to the selected profile.

Abstract: A handheld imaging device includes an image sensor for sensing an image; a multi-core processor for processing the sensed image; and a program memory provided external to the multi-core processor, and communicating therewith via a communication bus. The multi-core processor includes a bus interface for interfacing with the communication bus, and further includes an image sensor interface for interfacing with the image sensor separately from the communication bus and the bus interface. The multi-core processor includes a plurality of parallel processing units connected by a crossbar switch to form the multi-core.

Abstract: In a control device for an internal combustion engine, which includes control unit that has a processor with a plurality of cores and that computes various tasks associated with operation of the internal combustion engine, the control unit includes a selecting unit, that selects at least one core used in the computation from among the plurality of cores, a computing unit that distributes the tasks to the at least one core selected by the selecting unit to perform computation, and an acquisition unit that acquires an engine, rotational speed of the internal combustion engine, and, when the engine rotational speed acquired by the acquisition unit is higher than or equal to a predetermined threshold, the selecting unit increases the number of the cores selected as compared with when the acquired engine rotational speed is lower than the predetermined threshold.

Abstract: A method, system, and article of manufacture for solving ordinary differential equations described in a graphical model with nodes as blocks and dependencies as links using the processing of a computer with a plurality of processors. The method includes: generating segments of block with or without duplication for each block with an internal state and for each block without any output by traversing the graphical model from each block with an internal state to each block without any output; merging the segment to reduce duplication; compiling and converting each segment from the merged results in an executable code; and individually allocating the executable code for each segment to a plurality of processors for parallel execution.

Abstract: NUMA-aware reader-writer locks may leverage lock cohorting techniques to band together writer requests from a single NUMA node. The locks may relax the order in which the lock schedules the execution of critical sections of code by reader threads and writer threads, allowing lock ownership to remain resident on a single NUMA node for long periods, while also taking advantage of parallelism between reader threads. Threads may contend on node-level structures to get permission to acquire a globally shared reader-writer lock. Writer threads may follow a lock cohorting strategy of passing ownership of the lock in write mode from one thread to a cohort writer thread without releasing the shared lock, while reader threads from multiple NUMA nodes may simultaneously acquire the shared lock in read mode. The reader-writer lock may follow a writer-preference policy, a reader-preference policy or a hybrid policy.

Abstract: Provided is a parallel distributed processing method executed by a computer system comprising a parallel-distributed-processing control server, a plurality of extraction processing servers and a plurality of aggregation processing servers. The managed data includes at least a first and a second data items, the plurality of data items each including a value. The method includes a step of extracting data from one of the plurality of chunks according to a value in the second data item, to thereby group the data, a step of merging groups having the same value in the second data item based on an order of a value in the first data item of data contained in a group among groups, and a step of processing data in a group obtained through the merging by focusing on the order of the value in the first data item.

Abstract: System and methods for synchronizing redundant processing elements are provided. In certain embodiments, a self-checking pair of system on chips (SoCs) includes a first SoC configured to execute a first plurality of instructions; and a second SoC configured to execute a second plurality of instructions that are approximately identical; wherein the first SoC exchanges a first instruction count with the second SoC, the first instruction count identifying a number of instructions executed by the first SoC; wherein the second SoC exchanges a second instruction count with the first SoC, the second instruction count identifying a number of instructions executed by the second SoC; and wherein the first SoC executes a first single step execution utility to synchronize the first instruction count with the second instruction count and the second SoC executes a second single step execution utility to synchronize the first instruction count with the second instruction count.

Abstract: According to some embodiments, a data source is accessed from which data will be retrieved via a plurality of processing threads. The data source may have, for example, a plurality of records with each record being associated with a plurality of identifiers. Each of the plurality of identifiers may be dynamically evaluated as a potential range identifier, and the evaluation may be based at least in part on a number of distinct values present within each identifier. One of the potential range identifiers may be selected as a selected range identifier, and the plurality of records may be divided into ranges defined using the selected range identifier.

Abstract: A storage system which has flash memories constituting a storage area and a function of placing and handling the flash memories in on-line mode and off-line mode, and which stores and manages management information of the flash memories in on-line mode. The storage system includes: when some or all of flash memory in on-line mode is placed in off-line mode, creating a management area in off-line mode in the flash memories by moving existing data in an area used as the management area to an area other than the management area; writing the management information of the flash memories to the created management area; and placing the flash memories in off-line mode.

Abstract: Generating parallelized executable code from input code includes statically analyzing the input code to determine aspects of data flow and control flow of the input code; dynamically analyzing the input code to determine additional aspects of data flow and control flow of the input code; generating an intermediate representation of the input code based at least in part on the aspects of data flow and control flow of the input code identified by the static analysis and the additional aspects of data and control flow of the input code identified by the dynamic analysis; and processing the intermediate representation to determine portions of the intermediate representation that are eligible for parallel execution; and generating parallelized executable code from the processed intermediate representation.

Abstract: The systems and methods described herein may implement probabilistic counters and/or update mechanisms for those counters such that they are dependent on the value of a configurable accuracy parameter. The accuracy parameter value may be adjusted to provide fine-grained control over the tradeoff between the accuracy of the counters and the performance of applications that access them. The counters may be implemented as data structures that include a mantissa portion and an exponent portion that collectively represent an update probability value. When updating the counters, the value of the configurable accuracy parameter may affect whether, when, how often, or by what amount the mantissa portion and/or the exponent portion are updated. Updating a probabilistic counter may include multiplying its value by a constant that is dependent on the value of a configurable accuracy parameter. The counters may be accessible within transactions. The counters may have deterministic update policies.

Abstract: The present invention refers to a method for MapReducing the processing of an Electronic Data Interchange (EDI) document (1, the method comprising the following steps: a. mapping the EDI document (1) into a plurality of intermediate documents (10, 11); b. processing the intermediate documents (10, 11) to produce a plurality of intermediate results (20-23); c. reducing the plurality of intermediate results (20-23) to produce a plurality of reduced intermediate results (30, 31); and d. reducing the reduced intermediate results (30, 31) to produce a final result (2) representing the processed EDI document (1).

Abstract: Disclosed is an image forming apparatus which generates print data based on control data for page print inputted from an external device and which forms an image based on the print data. The image forming apparatus includes a control unit having a plurality of arithmetic processing units, and an image forming unit for forming an image based on the print data which is outputted by the control unit. The control unit generates the print data by carrying out a rasterizing process of one page in band unit constituted by a predetermined number of lines based on the control data for page print and sequentially outputs the generated print data at output timings defined in a constant output interval, wherein the rasterizing process is controlled based on whether or not generation of print data is to be completed within the output interval for all of bands in the one page.

Abstract: Provided is a method and system for dynamically parallelizing an application program. Specifically, provided is a method and system having multi-core control that may verify a number of available threads according to an application program and dynamically parallelize data based on the verified number of available threads. The method and system for dynamically parallelizing the application program may divide a data block to be processed according to the application program based on a relevant data characteristic and dynamically map the threads to division blocks, and thereby enhance a system performance.

Abstract: Described embodiments provide for restructuring a scheduling hierarchy of a network processor having a plurality of processing modules and a shared memory. The scheduling hierarchy schedules packets for transmission. The network processor generates tasks corresponding to each received packet associated with a data flow. A traffic manager receives tasks provided by one of the processing modules and determines a queue of the scheduling hierarchy corresponding to the task. The queue has a parent scheduler at each of one or more next levels of the scheduling hierarchy up to a root scheduler, forming a branch of the hierarchy. The traffic manager determines if the queue and one or more of the parent schedulers of the branch should be restructured. If so, the traffic manager drops subsequently received tasks for the branch, drains all tasks of the branch, and removes the corresponding nodes of the branch from the scheduling hierarchy.

Abstract: One embodiment of the present invention sets forth a technique for dynamically specifying a texture header and texture sampler using an index. The index corresponds to a particular register value that may be static or computed during execution of a shader program. Any texture operation instruction may specify an index value for each of the texture header and the texture sampler.

Abstract: Disclosed are methods, systems, paradigms and structures for processing data packets in a communication network by a multi-core network processor. The network processor includes a plurality of multi-threaded core processors and special purpose processors for processing the data packets atomically, and in parallel. An ingress module of the network processor stores the incoming data packets in the memory and adds them to an input queue. The network processor processes a data packet by performing a set of network operations on the data packet in a single thread of a core processor. The special purpose processors perform a subset of the set of network operations on the data packet atomically. An egress module retrieves the processed data packets from a plurality of output queues based on a quality of service (QoS) associated with the output queues, and forwards the data packets towards their destination addresses.

Abstract: An image processing apparatus includes: a receiving unit that receives job data of plural pages; plural RIP processors that interpret and expand the job data into raster images; and an allocating unit that allocates the plural pages of the job data to the plural RIP processors for RIP processing, the allocating unit dividing the job data based on a predetermined data size regardless of page breaks, allocating job data that is to be RIP processed to the plural RIP processors, sending job data, corresponding to and after the pages of the data that is to be RIP processed, to the plural RIP processors, and when a head part of the job data that is to be RIP processed allocated by the allocating unit is in the middle of a page, the plural RIP processors RIP processing the job data from the beginning of the next page.

Abstract: Described herein are methods, systems, apparatuses and products for multithreaded data merging for multi-core central and graphical processing units. An aspect provides for executing a plurality of threads on at least one central processing unit comprising a plurality of cores, each thread comprising an input data set (IDS) and being executed on one of the plurality of cores; initializing at least one local data set (LDS) comprising a size and a threshold; inserting IDS data elements into the at least one LDS such that each inserted IDS data element increases the size of the at least one LDS; and merging the at least one LDS into a global data set (GDS) responsive to the size of the at least one LDS being greater than the threshold. Other aspects are disclosed herein.

Abstract: Provided is a parallel distributed processing method executed by a computer system comprising a parallel-distributed-processing control server, a plurality of extraction processing servers and a plurality of aggregation processing servers. The managed data includes at least a first and a second data items, the plurality of data items each including a value. The method includes a step of extracting data from one of the plurality of chunks according to a value in the second data item, to thereby group the data, a step of merging groups having the same value in the second data item based on an order of a value in the first data item of data contained in a group among groups, and a step of processing data in a group obtained through the merging by focusing on the order of the value in the first data item.

Abstract: A mechanism is provided for redundant execution of a set of instructions. A redundant execution begin (rbegin) instruction to be executed by a first hardware thread on the first processor is identified in the set of instructions. The set of instructions immediately after the rbegin instruction are executed on the first hardware thread and on a second hardware thread. Responsive to both the first processor and the second processor ending execution of the set of instructions, responsive to a first set of cache lines in a first speculative store matching a second set of cache lines in a second speculative store, and responsive to a first set of register states in a first status register matching a second set of register states in a second status register, dirty lines in the first speculative store are committed thereby committing a redundant transaction state to an architectural state.

Abstract: A method of analyzing program source code prepared for a multithreading platform comprises analyzing a targeted source code set to extract a set of characteristic information for each wait operation; analyzing the targeted source code set to extract a set of characteristic information for each notification call to an application programming interface of the multithreading platform; identifying a one-way branching correspondence with a wait operation for each notification call by comparing the extracted set of characteristic information for the notification operation and the extracted set of characteristic information for each wait operation with a set of predefined asynchronous operation correspondence pattern information for notification and wait functions implemented by the application programming interface; extracting a set of information for each identified one-way branching correspondence; and storing the extracted set of information for each identified one-way branching correspondence in a data store.

Abstract: Some embodiments of the present invention provide a system that acquires a lock in a shared memory multiprocessor system. During operation, the system loads the lock into a cache associated with the thread and then reads a value of the lock. If the value indicates that the lock is currently held by another thread, the system periodically executes an instruction that tests a status of the lock. If the status indicates the lock is valid, the system continues to test the status of the lock. Otherwise, if the status indicates that the lock was invalidated by a store, the system attempts to acquire the lock by executing an atomic operation. On the other hand, if the status indicates that the lock was invalidated by an atomic operation, or that the lock is not present in the cache, the system repeats the loading and reading operations.

Abstract: A mechanism is provided for efficient communication of producer/consumer buffer status. With the mechanism, devices in a data processing system notify each other of updates to head and tail pointers of a shared buffer region when the devices perform operations on the shared buffer region using signal notification channels of the devices. Thus, when a producer device that produces data to the shared buffer region writes data to the shared buffer region, an update to the head pointer is written to a signal notification channel of a consumer device. When a consumer device reads data from the shared buffer region, the consumer device writes a tail pointer update to a signal notification channel of the producer device. In addition, channels may operate in a blocking mode so that the corresponding device is kept in a low power state until an update is received over the channel.

Abstract: A method for executing uniprocessor (UP) coded workloads in a computer capable of concurrent thread execution is disclosed. The method identifies threads in the uniprocessor coded workloads (UP-workloads) which can execute concurrently, and identifies threads in the UP-workloads which cannot execute concurrently. First threads which cannot execute concurrently are assigned to a first concurrency group. Second threads which cannot execute concurrently are assigned to a second concurrency group. Any thread in the first concurrency group can execute concurrently with any thread in the second concurrency group. The computer capable of concurrent thread execution then executes the UP-coded workloads in the first concurrency group at substantially the same time as executing the UP-coded workloads in the second concurrency group.

Abstract: General-purpose distributed data-parallel computing using high-level computing languages is described. Data parallel portions of a sequential program written in a high-level language are automatically translated into a distributed execution plan. Map and reduction computations are automatically added to the plan. Patterns in the sequential program can be automatically identified to trigger map and reduction processing. Direct invocation of map and reduction processing is also provided. One or more portions of the reduce computation are pushed to the map stage and dynamic aggregation is inserted when possible. The system automatically identifies opportunities for partial reductions and aggregation, but also provides a set of extensions in a high-level computing language for the generation and optimization of the distributed execution plan. The extensions include annotations to declare functions suitable for these optimizations.