Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.

Abstract: An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: Use of a smart proxy as a wrapper around a stub in a distributed system. Instead of receiving a stub as a result of a remote procedure call, a caller receives a smart proxy including the stub as an embedded object. The smart proxy performs predefined processing associated with a remote procedure call, the processing possibly occurring before, during, or after a response to the call.

Abstract: In a first aspect, a method includes, at a node of a Hadoop cluster, the node storing a first portion of data in HDFS data storage, executing a first instance of a data processing engine capable of receiving data from a data source external to the Hadoop cluster, receiving a computer-executable program by the data processing engine, executing at least part of the program by the first instance of the data processing engine, receiving, by the data processing engine, a second portion of data from the external data source, storing the second portion of data other than in HDFS storage, and performing, by the data processing engine, a data processing operation identified by the program using at least the first portion of data and the second portion of data.

Abstract: An approach to parallel access of data from a distributed filesystem provides parallel access to one or more named units (e.g., files) in the filesystem by creating multiple parallel data streams such that all the data of the desired units is partitioned over the multiple streams. In some examples, the multiple streams form multiple inputs to a parallel implementation of a computation system, such as a graph-based computation system, dataflow-based system, and/or a (e.g., relational) database system.

Abstract: An improved lookup service is provided that allows for the dynamic addition and deletion of services. This lookup service allows for the addition and deletion of services automatically, without user intervention. As a result, clients of the lookup service may continue using the lookup service and its associated services while the updates occur. Additionally, the lookup service provides a notification mechanism that can be used by clients to receive a notification when the lookup service is updated. By receiving such a notification, clients can avoid attempting to access a service that is no longer available and can make use of new services as soon as they are added to the lookup service.

Abstract: An improved lookup service is provided that allows for the dynamic addition and deletion of services. This lookup service allows for the addition and deletion of services automatically, without user intervention. As a result, clients of the lookup service may continue using the lookup service and its associated services while the updates occur. Additionally, the lookup service provides a notification mechanism that can be used by clients to receive a notification when the lookup service is updated. By receiving such a notification, clients can avoid attempting to access a service that is no longer available and can make use of new services as soon as they are added to the lookup service.

Abstract: In accordance with the present invention a process is provided for allocating and deallocating resources in a distributed processing system having a requester platform and a server platform. The process involves receiving a request from the requestor platform referring to a system resource and specifying a requested lease period, permitting shared access to the system resource for a lease period, sending a return call to the requestor platform advising of the lease period, and deallocating the system resource when the lease period expires.

Abstract: A system consistent with the present invention reduces the number of redundant class descriptors that are sent during remote method calls by using serialization contexts. “Serialization contexts” are dictionary objects that map a class descriptor to a corresponding integer handle. When possible, the integer handle, rather than the full class descriptor, is passed, saving processing time in RMI calls.

Abstract: A homogeneous execution environment operates within a heterogeneous client-server network. A client selects a server and transmits a procedure call with parameters. In response, a server dynamically and securely downloads code to a compute server; invokes a generic compute method; executes the code on the compute server; and returns the results to the calling client method, preserving the result on the compute server if requested. This technique is efficient in that it does not require multiple copies of code to be downloaded or compiled since server byte-codes can be executed on each of the different systems, therefore downloading or compiling multiple copies of code can be avoided. The code can be compiled once and downloaded as needed to the various servers as byte-codes and then executed.

Abstract: Use of a policy object for verification in a distributed system. A machine downloads a policy object containing a reference to code governing verification of data. The machine uses the reference to obtain the code and locally verify data or other information. As particular rules for the data change, the policy object may be updated to provide a reference to the code for the new rules when it is downloaded.

Abstract: A stub retrieval and loading subsystem is disclosed for use in connection with a remote method invocation system. The stub retrieval and loading subsystem controls the retrieval and loading of a stub for a remote method, into an execution environment, to facilitate invocation of the remote method by a program executing in the execution environment. The stub retrieval subsystem includes a stub retriever for initiating a retrieval of the stub and stub loader for, when the stub is received by the stub retriever, loading the stub into the execution environment, thereby to make the stub available for use in remote invocation of the remote method. In one embodiment, the stub retrieval and loading subsystem effects the retrieval and loading for a program operating in one address space provided by one computer, of stub class instances to effect the remote invocation of methods which are provided by objects operating in another address space, which may be provided by the same computer or a different computer.

Abstract: Use of a policy object for verification in a distributed system. A machine downloads a policy object containing a reference to code governing verification of data. The machine uses the reference to obtain the code and locally verify data or other information. As particular rules for the data change, the policy object may be updated to provide a reference to the code for the new rules when it is downloaded.