Abstract: A sparse dataset structure is created by creating column vectors for one or more columns in a dataset that have at least one significant value. Each column vector includes data values for columns of the dataset. Each column vector that is a sparse column vector includes a look-up index array and a value array. Entries in the look-up index array represent columns. The value array includes values for a row in a column. Each entry in the value array points to a row entry in the look-up index array. A side structure includes a row index and a column index. The row index includes a location for an entry for each row where entries point to a location in the column index that identifies a column that has a first significant entry for a row. Alternatively a sparse dataset could be constructed with sparse rows.

Abstract: Embodiments are directed to using object serialization to communicate references to shim objects and to managing memory on worker processes of a distributed software application. In one scenario, a computer system instantiates shim objects on one or more ranks of a distributed application. The shim objects are configured to store a reference to an instance of a native partitioned array, where the reference includes a unique identifier for the native partitioned array instance. The computer system then serializes the shim objects for communication of the stored references from the master rank of the distributed application to various other worker ranks of the distributed application. Then, upon serializing the shim objects, the computer system communicates the shim object's stored references to the other worker ranks of the distributed application.

Abstract: A system and method for generating real-time alert notifications includes a database for receiving in real-time at least one event, a processing engine for analyzing the at least one event with respect to a plurality of stored events, the processing engine also for determining whether the at least one event meets a defined condition, if the at least one event meets the defined condition, determining a prescriptive action and forwarding the prescriptive action to a user.

Abstract: Embodiments are directed to establishing registration objects for distributed processes, to managing memory on worker processes of a distributed software application and to using object serialization to communicate references to shim objects. In one scenario, a computer system accesses distributed process instances in a distributed runtime and creates a registration object for each of the process instances in the distributed runtime. The registration object includes a key value pair, where the key includes a unique identifier (ID) that identifies a distributed array instance associated with the distributed process, and the value includes a reference to a local portion of the distributed array instance. The computer system then maintains a mapping between the unique ID and the distributed array instance using the registration object. As such, the key value refers to the local portion of the same distributed array instance on each distributed process of the distributed runtime.

Abstract: Embodiments are directed to providing a generic software function for a plurality of data types in a distributed environment and to implementing type checking at compile time and at runtime. In one scenario, a computer system receives a portion of software code for a distributed software application, where the software code is written as a generic function in a computer programming language that supports reflection. The computer system presents a function signature at runtime to a just-in-time (JIT) compiler. The function signature describes which of multiple different input and output data types the generic function is configured to work with. The computer system also implements reflection at runtime to resolve each generic function to a function pointer after compilation by the JIT.

Abstract: Computerized singular value decomposition of an input complex matrix. A real-value matrix representation of the input complex matrix is provided to a singular value decomposition module, which correctly obtains a singular value representation of the real-value matrix representation. However, the result is not provided in a form for convenient conversion back into a valid singular value decomposition solution for the original input complex matrix, as the upper left half and lower right half of the diagonal of the diagonal matrix are not identical. A correction module corrects by formulating a corrected diagonal matrix that represents the value of the diagonal of the first diagonal matrix, but shuffled so that the upper left half of the diagonal of the second diagonal matrix is the same as the lower right half of the diagonal of the second diagonal matrix. Corrected unitary matrices may also be formed.

Abstract: A serial testing infrastructure includes the capability to execute a distributed test on multiple virtual processors. A test executable may be stored in a library and the test description, including the name of the test, the test library, and other test characteristics, may be stored in a separate test data file. The serial testing infrastructure initiates multiple distributed test executors that each launch an instance of the distributed test as a process that runs concurrently with other instances of the distributed test. Each distributed test executor monitors execution of it corresponding process until completion or timeout.

Abstract: Embodiments are directed to reshaping a partitioned data array. In an embodiment, a computer system identifies a block length parameter that describes the number of data blocks in the range-partitioned flattened representation of the array that appear consecutively in each locale. The computer system then identifies a stride parameter that describes the amount of separation between data blocks in the range-partitioned flattened representation of the array that appear consecutively in a plurality of locales. Based on the identified block length parameter and the stride parameter, the computer system determines which of the data blocks on the plurality of locales are to be sent to other locales to produce a local version of the reshaped array. The computer system then receives data blocks from the different locales in the distributed system and reconstructs the array based on the received blocks to create a local version of the reshaped array.

Abstract: Embodiments are directed to handling errors in a distributed software application. In one scenario, a computer system instantiates a distributed application which is configured for execution on multiple different computer systems. The computer system determines that an exception has occurred during the execution of the distributed application on at least one of the computer systems and translates the exception that occurred during distributed execution in a distributed environment into a serial exception that appears to have occurred during serial execution in a serial execution environment. The computer system then presents the translated serial exception to an application developer or other user.

Abstract: Embodiments are directed to establishing registration objects for distributed processes, to managing memory on worker processes of a distributed software application and to using object serialization to communicate references to shim objects. In one scenario, a computer system accesses distributed process instances in a distributed runtime and creates a registration object for each of the process instances in the distributed runtime. The registration object includes a key value pair, where the key includes a unique identifier (ID) that identifies a distributed array instance associated with the distributed process, and the value includes a reference to a local portion of the distributed array instance. The computer system then maintains a mapping between the unique ID and the distributed array instance using the registration object. As such, the key value refers to the local portion of the same distributed array instance on each distributed process of the distributed runtime.

Abstract: Embodiments are directed to establishing registration objects for distributed processes, to managing memory on worker processes of a distributed software application and to using object serialization to communicate references to shim objects. In one scenario, a computer system accesses distributed process instances in a distributed runtime and creates a registration object for each of the process instances in the distributed runtime. The registration object includes a key value pair, where the key includes a unique identifier (ID) that identifies a distributed array instance associated with the distributed process, and the value includes a reference to a local portion of the distributed array instance. The computer system then maintains a mapping between the unique ID and the distributed array instance using the registration object. As such, the key value refers to the local portion of the same distributed array instance on each distributed process of the distributed runtime.

Abstract: In an embodiment, a computer system accesses various different data entries in dense data array, where at least one of those data entries in the dense data array is invalid. The computer system creates an associated sparse data array that includes multiple data entries with zero values as well as data entries with non-zero values. The non-zero data entries are configured to store location information and data values for each of the invalid data entries in the dense array. The zero-value data entries are inferred from the location information of the non-zero data entries. The computer system stores the location information and data values of the non-zero data entries in the sparse data array. Those data values stored in the sparse array are proportional to the number of invalid values in the dense array.

Abstract: Embodiments are directed to providing a generic software function for a plurality of data types in a distributed environment and to implementing type checking at compile time and at runtime. In one scenario, a computer system receives a portion of software code for a distributed software application, where the software code is written as a generic function in a computer programming language that supports reflection. The computer system presents a function signature at runtime to a just-in-time (JIT) compiler. The function signature describes which of multiple different input and output data types the generic function is configured to work with. The computer system also implements reflection at runtime to resolve each generic function to a function pointer after compilation by the JIT.

Abstract: Embodiments are directed to using object serialization to communicate references to shim objects and to managing memory on worker processes of a distributed software application. In one scenario, a computer system instantiates shim objects on one or more ranks of a distributed application. The shim objects are configured to store a reference to an instance of a native partitioned array, where the reference includes a unique identifier for the native partitioned array instance. The computer system then serializes the shim objects for communication of the stored references from the master rank of the distributed application to various other worker ranks of the distributed application. Then, upon serializing the shim objects, the computer system communicates the shim object's stored references to the other worker ranks of the distributed application.

Abstract: Embodiments are directed to handling errors in a distributed software application and to handling errors in a distributed software application. In one scenario, a computer system instantiates a distributed application which is configured for execution on multiple different computer systems. The computer system determines that an exception has occurred during the execution of the distributed application on at least one of the computer systems and translates the exception that occurred during distributed execution in a distributed environment into a serial exception that appears to have occurred during serial execution in a serial execution environment. The computer system then presents the translated serial exception to an application developer or other user.

Abstract: Described is a technology by which Map-Reduce runtimes and distributed array runtimes are interoperable. Map-Reduce chunks are processed into array data for processing in a distributed array runtime based upon merge information. A staging Map-Reduce job tags a chunk with tag information that indicates a relative position of the chunk in an array. A distributed array framework imports files produced via a Map-Reduce framework and provides an array to an application of the distributed array framework for processing. An export mechanism may output one or more Map-Reduce files from the distributed array framework.

Abstract: A serial testing infrastructure includes the capability to execute a distributed test on multiple virtual processors. A test executable may be stored in a library and the test description, including the name of the test, the test library, and other test characteristics, may be stored in a separate test data file. The serial testing infrastructure initiates multiple distributed test executors that each launch an instance of the distributed test as a process that runs concurrently with other instances of the distributed test. Each distributed test executor monitors execution of it corresponding process until completion or timeout.

Abstract: In an embodiment, a computer system accesses various different data entries in dense data array, where at least one of those data entries in the dense data array is invalid. The computer system creates an associated sparse data array that includes multiple data entries with zero values as well as data entries with non-zero values. The non-zero data entries are configured to store location information and data values for each of the invalid data entries in the dense array. The zero-value data entries are inferred from the location information of the non-zero data entries. The computer system stores the location information and data values of the non-zero data entries in the sparse data array. Those data values stored in the sparse array are proportional to the number of invalid values in the dense array.

Abstract: Embodiments are directed to reshaping a partitioned data array. In an embodiment, a computer system identifies a block length parameter that describes the number of data blocks in the range-partitioned flattened representation of the array that appear consecutively in each locale. The computer system then identifies a stride parameter that describes the amount of separation between data blocks in the range-partitioned flattened representation of the array that appear consecutively in a plurality of locales. Based on the identified block length parameter and the stride parameter, the computer system determines which of the data blocks on the plurality of locales are to be sent to other locales to produce a local version of the reshaped array. The computer system then receives data blocks from the different locales in the distributed system and reconstructs the array based on the received blocks to create a local version of the reshaped array.

Abstract: Systems and methods to process vehicle operation data are described. A data module associated with a vehicle can collect a set of metrics relating to the operation of the vehicle, as well as events related to an operator's interaction with the vehicle. The data module can correlate the set of metrics with the events to generate a correlated set of data. A user can request various contexts in which to view the data, such as via a vehicle context or an operator context. The data module can generate, using the correlated set of data, a data view according to the request. Further, the correlated set of data and the various contexts can be updated on a real-time basis.