Abstract: This disclosure generally relates to data delivery in distributed applications. One example method includes identifying a data source associated with a shuffle operation, the data source configured provide data from a data set associated with the shuffle operation; identifying a data sink associated with the shuffle operation, the data sink configured to receive data provided by the data source; associating a shuffler component with the shuffle operation, the shuffler component configured to receive data from the data source and provide the data to the data sink; receiving, by the shuffler component, a first data portion from the data source; providing, by the shuffler component, the first data portion to the data sink; receiving, by the shuffler component, a second data portion from the data source, the second data portion being received from the data source prior to or concurrent with providing the first data portion to the data sink.

Abstract: This disclosure generally relates to data delivery in distributed applications. One example method includes identifying a data source associated with a shuffle operation, the data source configured provide data from a data set associated with the shuffle operation; identifying a data sink associated with the shuffle operation, the data sink configured to receive data provided by the data source; associating a shuffler component with the shuffle operation, the shuffler component configured to receive data from the data source and provide the data to the data sink; receiving, by the shuffler component, a first data portion from the data source; providing, by the shuffler component, the first data portion to the data sink; receiving, by the shuffler component, a second data portion from the data source, the second data portion being received from the data source prior to or concurrent with providing the first data portion to the data sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: Methods, systems, and apparatus for allocating, by a source of one or more sources, a segment of a data file of a transient memory for exclusive access by the source, the transient memory being a distributed in-memory file system that supports remote direct memory access; writing, by the source, data from an initial partition to one or more blocks within the allocated segment of the data file, wherein a portion of the initial partition is written to a first block of the one or more blocks; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by one or more sinks; and reading by a particular sink of the one or more sinks, a particular block of the published segment of the data file of the transient memory, wherein the particular block is associated with the particular sink.

Abstract: Embodiments are directed to securing data in the cloud, securely encrypting data that is to be stored in the cloud and to securely decrypting data accessed from the cloud. In one scenario, an instantiated trust service receives information indicating that a trust server is to be instantiated. The trust service instantiates the trust server, which is configured to store key references and encrypted keys. The trust service receives the public key portion of a digital certificate for each publisher and subscriber that is to have access to various specified portions of encrypted data. A data access policy is then defined that specifies which encrypted data portions can be accessed by which subscribers.

Abstract: Embodiments are directed to mapping encryption policies to data stored in a database using a policy identifier, and to accessing data stored in a database using a policy identifier. In one scenario, a computer system receives an indication that identifies which type of encryption is to be applied when encrypting a specified portion of data stored in a database. The database has a database schema identified by a database schema identifier, where the database schema defines relationships for data stored in the database. The computer system then accesses a namespace that identifies a set of databases in which the specified portion of data is accessed in the same manner. The computer system also generates a policy identifier, which contains information including the namespace and the database schema identifier.

Abstract: Embodiments are directed to mapping encryption policies to data stored in a database using a policy identifier, and to accessing data stored in a database using a policy identifier. In one scenario, a computer system receives an indication that identifies which type of encryption is to be applied when encrypting a specified portion of data stored in a database. The database has a database schema identified by a database schema identifier, where the database schema defines relationships for data stored in the database. The computer system then accesses a namespace that identifies a set of databases in which the specified portion of data is accessed in the same manner. The computer system also generates a policy identifier, which contains information including the namespace and the database schema identifier.

Abstract: Embodiments are directed to mapping encryption policies to user data stored in a database using a policy column uniform resource identifier (URI). In one scenario, a computer system receives the following: a database schema name that identifies the name of a specified schema within a relational database in which user data is stored, a table name that identifies a specified table within the relational database, a column name that identifies a specified column in the specified table and a namespace identifier that identifies a set of relational databases. The computer system also receives an indication that identifies which type of encryption is to be applied when encrypting the column of data specified by the column name. The computer system then generates a policy column URI that includes a hierarchical string comprising the namespace identifier, the database schema name, the table name and the column name.

Abstract: Embodiments are directed to mapping encryption policies to user data stored in a database using a policy column uniform resource identifier (URI). In one scenario, a computer system receives the following: a database schema name that identifies the name of a specified schema within a relational database in which user data is stored, a table name that identifies a specified table within the relational database, a column name that identifies a specified column in the specified table and a namespace identifier that identifies a set of relational databases. The computer system also receives an indication that identifies which type of encryption is to be applied when encrypting the column of data specified by the column name. The computer system then generates a policy column URI that includes a hierarchical string comprising the namespace identifier, the database schema name, the table name and the column name.

Abstract: Embodiments are directed to securing data in the cloud, securely encrypting data that is to be stored in the cloud and to securely decrypting data accessed from the cloud. In one scenario, an instantiated trust service receives information indicating that a trust server is to be instantiated. The trust service instantiates the trust server, which is configured to store key references and encrypted keys. The trust service receives the public key portion of a digital certificate for each publisher and subscriber that is to have access to various specified portions of encrypted data. A data access policy is then defined that specifies which encrypted data portions can be accessed by which subscribers.

Abstract: Various technologies and techniques are disclosed that automatically optimize package execution performance. A profiling phase executes each task in a control flow package and measures performance metrics, such as task execution length, task memory usage, task correlation to CPU versus input/output operations, network bandwidth, and running applications. An optimization phase optimizes subsequent executions of the package by using the performance metrics to make appropriate adjustments to the package. The profiling phase and optimizing phase are repeated over a period of time as appropriate to improve the package's performance without requiring a user to make adjustments to the tasks in the package.

Abstract: Various technologies and techniques are disclosed that automatically optimize package execution performance. A profiling phase executes each task in a control flow package and measures performance metrics, such as task execution length, task memory usage, task correlation to CPU versus input/output operations, network bandwidth, and running applications. An optimization phase optimizes subsequent executions of the package by using the performance metrics to make appropriate adjustments to the package. The profiling phase and optimizing phase are repeated over a period of time as appropriate to improve the package's performance without requiring a user to make adjustments to the tasks in the package.