Abstract: A system and method is disclosed for automated cross-node communication in a distributed directed acyclic graph. The method can include identifying a directed acyclic graph (“DAG”) overlaying a plurality of nodes and identifying the nodes underlying the DAG. A subordinate DAG can be generated in an entry vertex of the DAG. The subordinate DAG can include a vertex for each of the nodes underlying the DAG. Data and metadata can be received at the entry vertex. The data can be delivered to a next vertex in the DAG, and the metadata can be communicated to nodes underlying the DAG via the subordinate DAG.

Abstract: A method is disclosed for training and utilizing massively parallel neural networks. A distributed computing system may be configured to perform various operations. The distributed computing system may divide a directed acyclic graph (“DAG”) that comprises a plurality of vertices linked in pairwise relationships via a plurality of edges among a plurality of nodes. Each node may comprise a computing device. The distributed computing system may provide a map of the DAG that described a flow of data through the vertices to each of the vertices of the DAG. The distributed computing system may perform a topological sort of the vertices of the DAG and may traverse the DAG.

Abstract: A system and method is disclosed for automated cross-node communication in a distributed directed acyclic graph. The method can include identifying a directed acyclic graph (“DAG”) overlaying a plurality of nodes and identifying the nodes underlying the DAG. A subordinate DAG can be generated in an entry vertex of the DAG. The subordinate DAG can include a vertex for each of the nodes underlying the DAG. Data and metadata can be received at the entry vertex. The data can be delivered to a next vertex in the DAG, and the metadata can be communicated to nodes underlying the DAG via the subordinate DAG.

Abstract: A method is disclosed for synchronization of clone directed acyclic graphs. The method can include identifying a directed acyclic graph (“DAG”) including a plurality of vertices linked in pairwise relationships via a plurality of edges. At least one clone DAG can be created, which at least one clone DAG can be identical to at least a portion of the DAG. For each of the vertices of the DAG, a corresponding clone vertex from the clone vertices of the at least one clone DAG can be identified. Aggregate gradient data can be calculated based on gradient data from each of the clone vertices and its corresponding vertex in the DAG, and at least one weight of the DAG and of the at least one clone DAG can be updated based on the aggregate gradient data.