Abstract: Techniques are disclosed for generating a topology of components based on a set of components provided by a user. The system identifies, for each particular component of the first set of components, one or more characteristics. The characteristics may include at least one of: a rule associated with the particular component, a requirement associated with the particular component, a data input type corresponding to the particular component, and data output type corresponding to the particular component. Based on the characteristics, the system determines that an additional component not included in the first set of components is required for connecting the first set of components. The system selects the additional component and determines a topology of components that includes the first set of components and the additional component. The system also determines a dataflow between components in the topology of components.

Abstract: A system for generating a topology of components based on a set of components provided by a user. The system identifies, for each particular component of the first set of components, one or more characteristics. The characteristics may include at least one of: a rule associated with the particular component, a requirement associated with the particular component, a data input type corresponding to the particular component, and data output type corresponding to the particular component. Based on the characteristics, the system determines that an additional component not included in the first set of components is required for connecting the first set of components. The system selects the additional component and determines a topology of components that includes the first set of components and the additional component. The system also determines a dataflow between components in the topology of components.

Abstract: Techniques are disclosed for generating a topology of components based on a set of components provided by a user. The system identifies, for each particular component of the first set of components, one or more characteristics. The characteristics may include at least one of: a rule associated with the particular component, a requirement associated with the particular component, a data input type corresponding to the particular component, and data output type corresponding to the particular component. Based on the characteristics, the system determines that an additional component not included in the first set of components is required for connecting the first set of components. The system selects the additional component and determines a topology of components that includes the first set of components and the additional component. The system also determines a dataflow between components in the topology of components.

Abstract: Techniques for efficiently maintaining consistency of data items across storage partitions are disclosed using a hierarchical multi-level hash tree. Copies of a data item may be associated with corresponding attributes that are used to generate hash values for the data item. Hash values of the attributes may then be used to label nodes in a multi-level hash tree. Differences between the replicated copies of a data item may be quickly identified by comparing hash values associated with successively lower peer nodes in corresponding hash trees. Once identified, systems may update versions of a data item that are no longer current.

Abstract: Techniques for partitioning data from a data stream into batches and inferring schema for individual batches based on the field values of each batch are disclosed. The system may infer different schemas corresponding to different batches of data records even though the batches are received from a common data stream or a common data source. The system may infer a schema by determining whether a field contains single values or multiple values. Then the system determines the field type(s) associated with the values. These determinations are then stored in a dictionary generated for each batch.

Abstract: Techniques for efficiently maintaining consistency of data items across storage partitions are disclosed using a hierarchical multi-level hash tree. Copies of a data item may be associated with corresponding attributes that are used to generate hash values for the data item. Hash values of the attributes may then be used to label nodes in a multi-level hash tree. Differences between the replicated copies of a data item may be quickly identified by comparing hash values associated with successively lower peer nodes in corresponding hash trees. Once identified, systems may update versions of a data item that are no longer current.

Abstract: Techniques for partitioning data from a data stream into batches and inferring schema for individual batches based on the field values of each batch are disclosed. The system may infer different schemas corresponding to different batches of data records even though the batches are received from a common data stream or a common data source. The system may infer a schema by determining whether a field contains single values or multiple values. Then the system determines the field type(s) associated with the values. These determinations are then stored in a dictionary generated for each batch.