Abstract: First and second big data tables may be aligned into compacted and reordered tables, respectively, such that each row of the compacted table and the corresponding similarly-indexed row of the reordered table have equal keys. Advantageously, the time complexity of alignment scales linearly with the number of rows of the data tables, and the compacted and reordered tables may be subsequently joined by parsing once through the tables. Alignment may be extended to three or more big data tables.

Abstract: A hierarchical cluster analyzer identifies clusters in a big data set by identifying topological structure without distance-based metrics. The hierarchical cluster analyzer stochastically partitions the big data set to create pseudo-partitions of the big data set. The stochastic partitioning may be implemented with a random forest classifier that uses ensemble techniques to reduce variance and prevent overfitting. The hierarchical cluster analyzer implements random intersection leaves (RIL), a data mining technique that grows an intersection tree by intersecting candidate sets generated from the pseudo-partitions. The hierarchical cluster analyzer updates an association matrix according to co-occurrences of data points within each leaf node of the intersection tree. These co-occurring data points exhibit a high degree of similarity, which is recorded in the association matrix. A hierarchy of clusters may then be formed by finding community structure in the association matrix.