Abstract: A system standardizes values that occur in large datasets before the dataset is analyzed. The system identifies values in a dataset that are similar to each other and associates those values with each other to form groups. The system determines a canonical value for each group of associated values. Within each group, the system replaces values that have been associated with each other with the canonical value for the group. As a result, the dataset is transformed into a dataset that has standardized values, and the standardized dataset is provided as input for analysis by a data analysis system. By standardizing the dataset in this manner, the data analysis system can process a larger portion of the dataset.

Abstract: A data preprocessing system builds transformation scripts for preprocessing datasets for processing by a data analysis system. The data preprocessing system presents various representations of data of a dataset including visual representations, textual representations, or structural representations. The data preprocessing system receives selections of attributes or values based on these representations. The data preprocessing system generates recommendations of transformations based on the attributes or values selected. The data preprocessing system builds a transformation script based on the recommendations of the transformations. The transformation script can be used for preprocessing the dataset for analysis by a data analysis system.

Abstract: A data preprocessing system builds transformation scripts for preprocessing datasets for processing by a data analysis system. The data preprocessing system presents various representations of data of a dataset including visual representations, textual representations, or structural representations. The data preprocessing system receives selections of attributes or values based on these representations. The data preprocessing system generates recommendations of transformations based on the attributes or values selected. The data preprocessing system builds a transformation script based on the recommendations of the transformations. The transformation script can be used for preprocessing the dataset for analysis by a data analysis system.

Abstract: A system determines samples of datasets that are typically processed by big data analysis systems. The samples are for use for development and testing of transformations for preprocessing the datasets in preparation for analysis by big data systems. The system receives one or more transform operations input datasets for the transform operations. The system determines samples associated with the transform operations. According to a sampling strategy, the system determines samples that return at least a threshold number of records in the result set obtained by applying a transformation. According to another sampling strategy, the system receives criteria describing the result of the transform operations and determines sample sets that generate result sets satisfying the criteria as a result of applying the transform operations.

Abstract: A system determines samples of datasets that are typically processed by big data analysis systems. The samples are for use for development and testing of transformations for preprocessing the datasets in preparation for analysis by big data systems. The system receives one or more transform operations input datasets for the transform operations. The system determines samples associated with the transform operations. According to a sampling strategy, the system determines samples that return at least a threshold number of records in the result set obtained by applying a transformation. According to another sampling strategy, the system receives criteria describing the result of the transform operations and determines sample sets that generate result sets satisfying the criteria as a result of applying the transform operations.

Abstract: A system provides data profile information describing attributes of a dataset. The system determines relative frequency of occurrences of attribute values with respect to a set of bins from a histogram of another attribute. The system presents a user interface that presents statistical information describing attributes of a dataset based on the relative frequency of occurrences of attribute values. The system generates a transformation script based on the user interactions for transforming records of the dataset. The transformation script is configured to preprocess data of the dataset for further analysis.

Abstract: A system and method identifies rows in a file as uniform rows or outlier rows based on statistics from the file, and displays a sampling of uniform and outlier rows.

Abstract: A system and method parses one or more fields from a file by receiving example locations of the field in the file, fashioning rules that describe the field from the locations, and then scoring the rules against some or all of the file.