Abstract: The disclosed embodiments provide a system for processing data. During operation, the system obtains a validation configuration containing declarative specifications of fields in a data set and validation rules to be applied to the data set, wherein the validation rules include a field in the data set, a type of validation to be applied to the field, and a parameter for managing a validation failure during evaluation of the validation rules with the data set. Next, the system automatically applies the validation rules to the data set within a workflow for generating the data set to produce validation results indicating passing or failing of the validation rules by the data set. The system then outputs the validation results for use in managing the data set.

Abstract: The disclosed embodiments provide a system for performing profile-driven data validation. During operation, the system obtains a validation configuration containing declarative specifications of fields in a data set and validation rules to be applied to the data set. Next, the system analyzes the data set based on the validation configuration to produce a set of metrics related to the data set and stores the metrics in a profile for the data set. The system also matches a metric in the profile to the type of validation associated with a validation rule in the validation configuration. Finally, the system applies the validation rule to a value of the metric in the profile to produce a validation result for the validation rule.

Abstract: A method for processing activity records. The method includes obtaining an activity record, and generating an anonymization dictionary. Generating the anonymization dictionary includes detecting, in the activity record, a set of target entities to be anonymized, making a determination that a resource is associated with a subset of the target entities of the set of target entities, and after making the determination, assigning an anonymized identity to the subset of target entities, and generating an anonymization identifier for each target entity in the subset of target entities to obtain a set of anonymization identifiers, each including the anonymized identity. The method further includes processing the activity record using the anonymization dictionary to obtain an anonymized activity record and storing the anonymized activity record.

Abstract: A method for processing activity records. The method includes obtaining an activity record, and generating an anonymization dictionary. Generating the anonymization dictionary includes detecting, in the activity record, a set of target entities to be anonymized, making a determination that a resource is associated with a subset of the target entities of the set of target entities, and after making the determination, assigning an anonymized identity to the subset of target entities, and generating an anonymization identifier for each target entity in the subset of target entities to obtain a set of anonymization identifiers, each including the anonymized identity. The method further includes processing the activity record using the anonymization dictionary to obtain an anonymized activity record and storing the anonymized activity record.

Abstract: A method for processing activity records. The method includes obtaining an activity record, and generating an anonymization dictionary. Generating the anonymization dictionary includes detecting, in the activity record, a set of target entities to be anonymized, making a determination that a resource is associated with a subset of the target entities of the set of target entities, and after making the determination, assigning an anonymized identity to the subset of target entities, and generating an anonymization identifier for each target entity in the subset of target entities to obtain a set of anonymization identifiers, each including the anonymized identity. The method further includes processing the activity record using the anonymization dictionary to obtain an anonymized activity record and storing the anonymized activity record.

Abstract: A method for processing activity records. The method includes obtaining an activity record, and generating an anonymization dictionary. Generating the anonymization dictionary includes detecting, in the activity record, a set of target entities to be anonymized, making a determination that a resource is associated with a subset of the target entities of the set of target entities, and after making the determination, assigning an anonymized identity to the subset of target entities, and generating an anonymization identifier for each target entity in the subset of target entities to obtain a set of anonymization identifiers, each including the anonymized identity. The method further includes processing the activity record using the anonymization dictionary to obtain an anonymized activity record and storing the anonymized activity record.

Abstract: A database management system determines, in a single pass over an unordered database, the quantile information. The system sequentially compares each tuple in the data set to a test value, and then selectively inserts the tuple in a test set having a cardinality less than the cardinality of the data set based upon the comparison. The system next uses the quantile information to estimate the number of tuples in the database which satisfy a user-defined predicate to generate an efficient query plan.

Abstract: A method of index scanning involves scanning one or more selected indexes and determining the number of data transfers required to traverse all or a portion of a selected index for a selected number of buffer pool sizes. The number of page transfers to scan a whole table of interest versus the number of page transfers to scan each relevant index in accordance with the buffer pool size is determined for a query. The number of page transfers required in proportion to the selectivity of starting and stopping conditions is determined and scaled down in proportion to the selectivity of any starting and stopping conditions present in the search criteria in the query. A suitable correction factor is applied to the number of transfers to account for few rows remaining to be transferred after applying the search criteria, the buffer pool being large, or a low degree of clustering for an index,, as well as accounting for any remaining index sargable search criteria.

Abstract: Described herein is a system and method for operating a data processing system having a processing unit and physical storage in which data is stored. The storage is connected to the processing unit for transferring pages of data to the processing unit. The storage contains at least one database table stored on a plurality of pages. Each data base table has a plurality of rows of data, and one or more indexes composed of a sequence of entries which reference the rows. The processing unit has a memory associated with it. A portion of the memory comprises a buffer pool for caching data pages from the table. The method operates by scanning one or more selected indexes in determining the relationship of the number of data transfers required to traverse all or a portion of each selected index for a selected range of buffer pool sizes. Then, a description of this relationship is stored.

Abstract: A database management system determines, in a single pass over an unordered database, the quantile information. The system sequentially compares each tuple in the data set to a test value, and then selectively inserts the tuple in a test set having a cardinality less than the cardinality of the data set based upon the comparison. The system next uses the quantile information to estimate the number of tuples in the database which satisfy a user-defined predicate to generate an efficient query plan.