Abstract: This disclosure relates generally to a system and method to optimize a plurality of validations in an ETL workflow. The ETL workflow implements all types of validations expecting that there is erroneous data and that the validation would fail. Therefore, the actual execution of validation is made conditional based on a switch that can be turned on (validation is performed) or off (validation is skipped). The output of each instance of execution is recorded continuously at a repository. The means of identifying a specific validation that is eligible for being turned off requires a thorough analysis of all validations performed so far in the history. The thorough analysis includes how many times a validation has been performed on a particular field and to calculate an exposure value which is compared with a threshold value of exposure and determine specific validation that is eligible to be turned off.

Abstract: De-normalized data structure files generation systems and methods are provided. The system obtains files from sources wherein each file include records, parses files to validate records and attributes in the records, identifies a set of similar files from the validated files, and append two or more files from the set of similar files to obtain one or more consolidated files. Each of the one or more consolidate files corresponds to a specific category. The system further a predefined logic validation on each of the one or more consolidated files to obtain a logic validated file for each of the one or more consolidated files. Each logic validated file obtained for the one or more consolidated files include validated records. The system further generates a de-normalized data structure file including de-normalized records by merging each of the logic validated files to be used for generating intelligence reports.

Abstract: This disclosure relates generally to a system and method to optimize a plurality of validations in an ETL workflow. The ETL workflow implements all types of validations expecting that there is erroneous data and that the validation would fail. Therefore, the actual execution of validation is made conditional based on a switch that can be turned on (validation is performed) or off (validation is skipped). The output of each instance of execution is recorded continuously at a repository. The means of identifying a specific validation that is eligible for being turned off requires a thorough analysis of all validations performed so far in the history. The thorough analysis includes how many times a validation has been performed on a particular field and to calculate an exposure value which is compared with a threshold value of exposure and determine specific validation that is eligible to be turned off.

Abstract: De-normalized data structure files generation systems and methods are provided. The system obtains files from sources wherein each file include records, parses files to validate records and attributes in the records, identifies a set of similar files from the validated files, and append two or more files from the set of similar files to obtain one or more consolidated files. Each of the one or more consolidate files corresponds to a specific category. The system further a predefined logic validation on each of the one or more consolidated files to obtain a logic validated file for each of the one or more consolidated files. Each logic validated file obtained for the one or more consolidated files include validated records. The system further generates a de-normalized data structure file including de-normalized records by merging each of the logic validated files to be used for generating intelligence reports.