Abstract: With the availability of huge amount of data, it has becoming difficult to identify and manage duplicate data, especially when the data is in a plurality of columns. A method and system for identifying duplicate columns using statistical, semantics and machine learning techniques have been provided. The system provides a design framework to compare huge datasets at column level and identify potential duplicate columns, not based on the column title, but based on all of its values. The disclosure has ability to compare values in multiple columns and identify potential duplicate columns wherein comparison of values is not only for the exact match, but for semantic match, smart match, fuzzy match, and match after UOM conversion etc. using Statistical, semantics and machine learning techniques.

Abstract: This disclosure relates generally to system and method for hierarchical category classification of products. Generally in supervised hierarchical classification, the hierarchy structure is predefined. However, majority of the current machine learning methods either expect the model to learn the hierarchy from the data or requires separate models trained at each level taking the prediction of previous level as an additional input, thereby increasing latency in achieving training accuracy and/or requiring an explicit maintenance module to orchestrate inference and retrain multiple models (corresponding to the number of levels in the hierarchy). The disclosed method and system allows the predefined knowledge about hierarchy drive the learning process of a single model, which predicts all levels of the hierarchy. The disclosed multi-layer network model arrives at a consensus based on prediction at each level, thereby increasing the accuracy of prediction and reducing the training time.

Abstract: With the availability of huge amount of data, it has becoming difficult to identify and manage duplicate data, especially when the data is in a plurality of columns. A method and system for identifying duplicate columns using statistical, semantics and machine learning techniques have been provided. The system provides a design framework to compare huge datasets at column level and identify potential duplicate columns, not based on the column title, but based on all of its values. The disclosure has ability to compare values in multiple columns and identify potential duplicate columns wherein comparison of values is not only for the exact match, but for semantic match, smart match, fuzzy match, and match after UOM conversion etc. using Statistical, semantics and machine learning techniques.

Abstract: This disclosure relates generally to system and method for hierarchical category classification of products. Generally in supervised hierarchical classification, the hierarchy structure is predefined. However, majority of the current machine learning methods either expect the model to learn the hierarchy from the data or requires separate models trained at each level taking the prediction of previous level as an additional input, thereby increasing latency in achieving training accuracy and/or requiring an explicit maintenance module to orchestrate inference and retrain multiple models (corresponding to the number of levels in the hierarchy). The disclosed method and system allows the predefined knowledge about hierarchy drive the learning process of a single model, which predicts all levels of the hierarchy. The disclosed multi-layer network model arrives at a consensus based on prediction at each level, thereby increasing the accuracy of prediction and reducing the training time.