Abstract: Scale-out network attached storage (NAS) file systems can employ an Ingest, Transform, Store (ITS) framework for data processing. In one aspect, the ITS-NAS file systems comprise NAS nodes and high performance computing (HPC) nodes that operate under a common operating system and that are coupled to each other via a common high-bandwidth, low-latency private network infrastructure. The NAS nodes can present data to the HPC nodes as well as dispatch the execution of transform services to the HPC nodes. The ITS-NAS file systems enable massive parallelization of operations on files, for example, complex distributed operations on large files and/or simple parallel operations on large collections of small files, all within the same hardware and software architecture.

Abstract: Described herein are technologies pertaining to automatically summarizing contents of a dataset and visualizing a summary of the dataset together with summaries of other datasets. A schema that defines the structure and content of a dataset is received, and pre-processing is undertaken on the schema to generate an enriched schema. Portions of the enriched schema are selected to generate a semantic summary of the schema, which is included with at least one exemplary entry of the dataset to generate a summary of the dataset.

Abstract: A hot folder mechanism is employed to provide a truly integrated architecture for easy-to-use, easy-to-deploy scale-out computation and scale-out storage. A folder of an Ingest, Transform, Store (ITS)-Network attached storage (NAS) system can be configured as “hot.” The configured hot folder can then detect changes on its content, analyze such content, perform transform services on the content, and output the computation results as files on other specified output folders. In one aspect, file system nodes of the ITS-NAS can present the content to high performance computing (HPC) compute nodes of the ITS-NAS as well as to dispatch the execution of transform services to the HPC compute nodes.

Abstract: Described herein are technologies pertaining to automatically summarizing contents of a dataset and visualizing a summary of the dataset together with summaries of other datasets. A schema that defines the structure and content of a dataset is received, and pre-processing is undertaken on the schema to generate an enriched schema. Portions of the enriched schema are selected to generate a semantic summary of the schema, which is included with at least one exemplary entry of the dataset to generate a summary of the dataset.