Abstract: Embodiments for locating, identifying and categorizing data-assets through advanced machine learning algorithms implemented by profiler components across Hadoop and Hadoop Compatible File Systems, databases and in-memory objects automatically and periodically to provide a visual representation of the category of data infrastructure distributed across data-centers and multiple clusters, for the purposes of enriching data quality, enabling data discovery and improving outcomes from downstream systems.

Abstract: Embodiments for deploying services to multiple Hadoop clusters and providing user access to these services in a secure manner. A process allows authorized users to select a service, validate its entitlement to the organization and then install distributed components of the service onto multiple hosts on different Hadoop clusters. In order to enable this deployment and secure access of this service, an identity federation mechanism is used to ensure the user identity of the system is propagated to distributed clusters in a secure fashion thereby ensuring authorized access to clusters or services is provided in a seamless fashion.

Abstract: Embodiments for managing data in a large-scale computer network coupling one or more client computer to a server and having multiple clusters having respective applications, by: encoding web-based data of services to a web browser of a client computer; forwarding requests from the web browser to a cluster access subsystem that wraps the requests in a security protocol interaction that preserves an identity of a user of the client computer; deploying to deploy the applications using an application descriptor for each application of the deployed applications; and storing data about how each application can be accessed through service endpoints including a network address and port identifier for access by queries by any other component, application, or service in the network.