Abstract: Disclosed are systems, methods, and computer-readable storage media for infrastructure aware adaptive resource allocation. A resource manager can receive a request to allocate a first container instance in a network. The resource manager can determine a set of candidate computing nodes in the network that are capable of hosting the first container instance and request, from an infrastructure monitor that received infrastructure data from the network, health metrics for the set of candidate computing nodes. The resource manager can select, based on the health metrics for the set of candidate computing nodes, an optimal computing node to host the first container instance, and allocate the first container instance on the optimal computing node.

Abstract: Disclosed are systems, methods, and computer-readable storage media for infrastructure aware adaptive resource allocation. A resource manager can receive a request to allocate a first container instance in a network. The resource manager can determine a set of candidate computing nodes in the network that are capable of hosting the first container instance and request, from an infrastructure monitor that received infrastructure data from the network, health metrics for the set of candidate computing nodes. The resource manager can select, based on the health metrics for the set of candidate computing nodes, an optimal computing node to host the first container instance, and allocate the first container instance on the optimal computing node.

Abstract: Disclosed are systems, methods, and computer-readable storage media for infrastructure aware adaptive resource allocation. A resource manager can receive a request to allocate a first container instance in a network. The resource manager can determine a set of candidate computing nodes in the network that are capable of hosting the first container instance and request, from an infrastructure monitor that received infrastructure data from the network, health metrics for the set of candidate computing nodes. The resource manager can select, based on the health metrics for the set of candidate computing nodes, an optimal computing node to host the first container instance, and allocate the first container instance on the optimal computing node.

Abstract: The subject technology provides a drillable time-series heat map, which combines information of separate network element (e.g., switch, router, server or storage) and relates them together through impact zones to correlate network wide events and the potential impact on the other units in the network. The subject technology also brings together the network and its components, the distributed application(s) and a heat map controller to proactively communicate with one another to disseminate information such as failures, timeouts, new jobs, etc. For an individual job (e.g., for a distributed application), the subject technology may monitor consumption of resources during different phases of execution to provide individual job profile data that could be presented as a drillable heat map. The heat map, in this regard, nay include resource utilization heat metrics of resources such as CPU, Input/Output (I/O), memory, etc., in the heat map or graphs and presented along with network activity.

Abstract: Disclosed are systems, methods, and computer-readable storage media for infrastructure aware adaptive resource allocation. A resource manager can receive a request to allocate a first container instance in a network. The resource manager can determine a set of candidate computing nodes in the network that are capable of hosting the first container instance and request, from an infrastructure monitor that received infrastructure data from the network, health metrics for the set of candidate computing nodes. The resource manager can select, based on the health metrics for the set of candidate computing nodes, an optimal computing node to host the first container instance, and allocate the first container instance on the optimal computing node.

Abstract: Disclosed are systems, methods, and computer-readable storage media for infrastructure aware query optimization. A query optimizer can receive a first query for a first set of requested data and determine a set of computing nodes in a network that can provide the first set of requested data. The query optimizer can then request, from an infra monitor, health scores for the set of computing nodes that can provide the first set of requested data. The infra monitor can receive infrastructure data from the network. The query optimizer can select, based on the health scores, a subset of computing nodes from the set of computing nodes to service the first query, and request the first set of requested data from the subset of computing nodes.

Abstract: Systems, methods, and computer program products to perform an operation comprising receiving a priority of a distributed computing job, an intermediate traffic type of the distributed computing job, and a set of candidate compute nodes available to process the distributed computing job, the candidate compute nodes each available to process at least one input split of the distributed computing job, and selecting a mapper node from the candidate compute nodes, for one of the input splits, wherein the mapper node is selected based on the priority and the intermediate traffic type of the distributed computing job, wherein the mapper compute node is further selected upon determining that the mapper node is not affected by an error, and a resource utilization score for the mapper node does not exceed a utilization threshold.

Abstract: In one embodiment, data indicative of the size of an intermediate data set generated by a first resource device is received at a computing device. The intermediate data set is associated with a virtual machine to process the intermediate data set. A virtual machine configuration is determined based on the size of the intermediate data set. A second resource device is selected to execute the virtual machine based on the virtual machine configuration and on an available bandwidth between the first and second resource devices. The virtual machine is then assigned to the second resource device to process the intermediate data set.

Abstract: In one embodiment, a device receives information regarding a data set to be processed by a map-reduce process. The device generates a set of virtual clusters for the map-reduce process based on network bandwidths between nodes of the virtual clusters, each node of the virtual cluster corresponding to a resource device, and associates the data set with a map-reduce process task. The device then schedules the execution of the task by a node of the virtual clusters based on the network bandwidth between the node and a source node on which the data set resides.

Abstract: Systems, methods, and computer program products to perform an operation comprising receiving a priority of a distributed computing job, an intermediate traffic type of the distributed computing job, and a set of candidate compute nodes available to process the distributed computing job, the candidate compute nodes each available to process at least one input split of the distributed computing job, and selecting a mapper node from the candidate compute nodes, for one of the input splits, wherein the mapper node is selected based on the priority and the intermediate traffic type of the distributed computing job, wherein the mapper compute node is further selected upon determining that the mapper node is not affected by an error, and a resource utilization score for the mapper node does not exceed a utilization threshold.

Abstract: The subject technology provides a drillable time-series heat map, which combines information of separate network element (e.g., switch, router, server or storage) and relates them together through impact zones to correlate network wide events and the potential impact on the other units in the network. The subject technology also brings together the network and its components, the distributed application(s) and a heat map controller to proactively communicate with one another to disseminate information such as failures, timeouts, new jobs, etc. For an individual job (e.g., for a distributed application), the subject technology may monitor consumption of resources during different phases of execution to provide individual job profile data that could be presented as a drillable heat map. The heat map, in this regard, nay include resource utilization heat metrics of resources such as CPU, Input/Output (I/O), memory, etc., in the heat map or graphs and presented along with network activity.

Abstract: The subject technology provides a single drillable time-series heat map, which combines information of separate network element (e.g., switch, router, server or storage) and relates them together through impact zones to correlate network wide events and the potential impact on the other units in the network. The subject technology also brings together the network and its components (storage, ToR switches, servers, switches, etc.), the distributed application(s) and a heat map controller to proactively communicate with one another to quickly disseminate information such as failures, timeouts, new jobs, etc. Such communication ensures a more predictive picture of the network and enable better adaptive scheduling and routing, which may result in better utilization of resources. The subject technology uses impact zones to make better decisions to place data in the network, and measures network utilization through “Planned Metrics” to provide more realistic usage of network.

Abstract: In one embodiment, data indicative of the size of an intermediate data set generated by a first resource device is received at a computing device. The intermediate data set is associated with a virtual machine to process the intermediate data set. A virtual machine configuration is determined based on the size of the intermediate data set. A second resource device is selected to execute the virtual machine based on the virtual machine configuration and on an available bandwidth between the first and second resource devices. The virtual machine is then assigned to the second resource device to process the intermediate data set.

Abstract: In one embodiment, a device receives information regarding a data set to be processed by a map-reduce process. The device generates a set of virtual clusters for the map-reduce process based on network bandwidths between nodes of the virtual clusters, each node of the virtual cluster corresponding to a resource device, and associates the data set with a map-reduce process task. The device then schedules the execution of the task by a node of the virtual clusters based on the network bandwidth between the node and a source node on which the data set resides.