Abstract: A system provides systems and methods for visualizing a computational process. A portion of attributes of a computational process may be displayed on a face of a three-dimensional object. In response to rotation of the object by a user, a different face of the object may be displayed and a different portion of the attributes displayed thereon. Each object may represent a status of a computational cluster, specifically a set of jobs executing on the cluster. A cluster may be selected and the jobs represented by means of graphical indicators arranges on a time axis and a performance axis. The size of a graphical indicator may indicate an amount of processing time consume by the corresponding job. A color of the graphical indicator may indicate a status. A portion of the graphical indicator may be visually distinguished from a remaining portion to indicate a completion percentage of the job.

Abstract: Systems and methods are disclosed for aggregating data capable of diagnosing unique datacenter issues. Traffic statistic collection may be moved from intermediate, datacenter nodes to end hosts providing reports for aggregation and correlation with events at an analytic controller, uncovering implications for such events. To track metrics and/or diagnose datacenter issues not addressed in traffic statistics, information locally available to the end hosts may be combined and/or correlated with traffic statistics. Examples may involve information about: virtual and physical computing resources; a sub-cluster; an application and/or process utilized by a datacenter task; a task/job type; an implementation phase; an initiating user; a task priority; link utilization and/or other traffic statistics relative to the foregoing.

Abstract: Systems and methods are disclosed for traffic engineering and traffic flow control in a datacenter. Flow agents may be provided at hosts that may be operable to control a rate at which a traffic flow is sent from one point to another within a datacenter to facilitate differing levels of Quality of Service (QoS). The differing levels of QoS may be implemented by differing sizes of bandwidth allocations assigned remotely by a traffic engineering controller. The traffic engineering controller may distribute the differing allocations of bandwidth to the flow agents corresponding to the traffic flows to keep the traffic flow rates within the bandwidth allocation. An augmenting function may also be applied at the traffic engineering controller to maximize bandwidth utilization by checking for unutilized bandwidth and reallocating such bandwidth in accordance with one or more QoS.

Abstract: Systems and methods are disclosed for generating traffic statistics for a datacenter. Distributed, parallel processing may be used to generate traffic statistics from data sets about traffic in a datacenter. To reduce data sets from which such statistics are derived to manageable sizes and relevant processing times for distributed, parallel processing, traffic agents may be provided at end hosts in the datacenter. The traffic agents may summarize data traffic over large numbers of packets in terms of the various sockets over which they are transmitted. Reports on the various sockets may be sent by the various flow agents that monitor them to an analytic controller. The analytic controller may aggregate, provide flow-path information for, further reduce, and/or provision the resultant data for distributed parallel processing.

Abstract: Systems and methods are disclosed for generating traffic statistics for a datacenter. Distributed, parallel processing may be used to generate traffic statistics from data sets about traffic in a datacenter. To reduce data sets from which such statistics are derived to manageable sizes and relevant processing times for distributed, parallel processing, traffic agents may be provided at end hosts in the datacenter. The traffic agents may summarize data traffic over large numbers of packets in terms of the various sockets over which they are transmitted. Reports on the various sockets may be sent by the various flow agents that monitor them to an analytic controller. The analytic controller may aggregate, provide flow-path information for, further reduce, and/or provision the resultant data for distributed parallel processing.

Abstract: Systems and methods are disclosed for interacting with a file system. The file system is operable to reside in user space of a computing system. A module, also within user space, may provide a messaging service supporting requests from an application to the file system. By bypassing a System-Call Interface (SCI) of the computing system's kernel space, the module may support requests from the application to the file system with enhanced efficiency and/or customizable features not provided by the SCI. In some examples, the module may include a library in an independent layer within user space and below the application, allowing the library to provide an application-independent messaging service for different applications. Furthermore, in some examples, the module may include a segment of memory, within user space, shared between the application and the file system for passing data involved in requests and/or responses to and/or from the file system.

Abstract: Systems and methods are disclosed for classifying traffic flows. A traffic agent operable to collect classification information for one or more traffic flows may be deployed at an end host communicatively coupled to a data-center network. The traffic agent, deployed in a user space independent of the operating system, may compare the classification information for a given traffic flow to a metric value. Where the classification information achieves a certain threshold indicated by the metric value, the traffic agent may classify the traffic flow as an elephant flow. In some examples, a library may be included with the traffic agent that may include a modified send function. The modified send function may provide classification information to the traffic agent indexed to the traffic flow for which it is called so that the traffic agent may analyze the classification information to potentially provide a classification for the traffic flow.

Abstract: A system provides systems and methods for visualizing a computational process. A portion of attributes of a computational process may be displayed on a face of a three-dimensional object. In response to rotation of the object by a user, a different face of the object may be displayed and a different portion of the attributes displayed thereon. Each object may represent a status of a computational cluster, specifically a set of jobs executing on the cluster. A cluster may be selected and the jobs represented by means of graphical indicators arranges on a time axis and a performance axis. The size of a graphical indicator may indicate an amount of processing time consume by the corresponding job. A color of the graphical indicator may indicate a status. A portion of the graphical indicator may be visually distinguished from a remaining portion to indicate a completion percentage of the job.

Abstract: Systems and methods are disclosed for scheduling jobs processed in a distributed fashion to realize unharnessed efficiencies latent in the characteristics of the jobs and distributed processing technologies. A job store may be communicatively coupled to a job analyzer. The job analyzer may be operable to read information characterizing a job to identify multiple data blocks to be processed during the job at multiple locations in a cluster of nodes. A scheduling module may use information about the multiple data blocks, their storage locations, their status with respect to being provisioned to processing logic, data blocks to be processed by other jobs, data blocks in cache that have been pre-fetched for a prior job, quality-of-services parameters, and/or job characteristics, such as job size, to schedule the job in relation to other jobs.

Abstract: Systems and methods are disclosed for aggregating data capable of diagnosing unique datacenter issues. Traffic statistic collection may be moved from intermediate, datacenter nodes to end hosts providing reports for aggregation and correlation with events at an analytic controller, uncovering implications for such events. To track metrics and/or diagnose datacenter issues not addressed in traffic statistics, information locally available to the end hosts may be combined and/or correlated with traffic statistics. Examples may involve information about: virtual and physical computing resources; a sub-cluster; an application and/or process utilized by a datacenter task; a task/job type; an implementation phase; an initiating user; a task priority; link utilization and/or other traffic statistics relative to the foregoing.

Abstract: Systems and methods are disclosed for reducing latency in shuffle-phase operations employed during the MapReduce processing of data. One or more computing nodes in a cluster of computing nodes capable of implementing MapReduce processing may utilize memory servicing such node(s) to maintain a temporary file system. The temporary file system may provide file-system services for intermediate data generated by applying one or more map functions to the underlying input data to which the MapReduce processing is applied. Metadata devoted to this intermediated data may be provided to and/or maintained by the temporary file system. One or more shuffle operations may be facilitated by accessing file-system information in the temporary file system. In some examples, the intermediate data may be transferred from one or more buffers receiving the results of the map function(s) to a cache apportioned in the memory to avoid persistent storage of the intermediate data.

Abstract: Systems and methods are disclosed for reducing latency in processing data sets in a distributed fashion. A job-queue operable for queuing data-processing jobs run on multiple nodes in a cluster may be communicatively coupled to a job analyzer. The job analyzer may be operable to read the data-processing jobs and extract information characterizing those jobs in ways that facilitate identification of resources in the cluster serviceable to run the data-processing jobs and/or data to be processed during the running of those jobs. The job analyzer may also be coupled to a resource warmer operable to warm-up a portion of the cluster to be used to run a particular data-processing job prior to the running of the job. In some embodiments, mappers and/or reducers may be extracted from the jobs and converted into compute node identifiers and/or data units identifying blocks for processing, informing the warm-up operations of the resource warmer.

Abstract: In one exemplary aspect, a method of managing computer network traffic flow quality of service includes the step of configuring a configurable network device to provide a specified quality of service to a data packet with a specified quality of service configuration based on a quality of service classification port number in a data packet header of the data packet. At the source node, a data packet is generated. At the source node, replacing the destination port number in the data packet header with a quality of service classification port number. At the source node, the destination port number is included in an options field of the data packet's header. The data packet is communicated to the configurable network device. At the destination node, receiving the data packet, replacing the quality of service classification port number with the original destination port number, and forwarding the packet to a destination process.

Abstract: A system provides systems and methods for visualizing a computational process. A portion of attributes of a computational process may be displayed on a face of a three-dimensional object. In response to rotation of the object by a user, a different face of the object may be displayed and a different portion of the attributes displayed thereon. Each object may represent a status of a computational cluster, specifically a set of jobs executing on the cluster. A cluster may be selected and the jobs represented by means of graphical indicators arranges on a time axis and a performance axis. The size of a graphical indicator may indicate an amount of processing time consume by the corresponding job. A color of the graphical indicator may indicate a status. A portion of the graphical indicator may be visually distinguished from a remaining portion to indicate a completion percentage of the job.

Abstract: In one exemplary aspect, a method of managing computer network traffic flow quality of service includes the step of configuring a configurable network device to provide a specified quality of service to a data packet with a specified quality of service configuration based on a quality of service classification port number in a data packet header of the data packet. At the source node, a data packet is generated. At the source node, replacing the destination port number in the data packet header with a quality of service classification port number. At the source node, the destination port number is included in an options field of the data packet's header. The data packet is communicated to the configurable network device. At the destination node, receiving the data packet, replacing the quality of service classification port number with the original destination port number, and forwarding the packet to a destination process.

Abstract: In one exemplary aspect, a primary cache is maintained in a main memory of a computer system. The primary cache is populated with a set of data from a secondary data storage system. A secondary cache is maintained in another memory of the computer system. A subset of data is selected from the set of data in the primary cache. A trigger event is detected. The secondary cache is populated with the subset of data selected from the set of data in the primary cache. Optionally, a lifespan of each memory page in the primary cache can be estimated. Memory pages with lifespans within a specified lifespan range can be associated. A set of associated memory pages with lifespans within the specified lifespan range can be written to a block in the flash memory system. The main memory of the computer system can include a dynamic random-access memory (DRAM) memory system. The other memory of the computer system can include a flash memory system in a solid-state storage device.

Abstract: Systems and methods are disclosed for classifying traffic flows. A traffic agent operable to collect classification information for one or more traffic flows may be deployed at an end host communicatively coupled to a data-center network. The traffic agent, deployed in a user space independent of the operating system, may compare the classification information for a given traffic flow to a metric value. Where the classification information achieves a certain threshold indicated by the metric value, the traffic agent may classify the traffic flow as an elephant flow. In some examples, a library may be included with the traffic agent that may include a modified send function. The modified send function may provide classification information to the traffic agent indexed to the traffic flow for which it is called so that the traffic agent may analyze the classification information to potentially provide a classification for the traffic flow.

Abstract: Systems and methods are disclosed for traffic engineering and traffic flow control in a datacenter. Flow agents may be provided at hosts that may be operable to control a rate at which a traffic flow is sent from one point to another within a datacenter to facilitate differing levels of Quality of Service (QoS). The differing levels of QoS may be implemented by differing sizes of bandwidth allocations assigned remotely by a traffic engineering controller. The traffic engineering controller may distribute the differing allocations of bandwidth to the flow agents corresponding to the traffic flows to keep the traffic flow rates within the bandwidth allocation. An augmenting function may also be applied at the traffic engineering controller to maximize bandwidth utilization by checking for unutilized bandwidth and reallocating such bandwidth in accordance with one or more QoS.

Abstract: Systems and methods are disclosed for interacting with a file system. The file system is operable to reside in user space of a computing system. A module, also within user space, may provide a messaging service supporting requests from an application to the file system. By bypassing a System-Call Interface (SCI) of the computing system's kernel space, the module may support requests from the application to the file system with enhanced efficiency and/or customizable features not provided by the SCI. In some examples, the module may include a library in an independent layer within user space and below the application, allowing the library to provide an application-independent messaging service for different applications. Furthermore, in some examples, the module may include a segment of memory, within user space, shared between the application and the file system for passing data involved in requests and/or responses to and/or from the file system.

Abstract: Systems and methods are disclosed for generating traffic statistics for a datacenter. Distributed, parallel processing may be used to generate traffic statistics from data sets about traffic in a datacenter. To reduce data sets from which such statistics are derived to manageable sizes and relevant processing times for distributed, parallel processing, traffic agents may be provided at end hosts in the datacenter. The traffic agents may summarize data traffic over large numbers of packets in terms of the various sockets over which they are transmitted. Reports on the various sockets may be sent by the various flow agents that monitor them to an analytic controller. The analytic controller may aggregate, provide flow-path information for, further reduce, and/or provision the resultant data for distributed parallel processing.