Abstract: Techniques for analyzing performance and detecting anomalies for complex large-scale systems are provided herein. More specifically, the present disclosure provides the ability to identify events of interest in counter samples using markers and correlating metrics from different entities (switch, NIC, CPU, GPU, PCIe, memory). In contrast to previous visual representations, the correlated events of the current techniques enable an enhanced visual representation that focuses on the identified events and a timeline correlation of different metrics to understand job performance variations.

Abstract: A system receives, by a network device in a network fabric, a packet comprising at least one of a traffic-class value for a traffic-class type and an endpoint-class value for an endpoint-class type. The system extracts the traffic-class value and the endpoint-class value from the packet and determines a hierarchical class structure, which indicates: priorities associated with types of classes, comprising a first priority associated with the traffic-class type and a second priority associated with the endpoint-class type; and bandwidth allocation ratios for values in a respective class type, comprising first bandwidth allocation ratios for values of the traffic-class type and second bandwidth allocation ratios for values of the endpoint-class type. The system determines a bandwidth allocation for the packet based on the extracted values and the hierarchical class structure and forwards the packet based on the determined bandwidth allocation for the packet.

Abstract: A system determines paths through a hierarchical network from a source to a destination. The hierarchical network comprises layers, and each layer includes network devices. The system maps a path to a plurality of bits comprising one or more bits indicating a next-hop network device on the path. A network device in a first layer of the hierarchical network receives a packet indicating a traffic class and a destination address. If the traffic class corresponds to a first type, the system applies a deterministic forwarding algorithm by: identifying a first path mapped to a first plurality of bits, where the destination address includes the first plurality of bits; and forwarding, via the first path, the packet to a next-hop network device indicated by the first plurality of bits. If the traffic class corresponds to a second type, the system forwards the packet in accordance with an adaptive forwarding algorithm.

Abstract: One aspect of the instant application provides a system and method for extending a flow from a first network fabric into a second network fabric. During operation, the system may identify, at an egress edge node of the first network fabric, a flow to which a received packet belongs based on a first flow identifier associated with the packet. The egress edge node may forward the received packet to the second network fabric, where a respective node in the second network fabric may identify the flow based on a second flow identifier. The system may indicate to an upstream node of the forwarded packet exiting the first network fabric and keep the flow active within the first network fabric after all packets in the flow exit the first network fabric and until all packets arrive at a destination node.

Abstract: Systems and methods are provided for performing routing in a switch network or fabric. Switches can be configured in a hierarchical topology having a plurality of groups, where switches in a group are connected to one another, and groups are connected to other groups. Routing can be performed by maintaining per-group group load information. A packet can be routed between at least two groups using the per-group group load information to effect a set of routing decisions. The set of routing decisions can be biased towards or away one or more paths.

Abstract: A system receives, by a network device in a first network fabric, a to be forwarded flow over a link aggregation group (LAG) comprising a plurality of physical ports aggregated as a single logical port. The system determines loads associated with the LAG ports and selects a first LAG port based on a first load associated with the first LAG port. The system forwards the flow on a first path over the selected first LAG port. The system stores a state of the flow, wherein the flow is forwarded in a second network fabric. The system receives, from the first LAG port, a redirect acknowledgment (ACK) indicating that the flow is to be considered as a candidate flow to be rerouted. The system selects the flow from a plurality of candidate flows to be rerouted and forwards the selected flow on a second path over a second LAG port.

Abstract: A system generates, by a network device operating as an intermediate network device, a load metric for a respective flow of a first set of received flows. The system sends, to a first ingress network device, a redirect acknowledgment (ACK) including the load metric for the respective flow in response to the load metric being greater than a load value. The system forwards, by the network device operating as a second ingress network device, a second set of flows. The system receives, from a plurality of intermediate network devices, redirect ACKs corresponding to a plurality of flows of the second set of flows. A respective redirect ACK includes a load metric for a corresponding flow of the plurality of flows. The system selects, from the flows based on a set of rerouting conditions, a first flow to be rerouted. The system reroutes the first flow to a new path.

Abstract: A method, computer system, and computer-readable medium are provided which facilitate the use of link utilization as a load metric. The system monitors utilization of a plurality of links in a network. The system generates a link utilization load metric for a respective link of the plurality of links. The link utilization load metric for the respective link comprises an amount of data transmitted over the respective link in a most recent predetermined time period compared to a total available bandwidth of the respective link. The system receives data to be transmitted through the network. The system selects, from a set of valid paths, a path via which to forward the data based on a combination of the link utilization load metric for the respective link and a congestion metric associated with the respective link, causing a reduction in the number of flows to be rerouted.

Abstract: One aspect of the instant disclosure provides a method and system for identifying slow nodes among a plurality of nodes executing a distributed application. During operation, in response to receiving a trigger signal at a node, the system may monitor traffic to or from the node by measuring durations of one or more non-paused idle periods. In response to determining that a duration of a non-paused idle period falls within a predetermined idle-period duration range, the system may increment a corresponding counter. The system may generate a histogram for the node based on counter values corresponding to a plurality of idle-period duration ranges and identify one or more slow nodes based on histograms associated with the plurality of nodes.

Abstract: A system tracks, in a network device with multiple links, activity over a respective link for a predetermined amount of time. The activity comprises at least one of a number of idle periods or a duration of a respective idle period. The system divides a predetermined time interval into a number of bins. A bin is associated with a range of time. The system stores the tracked activity in data structure entries which each indicate the number of bins, a duration of time associated with a respective bin, and a count associated with the respective bin. The system indicates the tracked activity in the entry for the respective link by incrementing a count associated with a bin matching the duration of the respective idle period based on a first number of idle periods tracked for the matching duration. The system displays the stored tracked activity for the respective link.

Abstract: Systems and methods are provided for implementing encryption of data-in-motion and/or otherwise stored data using a key server and a secure enclave of a Network Interface Card (NIC). The NIC acts as a passthrough between the client device and the shared infrastructure of the supercomputer system to help ensure data security in a massively scaled and distributed system. For example, in response to an enrollment process that stores a decrypted key in the secure enclave of a NIC, the NIC can receive a data packet from a client device. The NIC can transmit a key request to a key server that includes an encrypted key corresponding to the decrypted key. The key server can look up the previously stored private/public key pair to authenticate the NIC. The key server can provide private/public key pair to the NIC to allow the NIC to later encrypt data-in-motion.

Abstract: Systems and methods of routing a data communication across a network having a plurality switches are provided by monitoring the operation of the plurality of global links to determine which of the plurality of global links provide working paths. A routing table indicative of a status for the plurality of links is maintained, where the routing table provides weighting for each of the working paths. When routing, a link using a weighted pseudo-random selection from the choices available in the routing table is selected. Routing along one of the working paths commensurate with the selected link is performed, and the weighting is updated based upon the operation of the plurality of links.

Abstract: A switch architecture for a data-driven intelligent networking system is provided. The system can accommodate dynamic traffic with fast, effective congestion control. The system can maintain state information of individual packet flows, which can be set up or released dynamically based on injected data. Each flow can be provided with a flow-specific input queue upon arriving at a switch. Packets of a respective flow are acknowledged after reaching the egress point of the network, and the acknowledgement packets are sent back to the ingress point of the flow along the same data path. As a result, each switch can obtain state information of each flow and perform flow control on a per-flow basis.

Abstract: A network interface controller (NIC) capable of efficient event management is provided. The NIC can be equipped with a host interface, a first memory device, and an event management module. During operation, the host interface can couple the NIC to a host device. The event management module can identify an event associated with an event queue stored in a second memory device of the host device. The event management module can insert, into a buffer, an event notification associated with the event. The buffer can be associated with the event queue and stored in the first memory device. If the buffer has met a release criterion, the event management module can insert, via the host interface, the aggregated event notifications into the event queue.

Abstract: One aspect of the disclosure can provide a method and system for application instrumentation. During operation, a node within a network may identify an application packet to be inserted with telemetry metadata, determine an execution phase of an application associated with the identified application packet, and insert application-specific telemetry metadata into an In-band Network Telemetry (INT) header of the identified application packet, the INT header comprising one or more metadata header fields and one or more metadata fields. The node may further insert at least one marker into the one or more metadata header fields, the marker indicating the determined execution phase of the application.

Abstract: A system tracks, in a network device with multiple links, activity over a respective link for a predetermined amount of time. The activity comprises at least one of a number of idle periods or a duration of a respective idle period. The system divides a predetermined time interval into a number of bins. A bin is associated with a range of time. The system stores the tracked activity in data structure entries which each indicate the number of bins, a duration of time associated with a respective bin, and a count associated with the respective bin. The system indicates the tracked activity in the entry for the respective link by incrementing a count associated with a bin matching the duration of the respective idle period based on a first number of idle periods tracked for the matching duration. The system displays the stored tracked activity for the respective link.

Abstract: In certain implementations, a method includes receiving, by a network interface controller (NIC), a request for inter-process communication associated with a sending process of a distributed application. The request includes a logical network address for a destination process of the distributed application. The method includes executing, by the NIC, a network address translation process to translate the logical network address for the destination process to a translated network address for the destination process. The network address translation process includes executing, using a first portion of the logical network address, a lookup of a network address translation table to determine a first address; determining a translation modifier by executing a translation algorithm using a second portion of the logical network address; and determining the translated network address using the first address and the translation modifier.

Abstract: An apparatus facilitating efficient key refresh in a node is provided. During operation, the apparatus can determine a collective operation initiated by the node. The node can include a processor and can be in a distributed system comprising a plurality of nodes. The collective operation can be performed by a subset of the plurality of nodes in conjunction with each other. The apparatus can generate a new key based on a previous key maintained at the apparatus. Here, a respective key can be used for encrypting an inter-node packet in the distributed system. The apparatus can maintain the new and previous keys for the duration of the collective operation. Either of the new and previous keys can be used for decrypting messages received at the apparatus from other nodes of the distributed system. Upon determining a threshold point of the collective operation, the apparatus can discard the previous key.

Abstract: A system allows a first endpoint to communicate with a plurality of endpoints in a connectionless network environment by enabling, by the first endpoint, a plurality of authorization keys, a respective endpoint of the plurality of endpoints being associated with a respective authorization key. The first endpoint receives, from a second endpoint of the plurality of endpoints, a first request comprising an action, a remote procedure call, and a first authorization key associated with the second endpoint. The system validates the first authorization key by matching the first authorization key to an enabled authorization key. The system performs the action indicated and, responsive to performing the action, returns the first authorization key to a completion event queue. The system obtains the first authorization key from the completion event queue and transmits, to the second endpoint based on the obtained first authorization key, data associated with the first request.

Abstract: In some examples, a system stores, in a first database having a first schema, metrics received from a network comprising communication nodes, the metrics relating to operations of the communication nodes, and the first database associating the metrics with metadata corresponding to hierarchical components in a topology of the network. In response to an alert relating to an issue in the network, the system computes a health measure based on the metrics, the health measure indicating a health status of a first component, performs a dynamic runtime mapping of the metadata associated with the metrics in the first database having the first schema with corresponding metadata in a second database having a second schema different from the first schema, where the second database contains information of a topology of the network, and initiates a management action to address the health status based on the health measure and the dynamic runtime mapping.