Abstract: A method, computer system, and a computer program for managing computer jobs in a queue is provided. This comprises extracting metadata from a new job received for processing and upon determining when a similar enriched metadata exists in a database. A job score and storage footprint may then be determined for the new job from the extracted metadata. It is then determined whether the new job can be grouped for processing with any other jobs already placed on a queue. The new job is then added to the queue based on the new job's score and footprint, and whether it can be grouped with other jobs. The queue is then updated and sent to a scheduler for further processing.

Abstract: A computer-implemented method, according to one embodiment, includes: determining, for each pair of HCI systems where each pair includes a first HCI system coupled to another HCI system, a federation relationship setting that corresponds to each pair. The federation relationship settings are used to control a flow of data, as well as to control a flow of workload scheduling, between the first HCI system and the other HCI systems in the respective pairs. Moreover, determining a federation relationship setting that corresponds to a pair includes: determining whether a risk score which corresponds to the pair is outside a predetermined range. In response to determining that the risk score is outside the predetermined range, a restrictive federation relationship setting is assigned to the pair, and in response to determining that the risk score is not outside the predetermined range, a permissive federation relationship setting is assigned to the pair.

Abstract: A mechanism is provided for managing an input/output device communication request. A first operating system passes a call from a first application intended for an input/output device in a second data processing system to a first host fabric interface controller in the first data processing system without processing the call. The first host fabric interface processes the call to determine the second data processing system with which the call is associated. The first host fabric interface initiates a connection to a second host fabric interface in the second data processing system and transfers the call to a second operating system associated with the input/output device in the second data processing system via the connection to the second host fabric interface. The second operating system then processes the call intended for the input/output device without assistance from any application running on the second data processing system.

Abstract: A mechanism is provided for managing a process-to-process inter-cluster communication request. A call from a first application is received in a first operating system in a first data processing system. The first operating system passes the call from the first operating system to a first host fabric interface controller in the first data processing system without processing the call. The first host fabric interface processes the call to determine a second data processing system in the plurality of data processing systems with which the call is associated, wherein the call is processed by the first host fabric interface without intervention by the first operating system. The first host fabric interface initiates an inter-cluster connection to a second host fabric interface in the second data processing. The call is then transferred to the second host fabric interface in the second data processing system via the inter-cluster connection.

Abstract: Disclosed are a method, a system and a computer program product for automatically allocating and de-allocating resources for jobs executed or processed by one or more supercomputer systems. In one or more embodiments, a supercomputing system can process multiple jobs with respective supercomputing resources. A global resource manager can automatically allocate additional resources to a first job and de-allocate resources from a second job. In one or more embodiments, the global resource manager can provide the de-allocated resources to the first job as additional supercomputing resources. In one or more embodiments, the first job can use the additional supercomputing resources to perform data analysis at a higher resolution, and the additional resources can compensate for an amount of time the higher resolution analysis would take using originally allocated supercomputing resources.

Abstract: A mechanism is provided for managing a process-to-process communication request. A call is received in an operating system from an application in the data processing system. The operating system passes the call to a host fabric interface controller in the data processing system without processing the call. The host fabric interface controller processes the call using state information associated with the call. The call is processed by the host fabric interface controller without intervention by the operating system.

Abstract: A mechanism is provided for managing an application communication request. A first operating system passes a call from a first application in a first data processing system intended for a second application in a second data processing system to a first host fabric interface controller in the first data processing system without processing the call. The first host fabric interface processes the call using state information associated with the call to determine the second data processing system with which the call is associated. The first host fabric interface initiates a connection to a second host fabric interface in the second data processing system and transfers the call to a second operating system in the second data processing system via the connection to the second host fabric interface. The second data processing system then processes the call intended for the second application without assistance from the second application.

Abstract: A mechanism is provided for managing a process-to-process communication request. A call is received in an operating system from an application in the data processing system. The operating system passes the call to a host fabric interface controller in the data processing system without processing the call. The host fabric interface controller processes the call using state information associated with the call. The call is processed by the host fabric interface controller without intervention by the operating system.

Abstract: Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner. The mechanisms perform a data scrubbing operation, in response to a determination to perform data scrubbing, to identify any missing data elements in the plurality of data elements written to the file system and assemble the plurality of data elements into a plurality of data streams in response to results of the data scrubbing indicating that there are no missing data elements.

Abstract: A mechanism is provided for managing a process-to-process communication request. A call is received in an operating system from an application in the data processing system. The operating system passes the call to a host fabric interface controller in the data processing system without processing the call. The host fabric interface controller processes the call using state information associated with the call. The call is processed by the host fabric interface controller without intervention by the operating system.

Abstract: A mechanism is provided for managing a process-to-process intra-cluster communication request. A call from a first application is received in a first operating system in a first data processing system. The first operating system passes the call from the first operating system to a first host fabric interface controller in the first data processing system without processing the call. The first host fabric interface controller processes the call without intervention by the first operating system to determine a second data processing system in the plurality of data processing systems with which the call is associated. The first host fabric interface controller initiates an intra-cluster connection to a second host fabric interface controller in the second data processing system. The first host fabric interface controller then transfers the call to the second host fabric interface controller in the second data processing system via the intra-cluster connection.

Abstract: In a multinode data processing system in which the nodes communicate with one another via communication adapters over a network or switch, the adapters are provided with a dual register mechanism for tracking microcode task status. Upon the issuance of a disruptive command that requires attention from one of the nodes, the task status maintained in one register is copied to the snapshot register. As tasks within the adapter are completed, both registers are updated, thus providing a mechanism for the nodes to determine that all tasks active at the time of the disruptive command have completed. This means that the nodes now have a mechanism for determining, as soon as possible, that all tasks that are active when a disruptive command occurs have completed, thus allowing the data processing node to perform such operations as releasing system memory that is associated with the disruptive command, thus eliminating temporal overhead that can affect performance.

Abstract: Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms further perform a data scrubbing operation to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system and assemble the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements. In addition, the mechanisms release the assembled plurality of data streams for access via the file system.

Abstract: A mechanism is provided for managing a process-to-process communication request. A call is received in an operating system from an application in the data processing system. The operating system passes the call to a host fabric interface controller in the data processing system without processing the call. The host fabric interface controller processes the call using state information associated with the call. The call is processed by the host fabric interface controller without intervention by the operating system.

Abstract: A synchronization optimized queuing method and device to minimize software/hardware interaction in network interface hardware during an end-of-initiative process, including network adapter queue implementations for network interface hardware for optimized communication in a computer system. An end-of-initiative procedure to ensure that the network interface hardware has received an interrupt enable and to recheck the interrupt queue is unnecessary in the present invention.

Abstract: Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner. The mechanisms perform a data scrubbing operation, in response to a determination to perform data scrubbing, to identify any missing data elements in the plurality of data elements written to the file system and assemble the plurality of data elements into a plurality of data streams in response to results of the data scrubbing indicating that there are no missing data elements.

Abstract: Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner. The mechanisms perform a data scrubbing operation, in response to a determination to perform data scrubbing, to identify any missing data elements in the plurality of data elements written to the tile system and assemble the plurality of data elements into a plurality of data streams in response to results of the data scrubbing indicating that there are no missing data elements.

Abstract: Mechanisms for performing a backend operation in a file system are provided. A backend operation on a portion of the file system is initiated. At least one indirect transition table data structure is created for performing the backend operation. Metadata corresponding to the portion of the file system is linked to the at least one indirect transition table data structure. The backend operation is performed on data in a sub-portion of the portion of the file system and the at least one indirect transition table data structure is updated with pointers to new locations of the data in the sub-portion as transitions of the data are completed. At least one data access operation is performed to the portion of the file system at substantially a same time as performing the backend operation on the data in the sub-portion of the portion of the file system.

Abstract: A barrier synchronization register, accessible to the nodes in a distributed data processing system, has portions thereof allotted to threads which are present in multiple groups. The barrier synchronization register portion allotted to a given thread has stored therein, over time, group identifier numbers. In this way the state space of a barrier synchronization register is shared over more than one group of process threads.

Abstract: Disclosed are a method, a system and a computer program product for dynamically allocating and/or de-allocating resources and/or partitions that provide I/O and/or active storage access services in a supercomputing system. The supercomputing system can include multiple compute nodes, high performance computing (HPC) switches coupled to the compute nodes, and active non-volatile storage devices coupled to the compute nodes. Each of the compute nodes can be configured to communicate with another compute node through at least one of the HPC switches. In one or more embodiments, each of at least two compute nodes includes a storage controller and is configured to dynamically allocate and de-allocate a storage controller partition to provide storage services to the supercomputing system, and each of at least two compute nodes includes an I/O controller and is configured to dynamically allocate and de-allocate an I/O controller partition to provide I/O services to the supercomputing system.