Abstract: For each source process running on a source node, a rate of high-priority operations initiated by the first process during the period that require accessing a target node can be calculated. An average rate (e.g., an exponential moving average (EMA)) of high-priority operations initiated by the first process over a larger period of time that require accessing the target node can be calculated. Similarly, a rate of high-priority operations initiated by any process executing on a processing node during the period that require accessing the target node can be calculated, and an average rate (e.g., an EMA) of high-priority operations initiated by any such process over the larger period of time that require accessing the target node can be calculated. If one or more of the rates for the period or the average rates reach or exceed respective thresholds, corrective actions can be taken.

Abstract: Improved mechanisms and techniques for recording and aggregating trace information from multiple computing modules of a storage system may be provided. On a storage system having multiple computing modules, where each computing module has multiple processing cores, processing cores may record trace information for I/O operations in dedicated local memory—i.e., memory in the same computing module as the processing core that is dedicated to the computing module. One of the processing cores may be configured to aggregate trace information from across multiple computing modules into its dedicated local memory by accessing trace information from the dedicated local memories of the other computing modules in addition to its own. The aggregated information in one dedicated local memory then may be analyzed for functionality and/or performance and additional action taken based on the analysis.

Abstract: A node mask of a global metadata structure entry representing an MD portion in GM may be updated when an entry of a local MD table representing the MD portion is removed in response to receiving an IO operation, without negatively impacting performance of the IO operation. An update process that is independent of the process executing the IO operation may update the node mask so that performance of the IO operation is not negatively affected. In response to the entry for the MD portion being removed from the local MD table, an entry may be added to a queue. The update process may include accessing the entries in the queue, and, for each entry, updating the node mask (e.g., clearing a bit representing the processing node) and removing the entry from the queue.

Abstract: A node mask of a global metadata structure entry representing an MD portion in GM may be updated when an entry of a local MD table representing the MD portion is removed in response to receiving an IO operation, without negatively impacting performance of the IO operation. An update process that is independent of the process executing the IO operation may update the node mask so that performance of the IO operation is not negatively affected. In response to the entry for the MD portion being removed from the local MD table, an entry may be added to a queue. The update process may include accessing the entries in the queue, and, for each entry, updating the node mask (e.g., clearing a bit representing the processing node) and removing the entry from the queue.

Abstract: Improved mechanisms and techniques for recording and aggregating trace information from multiple computing modules of a storage system may be provided. On a storage system having multiple computing modules, where each computing module has multiple processing cores, processing cores may record trace information for I/O operations in dedicated local memory—i.e., memory in the same computing module as the processing core that is dedicated to the computing module. One of the processing cores may be configured to aggregate trace information from across multiple computing modules into its dedicated local memory by accessing trace information from the dedicated local memories of the other computing modules in addition to its own. The aggregated information in one dedicated local memory then may be analyzed for functionality and/or performance and additional action taken based on the analysis.

Abstract: Accessing data memory includes writing data to a first memory location and to a second memory location in response to a request to write data to a memory address that corresponds to both locations, where the first and second memory locations are mirrored, in response to a request to read data from the memory address, reading data from the first memory location or the second memory location based on load balancing, and accessing data from the second memory location in response to a request to access data at the memory address when memory hardware corresponding to the first memory location has failed. Accessing the data memory may include requesting access to a specific one of the first and second memory locations. The memory address may contain a portion that is common to both the first memory location and the second memory location. Hardware coupled to the memory may cause data written using the memory address to be automatically written to the first memory location and the second memory location.

Abstract: Handling a faulting memory of a pair of mirrored memories includes initially causing a non-faulting memory of the pair of mirrored memories to service all read and write operations for the pair of mirrored memories, determining that hardware corresponding to the faulting memory of the pair of mirrored memories has been successfully replaced to provide a new memory, in response to the new memory being provided, causing data to be copied from the non-faulting memory to the new memory while data is being read to and written from the non-faulting memory, and, in response to successful copying to the new memory, causing writes to be performed to both memories of the pair of mirrored memories and selecting one of the pair of mirrored memories for read operations when one or more read operations are performed.