Abstract: Operating at least one hypervisor includes running a first hypervisor as a first thread of an underlying operating system, running a second hypervisor as a second thread of the underlying operating system, loading a first guest operating system using the first hypervisor based on the first thread of the underlying operating system, loading a second guest operating system using the second hypervisor based on the second thread of the underlying operating system, and scheduling sharing of resources of the underlying system between the first hypervisor and the second hypervisor according to a scheduler of the underlying operating system, where the first hypervisor and the second hypervisor run independently of each other. The scheduler of the underlying operating system may schedule fractional time shares for the first hypervisor and the second hypervisor to access the same resource.

Abstract: Various embodiments are directed toward improving computing performance for data intensive computing applications by arranging a physical machine into a single computing appliance that includes tightly coupled dedicated computation and storage resources that may scale to multiple peta-byte or greater storage in concert with providing a similarly scaling computing and data caching platform, monitoring co-resident guest applications, and data access applications. Generally, the computing appliance may be arranged to execute compute applications and data storage applications in a single self-contained computing device. The data storage applications may execute within virtual machines (VM's) and the compute applications may execute in other co-resident VM's residing one or more computing appliances.

Abstract: A multi-core processor of a network attached storage system processes requests from host computers for services of a file system service. Each core maintains endpoints of respective connection-layer connections to the hosts to affine respective streams of network traffic with the core, and dynamically and preferentially assigns execution threads of the core to process file system service requests of the streams affined with the core. Each core also co-operates with the other cores to dynamically and non-preferentially (a) assign execution threads of the core to process file system service requests of the streams affined with the other cores, and (b) assign execution threads of the other cores to process file system service requests of the streams affined with the core, promoting efficient use of the cores for the processing workload of the file system service.

Abstract: A burst buffer appliance is adapted for coupling between a computer system and a file system. The burst buffer appliance comprises a first memory, at least one additional memory, and a controller. The controller is configured to detect a remote direct memory access operation between a given compute node of the computer system and the additional memory, and to execute the detected remote direct memory access operation in a manner that bypasses utilization of an operating system associated with the first memory. The first and additional memories may comprise respective main and flash memories. The controller may comprise a virtual machine arranged between a host channel adaptor of the burst buffer appliance and the additional memory, with the virtual machine configured to identify a remote direct memory access operation directed to an address space of the additional memory and to control the flow of data associated with that operation.

Abstract: A virtual switch implemented in a NIC routes packets to a specific virtual machine of a group of supported virtual machines based on a unique MAC address assigned to that virtual machine. Packets which are addressed to the broadcast or a multicast address may be copied to all virtual machines associated with the virtual switch or may be passed to the NIC device driver so that it may be copied by software. The NIC operates in a selective mode whereby packets that are not addressed to a supported virtual machine are rejected or discarded. Further, the virtual switch routes the packets to the virtual machines by performing a DMA copy operation to dedicated memory associated with the virtual machine to which the packet is addressed. The NIC may also include dedicated memory for each supported MAC address in order to facilitate operation.

Abstract: Various embodiments are directed toward improving Virtual Machine performance by reducing the amount of communication protocol processing performed between co-resident Virtual Machine guest applications executing on the same physical machine. Overhead can be reduced by short-circuiting the full communication protocol stack when a plurality of Virtual Machine guest applications are hosted and executing on the same physical machine such as a data storage node or a compute accelerator node. Improvements may also be realized by enabling the sharing in virtual memory of physical memory pages between co-resident Virtual Machine guest applications, and employing a low latency high bandwidth network for enabling guest applications to communicate with each other that are located in different nodes in a data storage cluster.

Abstract: A device for providing a plurality of virtual machines utilizes a multi-core processor having a plurality of cores, each with a memory cache, and a shared memory resource in communication with the cores. The device utilizes a cooperative, multi-level scheduler. The multi-level scheduler includes a primary scheduler and a plurality of secondary schedulers, each supporting a subset of the physical cores. The primary scheduler assigns a group of threads to one of the processor cores. The secondary scheduler associated with the processor core to which the group of threads was assigned schedules execution of individual ones of the threads. The secondary scheduler also provides an indication of lock status to the primary scheduler. The lock status information can be used by the primary scheduler to avoid preempting a thread that holds a lock.

Abstract: Based on a count of the number of dirty pages in a cache memory, the dirty pages are written from the cache memory to a storage array at a rate having a component proportional to the rate of change in the number of dirty pages in the cache memory. For example, a desired flush rate is computed by adding a first term to a second term. The first term is proportional to the rate of change in the number of dirty pages in the cache memory, and the second term is proportional to the number of dirty pages in the cache memory. The rate component has a smoothing effect on incoming I/O bursts and permits cache flushing to occur at a higher rate closer to the maximum storage array throughput without a significant detrimental impact on client application performance.

Abstract: Interrupts and code threads are assigned in a particular way to the core CPUs of a network file server in order to reduce latency for processing client requests for file access. Threads of the network stack are incorporated into real time threads that are scheduled by a real-time scheduler and executed exclusively by a plurality of the core CPUs that are not interrupted by disk adapter interrupts so that the disk adapter interrupts do not interrupt execution of the network stack. Instances of a storage access driver are hard affinity threads, and soft affinity threads include a multitude of instances of a thread of the file system stack for file access request processing so that file access request processing for a multitude of concurrent file access requests is load balanced over the core CPUs.

Abstract: A file server system has a cluster of server computers that share access to a file system in shared storage. One of the server computers has primary responsibility for management of access to the file system. In order to reduce the possibility of primary server overload when a large number of the clients happen to concurrently access the same file system, most metadata processing operations are offloaded to secondary server computers. This also facilitates recovery from failure of a primary server computer since only a fraction of the ongoing metadata operations of a primary server computer is interrupted by a failure of the primary server computer. For example, a secondary data mover may truncate, delete, create, or rename a file in response to a client request.

Abstract: Access to movies ranging from very popular movies to unpopular movies is managed by configuring sets of data movers for associated movie rankings, reserving data mover local cache resources for the most popular movies, reserving a certain number of streams for popular movies, negotiating with a client for selection of available movie titles during peak demand when resources are not available to start any freely-selected movie in disk storage, and managing disk bandwidth and primary and local cache memory and bandwidth resources for popular and unpopular movies. The assignment of resources to movie rankings may remain the same while the rankings of the movies are adjusted, for example, during off-peak hours. A movie locked in primary cache and providing a source for servicing a number of video streams may be demoted from primary cache to disk in favor of servicing one or more streams of a higher-ranking movie.

Abstract: A file server system has a cluster of server computers that share access to a file system in shared storage. One of the server computers has primary responsibility for management of access to the file system. In order to reduce the possibility of primary server overload when a large number of the clients happen to concurrently access the same file system, most metadata processing operations are offloaded to secondary server computers. This also facilitates recovery from failure of a primary server computer since only a fraction of the ongoing metadata operations of a primary server computer is interrupted by a failure of the primary server computer. For example, a secondary data mover may truncate, delete, create, or rename a file in response to a client request.

Abstract: A system for producing multiple concurrent real-time video streams from stored MPEG video clips includes a video server and at least one MPEG decoder array. The decoder array has multiple decoder pairs, each pair having a video switch for switching from one decoder in the pair to the other at a specified time. Switching may occur from a specified Out-point frame to a specified In-point frame, and the specified frames can be any frame type at any location in the group of pictures (GOP) structure. In a preferred construction, the video server has a controller server linked to a series of data mover computers, each controlling one or more respective decoder arrays. The data mover computers use a control protocol to control the decoder arrays, and each decoder uses a data protocol to request data from a respective data mover computer.

Abstract: Transferring data from at least one local data storage device to at least one remote data storage device includes subdividing the data into portions, assigning a sequence number to each of the portions, where writes to the at least one local storage device for a first set of portions having a first sequence number are begun before writes for a second set of portions having a second sequence number that is assigned after to the first sequence number, and updating the sequence number in response to a particular sequence number having been used before and there being data to send from the at least one local storage device to the at least one remote storage device. The at least one local storage device may include a plurality of storage entities that form a consistency group.

Abstract: For each high-level protocol, a respective mesh of Transmission Control Protocol (TCP) connections is set up for a cluster of server computers for the forwarding of client requests. Each mesh has a respective pair of TCP connections in opposite directions between each pair of server computers in the cluster. The high-level protocols, for example, include the Network File System (NFS) protocol, and the Common Internet File System (CIFS) protocol. Each mesh can be shared among multiple clients because there is no need for maintenance of separate TCP connection state for each client. The server computers may use Remote Procedure Call (RPC) semantics for the forwarding of the client requests, and prior to the forwarding of a client request, a new unique transaction ID can substituted for an original transaction ID in the client request so that forwarded requests have unique transaction IDs.

Abstract: Metadata for splicing of an encoded digital motion video stream (such as an MPEG Transport Stream) is prepared in real time while recording at the encoding bit rate and faster than encoded bit rate for off line encoding independent of the bit rate and mechanisms for ingestion of the data stream into data storage. Preprocessing is performed during a metered file transfer protocol (FTP) and includes pseudo real-time encoding. The preprocessing includes Group of Pictures (GOP) level pre-processing of splicing In Points and results in an intimate linkage between metadata and the file system in which the video data is stored. The preferred file system enables access to metadata in parallel to writing the data on disk. The pre-processing is performed simultaneous to writing the data to the disk using a carousel type buffer mechanism.

Abstract: Coded video from an on-line MPEG video encoder is stored as a clip in a video server or is otherwise received in the video server and prepared or used for splicing. In order to reduce apparent frame inaccuracy that may result from the splicing process, the on-line MPEG video encoder and the server are coordinated so that the group-of-picture (GOP) structure in the encoder provides specified In-points and Out-points that are valid and desirable for splicing. An encoder control protocol is also provided for remote control of the on-line MPEG video encoder in order to coordinate the on-line MPEG video encoder with the video server.

Abstract: Original-quality MPEG coded video is processed to produce reduced-quality MPEG coded video for trick mode operation by removing non-zero AC DCT coefficients from the 8×8 blocks of I-frames of the MPEG coded video to produce I-frames of reduced-quality MPEG coded video, and inserting freeze frames in the reduced-quality MPEG coded video. Preferably, the coded video is stored in a main file, a fast-forward file and a fast-reverse file. The fast forward file and the fast reverse files contain reduced-quality I frames corresponding to original-quality I frames in the main file. A reading of the main file produces an MPEG transport stream for an audio-visual presentation at a normal rate, a reading of the fast-forward file produces an MPEG transport stream of the audio-visual presentation in a forward direction at a fast rate, and a reading of the fast-reverse file produces an MPEG transport stream of the audio-visual presentation in a reverse direction at a fast rate.

Abstract: Time stamps in an encoded digital motion video data stream (such as an MPEG Transport Stream) are re-stamped for seamless splicing. The re-stamping includes attainment of Presentation Time Stamp (PTS) agreement in the video stream, alignment of audio stream and video stream Presentation Time Stamps, and re-stamping of the Program Clock Reference (PCR) to eliminate jitter and to comply with MPEG standards. The MPEG compliance is achieved by padding, stuffing and insertion of freeze B and P frames and pre-encoded black I frames. Freeze frame insertion can be also achieved by replacing B and P frames by, smaller in size, freeze B and P frames respectively. No frames are discarded such that both spatial and temporal frame accuracy (no extra frame insertion assumed) is achieved.

Abstract: Predictive analysis is performed upon encoded digital motion video (such as an MPEG Transport Stream) to facilitate real-time splicing. The predictive analysis includes estimation of upper and lower bounds of the data levels in a decoder's video and audio buffers for splicing in such a way as to prevent buffer overflow and underflow. This enables buffer management techniques including padding or stuffing, micro-restamping, freeze or repeat of frames, skip or drop of frames, alignment of audio with video. The predictive analysis also includes analysis of the structure of the encoded audio including audio access units (AAUs) and compression windows (AFs), prediction in the compressed domain of initial conditions of the decoder buffer levels for every single Elementary Stream (ES) component of a Transport Stream (TS), and identification of valid splicing In Points and Out Points based on the predicted buffer levels without any special encoder.