Abstract: A file server system having solid-state drive primary storage and magnetic disk drive secondary storage aggressively moves newly created files from the primary storage to selected file systems in the secondary storage to match expected access patterns upon the files to different configurations of the file systems and for load balancing upon the file systems in the secondary storage. Upon read access to a file that has been moved to the secondary storage, or upon migration of a newly created file that was read in primary storage after creation, a corresponding stub file containing file mapping metadata is created in the primary storage. The file mapping metadata in the stub file maps the extent of the file to logical storage addresses in the secondary storage.

Abstract: Based on a count of the number of dirty pages in a cache memory, the dirty pages are written from the cache memory to a storage array at a rate having a component proportional to the rate of change in the number of dirty pages in the cache memory. For example, a desired flush rate is computed by adding a first term to a second term. The first term is proportional to the rate of change in the number of dirty pages in the cache memory, and the second term is proportional to the number of dirty pages in the cache memory. The rate component has a smoothing effect on incoming I/O bursts and permits cache flushing to occur at a higher rate closer to the maximum storage array throughput without a significant detrimental impact on client application performance.

Abstract: Embodiments of the invention relate to maintaining reference counts in a file system to track when a block is available to be freed. When a snapshot copy of a file is created, rather than copying all the blocks of the file for the snapshot copy, the snapshot inode is given pointers to the blocks of the file. Rather than updating the reference counts for all these blocks to indicate that they are now in use by more than one file, the update of reference counts for at least some of these blocks may be deferred until a later time.

Abstract: Embodiments of the invention relate to maintaining reference counts in a file system to track when a block is available to be freed. When a snapshot copy of a file is created, rather than copying all the blocks of the file for the snapshot copy, the snapshot inode is given pointers to the blocks of the file. Rather than updating the reference counts for all these blocks to indicate that they are now in use by more than one file, the update of reference counts for at least some of these blocks may be deferred until a later time.

Abstract: A method is used in managing metadata. Data of an object is stored in a magnetic hard disk set in an object addressable data storage system. Metadata for the object is stored in an SSD set in the object addressable data storage system. The metadata includes information necessary for determining the location of the data of the object in the magnetic hard disk set.

Abstract: A file server includes active storage containing frequently accessed files, and active/inactive disk drives for containing infrequently accessed files. Groups of the files having become inactive in the active storage are successively migrated to respective evacuated active/inactive disk drives so that each active/inactive disk drive is loaded with files having a similar probability of access when access to the active/inactive disk drive reaches an inactivity threshold for powering down the active/inactive disk drive. Storage of the active/inactive disk drives is reclaimed when an oldest group of the files is archived or when an active/inactive disk drive is evacuated by migrating files from the active/inactive disk drive to storage having been released in other disk drives by promotion of files for client access to the promoted files. Therefore, recovery of storage can be planned and scheduled in advance and performed efficiently in a background process.

Abstract: A method is used in managing metadata. Data of an object is stored in a magnetic hard disk set in an object addressable data storage system. Metadata for the object is stored in an SSD set in the object addressable data storage system. The metadata includes information necessary for determining the location of the data of the object in the magnetic hard disk set.

Abstract: A write interface in a file server provides permission management for concurrent access to data blocks of a file, ensures correct use and update of indirect blocks in a tree of the file, preallocates file blocks when the file is extended, solves access conflicts for concurrent reads and writes to the same block, and permits the use of pipelined processors. For example, a write operation includes obtaining a per file allocation mutex (mutually exclusive lock), preallocating a metadata block, releasing the allocation mutex, issuing an asynchronous write request for writing to the file, waiting for the asynchronous write request to complete, obtaining the allocation mutex, committing the preallocated metadata block, and releasing the allocation mutex. Since no locks are held during the writing of data to the on-disk storage and this data write takes the majority of the time, the method enhances concurrency while maintaining data integrity.

Abstract: Read-only and read-write snapshot copies of a production file in a Unix-based file system are organized as a version set of file inodes and shared file blocks. Version pointers and branch pointers link the inodes. Initially the production file can have all its blocks preallocated or it can be a sparse file having only an inode and its last data block. A protocol is provided for creating read-only and read-write snapshots, deleting snapshots, restoring the production file with a specified snapshot, refreshing a specified snapshot, and naming the snapshots. Block pointers are marked with a flag indicating whether or not the pointed-to block is owned by the parent inode. A non-owner marking is inherited by all of the block's descendants. The block ownership controls the copying of indirect blocks when writing to the production file, and also controls deallocation and passing of blocks when deleting a read-only snapshot.

Abstract: Access to movies ranging from very popular movies to unpopular movies is managed by configuring sets of data movers for associated movie rankings, reserving data mover local cache resources for the most popular movies, reserving a certain number of streams for popular movies, negotiating with a client for selection of available movie titles during peak demand when resources are not available to start any freely-selected movie in disk storage, and managing disk bandwidth and primary and local cache memory and bandwidth resources for popular and unpopular movies. The assignment of resources to movie rankings may remain the same while the rankings of the movies are adjusted, for example, during off-peak hours. A movie locked in primary cache and providing a source for servicing a number of video streams may be demoted from primary cache to disk in favor of servicing one or more streams of a higher-ranking movie.

Abstract: A cached disk array includes a disk storage array, a global cache memory, disk directors coupling the cache memory to the disk storage array, and front-end directors for linking host computers to the cache memory. The front-end directors service storage access requests from the host computers, and the disk directors stage requested data from the disk storage array to the cache memory and write new data to the disk storage. At least one of the front-end directors or disk directors is programmed for block resolution of virtual logical units of the disk storage, and for obtaining, from a storage allocation server, space allocation and mapping information for pre-allocated blocks of the disk storage, and for returning to the storage allocation server requests to commit the pre-allocated blocks of storage once data is first written to the pre-allocated blocks of storage.

Abstract: A system for producing multiple concurrent real-time video streams from stored MPEG video clips includes a video server and at least one MPEG decoder array. The decoder array has multiple decoder pairs, each pair having a video switch for switching from one decoder in the pair to the other at a specified time. Switching may occur from a specified Out-point frame to a specified In-point frame, and the specified frames can be any frame type at any location in the group of pictures (GOP) structure. In a preferred construction, the video server has a controller server linked to a series of data mover computers, each controlling one or more respective decoder arrays. The data mover computers use a control protocol to control the decoder arrays, and each decoder uses a data protocol to request data from a respective data mover computer.

Abstract: Metadata for splicing of an encoded digital motion video stream (such as an MPEG Transport Stream) is prepared in real time while recording at the encoding bit rate and faster than encoded bit rate for off line encoding independent of the bit rate and mechanisms for ingestion of the data stream into data storage. Preprocessing is performed during a metered file transfer protocol (FTP) and includes pseudo real-time encoding. The preprocessing includes Group of Pictures (GOP) level pre-processing of splicing In Points and results in an intimate linkage between metadata and the file system in which the video data is stored. The preferred file system enables access to metadata in parallel to writing the data on disk. The pre-processing is performed simultaneous to writing the data to the disk using a carousel type buffer mechanism.

Abstract: Servers in a storage system store a nested multilayer directory structure, and a global index that is an abstract of the directory structure. The global index identifies respective portions of the directory structure that are stored in respective ones of the servers, and the global index identifies paths through the directory structure linking the respective portions. Upon performing a top-down search of the directory structure in response to a client request and finding that a portion of it is offline, the global index is searched to discover portions of the directory structure that are located below the offline portion. The global index may also identify the respective server storing each of the respective portions of the directory structure, and may indicate whether or not each of the respective portions of the directory structure is known to be offline.

Abstract: Coded video from an on-line MPEG video encoder is stored as a clip in a video server or is otherwise received in the video server and prepared or used for splicing. In order to reduce apparent frame inaccuracy that may result from the splicing process, the on-line MPEG video encoder and the server are coordinated so that the group-of-picture (GOP) structure in the encoder provides specified In-points and Out-points that are valid and desirable for splicing. An encoder control protocol is also provided for remote control of the on-line MPEG video encoder in order to coordinate the on-line MPEG video encoder with the video server.

Abstract: Read-only and read-write snapshot copies of a production file in a Unix-based file system are organized as a version set of file inodes and shared file blocks. Version pointers and branch pointers link the inodes. Initially the production file can have all its blocks preallocated or it can be a sparse file having only an inode and its last data block. A protocol is provided for creating read-only and read-write snapshots, deleting snapshots, restoring the production file with a specified snapshot, refreshing a specified snapshot, and naming the snapshots. Block pointers are marked with a flag indicating whether or not the pointed-to block is owned by the parent inode. A non-owner marking is inherited by all of the block's descendants. The block ownership controls the copying of indirect blocks when writing to the production file, and also controls deallocation and passing of blocks when deleting a read-only snapshot.

Abstract: A write interface in a file server provides permission management for concurrent access to data blocks of a file, ensures correct use and update of indirect blocks in a tree of the file, preallocates file blocks when the file is extended, solves access conflicts for concurrent reads and writes to the same block, and permits the use of pipelined processors. For example, a write operation includes obtaining a per file allocation mutex (mutually exclusive lock), preallocating a metadata block, releasing the allocation mutex, issuing an asynchronous write request for writing to the file, waiting for the asynchronous write request to complete, obtaining the allocation mutex, committing the preallocated metadata block, and releasing the allocation mutex. Since no locks are held during the writing of data to the on-disk storage and this data write takes the majority of the time, the method enhances concurrency while maintaining data integrity.

Abstract: Original-quality MPEG coded video is processed to produce reduced-quality MPEG coded video for trick mode operation by removing non-zero AC DCT coefficients from the 8×8 blocks of I-frames of the MPEG coded video to produce I-frames of reduced-quality MPEG coded video, and inserting freeze frames in the reduced-quality MPEG coded video. Preferably, the coded video is stored in a main file, a fast-forward file and a fast-reverse file. The fast forward file and the fast reverse files contain reduced-quality I frames corresponding to original-quality I frames in the main file. A reading of the main file produces an MPEG transport stream for an audio-visual presentation at a normal rate, a reading of the fast-forward file produces an MPEG transport stream of the audio-visual presentation in a forward direction at a fast rate, and a reading of the fast-reverse file produces an MPEG transport stream of the audio-visual presentation in a reverse direction at a fast rate.

Abstract: Time stamps in an encoded digital motion video data stream (such as an MPEG Transport Stream) are re-stamped for seamless splicing. The re-stamping includes attainment of Presentation Time Stamp (PTS) agreement in the video stream, alignment of audio stream and video stream Presentation Time Stamps, and re-stamping of the Program Clock Reference (PCR) to eliminate jitter and to comply with MPEG standards. The MPEG compliance is achieved by padding, stuffing and insertion of freeze B and P frames and pre-encoded black I frames. Freeze frame insertion can be also achieved by replacing B and P frames by, smaller in size, freeze B and P frames respectively. No frames are discarded such that both spatial and temporal frame accuracy (no extra frame insertion assumed) is achieved.

Abstract: Predictive analysis is performed upon encoded digital motion video (such as an MPEG Transport Stream) to facilitate real-time splicing. The predictive analysis includes estimation of upper and lower bounds of the data levels in a decoder's video and audio buffers for splicing in such a way as to prevent buffer overflow and underflow. This enables buffer management techniques including padding or stuffing, micro-restamping, freeze or repeat of frames, skip or drop of frames, alignment of audio with video. The predictive analysis also includes analysis of the structure of the encoded audio including audio access units (AAUs) and compression windows (AFs), prediction in the compressed domain of initial conditions of the decoder buffer levels for every single Elementary Stream (ES) component of a Transport Stream (TS), and identification of valid splicing In Points and Out Points based on the predicted buffer levels without any special encoder.