Abstract: The techniques introduced here provide for efficient management of storage resources in a modern, dynamic data center through the use of virtual storage appliances. Virtual storage appliances perform storage operations and execute in or as a virtual machine on a hypervisor. A storage management system monitors a storage system to determine whether the storage system is satisfying a service level objective for an application. The storage management system then manages (e.g., instantiates, shuts down, or reconfigures) a virtual storage appliance on a physical server. The virtual storage appliance uses resources of the physical server to meet the storage related needs of the application that the storage system cannot provide. This automatic and dynamic management of virtual storage appliances by the storage management system allows storage systems to quickly react to changing storage needs of applications without requiring expensive excess storage capacity.

Abstract: A distributed object store in a network storage system uses location-independent global object identifiers (IDs) for stored data objects. The global object ID enables a data object to be seamlessly moved from one location to another without affecting clients of the storage system, i.e., “transparent migration”. The global object ID can be part of a multilevel object handle, which also can include a location ID indicating the specific location at which the data object is stored, and a policy ID identifying a set of data management policies associated with the data object. The policy ID may be associated with the data object by a client of the storage system, for example when the client creates the object, thus allowing “inline” policy management. An object location subsystem (OLS) can be used to locate an object when a client request does not contain a valid location ID for the object.

Abstract: A distributed object store in a network storage system uses location-independent global object identifiers (IDs) for stored data objects. The global object ID enables a data object to be seamlessly moved from one location to another without affecting clients of the storage system, i.e., “transparent migration”. The global object ID can be part of a multilevel object handle, which also can include a location ID indicating the specific location at which the data object is stored, and a policy ID identifying a set of data management policies associated with the data object. The policy ID may be associated with the data object by a client of the storage system, for example when the client creates the object, thus allowing “inline” policy management. An object location subsystem (OLS) can be used to locate an object when a client request does not contain a valid location ID for the object.

Abstract: Methods and system for securely capturing workloads at a live network for replaying at a test network. The disclosed system captures file system states and workloads of a live server at the live network. In one embodiment the captured data is anonymized to protect confidentiality of the data. A file system of a test server at the test network is mirrored from a captured state of the live server. An anonymized version of the captured workloads is replayed as a request to the test server. A lost or incomplete command is recreated from the states of the live server. An order of the commands during replay can be based on an order in the captured workload, or based on a causal relationship. Performance characteristics of the live network are determined based on the response to the replayed command.

Abstract: A method and apparatus for proxying search requests for a storage system and maintaining a central index for performing the search requests is described herein. An index manager on the storage system may initially produce the central index by examining each file in a file system and update the central index thereafter by examining only those files that have changed since the central index was initially produced or last updated. The index manager may receive a changed file list from a differencing layer configured for comparing snapshots of the file system at different time points to produce changed file lists. A search proxy module may receive search requests in a search protocol and proxy the search requests to a search engine by converting the search requests to another search protocol compatible with the search engine. The search engine may then use the central index for performing the search request.

Abstract: Methods and system for securely capturing workloads at a live network for replaying at a test network. The disclosed system captures file system states and workloads of a live server at the live network. In one embodiment the captured data is anonymized to protect confidentiality of the data. A file system of a test server at the test network is mirrored from a captured state of the live server. An anonymized version of the captured workloads is replayed as a request to the test server. A lost or incomplete command is recreated from the states of the live server. An order of the commands during replay can be based on an order in the captured workload, or based on a causal relationship. Performance characteristics of the live network are determined based on the response to the replayed command.

Abstract: Described herein is a flash remapping (FR) layer in a storage operating system for utilizing flash memory as a secondary permanent storage device in a storage system. The FR layer collects particular information (specified by collection parameters) of received access requests for data stored on primary storage devices of the storage system. Based on the collected information and a predetermined access pattern (specified by pattern parameters), the FR layer selects data sets on the primary storage devices to be transferred permanently to flash memory, whereby subsequent access requests to the selected data sets are redirected to flash memory. New parameters may be received by the FR layer (from a user or program) to dynamically reconfigure the functions of the FR layer. The FR layer may be implemented in the operating system without requiring other code of the storage operating system to be modified.

Abstract: A method and apparatus for proxying search requests for a storage system and maintaining a central index for performing the search requests is described herein. An index manager on the storage system may initially produce the central index by examining each file in a file system and update the central index thereafter by examining only those files that have changed since the central index was initially produced or last updated. The index manager may receive a changed file list from a differencing layer configured for comparing snapshots of the file system at different time points to produce changed file lists. A search proxy module may receive search requests in a search protocol and proxy the search requests to a search engine by converting the search requests to another search protocol compatible with the search engine. The search engine may then use the central index for performing the search request.

Abstract: Example embodiments provide various techniques for fast and efficient search of attributes stored in data structures. The attributes are organized following a hierarchical structure of the file system and, in an example, the attributes are stored in a data structure where the hierarchical structure is maintained. As a result, a search within such data structure may follow one or more paths along the hierarchical structure of the file system. Attributes associated with directories and files outside of the path can be excluded from the search. Example embodiments also provide various techniques for updating signatures associated with the attributes. In an example, updates to the signatures can be made incrementally. For example, signatures can be updated when the attributes change.

Abstract: Embodiments of the present invention provide mechanisms that overcome limitations of existing indexes by creating two-dimensional (2D) spatial indexes to quickly locate data containers that match two or more predicates. This is accomplished by representing metadata attributes describing a data container as dimensions in a 2D space so that a data container can be expressed as a point or a cell in a 2D space with coordinates being a pair of values of the selected attributes. A space filling curve is used to traverse the 2D space and convert each pair of the 2D coordinates to a single space filling curve value. A 2D spatial index is then created based on the computed space filling curve values so that one value can be associated with one or more points (data containers) in the index. Advantageously, the created spatial index provides for searching and processing fewer metadata entries, thereby decreasing the time typically used to search for data.

Abstract: A network storage server system includes a distributed object store, a presentation layer, a metadata subsystem, and a content management subsystem. The object store has no namespace and provides location-independent addressing of data objects. The presentation layer provides multiple interfaces for accessing data stored in the object store, including a NAS interface and a Web Service interface, and provides at least one namespace for accessing data via the NAS interface or the Web Service interface. The Web Service interface allows access to stored data via the namespace or without using the namespace (“raw object” access). The metadata subsystem stores user-specified and/or system-generated metadata relating to data objects and allows data objects to be identified and retrieved by searching on the metadata.

Abstract: A facility for comparing two datasets and identifying metadata differences between the two datasets irrespective of the manner in which the data is stored. In some embodiments, the facility includes a comparison unit and a catalog unit. The comparison unit compares a hierarchical hash of a first dataset with a hierarchical hash of a second dataset, the hierarchical hashes each including a plurality of hierarchical hash values, to identify differences in metadata of the first and second datasets by progressively comparing the hierarchical hash values of the first and second hierarchical hashes without comparing the metadata of the first and second datasets. The catalog unit generates a catalog of differences between the first and second datasets, the catalog indicating differences in metadata of the first and second datasets.

Abstract: A system and method reclaims unused storage space from a data container, such as a logical unit number (LUN) of a storage system. In particular, a novel technique is provided that allows a storage system to reclaim storage space not used by a client file system for which the storage system maintains storage, without requiring assistance from the client file system to determine storage usage. In other words, storage system may independently reclaim storage space not used by the client file system, without that file system's intervention.

Abstract: A network storage server implements a method to perform fast crawling of a hierarchical storage structure. The hierarchical storage structure contains data entities stored by a network storage server. The hierarchical storage structure can be recursively divided into a plurality of sections. A plurality of parallel-processing threads can be used to process the plurality of sections. Each thread selects and processes one of the plurality of sections at a time to generate a sorted list of metadata corresponding to the section of the hierarchical storage structure. The sorted lists generated by the plurality of threads are merged to a baseline list. The baseline list contains sorted metadata for entities managed by the hierarchical storage structure. The baseline list can then be outputted as a representation of the state of data stored by the network storage server.

Abstract: A system and method accelerates update of a metadata search database using PCPI differencing. After first populating the search database, a search agent generates a PCPI and utilizes a PCPI differencing technique to quickly identify changes between inode files of first and second PCPIs. The differences are noted as modified metadata and are written to a log file, which is later read by the search agent to update the search database.

Abstract: A system and method for nearly in-band search indexing. A network switch (or other intermediate network device) is configured to provide port mirroring so that data access requests directed to a storage system are forwarded to both the storage system and to a search appliance. The search appliance collects index information from the received data access requests to update a search index. As the search appliance is nearly in-band, i.e., not directly in-line of the data access request path, no increase of latency occurs for processing data access requests by the storage system.

Abstract: Method and system for processing a plurality of requests for accessing “small files” stored at a storage device is provided. A user may define file size and each small file may include one or more blocks of data. The requests are sorted based on an address of a first data block for each small file. The sorted requests are then used to access the stored files, instead of accessing the requests based on when a request was received.

Abstract: A system and method for nearly in-band search indexing. A network switch (or other intermediate network device) is configured to provide port mirroring so that data access requests directed to a storage system are forwarded to both the storage system and to a search appliance. The search appliance collects index information from the received data access requests to update a search index. As the search appliance is nearly in-band, i.e., not directly in-line of the data access request path, no increase of latency occurs for processing data access requests by the storage system.

Abstract: A system and method accelerates update of a metadata search database using PCPI differencing. After first populating the search database, a search agent generates a PCPI and utilizes a PCPI differencing technique to quickly identify changes between inode files of first and second PCPIs. The differences are noted as modified metadata and are written to a log file, which is later read by the search agent to update the search database.

Abstract: One embodiment of the present invention provides a system that facilitates delayed block allocation in a distributed file system. During operation, the system receives a write command at a client, wherein the write command includes a buffer containing data to be written and a file identifier. In response to receiving the write command, the system reserves a set of disk blocks for the file from a virtual pool of disk blocks allocated to the client. The system also transfers the data to be written to the kernel of the client where the data waits to be transferred to the disk.