Abstract: A global-level manager receives a work order referencing at least one executable file and one or more input files. The one or more input files include multiple input resources. A first type of input resource is identified in the one or more input files and a second type of input resource in the one or more input files. The first type of input resource is split into a plurality of first-type input shards. The second type of input resource is split into a plurality of second-type input shards. The plurality of second-type input shards are associated with each of the first-type input shards. For each of the first-type input shards, the global-level manager distributes the first-type input shards, the associated second-type input shards, and the executable file to a single.

Abstract: A central storage configured to store one or more input files and an executable file. A work order frontend is configured to receive, from the client, a representational state transfer (RESTful) message that contains a reference to the one or more input files in the central storage. The work order frontend is further configured to transmit, to a global-level manager, a work order. The global-level manager is configured to access the work order. The global-level manager is further configured to partition the one or more input files into multiple shards, the work order into multiple jobs, each job being associated with one or more of the multiple shards and the executable file. The global-level manager is further configured to distribute the jobs among a plurality of clusters.

Abstract: Techniques for encrypting documents in a search index may include: receiving a document for inclusion in a search index of a search system, where the document has an associated access control list (ACL), and the ACL includes data for use in restricting access to the document to users of the search system having credentials that match corresponding data in the ACL; encrypting the document using a first key to produce an encrypted document; generating a wrapped key for the document by encrypting both the first key and the ACL using a second key; and storing, along with the search index, the encrypted document in association with the wrapped key and an identifier for the document.

Abstract: A data storage system where a RAID-X format is implemented at a file level includes storage devices, and one or more nodes that access files from the storage devices. N represents a maximum limit of failures beyond which reconstruction of a file stored using the file level implementation is not possible. A directory is encoded for each of a plurality of files, and a map is encoded that includes a list of storage devices where components of the directory associated with the map reside. The encoding of the directory is performed using a RAID-X format that can tolerate at least N+1 failures prior to being unable to reconstruct the directory.

Abstract: In a file system recovery (FSRC) process, an FSRC module performs dynamic checks on file system objects to verify parent-child relationships among various objects in the system. The FSRC module stores a list of objects in the system and relevant object-related information for each object in a memory. A dynamic check on each listed object is then performed using the object-related information to verify parent-child relationships for that object. After the dynamic check is over, the object is removed from the list and its object-related information is deleted from the memory.

Abstract: In a distributed object-based storage system that includes a plurality of object storage devices and one or more clients that access distributed, object-based files from the object storage devices, each of the files being comprised of a plurality of object components residing on different object storage device, systems and methods that use pointers stored as object attributes for file analysis and monitoring.

Abstract: In a file system recovery (FSRC) process, an FSRC module performs dynamic checks on file system objects to verify parent-child relationships among various objects in the system. The FSRC module stores a list of objects in the system and relevant object-related information for each object in a memory. A dynamic check on each listed object is then performed using the object-related information to verify parent-child relationships for that object. After the dynamic check is over, the object is removed from the list and its object-related information is deleted from the memory. Objects in the system may be divided into a number of segments and dynamic checks can be performed—one segment at a time. A link database for segment X keeps links (forward or backward pointers) from segments Y (Y≠X) such that the destination object of each link is in segment X.