Abstract: Systems and methods are described for obtaining, for insertion into a schema-less database, a data object that comprises a plurality of key and value pairs. The method also includes hashing the keys associated with the plurality of key and value pairs. The hashing includes executing a hash function to generate a hashed data object. The method also includes comparing the hashed data object to a plurality of first hashed data objects and determining that the hashed data object does not match any of the first hashed data objects. Based on the determining, the method includes associating the hashed data object with the first hashed data objects to generate second hashed data objects and generating a schema for the database. The schema includes a hierarchy of keys that represent the second hashed data objects.

Abstract: A distributed system and method for replicated storage of structure data records. According to one embodiment, a system may include storage hosts each configured to store and retrieve structured data records, and a data store manager configured to receive a request from a client to store a structured data record within a table. In response to receiving the request, the data store manager may be further configured to map the structured data record to a block according to a partition key value of the structured data record and an identifier of the table and to map the block to a subset comprising at least two of the plurality of storage hosts. Upon successfully storing the structured data record to the block within at least two storage hosts within the subset, the data store manager may be further configured to return to the client an indication that said request is complete.

Abstract: Technologies related to shared secret identification for secure communication are generally described. In some examples, devices may exchange hashes, such as file deduplication hashes, to identify a matching hash. The identified matching hash represents a shared data item which may be used as a shared secret to encrypt and/or decrypt subsequent secure communications between the devices. Each device retrieves the shared data item from its respective secure memory and may use the shared data item to encrypt and/or decrypt subsequent secure communications. An eavesdropper may observe the hash exchange, but will not be able to decrypt the secure communications without access to the shared data item, because hashes may be effectively non-invertible.

Abstract: Method and apparatus for recording associations with logs are described. In one embodiment, the method comprises obtaining a first media identifier corresponding to a first media; accessing a log that contains one or more entries that associate two or more media identifiers with each other; and determining that the first media is related to a second media based on the log indicating that a second media identifier is associated with, but different from, the first media identifier.

Abstract: Embodiments of the invention include systems and methods for providing block-level verification of replicated file systems. Embodiments operate in context of data storage environments, which may typically have multiple file systems, snapshots of file systems, and replicas of file systems. In one illustrative scenario, a replica is created of a file system having multiple associated snapshots, and a user desires to verify the accuracy of the replica. A signature is created for each of the source active file system and the target replica file system, so that each signature includes records of both block-level signatures and block-level allocations. The signatures are compared to discover any differences. The differences may then be reconciled, where possible, to determine whether the differences indicate a corrupt or otherwise invalid replica.

Abstract: An information processing device reduces a time of processing for adding an attribute name performed in each node. The device memorizes its own segment. This segment is at least one segment among each segment made by dividing a range of a key into a plurality of segments, so as to make the segments neighbor with each other. The key is generated about at least two attribute names using an attribute name and an attribute value based on a specified order relation and is of size-comparable form. The device judges whether a key generated from an attribute name and an attribute value is included in its own segment or not.

Abstract: Embodiments relate to systems and methods for a cloud-based directory system based on hashed values of parent and child storage locations. Platforms and techniques are provided to store a data object to cloud storage resources in two or more locations recorded in a consistent hash structure. A file management tool can store one copy of the data object to a location corresponding to the hashed value of the file path or name, and a second copy to a location corresponding to the hashed value of the parent directory of the data object. All files sharing a common parent directory or other location therefore have at least one copy stored to the same location, in common with the parent. Directory-wide read, write, and/or search operations can therefore be performed more efficiently, since the constituent files of a directory or other location can be accessed from one location rather than distributed locations.

Abstract: Systems and methods enabling parallel processing of hash functions are provided. A data string including a plurality of pieces arranged in an order is hashed using a hash function to determine a plurality of authentication checkpoint hashes associated with the pieces. To authenticate the data string, the pieces are grouped into sets, and the authentication checkpoint hash associated with the piece following all other pieces of that set in the order is associated with that set. The system simultaneously performs a separate hash process on each set. That is, the system hashes the pieces of that set using the hash function to determine a result hash, and compares that result hash with the authentication checkpoint hash associated with that set. The initial input to the hash function for the hash process for each set includes one of the pieces and either a default seed or an authentication checkpoint hash.

Abstract: A network packet includes a packet key that includes one or more source-destination field pairs. Each source-destination field pair that is included in the one or more source-destination field pairs includes a source field and a destination field. For each selected source-destination field pair, included in the one or more source-destination field pairs, a first section and a second section are selected in the packet key. A source field value is extracted from the source field and a destination field value is extracted from the destination field of the selected source-destination field pair.

Abstract: For a network that includes several managed edge switching elements and several managed non-edge switching elements that are for implementing a logical switching element, some embodiments provide a method of distributing packet processing across the several managed non-edge switching elements. The method receives a packet for processing through the logical switching element. Based on a determination that the packet needs to be processed by a managed non-edge switching element, the method determines a particular managed non-edge switching element of the several managed non-edge switching elements to forward the packet. The method forwards the packet to the particular managed non-edge switching element for the particular managed non-edge switching element to process the packet.

Abstract: A robust hashing method is applied to media data (e.g., video, image, and/or audio data), producing a hash output that is robust with respect to at least one attribute of the media data. A histogram is generated for the media data and the histogram is hashed using a weighted hashing procedure. The histogram can be derived from a plurality of randomized versions of the media file, each randomized version of the media file altered to a random extent with respect to the attribute. The histogram can also be derived from a plurality of feature descriptors computed for the media data that are coarsely encoded with respect to the attribute. The weighted hashing procedure includes assigning a weight to components of the histogram and applying a plurality of hash functions to a number of versions of each component, the number of versions based on the assigned weight.

Abstract: Techniques for data migration of a storage system are described herein. According to one embodiment, a candidate index is generated to store information identifying a plurality of segments associated with one or more candidate files of a source storage tier to be migrated to a target storage tier of a storage system. Segments of the one or more candidate files are then migrated from the source storage tier to the target storage tier according to a sequential order, and based on the candidate index.

Abstract: The present invention refers to a computer-implemented method of anonymizing data of a database (2), wherein the method comprises the following steps: a. exporting at least one data record (10) from the database (2), wherein the data record (10) has a structure and comprises content (100); and b. anonymizing at least part of the content (100) to produce at least one anonymized data record (3); c. wherein the anonymized data record (3) has the same structure than the data record (10) read from the database (2).

Abstract: Techniques for data migration of a storage system are described herein. According to one embodiment, a list of files stored in a source storage tier is identified as candidate files to be migrated to a target storage tier based on a migration policy. One of a regular data movement method and a bulk data movement method is selected based on a characteristic of the candidate files. The candidate files are then migrated from the source storage tier to the target storage tier using the selected data movement method.

Abstract: A method and apparatus for preventing a user from interpreting optional stored data information even when the user extracts the optional stored data, by managing data associated with a flash memory in a flash translation layer, the method comprising searching at least one page of the flash memory when writing data to the flash memory, determining whether authority information corresponding to respective searched pages includes an encryption storage function, generating, corresponding to respective searched pages, a page key according to an encrypting function when the authority information includes the encryption storage function encrypting the data using the generated page key and storing the encrypted data in the respective searched pages, and storing the data in the respective searched pages without encryption when the authority information does not include the encryption storage function.

Abstract: A system and method for efficiently removing duplicate data blocks at a fine-granularity from a storage array. A data storage subsystem supports multiple deduplication tables. Table entries in one deduplication table have the highest associated probability of being deduplicated. Table entries may move from one deduplication table to another as the probabilities change. Additionally, a table entry may be evicted from all deduplication tables if a corresponding estimated probability falls below a given threshold. The probabilities are based on attributes associated with a data component and attributes associated with a virtual address corresponding to a received storage access request. A strategy for searches of the multiple deduplication tables may also be determined by the attributes associated with a given storage access request.

Abstract: The distributed computing backup and recovery (DCBR) system and method provide backup and recovery for distributed computing models (e.g., NoSQL). The DCBR system extends the protections from server node-level failure and introduces persistence in time so that the evolving data set may be stored and recovered to a past point in time. The DCBR system, instead of performing backup and recovery for an entire dataset, may be configured to apply to a subset of data. Instead of keeping or recovering snapshots of the entire dataset which requires the entire cluster, the DCBR system identifies the particular nodes and/or archive files where the dataset resides so that backup or recovery may be done with a much smaller number of nodes.

Abstract: This invention pertains generally to a method for resolving and managing eventual consistency conflicts when using a Prefix Hash Tree (“PHT”) in which the underlying storage system is a Distributed Hash Table (“DHT”). An embodiment according to the within invention comprises a computer implemented method for resolving eventual consistency conflicts when using a PHT and DHT.

Abstract: A system for and method of partial file hashing is presented. The system and method may generate a hash table file using a hash algorithm on portions of a file before transmitting the file. The receiving system may calculate hash values on one or more portions of a file and compare the values to the values in the hash table file.

Abstract: An encrypted file storage solution consists of a cluster of processing nodes, external data storage, and a software agent (the “File System Watcher”), which is installed on the application servers. Cluster sizes of one node up to many hundreds of nodes are possible. There are also remote “Key Servers” which provide various services to one or more clusters. The preceding describes a preferred embodiment, though in some cases it may be desirable to “collapse” some of the functionality into a smaller number of hardware devices, typically trading off cost versus security and fault-tolerance.

Abstract: In a hierarchical switching architecture that includes at least one lower level managed switching element that connects to several higher level managed switching elements, some embodiments provide a method of identifying a higher level managed switching element to which the lower level managed switching element forwards a packet for further processing. The method computes a value based on a set of attributes of the packet. The method identifies a record from a hierarchy traversal table based on the computed value. The record specifies (1) a first higher level managed switching element as a primary higher level managed switching element and (2) a second higher level managed switching element as a secondary higher level managed switching element. The primary and secondary higher level managed switching elements are for forwarding the packet for further processing. The method forwards the packet to one of the higher level managed switching elements.

Abstract: A computer-implemented method and apparatus are disclosed for decoding an encoded data signal. In one embodiment, the method includes accessing, in a memory, a set of signal elements. The encoded data signal is received at a computing device. The signal includes signal fragments each having a projection value and an index value. The projection value has been calculated as a function of at least one signal element of the set of signal elements and at least a portion of the data signal. The index value associates its respective signal fragment with the at least one signal element used to calculate the projection value. The computing device determines amplitude values based on the projection values in the signal fragments. The decoded signal is determined using the amplitude values and the signal elements associated with the at least some of the signal fragments.

Abstract: A method and apparatus is disclosed herein for synchronizing distributed work. In one embodiment, the method comprises receiving first and second metadata entries, adding the first and second metadata entries to a set corresponding to a digital object, and providing access to first and second unique identifiers used for referencing the first and second metadata entries respectively, where the first and second unique identifiers are based on contents of the first and second metadata entries respectively.

Abstract: An electronic lock, system and method for dynamic controlled access, without the lock communicating with or connected to a code server, are provided. The lock includes a locking mechanism, a clock, a microprocessor, and a memory storing a hash function and programmed instructions for the microprocessor to perform certain operations. The microprocessor and memory may be comprised in a microcontroller. When an access code is entered into the lock, the lock microcontroller hashes currently valid access start date/duration combinations with lock identifying data to return valid access codes. If the entered access code equals any of the valid access codes, the locking mechanism is opened. The lock identifying data may include data stored by a lock manufacturer and/or data created by a lock owner. Static access codes may also be programmed into the lock if desired. Caching of valid access codes may be used to reduce processing time.

Abstract: A computer-implemented method for distributed data deduplication may include (1) identifying a deduplicated data system, the deduplicated data system include a plurality of nodes, wherein each node within the plurality of nodes is configured to deduplicate data stored on the node, (2) identifying a data object to store within the deduplicated data system, (3) generating a similarity hash of the data object, the similarity hash representing a probabilistic dimension-reduction of the data object, (4) selecting, based at least in part on the similarity hash, a target node from the plurality nodes on which to store the data object, and then (5) routing the data object for storage on the target node based on the selection of the target node. Various other methods, systems, and computer-readable media are also disclosed.

Abstract: Systems and methods for facilitating combining media fingerprinters are provided. In one implementation, a system can include a memory and a microprocessor. The system can also include: a communication component that receives media; a wavelet-based hash generation component that generates a plurality of hashes based on wavelet descriptors associated with the media; a local features information generation component that generates local descriptors associated with the media; a grouping component that groups the plurality of hashes into two or more subgroups, and generates a combined group of descriptors. The combined group of descriptors can include quantized local descriptors and groups of hashes. The system can also include a hash generation component that hashes the combined group of descriptors.

Abstract: Mechanisms are provided for accelerated data deduplication. A data stream is received an input interface and maintained in memory. Chunk boundaries are detected and chunk fingerprints are calculated using a deduplication accelerator while a processor maintains a state machine. A deduplication dictionary is accessed using a chunk fingerprint to determine if the associated data chunk has previously been written to persistent memory. If the data chunk has previously been written, reference counts may be updated but the data chunk need not be stored again. Otherwise, datastore suitcases, filemaps, and the deduplication dictionary may be updated to reflect storage of the data chunk. Direct memory access (DMA) addresses are provided to directly transfer a chunk to an output interface as needed.

Abstract: A document archiving and retrieval system enables third parties to verify that an electronic document had been archived on a particular date and time. For each document to be archived, a hash function generates a digital fingerprint of the document, and inserts it into a hash list file, together with an indicator of the location of the file in the archive. The hash list file is submitted to an electronic time stamping authority to obtain a corresponding digital time stamp. Upon request, authentication of the hash list file with the digital time stamp, and verification that the current hash of the file matches that in the hash list file proves that the document was archived on the date indicated by the digital time stamp, and had not subsequently been altered.

Abstract: Disclosed herein are a big data extraction system and method. The big data extraction system includes a data buffer unit for hooking the file message of an operating system, extracting some data from the original data based on the hooked file message, and storing the extracted some data in memory, a data generation unit for generating hash data of the stored some data, verifying the hash data of the stored some data, and generating regeneration data corresponding to the original data based on a result of the verification, and a data storage unit for storing the regeneration data.

Abstract: A hash table is created in parallel without requiring a lock or random accesses to memory. The hash table of a database system is logically partitioned and a separate thread is assigned to each partition of the hash table. As many separate threads as can fit their corresponding hash table partitions into the processor's cache are executed in parallel with other threads without a lock. Execution of a number of separate threads includes: scanning an input data table for a thread's partition and applying a hash function to each key, inserting data of keys that hash to the thread's partition into the thread's partition, and ignoring keys that do not hash to the thread's partition.

Abstract: A method includes receiving a plurality of trusted assets, generating a first signature set for a known software application, and generating a second signature set for a subject software application. Each trusted asset is associated with at least a threshold number of trusted authors. Each signature in the first signature set corresponds to a known asset that is associated with the known software application. Each signature in the second signature set corresponds to a subject asset that is associated with the subject software application. The method further includes generating first and second filtered signature set based on the first and second signature sets, respectively, by excluding signatures corresponding to the trusted assets. The method also includes generating a similarity rating for the subject application based on a comparison of the first filtered signature set and the second filtered signature set.

Abstract: In one embodiment, a method of identifying the presence of matching information items in a network includes using a hashing scheme to generate a set of first hash values from a respective set of first information items stored at a first node and transmitting the set of first hash values over the network to a second node. The set of first hash values is compared at the second node with a set of second hash values generated, using the hashing scheme, from a respective set of second information items stored in the network, to thereby determine at least one matching hash value between the set of first hash values and the set of second hash values. The determined matching hash value is used to identify the presence of at least one matching information item between the set of first information items and the set of second information items.

Abstract: A method of managing read/write requests in a non-transitory data storage system includes receiving a read/write request for data stored in the data storage system and processing the read/write request on at least one computing processor. For a read request, the method includes computing a first checksum of the data, comparing the first checksum with a second checksum associated with the data, and allowing a read operation of the read/write request on the data when the first and second checksums match. For a write request, the method includes allowing a write operation of the read/write request on the data, computing a third checksum of the modified data, and associating the third checksum with the modified data.

Abstract: A client device is configured to generate checksums for attributes of product records to be transmitted to a marketplace operator system that provides an e-commerce marketplace. On subsequent transmissions of the product records, the client device can utilize the checksums to determine if the attributes of the product records have been modified since a previous transmission. If the attributes have not been modified, the client device does not re-transmit the unchanged attributes to the e-commerce marketplace. The client device may also be configured to retrieve a checksum configuration file from the marketplace operator system that includes data defining the manner in which the client device should compute the checksums.

Abstract: An embodiment of the disclosure can receive a composite resource document containing at least one resource. An updated manifest resource can be obtained. The updated manifest resource can list all resources in the composite resource document. A set of zero or more (0 . . . N) resources can be indicated. Each indicated resource is one that is to be subtracted from the list of resources in the updated manifest resource in order to create a generated signature reference list of identified resources to be signed. A hash token can be generated using the resources identified in the generated signature reference list to form a signature hash token. The signature hash token can be encrypted with a secret key.

Abstract: A system and methods for reconciling data and metadata in a cloud storage system while the cloud storage system is fully operational are provided. The method comprises scanning for broken references in a metadata database containing metadata of blocks stored in the cloud storage system, wherein the scanning for the broken references is performed as a background process; and synchronously verifying blocks for at least existence of the blocks in the object storage system, wherein the synchronous block verification is performed using a foreground process as blocks are requested.

Abstract: There is realized a data conversion device that performs generation of a hash value with improved analysis resistance and a high degree of safety. There are provided a stirring processing section performing a data stirring process on input data; and a compression processing section performing a data compression process on input data including data segments which are divisions of message data, the message data being a target of a data conversion. Part of multi-stage compression subsections is configured to perform a data compression process based on both of output of the stirring processing section and the data segments in the message data. There is provided such a configuration that the stirring process is executed at least on fixed timing of a compression processing round of plural rounds and thus, there is realized a data conversion device that performs generation of a hash value with improved analysis resistance and a high degree of safety.

Abstract: An encrypted file storage solution consists of a cluster of processing nodes, external data storage, and a software agent (the “File System Watcher”), which is installed on the application servers. Cluster sizes of one node up to many hundreds of nodes are possible. There are also remote “Key Servers” which provide various services to one or more clusters. The preceding describes a preferred embodiment, though in some cases it may be desirable to “collapse” some of the functionality into a smaller number of hardware devices, typically trading off cost versus security and fault-tolerance.

Abstract: An approach is provided in which a computer system generates a current hash value of a computer program in response to receiving a request to execute the computer program. Next, the computer system determines that the current hash value fails to match a reference hash value that was previously generated subsequent to installing the computer program on the computer system. Since the two hash values do not match each other, the computer system matches the current hash value to an updated hash value that was previously generated in response to modifying the computer program on the computer system. In turn, the computer system executes the computer program when the current hash value matches the updated hash value.

Abstract: Systems and methods for data management are disclosed. One method can comprise receiving first data from one or more data sources and forming a classification group from the one or more data sources based upon one or more classifiers. The method can also comprise generating an identifier for the classification group by applying a function to the first data and/or the one or more classifiers. Second data can be received from the one or more data sources of the classification group and the second data can be associated with the identifier of the classification group.

Abstract: A method includes generating an N-bit generated hash key for an M-bit search key. A data value associated with a matching hash key stored in a content addressable memory is retrieved. The matching hash key is then validated. The content addressable memory supports 2k entries, wherein M>N>k.

Abstract: A method for estimating aggregates over a stream of axis-aligned rectangles, includes: decomposing the stream along one-dimensional intervals, wherein vertices for the rectangle are located in a predetermined grid; assigning each grid row to buckets, wherein the one-dimensional intervals are placed into buckets according to the corresponding rows in which the one-dimensional intervals are positioned; and estimating a sum of a number of grid points touched by at least one of the rectangles in each row of the grid to approximate a volume of the axis-aligned rectangles by: using pairwise-independent hash functions in a multi-dimensional algorithm to determine buckets that include a first interval corresponding to a given rectangle, wherein the interval has hash function results that meet a predetermined threshold; and inserting a second interval for the rectangle corresponding to the first interval into a one-dimensional algorithm for the corresponding bucket meeting the predetermined threshold.

Abstract: Provided are systems and methods for use in data archiving. In one arrangement, compression techniques are provided wherein an earlier version of a data set (e.g., file folder, etc) is utilized as a dictionary of a compression engine to compress a subsequent version of the data set. This compression identifies changes between data sets and allows for storing these differences without duplicating many common portions of the data sets. For a given version of a data set, new information is stored along with metadata used to reconstruct the version from each individual segment saved at different points in time. In this regard, the earlier data set and one or more references to stored segments of a subsequent data set may be utilized to reconstruct the subsequent data set.

Abstract: An information processing apparatus including a processor configured to divide print data into a plurality of separated data pieces, and generate a transmitting unit for each of the plurality of separated data pieces. Each of the transmitting units is generated by combining a hash value and the separated data piece, and the hash value for a first transmitting unit is generated using an electronic signature and the hash value for each of transmitting units that is not the first transmitting unit is generated using the previous separated data piece.

Abstract: File comparing device includes a comparator for comparing a first and second content identifiers, of respective files, with each other. Each file having at least two content identifiers and each content identifier is defined by at least one parameter. In addition a selector is included for selecting, using the parameters of the content identifiers, sub-groups of content identifiers, to increase flexibilities. Calculators calculate functions of the values of the parameters and further comparators compare the functions with thresholds. Content identifiers may be defined by cost and accuracy parameters. Then, calculators calculate first functions of the values of the cost parameters and calculate second functions of the values of the accuracy parameters and further comparators compare the first functions with first thresholds and compare the second functions with second thresholds.

Abstract: Various methods and systems for securing access to hash-based storage systems are disclosed. One method involves receiving information to be stored in a storage system from a storage system client and then generating a key. The key identifies the information to be stored. The value of the key is dependent upon a secret value, which is associated with the storage system. The key is generated, at least in part, by applying a hash algorithm to the information to be stored. The key can then be returned the key to the storage system client. The storage system client can then use the key to retrieve the stored information.

Abstract: Techniques for detecting unwanted data are described herein. In one embodiment, a request is received for storing a data object in a storage system from a client over a network, where the request includes first representative data representing the data object without including actual content of the data object. It is detected whether the data object contains unwanted content by comparing the first representative data with second representative data without accessing the actual content of the data object, where the second representative data represents the unwanted content. A response is transmitted to the client over the network indicating whether the data object is likely to contain the unwanted object based on comparison of the first and second representative data.

Abstract: A method for detecting and combating an attack in an industrial control system includes sending a command stream from a protection network of an industrial control system to at least one zone, the command stream comprising at least one command; concatenating the at least one command into at least one sequential command package comprising units of work; passing the at least one sequential command package to a crypto hash generator; generating at least one of unit of work hash codes or sequence hash codes; comparing the generated hash codes against a database of existing valid unit of work hash codes and sequence hash codes; and if a command stream fault is detected, generating an alert and accessing a database comprising emergency procedures.

Abstract: A method for detecting and combating an attack in an industrial control system includes sending a command stream from a protection network of an industrial control system to at least one zone, the command stream comprising at least one command; concatenating the at least one command into at least one sequential command package comprising units of work; passing the at least one sequential command package to a crypto hash generator; generating at least one of unit of work hash codes or sequence hash codes; comparing the generated hash codes against a database of existing valid unit of work hash codes and sequence hash codes; and if a command stream fault is detected, generating an alert and accessing a database comprising emergency procedures.

Abstract: A client device can be configured to perform a local index search and a server index search to automatically identify and upload content items on the client device that have not been uploaded to an online content management system. A local index search can include creating a unique local identifier of a content item and searching a local upload index that includes the unique local identifier of each content item that has been uploaded. A server index search can include creating a unique server identifier of the content item and searching a server upload index that includes the unique server identifier of each content item stored on the online content management system. Content items that are determined to have not been uploaded to the online content management system based on the results of the two searches, can be uploaded to the content management system by the client device.