Abstract: Described herein is a method and system for distributing whole and fragmented requests and responses across a multi-core system. Each core executes a packet engine that further processes data packets and data packet fragments allocated to that core. A flow distributor executing within the multi-core system forwards client requests to a packet engine on a core that is selected based on a value generated when a hash is applied to a tuple comprising a client IP address, a client port, a server IP address and a server port identified in the request. The packet engine maintains each element of the tuple and forwards the request to the selected core. The packet engine can also process data packet fragments by assembling the fragments prior to transmitting them to the selected core, or by transmitting the data packet fragments to the selected core.

Abstract: Some embodiments relate to a migration policy that controls migration of content from a transactional storage system to a fixed content storage system. In some embodiments, content may be migrated from the transactional storage system to the fixed content storage system based on at least one criteria that relates to whether at least a portion of the content is binding.

Abstract: There is disclosed a system and method for executing multiple distinct aggregate queries. In an embodiment, the method comprises: providing at least one Counting Bloom Filter for each distinct column of an input data stream; reviewing count values in the at least one Counting Bloom Filter for the existence of duplicates in each distinct column; and if necessary, using a distinct hash operator to remove duplicates from each distinct column of the input data stream, thereby removing the need for replicating the input data stream and minimizing distinct hash operator processing. Also, the use of Counting Bloom Filters for monitoring data streams allow an early duplicate removal of the input stream of data, resulting in savings in computation time and memory resources.

Abstract: A hash table controller may include a hash calculator configured to receive a key and to determine, based thereon, a first entry in a first bank of a hash table for a value associated with the key and determine a second entry in a second bank of the hash table for the value. The hash table controller also may include a table operations manager configured to determine that the first entry and the second entry are empty, and to store the value and a duplicate of the value at both the first entry and the second entry, respectively.

Abstract: A number of hash tables are accessed concurrently with a different computed index based on a single search key for each hash table. Each index identifies a location in one of the hash tables capable of storing at least one entry. If all indexed locations are used, the entries stored in the lookup table can be reordered so that the new entry can be inserted in one of the locations identified by the computed indexes.

Abstract: Data deduplication compression in a streaming storage application, is provided. The disclosed deduplication process provides a deduplication archive that enables storage of the archive to, and extraction from, a streaming storage medium. One implementation involves compressing fully sequential data stored in a data repository to a sequential streaming storage, by: splitting fully sequential data into data blocks; hashing content of each data block and comparing each hash to an in-memory lookup table for a match, the in-memory lookup table storing all hashes that have been encountered during the compression of the fully sequential data; for each data block without a hash match, adding the data block as a new data block for compression of fully sequential data; and encoding duplicate data blocks using the in-memory lookup table into data segments.

Abstract: Efficient reconciliation of different versions of a target file or dataset can be facilitated using a hash hierarchy of a master version of the dataset. Each level of the hash hierarchy has hashes of blocks of the master version that may be sub-blocks of the hashes in the level above. The top level of the hash hierarchy, having the hashes of the largest blocks of the master, may be transmitted on a first communication channel, possibly a one-way or broadcast communication channel. Streams of encodings of each lower level of the hash hierarchy may be transmitted on respective communication channels. The encodings for a level of the hierarchy may be combinations, such as random linear combinations, of the hashes of that level. A receiver with a target dataset can receive the top level hashes from the first channel and use them to determine which lower hashes are needed to identify parts of the master missing from the target.

Abstract: A method, system and program are disclosed for accelerating data storage in a cache appliance that transparently monitors NFS and CIFS traffic between clients and NAS subsystems and caches files in a cache memory by using a dual hash technique to rapidly store and/or retrieve connection state information for cached connections in a plurality of index tables that are indexed by hashing network protocol address information with a pair of irreducible CRC hash algorithms to obtain an index to the memory location of the connection state information.

Abstract: The present invention is directed toward methods and systems for data de-duplication. More particularly, in various embodiments, the present invention provides systems and methods for data de-duplication that may utilize a vectoring method for data de-duplication wherein a stream of data is divided into “data sets” or blocks. For each block, a code, such as a hash or cyclic redundancy code may be calculated and stored. The first block of the set may be written normally and its address and hash can be stored and noted. Subsequent block hashes may be compared with previously written block hashes.

Abstract: A method for operating a communication system is provided. The method includes receiving an arrival rate of a plurality of real-time packets, and receiving a real-time packet including a first plurality of identifiers, for transmission on a first link or a second link, where the first link has a first bandwidth. The method also includes processing the real-time packet to select a first selected link from the first link and the second link based on the first plurality of identifiers, the arrival rate of the plurality of real-time packets, and the first bandwidth, and transmitting the real-time packet on the first selected link.

Abstract: A configuration management system and related methods are provided to facilitate intersection-based management of configuration information for network devices. In one example, a method includes polling a plurality of network devices to determine address entries stored by address tables of the network devices. The method also includes applying a hash function to the address entries to determine a plurality of key-value associations. The method further includes maintaining a hash table comprising the key-value associations. Each key of the hash table corresponds to an address entry stored by one or more of the address tables. Each value of the hash table identities which of the address tables are storing the address entry corresponding to the associated key.

Abstract: The present invention overcomes the disadvantages of the prior art by providing a technique that stripes data containers across volumes of a striped volume set (SVS) using one of a plurality of different data placement patterns to thereby reduce the possibility of hotspots arising due to each data container using the same data placement pattern within the SVS. The technique is illustratively implemented by calculating a first index value, an intermediate index value and calculating a hash value of an mode associated with a data container to be accessed within the SVS. A final index value is calculated by multiplying the intermediate index value by the hash value, modulo the number of volumes of the SVS. Further, a Locate( ) function may be used to compute the location of data container content in the SVS to which a data access request is directed to ensure consistency of such content.

Abstract: A multi-core system that includes a 64-bit cache storage and a 32-bit memory storage that stores a 32-bit cache object directory. One or more cache engines execute on cores of the multi-core system to retrieve objects from the 64-bit cache, create cache directory objects, insert the created cache directory object into the cache object directory, and search for cache directory objects in the cache object directory. When an object is stored in the 64-bit cache, a cache engine can create a cache directory object that corresponds to the cached object and can insert the created cache directory object into an instance of a cache object directory. A second cache engine can receive a request to access the cached object and can identify a cache directory object in the instance of the cache object directory, using a hash key calculated based on one or more attributes of the cached object.

Abstract: In general, this disclosure describes techniques of storing data in and retrieving data from a cache of a computing device. More specifically, techniques are described for utilizing a “perfect hash” function to implement an associative cache within a computing device. That is, the associative cache implements a fully associative map between a predetermined set of addresses and data values, employing only a single tag fetch comparison.

Abstract: A method and system having a probabilistic offload engine for distributed hierarchical object storage devices is disclosed. According to one embodiment, a system comprises a first storage system and a second storage system in communication with the first storage system. The first storage system and the second storage system are key/value based object storage devices that store and serve objects. The first storage system and the second storage system execute a probabilistic algorithm to predict access patterns. The first storage system and the second storage system execute a probabilistic algorithm to predict access patterns and minimize data transfers between the first storage system and the second storage system.

Abstract: A mechanism for live migration of virtual machines (VMs) with memory optimizations is disclosed. A method of embodiments of the invention includes receiving a hash value for each of one or more memory pages of a migrating VM from a source host machine, obtaining a hash value for each of one or more memory pages hosted by a target host machine, and comparing the received hash values with the obtained hash values for matches. The method further comprises for each of the received hash values that do not match any of the obtained hash values, sending a negative acknowledgment to the source host machine for that hash value, and for each of the received hash values that do match any of the obtained hash values, using a memory page associated with the matching hash value at the target host machine for the migrating VM.

Abstract: A measurement engine performs active platform observation. A program includes an integrity manifest to indicate an integrity check value for a section of the program's source code. The measurement engine computes a comparison value on the program's image in memory and determines if the comparison value matches the expected integrity check value. If the values do not match, the program's image is determined to be modified, and appropriate remedial action can be triggered. The integrity manifest can include a secure signature to verify the validity of the integrity manifest.

Abstract: A computer system includes a disk space comprising at least one type of memory and an operating system for controlling allocations and access to the disk space. A runtime machine runs applications through at least one of the operating system or directly on at least one processor of the computer system. In addition, the runtime machine manages a selected runtime disk space allocated to the runtime machine by the operating system and manages a separate method cache within the selected virtual disk space. The virtual machine controls caching within the method cache of a separate result of at least one method of the application marked as cache capable. For a next instance of the method detected by the runtime machine, the runtime machine accesses the cached separate result of the method in lieu of executing the method again.

Abstract: A method and system for controlling data flow are provided. The method includes: selecting optional reference data of a flow algorithm to generate configuration information; and performing traffic calculation based on the configuration information to determine hash distribution of the data flow to load balancing links. The flow algorithm is a HASH algorithm.

Abstract: A computer implemented method, apparatus and program product automatically optimizes hash function operation by recognizing when a first hash function results in an unacceptable number of cache misses, and by dynamically trying another hash function to determine which hash function results in the most cache hits. In this manner, hardware optimizes hash function operation in the face of changing loads and associated data flow patterns.

Abstract: A system, method, and computer program product for harvesting an image from a local disk of a managed endpoint to an image library is provided. In an embodiment of the method for harvesting an image, a managed endpoint is provided with a boot image that causes the endpoint to instantiate a RAM disk and execute the boot image from the RAM disk. The boot image is used to harvest an image by determining data on a local disk of the managed endpoint to be included in the image that are not already stored in the image library. In one embodiment, this is done by comparing hashes calculated on the data on the local disk to hashes of data in the image library. The data not already stored in the image library are then copied to the image library.

Abstract: A distributed storage network received a data segment. The data segment is partitioned into two or more portions. A first portion hash is calculated from the first portion of data and used to encrypt the second portion of data. A hash of the encrypted second portion of data is then used to either encrypt the next portion of data (in this case, a third portion of data) or to circle back to the beginning and encrypt the first portion of the data if the second portion of data is the last in data segment. This iterative process continues until all portions of the data segment are encrypted in a sequence. In essence, the data portions of the segment are sequentially processed in some order to encrypt the various portions in that progressing order. A reverse order is used to derive the hash values and decrypt the encrypted data portions into decrypted original data to recreate the data segment.

Abstract: A technique for routing data for improved deduplication in a storage server cluster includes computing, for each node in the cluster, a value collectively representative of the data stored on the node, such as a “geometric center” of the node. New or modified data is routed to the node which has stored data identical or most similar to the new or modified data, as determined based on those values. Each node stores a plurality of chunks of data, where each chunk includes multiple deduplication segments. A content hash is computed for each deduplication segment in each node, and a similarity hash is computed for each chunk from the content hashes of all segments in the chunk. A geometric center of a node is computed from the similarity hashes of the chunks stored in the node.

Abstract: An input data enlarging unit (100) derives a first enlargement unit output and a second enlargement unit output that are uniquely specified by input data (103) to output the same. The first enlargement unit output and the second enlargement unit output are elements of output data set B which forms a group. An ??? hash function calculation unit (101) receives as input the first enlargement unit output to calculate an h? function which is specified by hash-function-specifying data (104) and an element of the H? function set. The function set H? is such that the number of h?H ? which satisfies h(x)?h(y)=d for an arbitrary element d of the output data set B and two different elements x and y of the output data set B is equal to or smaller than |H?|·?. An adding unit (102) adds together the result of calculation of the function H? and the second enlargement unit output to output a result of the addition.

Abstract: A memory subsystem includes a master controller that includes a pseudo random bit sequence (PRBS) generator having a plurality of output taps and an exclusive-OR (XOR) unit. The memory subsystem also includes a memory device that is coupled to the master controller via a plurality of single ended bidirectional data paths. The master controller may scramble a plurality of data bits using the PRBS generator and the XOR unit prior to writing the plurality of data bits to the memory device. In addition, the master controller may perform an XOR between each bit of the plurality of data bits and a respective output tap of the PRBS generator prior to conveyance on a respective path of the plurality of single ended bidirectional data paths.

Abstract: A linked list traversal system identifies when a linked list has become inefficient, either through attack or an undue multiplicity of collisions. A data unit is parsed to extract a key. A first hash result associated with the key is calculated based on a first hash function. A first linked list is identified based on the first hash result. It is determined whether the first linked list has been compromised. A second hash result associated with the key is calculated based on a second hash function when the first linked list has been compromised. A second linked list is established based on the second hash result, where the second hash result is different from the first hash result.

Abstract: A method for managing a virtual memory system configured to allow multiple page sizes is described. Each page size has at least one table associated with it. The method involves maintaining entries in the tables to keep track of the page size for which the effective address is mapped. When a new effective address to physical address mapping needs to be made for a page size, the method accesses the appropriate tables to identify prior mappings for another page size in the same segment. If no such conflicting mapping exists, it creates a new mapping in the appropriate table. A formula is used to generate an index to access a mapping in a table.

Abstract: A data processing apparatus is provided comprising a plurality of master devices configured to issue memory access requests including virtual addresses. A memory management unit is configured to receive memory access requests and to translate a virtual address included in a memory access request from a requesting master device into a physical address indicating a storage location in memory. The memory management unit has an internal storage unit having a plurality of entries wherein indications of corresponding virtual address portions and physical address portions are stored. The memory management unit is configured to select an entry of the internal storage unit in dependence on the virtual address and an identifier of the requesting master device. Conflict between the master devices in their usage of the internal storage unit is thus avoided.

Abstract: A method for protecting a privilege level of a system management mode (SMM) of a computer system is disclosed. A SMM program is loaded into a special memory (SMRAM) area within a system memory of a computer. A first program, a second program, and a vector table are loaded into a general area of the system memory. Before the booting process of the computer has been completed, a reference hash value of the first program is determined by the SMM program, and the reference hash value is stored in the SMRAM area. A hash value of the first program is the computed by the SMM program. After the computer has been operating under an operating environment of an operating system, the computed hash value is compared to the reference hash value. When the computed hash value matches the reference hash value, the first program is called by the SMM program.

Abstract: Described embodiments provide a media controller that determines the size of a cache of data being transferred between a host device and one or more sectors of a storage device. The one or more sectors are segmented into a plurality of chunks, and each chunk corresponds to at least one sector. The contents of the cache are managed in a cache hash table. At startup of the media controller, a buffer layer module of the media controller initializes the cache in a buffer of the media controller. During operation of the media controller, the buffer layer module determines a number of chunks allocated to the cache. Based on the number of chunks allocated to the cache, the buffer layer module updates the size of the of the cache hash table.

Abstract: A lookup unit matrix combines a plurality of lookup units to provide a longest prefix match for a search key longer than the lookup unit's mapper key. A portion of the search key is provided to each of the plurality of lookup units in a single search request issued to the lookup unit matrix. Each lookup unit in the lookup unit matrix performs a multi-level search for the result value based on the portion of the search key forwarded as the mapper key and the result of a multilevel search in the previous lookup unit. The search results in a value corresponding to the search key stored in a single location in one of the lookup units.

Abstract: An exact match lookup system includes a hash function that generates a hash value in response to an input hash key. The hash value is used to retrieve a hash bucket index value from a hash bucket index table. The hash bucket index value is used to retrieve a plurality of hash keys from a plurality of hash bucket tables, in parallel. The retrieved hash keys are compared with the input hash key to identify a match. Hit logic generates an output index by concatenating the hash bucket index value with an address associated with the hash bucket table that provides the matching hash key. An exact match result is provided in response to the output index. A content addressable memory (CAM) may store hash keys that do not fit in the hash bucket tables.

Abstract: An amount of storage capacity used during mirroring operations is reduced by applying de-duplication operations to the mirror volumes. Data stored to a first volume is mirrored to a second volume. The second volume is a virtual volume having a plurality of logical addresses, such that segments of physical storage capacity are allocated for a specified logical address as needed when data is stored to the specified logical address. A de-duplication operation is carried out on the second volume following a split from the first volume. A particular segment of the second volume is identified as having data that is the same as another segment in the second volume or in the same consistency group. A link is created from the particular segment to the other segment and the particular segment is released from the second volume so that physical storage capacity required for the second volume is reduced.

Abstract: An apparatus, system, and method are disclosed for improved deduplication. The apparatus includes an input module, a hash module, and a transmission module that are implemented in a nonvolatile storage device. The input module receives hash requests from requesting entities that may be internal or external to the nonvolatile storage device; the hash requests include a data unit identifier that identifies the data unit for which the hash is requested. The hash module generates a hash for the data unit using a hash function. The hash is generated using the computing resources of the nonvolatile storage device. The transmission module sends the hash to a receiving entity when the input module receives the hash request. A deduplication agent uses the hash to determine whether or not the data unit is a duplicate of a data unit already stored in the storage system that includes the nonvolatile storage device.

Abstract: The application describes a data processor operable to process data, and comprising: a cache in which a storage location of a data item within said cache is identified by an address, said cache comprising a plurality of storage locations and said data processor comprising a cache directory operable to store a physical address indicator for each storage location comprising stored data; a hash value generator operable to generate a generated hash value from at least some of said bits of said address said generated hash value having fewer bits than said address; a buffer operable to store a plurality of hash values relating to said plurality of storage locations within said cache; wherein in response to a request to access said data item said data processor is operable to compare said generated hash value with at least some of said plurality of hash values stored within said buffer and in response to a match to indicate a indicated storage location of said data item; and said data processor is operable to access

Abstract: Backup and restore technology comprising a backup engine, one or more client backup modules coupled to the backup engine via a backup protocol, and a backup database coupled to the backup engine, the backup database including a set of clusters, the set of clusters forming one or more backups, wherein each cluster of the set of clusters is unique such that single-instance storage across clients is achieved.

Abstract: Methods and systems for targeted data delivery are described. A user profile that includes information about a user is accessed. A root hash of a hash-based directed acyclic graph (HDAG) is computed. The HDAG includes hashed values of items of information in the user profile. The root hash is used in proving that the user profile satisfies selection criteria associated with an offer to deliver data. The user is eligible to be presented with the offer of data provided the user profile satisfies the selection criteria. The data is targeted to the user based on the user profile without requiring a release of any of the information in the user profile.

Abstract: A method and system for reinserting a chain into a hash table of a computer system are described. The chain includes a plurality of keys. The hash table includes a key fields and link fields associated with the key fields. The hash table utilizes an initial collision resolution scheme that precludes coalescing of chains. The method and system include providing a binary tree including nodes corresponding to the keys of the chain. The nodes include a root node corresponding to a root key and at least one child node corresponding to at least one child key. The method and system also include storing a pair of child keys in two empty adjacent key fields and storing a first empty adjacent key value of one of the two empty adjacent key fields in the home link field if the root node is the parent node of more than one child node.

Abstract: In one embodiment, a method includes accessing a condition test vector, selecting a key from a plurality of keys, and determining whether the key selected and a condition value satisfy a condition relation. The accessing being based on an index value. The condition test vector including a first plurality of bit values defining the condition relation, a second plurality of bit values defining a key selector, and a third plurality of bit values defining the condition value. The selecting being based on the second plurality of bit values. Each key from the plurality of keys including a combination of bit values representing a portion of a data packet. A result is defined based on the determining.

Abstract: Aspects of a method and system for hash table based routing via table and prefix aggregation are provided. Aspects of the invention may enable aggregating prefixes of varying lengths into a single hash table, wherein each entry in the hash table comprises one or more encoded bits to uniquely identify said prefixes. Additionally, an entry in a hash table may be formatted based on a length of one or more representations of said prefixes in the entry. Aggregating prefixes into a hash table may comprise truncating the prefixes to a common length. In this regard, the encoded bits may indicate the length of the prefixes prior to and/or subsequent to truncation. Additionally, the encoded bits may represent bits removed from the prefix during truncation. In this regard, an encoded bit may represent a possible combination of removed bits and may be asserted when the removed bits are equal to that combination.

Abstract: A request is received from a requester to run a first software module at a first privilege level, where the requester is associated with a second privilege level, and wherein the first privilege level is higher than the second privilege level. It is determined whether the first software module is valid to run at the first privilege level by checking for predefined content associated with the first software module. The first software module is run at the first privilege level on the computer in response to detecting the predefined content.

Abstract: A non-transitory computer-readable medium storing an data storing program executed by an archive device including a first storage unit for storing data and a second storage unit for storing hash value determined from the data, the program causing the archive device to execute a process includes receiving hash value determined from data to be stored from an external device which requests storage of the data, comparing the received hash value with the hash value stored in the second storage unit, and transmitting request information for transmitting the data corresponding to the received hash value to the external device which transmits the hash value when the received hash value has not been stored in the second storage unit.

Abstract: A method and apparatus for managing memory allocation using memory pages. A first physical memory page is compared with a second physical memory page, wherein the first physical memory page is associated with a first page table and the second physical memory page is associated with a second page table. If the second physical memory page matches the first physical memory page, the second physical memory page is deallocated, and the second page table is associated with the first physical memory page.

Abstract: One aspect of the present invention includes a configuration of a storage management system that enables the performance of deduplication activities at both the client (source) and at the server (target) locations. The location of deduplication operations can then be optimized based on system conditions or predefined policies. In one embodiment, seamless switching of deduplication activities between the client and the server is enabled by utilizing uniform deduplication process algorithms and accessing the same deduplication index (containing information on the hashed data chunks). Additionally, any data transformations on the chunks are performed subsequent to identification of the data chunks. Accordingly, with use of this storage configuration, the storage system can find and utilize matching chunks generated with either client- or server-side deduplication.

Abstract: A compression device recognizes patterns of data and compressing the data, and sends the compressed data to a decompression device that identifies a cached version of the data to decompress the data. Both the compression device and the decompression device cache the data in packets they receive. Each device has a disk, on which each device writes the data in the same order. The compression device looks for repetitions of any block of data between multiple packets or datagrams that are transmitted across the network. The compression device encodes the repeated blocks of data by replacing them with a pointer to a location on disk. The decompression device receives the pointer and replaces the pointer with the contents of the data block that it reads from its disk.

Abstract: A “Vector Graphics Encoder” encodes vector graphics in a randomly accessible format. This encoding format enables particular portions of encoded images to be directly accessed, at any desired level of zoom, without processing or otherwise decoding the entire image. This random-access format is based on a coarse image grid of partially overlapping cells wherein each cell is defined by a “texel program.” Unlike fixed-complexity cells used by conventional vector images, each cell defined by a texel program is locally specialized without requiring global constraints on the complexity of each cell. The texel program for each cell is provided as a variable-length string of tokens representing a locally specialized description of one or more of layers of graphics primitives overlapping the cell. Images are then rendered by interpreting the texel programs defining one or more cells.

Abstract: An amount of storage capacity used during mirroring operations is reduced by applying de-duplication operations to the mirror volumes. Data stored to a first volume is mirrored to a second volume. The second volume is a virtual volume having a plurality of logical addresses, such that segments of physical storage capacity are allocated for a specified logical address as needed when data is stored to the specified logical address. A de-duplication operation is carried out on the second volume following a split from the first volume. A particular segment of the second volume is identified as having data that is the same as another segment in the second volume or in the same consistency group. A link is created from the particular segment to the other segment and the particular segment is released from the second volume so that physical storage capacity required for the second volume is reduced.

Abstract: Methods for creation of a content association system in which a client application communicates to a distributed hash table network an association between one piece of content in a client's library and one other piece of content in the client's library are described. The client creates a hash for a first file in the library and a reference to a second file in the library. The client stores the reference to the second file in the distributed hash table at a node corresponding to the hash. To discover files associated with a file in a peer's library, the peer creates a hash for a file in its library and retrieves references to other files from the distributed hash table.

Abstract: A system and method for space and time efficient bound calculation is disclosed. The method comprises inserting a plurality of key/value pairs into a “Bloom bounder”, each key/value pair comprising a key and a value. For each pair, the inserting includes calculating a plurality of hash values, each calculated by applying a different one of a plurality of hash functions to the key, and selectively updating one or more data arrays based on the plurality of hash values and the value received key/value pair. A bound may then be determined for a given query key by analyzing information in the one or more data arrays to determine a bound value, such that for every received key/value pair with a key matching the query key, the corresponding value is less than or equal to the bound value.

Abstract: A system, method, and computer-readable medium for optimized processing of queries that feature maximum or minimum equality conditions are provided. The disclosed mechanisms provide for a single-scan of the table on which the group-by query is applied. When the table is scanned, each processing module dynamically keeps track of the row(s) having a value of the attribute on which the equality condition is applied that equals or exceeds the maximum attribute value (assuming a maximum equality condition is applied) previously encountered by the processing module. Subsequently, a global aggregation process is then performed to compute the query's result without rescanning the table. Queries featuring a minimum equality condition are similarly processed in accordance with the disclosed embodiments.