Abstract: A system and method for setting a compression ratio for utilizing a compressed memory pool (which is backed by pinned memory) by a virtual memory manager (VMM). Compression of pages of corresponding segments can be tracked as part of a VMM paging algorithm that compresses pages to store in a compressed memory pool. A segment having pages with an average compression ratio below a threshold is identified. The identified segment pages are prevented from utilizing the compressed memory pool resulting in optimizing the use of the compressed memory pool.

Abstract: A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions.

Abstract: A hybrid deduplication system operates to detect variable-sized deduplication matches, while performing the storage deduplication on fixed-size segments of data. The hybrid deduplication system calculates unique identifiers for variable-sized sections of data within a data stream being written to a deduplicated data store. The hybrid deduplication system then compares those newly-calculated identifiers to identifiers of variable-sized sections of data that have already been stored within the deduplicated data store. If a match is found, the hybrid deduplication system identifies the location of each of the fixed-size data segment(s), already stored in the deduplicated data store, that include the identified variable-sized section of data.

Abstract: Document fragments are managed by the use of fragment objects. Fragment objects contain a reference to a portion of a source document. Source document deletion is controlled by storing the references to the source document by fragment objects so that no fragment object has a reference to a non-existent source document. A process for organizing document fragments within a document management system wherein source documents comprise duplicates of the document fragments includes replacing a duplicate fragment with a reference to a single fragment object the fragment object including a reference to a fragment of a source document and an object class method for deletion of the fragment object including instructions to remove the reference to the fragment object from the metadata of the source document to which the fragment object refers.

Abstract: A method of operating a data compression circuit includes receiving and storing a plurality of data blocks until a cache is full and writing the data blocks that have been stored in the cache to a buffer memory when the cache is full. The method also includes performing forced literal/literal encoding on each of the data blocks regardless of repetitiveness of each data block when the cache is full.

Abstract: A query may be provided to aggregators at hierarchical levels in an in-memory data storage module. The query may be provided to leaf nodes of the in-memory data storage module. The leaf nodes may execute the query, returning results of the query to the aggregators. One or more aggregations may be performed based on the results. In an embodiment, log entries associated with a logged event may be serialized and divided into distributed chunks for storage in the leaf nodes. A leaf node, from the leaf nodes, having storage capacity for a distributed chunk may be identified. The distributed chunk may be stored in the leaf node.

Abstract: A data compression apparatus includes a memory and a processor. The processor extracts a second character string as a matching string from a character string after a first character string in a character string of data before compression that is stored in the memory, the second character string being identical with the first character string, and identifies a length of the matching string, and a relative position indicating how many addresses the first character string precedes the second character string by. The processor extracts a third character string having a length that is less than the relative position from the extracted second character string. The processor encodes a length of the third character string. The processor encodes the relative position.

Abstract: A method and system for efficient processing of encoded data streams communicated through a wireless network are disclosed. A system may include a receiver component, an iterator component, and a return component. The receiver component receives an encoded message including a plurality of objects. The message may be encoded according to a protocol. The iterator component traverses, without decoding the encoded message, the plurality of objects within the encoded message to locate a selected object of the plurality of objects. The selected object is selected by a client application. The return component returns data associated with the selected object to the client application.

Abstract: Although a video encoding device multiplexes ctu_pcm_flag syntax indicating whether all image blocks belonging to a coding tree unit as a set of image blocks are PCM blocks, at the top of the coding tree unit, when all image blocks belonging to the coding tree unit are PCM blocks, at least cu_split_flag syntax indicative of the splitting pattern of the coding tree unit is not multiplexed to transmit PCM data efficiently.

Abstract: Data replication with delta compression is disclosed. A primary system and a replica system are determined to both have an identical first data segment that is similar to a second data segment. The second data segment is encoded, wherein the encoding refers to the first data segment.

Abstract: A system and method for multiple dimensional encoding of a message, the method may include repeating, for each data unit that belongs to the message the stages of: (a) receiving by a processor the data unit; (b) executing by the processor, for each dimension of a plurality of dimensions of the multiple dimension, the stages of: (b.1) restoring a last state of an encoder during an encoding process of a packet that comprises the data unit, wherein the encoding process corresponds to the dimension; (b.2) encoding the data unit by the encoder in correspondence to the dimension to provide an updated state of the encoder; and (b.3) storing the updated state of the encoder as a last state of the encoder; wherein the updated state of the encoder following the encoding of all data units of the message represents redundancy bits; and (c) adding the redundancy bits to the message to provide a multiple dimensional encoded message.

Abstract: Systems, devices and methods for data compression using history search for dictionary based compression. Systems, devices and methods may use parallel processing techniques for data compression and encoding. Systems, devices and methods may provide memory search techniques for hardware.

Abstract: Base64 encoding and decoding advances are characterized. Some embodiments rely on only a single bit shifting operation; some have no reliance on bit-shifting. Some access data source bytes with multi-byte reads, and use those data source bytes as indexes into an encoding lookup table. Some use a table with four quadrants, each of which contains a copy of the same 64 encoding values. Some use a negative value as both a counter and an index. Some exclude BSWAP instructions even with a little-endian processor. Some partition a data source into N partitions, which are then encoded or decoded concurrently. In some, partition length is based on expected processor performance. Unrolled loops, exemplary assembly language code and coding choices, division using MagicNumber multiplication, terminal null addition, encoding/decoding applications, and other examples are also presented.

Abstract: In a compression processing storage system, using a pool of compression cores, the compression cores are assigned to process either compression operations, decompression operations, or decompression and compression operations, which are scheduled for processing. A minimal number of the compression cores are allocated for processing the compression operations, thereby increasing compression latency. Upon reaching a throughput limit for the compression operations that causes the minimal number of the plurality of compression cores to reach a busy status, the minimal number of the plurality of compression cores for processing the compression operations is increased.

Abstract: Provided are a computer program product, system, and method for generating a code alphabet for use by a deployed program to determine codewords for words. A first code alphabet has a first number of symbols that provide variable length codings of the words. A second code alphabet is generated having a second number of symbols formed by merging the symbols in the first code alphabet, wherein the second code alphabet comprises the code alphabet used by the deployed program.

Abstract: A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions.

Abstract: A method includes: setting a first and a second storage regions; first creating a first compression code of a compression target data in a file using a identifier indicating the data in the first storage region when a predetermined first consistency between the compression target data and the data in the first storage region is detected; comparing the compression target data with data in the second storage region when the predetermined first consistency between the compression target data and the data in the first storage region is not detected, the compression target data being moved to the second storage region after the comparing; and storing the compression target data into the first storage region associated with a identifier indicating the data in the first storage region when a predetermined second consistency between the compression target data and the data in the second storage region is detected.

Abstract: Systems, devices and methods for data compression using history search for dictionary based compression. Systems, devices and methods may use parallel processing techniques for data compression and encoding. Systems, devices and methods may provide memory search techniques for hardware.

Abstract: A system and method are disclosed for storing data in a hash table. The method includes receiving data, determining a location identifier for the data wherein the location identifier identifies a location in the hash table for storing the data and the location identifier is derived from the data, compressing the data by extracting the location identifier; and storing the compressed data in the identified location of the hash table.

Abstract: A method and system for data compression of images using a shared dictionary are described herein. According to one embodiment, a server identifies a current images that is part of a stream of images generated by an application hosted by the server. The stream of images includes images previously transmitted to the client. For each segment of pixels in the current image, the server searches a dictionary containing data for the stream of images. If the dictionary includes data corresponding to the segment of pixels in the current image, the server determines metadata for the segment of pixels in the current image using the corresponding data from the dictionary, and transmits the metadata to the client without transmitting the segment of pixels from the current image.

Abstract: Approaches for staged data compression are provided, where each stage reflects a progressive increase in granularity, resulting in a scalable approach that exhibits improved efficiency and compression performance. The first stage comprises a long-range block-level compressor that determines redundancies on a block-level basis (based on entire data blocks, as opposed to partial segments within data blocks). The second stage comprises a long-range byte-level compressor that compresses an uncompressed block based on byte segments within the block that match previously transmitted segments. The duplicate segments are replaced with pointers to matching segments within a decompressor cache. Nonmatching segments of the data block are left uncompressed and passed to a third stage short-range compressor (e.g., a grammar-based compressor). The staged progression in granularity provides advantages of maximizing the compression gain while minimizing processing and storage requirements of the compressor and decompressor.

Abstract: A system, method, and computer program for compressing packet data is provided. In exemplary embodiments, one or more prefix arrays may be generated for retrieved data, and used as the basis for predicting subsequent data. The packet data may be compressed based, at least partially, on the predicted subsequent data. Accordingly, the compressed packet data may be transferred over a communication network.

Abstract: A system to collect and store in a special data structure arranged for rapid searching massive amounts of data. Performance metric data is one example. The performance metric data is recorded in time-series measurements, converted into unicode, and arranged into a special data structure having one directory for every day which stores all the metric data collected that day. The data structure at the server where analysis is done has a subdirectory for every resource type. Each subdirectory contains text files of performance metric data values measured for attributes in a group of attributes to which said text file is dedicated. Each attribute has its own section and the performance metric data values are recorded in time series as unicode hex numbers as a comma delimited list. Analysis of the performance metric data is done using regular expressions.

Abstract: Image data is subject to compression and decompression when it is respectively written to and read from a frame buffer. If a portion of the image data is identified as static (subject to less than a threshold amount of change for greater than a threshold time), then compression control parameters used for compression of that portion of the image are adjusted so as to increase the compression ratio achieved, hold the degree of lossiness substantially constant and increase the energy consumed while compressing that portion. The increased energy consumption during this high compression ratio compression is likely compensated for by a reduction in energy subsequently consumed when writing that frame-buffer image data to the frame buffer and reading that frame-buffer image data multiple times from the frame buffer. The compression characteristics varied may be to increase the block size used in the compression.

Abstract: According to embodiments of the present invention, machines, systems, methods and computer program products for hierarchical compression of data are presented comprising creating a compression hierarchy of compression nodes, wherein each compression node is associated with a compression operation to produce compressed data. An output of any of the compression nodes may be compressed by another compression node or the same compression node. A path of one or more compression nodes is determined through said compression hierarchy based upon compression statistics to compress data, and the data is compressed by the compression nodes of the path. Various computational techniques are presented herein for manipulating the compression hierarchy to defer or reduce computation during query evaluation.

Abstract: An invention is disclosed for compressing data. In embodiments of the invention, input data is processed to determine one or more hints, or meta-data about how to compress the data. These hints may be produced based on the contents of the data itself, or by the process which generated the data. A hint may comprise an indication that a portion of the data matches another portion of the data, an indication that the portion of the data should not be compressed by an associated compressor, or an indication that the portion of the data should be compressed normally by the associated compressor. The source data and these hints may then be sent to a compressor, which traverses both the source data and the hints, and encodes the source data accordingly to produce an encoded data output stream.

Abstract: A method and system for securing data transmitted between a client device and a server by obtaining input text at an intermediate module, processing the input text to obtain processed text, and transmitting the processed text to the server. Embodiments of the invention include securing data between a client device and a server by processing the input text at the intermediate module by applying an order-preserving transformation, the order-preserving transformation comprising: generating order information based on the input text, the order information indicative of a relative order of the input text within a set of possible input texts according to a collation rule.

Abstract: Methods, apparatus, systems, and computer program products for utilizing a split dictionary are described. A mean and a standard deviation of a length of a plurality of records in a dictionary may be computed. A split condition may be determined based on the computed mean and the computed standard deviation. The dictionary may be split into a plurality of split dictionaries based on the split condition.

Abstract: A plurality of aligned or unaligned data packets are received in a data storage device. A data bundle is constructed by concatenating different ones of the plurality of unaligned data packets. Data loss protection identifiers are utilized to track the construction of the data bundle. The data loss protection identifiers are employed to prevent at least one of packet data loss or metadata loss in response to detecting a state reset of the data storage device.

Abstract: Systems and methods for transforming a logical data object for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimization and restoring thereof. Transforming the logical data object comprises creating in the storage device a transformed logical data object comprising one or more allocated storage sections with a predefined size; transforming one or more sequentially obtained chunks of obtained data corresponding to the transforming logical data object; and sequentially storing the processed data chunks into said storage sections in accordance with a receive order of said chunks, wherein said storage sections serve as atomic elements of transformation/de-transformation operations during input/output transactions on the logical data object.

Abstract: Techniques related to graphics rendering including techniques for compression and/or decompression of graphics data by use of pixel region bit values are described.

Abstract: A method and system for anonymizing data to be transmitted to a destination computing device is disclosed. Data to be transmitted is received from a user computer. The data includes one or more characters. The data is replaced with a token representative of the data. The token is transmitted to the destination computing device over a network.

Abstract: A communication channel structure and a decoding method supported by such a communication channel structure are disclosed. Such a communication channel includes a digital filter configured for filtering an input signal and two quantizer configured for quantizing the filtered signal. A first quantizer is utilized to quantize the filtered signal to produce a first quantized sample having a first precision and a second quantizer is utilized to quantize the filtered signal to produce a second quantized sample having a second precision, wherein the second precision is different from the first precision. The communication channel also includes an iterative decoder configured for utilizing the first quantized sample for a first global iteration of a decoding process and utilizing the second quantized sample for at least one subsequent global iteration of the decoding process.

Abstract: Embodiments of the present invention provide systems, methods, and computer storage media directed to hosting a plurality of copies of a digital content. A common component and one or more individual components from one or more copies of the digital content are generated. As such, the common component and the one or more individual components are stored, such that each individual component in conjunction with the common component represents a separate copy of the digital content. In some implementations, a compression ratio may be customized for determining the sizing of the common component and individual component.

Abstract: A method includes performing, by a microprocessor of a computing device, a coding to obtain a residual of a residual data block from an original data block. The method includes pre-defining a default order of symbols in the residual data block. The method includes re-ordering the symbols in the residual data block in an order other than the default order, such that a first set of most frequently used symbols are in a first location and a second set of least frequently used symbols are in a second location, producing a multi-symbol run-length code based on a group of symbols from the first set of most frequently used symbols, and detecting the second set of least frequently used symbols based on the multi-symbol run-length code.

Abstract: Systems and methods for compressing a raw logical data object (201) for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimizatic in and restoring thereof. Compressing the raw logical data object (201) comprises creating in the storage device a compressed logical data object (203) comprising a header (204) and one or more allocated compressed sections with predefined size (205-1-205-2); compressing one or more sequentially obtained chunks of raw data (202-1-202-6) corresponding to the raw logical data object (201) thus giving rise to the compressed data chunks (207-1-207-6); and sequentially accommodating the processed data chunks into: said compressed sections (205-1-205-2) in accordance with an order said chunks received, wherein said compressed sections serve as atomic elements of compression/decompression operations during input/output transactions on the logical data object.

Abstract: Systems and methods for iterative near lossless image compression are provided. An exemplary computer-implemented method of compressing image data can include performing a plurality of compression iterations. Each compression iteration can include at least one decision regarding a loss of image data. The method can also include updating an entropy model following each compression iteration. The entropy model can describe an entropy associated with selected of a plurality of data blocks. The at least one decision regarding the loss of image data can be decided, for each compression iteration, based on the entropy model as updated following the previous compression iteration. Further, a total loss of image data from each data block can remain within an acceptable loss bound associated with the pixel described by such data block. An exemplary system can include a loss determination module, an entropy modeling module, a compression module, and an entropy coding module.

Abstract: A maximum a posteriori (MAP) decoding apparatus and method are provided, which select a path having a maximum value without calculating all path metrics by generating metric difference vectors based on generated tables, calculating metrics of a path formed through a combination of a state and a branch each having a maximum metric, and calculating an index and a metric of a current state to which a calculation value is to be applied based on a result of comparing a state metric difference vector and a branch metric difference vector.

Abstract: A re-encryptor compares hashed digests of updated segments and original segments to located changed segments that must be re-encrypted. A new initialization vector is input to a block cipher engine for each changed segment. Since only changed segments need to be re-encrypted, transmission bandwidth to remote encrypted storage may be reduced. The amount of cipher text that is changed by a single update is reduced to a segment. Segments have a variable length and are bound by bits matching a segment delimiter. Each segment may have many fixed-length blocks that are encrypted by the block cipher engine with the same initialization vector for that segment. The segment delimiter is a randomly-generated word that is included with the initialization vectors in the metadata. Variable-length segments limit update disruption of the cipher text while fixed-length blocks are more efficiently encrypted. Combining segments and blocks provides for better re-encryption of updates.

Abstract: Unified binarization for CABAC/CAVLC entropy coding. Scalable entropy coding is implemented in accordance with any desired degree of complexity (e.g., entropy encoding and/or decoding). For example, appropriately implemented context-adaptive variable-length coding (CAVLC) and context-adaptive binary arithmetic coding (CABAC) allow for selective entropy coding in accordance with a number of different degrees of complexity. A given device may operate in accordance with a first level complexity a first time, a second level complexity of the second time, and so on. Appropriate coordination and signaling between an encoder/transmitter device and a decoder/receiver device allows for appropriate coordination along a desired degree of complexity.

Abstract: A non-transitory computer-readable storage medium comprising instructions stored thereon. When executed by at least one processor, the instructions may be configured to cause a computing system to at least determine, for each of a plurality of current symbols in a data block, frequencies of succeeding symbols within the data block, generate a plurality of clusters, each of the plurality of clusters including a subset of the plurality of current symbols, generate, for each of the clusters, a code, the code including variable length codewords for each of the succeeding symbols of the current symbols included in the respective cluster, and encode each of the succeeding symbols in the data block based on the code for the cluster that includes the succeeding symbol's respective current symbol.

Abstract: A disclosed data structure of a phrase tree includes, as data of each node in the phrase tree, a data block including a first area for holding a code for a character or a character string, a second area for holding, for each character that will be used, an appearance sequential number of a second character that appears next to the character or the character string, and a third area for holding, according to the appearance sequential number, a number of appearances or a code corresponding to a character string comprising the character or the character string and the second character when the number of appearances exceeds a threshold.

Abstract: A non-transitory computer-readable recording medium has stored therein a compression program that causes a computer to execute a process.

Abstract: A method is provided for compressing results of a join query. A join order of a result set is determined from the join query, where the result set includes a plurality of tuples. A plurality of dictionary entries for the result set is received. A nested hierarchy of dictionaries is created based on the join order and the dictionary entries. A plurality of encoded tuples is received. The nested hierarchy of dictionaries is used by a processor to decode the plurality of encoded tuples so as to produce the plurality of tuples of the result set.

Abstract: A computer implemented method of performing data compression includes applying, with a computing device, a hash function to a selected part of a character string to calculate a hash value; searching, using the hash value, through entries in a bucket chain having the hash value previously registered in a hash table, and finding a longest matching character string; acquiring, an index indicating that a longest matching character string cannot be found in the search through the entries and thus the search operation is wasted; and switching the hash function to a different hash function for expanding the selected part of the character string, without reconstructing the hash table, when the index exceeds a predetermined threshold.

Abstract: The present invention concerns coding an XML-type structured document. The structured document includes values to be coded. The coding uses at least one indexing dictionary which includes entries. At least one entry value of the dictionary linked to the value to be coded is identified, by matching between at least part of one of the values and a subpart of the other value. The value to be coded is coded by reference, according to the matching, to at least the identified entry of the dictionary.

Abstract: Concurrently writing an uncompressed data element, if the uncompressed data element comprises an indication that it is valid, in a main hash table using a first address generated by a first hash function, and reading a first data element from the main hash table using the first address. Introducing a first pipeline delay for maintaining the uncompressed data element in a first data path until the first data element is read. Concurrently writing the first data element to a victim hash table, if the first data element comprises an indication that it is valid, using a second address generated by a second hash function, and reading a second data element from the victim hash table using a third address generated by the second hash function. Introducing a second pipeline delay for maintaining the uncompressed data element in the first data path until the second data element is read.

Abstract: Methods and apparatus for processing bitstreams and byte streams. According to one aspect, bitstream data is compressed using coalesced string match tokens with delayed matching. A matcher is employed to perform search string match operations using a shortened maximum string length search criteria, resulting in generation of a token stream having <len, distance> data and literal data. A distance match operation is performed on sequentially adjacent tokens to determine if they contain the same distance data. If they do, the len values of the tokens are added through use of a coalesce buffer. Upon detection of a distance non-match, a final coalesced length of a matching string is calculated and output along with the prior matching distance as a coalesced token.

Abstract: An output sequence of data elements is processed. The output sequence of data elements represents a sequence of input data elements in a compressed format. An output data element comprises a backward reference for each string in the input data elements that occurs again in an input data element that is used to produce the output data element. A backward reference identified in a selected output data element is used for selecting the string to which it refers in the stored input data elements. The selected string is combined with strings of one or more subsequent output data elements. A matching sequence in the stored input data elements matching at least part of one or more combined strings is found. A combined string of the one or more combined strings having the longest matching sequence is selected. The backward reference is redefined to indicate the longest matching sequence.

Abstract: A method of operating a data compression device includes analyzing data using an analyzer and generating a result of the analysis, while the data is buffered by an input buffer, and selectively compressing the buffered data according to the result of the analysis. A data compression device includes a data pattern analyzer configured to analyze data transmitted to an input buffer, and generate an analysis code based on the analysis of the data; and a data compression manager configured to selectively compress the data in the input buffer based on the analysis code.