Abstract: A system and method for data compression with protocol adaptation, that utilizes a codebook generator which leverages one or more machine/deep learning algorithms trained on at least a plurality of protocol policies in order to generate a protocol appendix and codebook, wherein original data is encoded by an encoder according to the codebook and sent to a decoder, but instead of just decoding the data according to the codebook to reconstruct the original data, data manipulation rules such as mapping and transformation are applied at the decoding stage to transform the decoded data into protocol formatted data.

Abstract: A system and method for encoding personal health monitor data using a plurality of encoding libraries. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction or on some other criteria for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.

Abstract: A system and method for data compression with homomorphic encryption, which enables secure storage of private information in a database, and which enables searching and comparison of encrypted data within the database, comprising a stream condition system configured to optimize the contents of received data for lossless compression by a data encoder, a data encoder to perform the lossless compression, and an encrypted search engine configured to encrypt the compressed data according to a homomorphic encryption scheme and store the encrypted data in a database. The system may receive a data query and encrypt the data query according to the homomorphic encryption scheme. The encrypted data query may be compared against an encrypted element in the database and an encryption score generated. The encryption score may be compared against a set of criteria to determine if a match is found. Matched data may be returned to the requesting entity.

Abstract: A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system further comprises a signature generating component which correlates anomalous event data with known vulnerabilities and exploits to create a signature based on statistical information of the anomalous event. Computed statistics may be compared against a signature database to determine if a data intrusion has occurred.

Abstract: Compression of medical imaging data using codebooks and entropy encoding. Medical imaging data such as tomosynthesis imagery data may be compressed using codewords based on frequency analysis. In an implementation sequential registration technique may be applied to the medical imaging data to create a plurality transformation matrices. The plurality of transformation matrices may be compressed using a matrix codebook. The compressed medical imaging data may be represented as an image codebook and the matrix codebook, providing secure storage and lossless compression of sensitive medical information.

Abstract: Techniques for compressing data that use mismatch probability estimation to improve entropy encoding methods to account for, and efficiently handle, previously unseen data in data to be compressed. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Entropy encoding is used to generate codebooks comprising codewords for data sourceblocks based on the frequency of occurrence of each sourceblock. A “mismatch codeword” is inserted into the codebook based on the mismatch probability estimate to represent those cases when a block of data to be encoded does not have a codeword in the codebook.

Abstract: A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by transforming the codebook into a hierarchical representation and then traversing the representation scanning for specific codewords associated with a data query request. In an embodiment, an estimator module is present and configured to utilize cardinality estimation to determine a starting codeword to begin searching the compacted data file for the data associated with the data query. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.

Abstract: A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams.

Abstract: A system and method for data compaction with codebook statistical estimates to improve entropy encoding methods to account for, and efficiently handle, previously-unseen data in data to be compacted. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Entropy encoding is used to generate codebooks comprising codewords for data sourceblocks based on the frequency of occurrence of each sourceblock. A “mismatch codeword” is inserted into the codebook based on the mismatch probability estimate to represent those cases when a block of data to be encoded does not have a codeword in the codebook. During encoding, if a mismatch occurs, a secondary encoding process is used to encode the mismatched sourceblock.

Abstract: A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by accessing a reference codebook associated with the compacted data file, a frequency table used to construct the reference codebook, and data query details. A data read request is supported by random-access search capabilities that may enable the locating and decoding of the bits corresponding to data query details. A random-access engine facilitates data write processes. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.

Abstract: A system and method for extracting data from a compressed and encrypted data stream employing lossless compression and a codeword-to-byte mapping enables selective random extraction from compressed and encrypted data. Initially, data is compressed to reduce its size without information loss. A codeword-to-byte mapping table is created, associating codewords with specific positions in the compressed data. When a request for a specific byte range is received, the system uses the mapping to identify the codewords linked to that range within the compressed data. The system then selectively decompresses the identified codewords and subsequently decrypts the decompressed data. This process ensures that only the relevant data is processed, eliminating the need to decompress and decrypt the entire dataset.

Abstract: A system and method for encoding personal health monitor data using a plurality of encoding libraries. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction or on some other criteria for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.

Abstract: A system and method for data compression with protocol adaptation, that utilizes a codebook generator which leverages one or more machine/deep learning algorithms trained on at least a plurality of protocol policies in order to generate a protocol appendix and codebook, wherein original data is encoded by an encoder according to the codebook and sent to a decoder, but instead of just decoding the data according to the codebook to reconstruct the original data, data manipulation rules such as mapping and transformation are applied at the decoding stage to transform the decoded data into protocol formatted data.

Abstract: A system and method for codebook-based data encoding. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.

Abstract: A system and method for distributed node-based data compaction. The system uses machine learning on data chunks to generate codebooks which compact the data to be stored, processed, or sent with a smaller data profile than uncompacted data. The system uses a data compaction in an existing blockchain fork or implemented in a new blockchain protocol from which nodes that wish to or need to use the blockchain can do so with a reduced storage requirement. The system uses network data compaction across all nodes to increase the speed of and decrease the size of a blockchain's data packets. The system uses data compaction firmware to increase the efficiency at which mining rigs can computationally validate new blocks on the blockchain. The system can be implemented using any combination of the three data compaction services to meet the needs of the desired blockchain technology.

Abstract: A system and method for scriptable selective obfuscation of records comprising a data obfuscation module configured to identify and perform data anonymization on personal identifiable information (PII) contained within a plurality of records to create a partially-blurred dataset, and further comprising an encoder which receives the partially-blurred dataset and performs data compaction on the partially-blurred dataset before storing the compacted dataset in a data storage system. In some implementations, the data storage system is a blockchain database and the system functions as a clearinghouse to validate and monitor transactions involving data access rights between record owners and third-party entities. The system can further broker such transactions and direct payment form the third-party entity to the record owner when access rights have been purchased.

Abstract: A system and method for data compression with homomorphic encryption, which enables secure storage of private information in a database, and which enables searching and comparison of encrypted data within the database, comprising a stream condition system configured to optimize the contents of received data for lossless compression by a data encoder, a data encoder to perform the lossless compression, and an encrypted search engine configured to encrypt the compressed data according to a homomorphic encryption scheme and store the encrypted data in a database. The system may receive a data query and encrypt the data query according to the homomorphic encryption scheme. The encrypted data query may be compared against an encrypted element in the database and an encryption score generated. The encryption score may be compared against a set of criteria to determine if a match is found. Matched data may be returned to the requesting entity.

Abstract: A system and methods for data compression with encryption using asymmetric codebooks, that utilizes machine learning to generate asymmetric encoder and decoder codebooks for data encoding and compression. The use of asymmetric codebooks enables a variety of data obfuscation techniques such that output data remains useful to the decoder and any downstream entity, while concealing the original sourceblock data. The asymmetric codebooks may be utilized in reverse order to restore the original data as needed, facilitating a two-way reversible operation.

Abstract: A system and method for manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by accessing a reference codebook associated with the compacted data file, a frequency table used to construct the reference codebook, and data query details. A data read request is supported by random-access search capabilities that may enable the locating and decoding of the bits corresponding to data query details. A random-access engine facilitates data write processes. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.

Abstract: A system and method for data compaction utilizing distributed codebook encoding to improve entropy encoding methods to account for, and efficiently handle, previously-unseen data in data to be compacted, allow for distributed encoding and decoding capabilities, and allow for parametrized codebook encoding methods. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Further, a codebook and a behavior codebook may both be maintained or altered in a distributed fashion across multiple devices or services, for widespread, or permission-based, or parametrized codebook encoding.