Abstract: A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by transforming the codebook into a hierarchical representation and then traversing the representation scanning for specific codewords associated with a data query request. In an embodiment, an estimator module is present and configured to utilize cardinality estimation to determine a starting codeword to begin searching the compacted data file for the data associated with the data query. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.

Abstract: A system and method for event-driven data communication using codebooks with protocol prediction and translation. This invention presents an advanced adaptive communication system that dynamically optimizes network protocols using machine learning-driven prediction and translation modules. The system analyzes real-time traffic patterns and historical data to anticipate communication needs, proactively switching to optimal protocols when beneficial. A sophisticated translation module, powered by large language models, enables seamless communication between systems using different protocols, including legacy systems. This approach enhances network efficiency, ensures backward compatibility, and future-proofs communication infrastructures, making it particularly valuable in complex, heterogeneous network environments.

Abstract: A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system further comprises a signature generating component which correlates anomalous event data with known vulnerabilities and exploits to create a signature based on statistical information of the anomalous event. Computed statistics may be compared against a signature database to determine if a data intrusion has occurred.

Abstract: A system and methods for bandwidth-efficient data encoding comprising a sequence analyzer configured to: analyze a received sequence dataset to determine a sequence dataset file type, scan the sequence dataset to maintain a count of unique characters contained therein, identify positions where the unique character count increases by a power of two, deconstruct the sequence dataset into a plurality of sourceblocks at the identified positions, and encode the plurality of sourceblocks using a data deconstruction engine and library management module to assign each sourceblock a reference code.

Abstract: A system and method for implementing multilayer security in documents using multiple codebooks. The system encodes different sections of a document using codebooks associated with specific security clearance levels. Users can only access sections of the document for which they have appropriate clearance, with unauthorized sections appearing redacted. The system provides granular, section-level security while also offering data compression benefits. Additional features include dynamic reclassification, hierarchical codebooks, expiring access, multi-factor authentication, and user-specific watermarking.

Abstract: A system and method for securing high-speed communications between processing units on computer chips, wherein a training data set is used to find patterns and associated smaller indices, or codewords, which are stored in a reference codebook library, and where reconstruction and deconstruction algorithms are used to encode and decode data as it is received. The codebook and algorithms may be stored in the firmware of a semiconductor which enable reduced resources and cost when transmitting data between or among devices that utilize such semiconductors.

Abstract: A system and method for securing high-speed communications between processing units on computer chips, wherein a training data set is used to find patterns and associated smaller indices, or codewords, which are stored in a reference codebook library, and where reconstruction and deconstruction algorithms are used to encode and decode data as it is received. The codebook and algorithms may be stored in the firmware of a semiconductor which enable reduced resources and cost when transmitting data between or among devices that utilize such semiconductors.

Abstract: A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system further comprises a signature generating component which correlates anomalous event data with known vulnerabilities and exploits to create a signature based on statistical information of the anomalous event. Computed statistics may be compared against a signature database to determine if a data intrusion has occurred.

Abstract: A system and method for data compression with encryption, that produces a conditioned data stream by replacing data blocks within an input data stream to bring the frequency of each data block closer to an ideal value, produces an error stream comprising the differences between the original data and the encrypted data, and compresses the conditioned data.

Abstract: A system and methods for bandwidth-efficient data encoding comprising a sequence analyzer configured to: analyze a received sequence dataset to determine a sequence dataset file type, scan the sequence dataset to maintain a count of unique characters contained therein, identify positions where the unique character count increases by a power of two, deconstruct the sequence dataset into a plurality of sourceblocks at the identified positions, and encode the plurality of sourceblocks using a data deconstruction engine and library management module to assign each sourceblock a reference code.

Abstract: A system and method for random-access manipulation of compacted data files, utilizing a reference codebook, a random-access engine, a data deconstruction engine, and a data deconstruction engine. The system may receive a data query pertaining to a data read or data write request, wherein the data file to be read from or written to is a compacted data file. A random-access engine may facilitate data manipulation processes by transforming the codebook into a hierarchical representation and then traversing the representation scanning for specific codewords associated with a data query request. In an embodiment, an estimator module is present and configured to utilize cardinality estimation to determine a starting codeword to begin searching the compacted data file for the data associated with the data query. The random-access engine may encode the data to be written, insert the encoded data into a compacted data file, and update the codebook as needed.

Abstract: A system and method for event-driven data communication using codebooks with protocol prediction and translation. This invention presents an advanced adaptive communication system that dynamically optimizes network protocols using machine learning-driven prediction and translation modules. The system analyzes real-time traffic patterns and historical data to anticipate communication needs, proactively switching to optimal protocols when beneficial. A sophisticated translation module, powered by large language models, enables seamless communication between systems using different protocols, including legacy systems. This approach enhances network efficiency, ensures backward compatibility, and future-proofs communication infrastructures, 5 10 making it particularly valuable in complex, heterogeneous network environments.

Abstract: A system and method are disclosed for low-light image enhancement using denoising preprocessing with wavelet decomposition AI-based techniques to enhance image quality of low-light images. Subsampled images are created from a raw input image. A wavelet decomposition process is performed on each subimage to create multiple frequency domain subimages. Each frequency domain subimage is input into a corresponding neural network. The output of each corresponding network is input to an inverse wavelet module. The output of the inverse wavelet module is a denoised image that is input to an image signal processing pipeline, where additional processing may be performed on the denoised image.

Abstract: A system and method for codebook management is disclosed. Training datasets are obtained from various data sources. A similarity score is generated for each training dataset with reference to the other training datasets. In response to detecting a similarity score above a predetermined threshold for one or more of the other training datasets, a combined codebook is created based on training datasets that have a similarity score above a predetermined threshold. Based on the similarity score, multiple data sources are combined into a group, and the combined codebook is used for the data sources within the group. A mismatch performance metric can be computed for the combined codebook, and a revised combined codebook can be regenerated in response to the mismatch performance metric being above a predetermined threshold.

Abstract: Data compaction using codebooks, that applies delta encoding methods to entropy encoding methods to improve data compaction of entropy encoding methods under certain conditions and when compacting data having certain characteristics. Delta encoding may be applied to entropy encoding methods to further compact data sets by reducing the number of sourceblocks included in a codebook to those most commonly encountered in data to be encoded and, where mismatches occur during encoding, using delta encoding of bit differences with existing sourceblocks in the codebook rather than adding new sourceblocks to the codebook.

Abstract: A system and method for data compaction optimization which leverages a neural network to predict optimal block sizes for data encoding, enhancing efficiency and adaptability in various applications. It begins with data preprocessing, extracting features, and creating labeled datasets for training. The neural network architecture is carefully designed, allowing it to learn complex relationships between data characteristics and optimal block sizes. During training, the network is fine-tuned and optimized using appropriate loss functions and regularization techniques. Once deployed, it continuously monitors incoming data streams for shifts in data patterns and adapts predictions accordingly. By predicting multiple block sizes, the system accommodates diverse compression needs. This versatile system offers real-time adaptability, ensuring optimal encoding performance as data patterns evolve over time.

Abstract: Compression of medical imaging data using codebooks and entropy encoding. Medical imaging data such as tomosynthesis imagery data may be compressed using codewords based on frequency analysis. In an implementation sequential registration technique may be applied to the medical imaging data to create a plurality transformation matrices. The plurality of transformation matrices may be compressed using a matrix codebook. The compressed medical imaging data may be represented as an image codebook and the matrix codebook, providing secure storage and lossless compression of sensitive medical information.

Abstract: A system and methods for multi-type data compression or decompression with a virtual management layer, comprising. It incorporates a virtual management layer to organize incoming data types and select a compression or decompression system that utilizes a technique best suited for a particular data type. Associated data sets may be flagged prior to compression or decompression so that associated types may be preserved together after the compression or decompression process is complete. This approach allows each data type to be compressed or decompressed using a technique that is the most efficient for a particular data type. Additionally, the approach allows all information associated with a particular data set to be compressed or decompressed in some way.

Abstract: A system and method for codebook management is disclosed. Training datasets are obtained from various data sources. A similarity score is generated for each training dataset with reference to the other training datasets. In response to detecting a similarity score above a predetermined threshold for one or more of the other training datasets, a combined codebook is created based on training datasets that have a similarity score above a predetermined threshold. Based on the similarity score, multiple data sources are combined into a group, and the combined codebook is used for the data sources within the group. A mismatch performance metric can be computed for the combined codebook, and a revised combined codebook can be regenerated in response to the mismatch performance metric being above a predetermined threshold.

Abstract: A system and method for codebook management is disclosed. Training datasets are obtained from various data sources. A similarity score is generated for each training dataset with reference to the other training datasets. In response to detecting a similarity score above a predetermined threshold for one or more of the other training datasets, a combined codebook is created based on training datasets that have a similarity score above a predetermined threshold. Based on the similarity score, multiple data sources are combined into a group, and the combined codebook is used for the data sources within the group. A mismatch performance metric can be computed for the combined codebook, and a revised combined codebook can be regenerated in response to the mismatch performance metric being above a predetermined threshold.