Abstract: Herein are spatially scalable techniques for anisotropic compression of shared entropy between alternate representations of same data. In an embodiment, a computer compresses an uncompressed independent column into a compressed independent column. Based on the compressed independent column, an uncompressed dependent column is compressed into a compressed dependent column. The compressed independent column and the compressed dependent column are stored in a same file. In an embodiment, a computer stores, in metadata, an encrypted private key for decrypting an encrypted column. The encrypted column and the metadata are stored in a file. A request to read the encrypted column is received. Based on a public key and the file, the encrypted private key is decrypted into a decrypted private key. The public key is contained in the request and/or the file. The request is executed by decrypting, based on the decrypted private key and the file, the encrypted column.

Abstract: Herein are spatially scalable techniques for anisotropic compression of shared entropy between alternate representations of same data. In an embodiment, a computer compresses an uncompressed independent column into a compressed independent column. Based on the compressed independent column, an uncompressed dependent column is compressed into a compressed dependent column. The compressed independent column and the compressed dependent column are stored in a same file. In an embodiment, a computer stores, in metadata, an encrypted private key for decrypting an encrypted column. The encrypted column and the metadata are stored in a file. A request to read the encrypted column is received. Based on a public key and the file, the encrypted private key is decrypted into a decrypted private key. The public key is contained in the request and/or the file. The request is executed by decrypting, based on the decrypted private key and the file, the encrypted column.

Abstract: Herein is a probabilistic indexing technique for searching semi-structured text documents in columnar storage formats such as Parquet, using columnar input/output (I/O) avoidance, and needing minimal storage overhead. In an embodiment, a computer associates columns with text strings that occur in semi-structured documents. Text words that occur in the text strings are detected. Respectively for each text word, a bitmap, of a plurality of bitmaps, that contains a respective bit for each column is generated. Based on at least one of the bitmaps, some of the columns or some of the semi-structured documents are accessed.

Abstract: Techniques are described herein for indexing personal information in columnar data storage format based files. In an embodiment, row groups of rows that comprise a plurality of columns are stored in a set of files. Each column of a row group is stored in a chunk of column pages in the set of files. A regular expression index that indexes a particular column in the set of files is stored for each row group. The regular expression index identifies column pages in the chunk of the particular column that include a particular column value that satisfies a regular expression specified in a query. The regular expression specified in the query in evaluated against the particular column using the regular expression index.

Abstract: Herein is a probabilistic indexing technique for searching semi-structured text documents in columnar storage formats such as Parquet, using columnar input/output (I/O) avoidance, and needing minimal storage overhead. In an embodiment, a computer associates columns with text strings that occur in semi-structured documents. Text words that occur in the text strings are detected. Respectively for each text word, a bitmap, of a plurality of bitmaps, that contains a respective bit for each column is generated. Based on at least one of the bitmaps, some of the columns or some of the semi-structured documents are accessed.

Abstract: Techniques are described herein for indexing personal information in columnar data storage format based files. In an embodiment, row groups of rows that comprise a plurality of columns are stored in a set of files. Each column of a row group is stored in a chunk of column pages in the set of files. A regular expression index that indexes a particular column in the set of files is stored for each row group. The regular expression index identifies column pages in the chunk of the particular column that include a particular column value that satisfies a regular expression specified in a query. The regular expression specified in the query in evaluated against the particular column using the regular expression index.

Abstract: Techniques are described for generating a map of operations to ingest external data. In an embodiment, DBMS receives a query schema that specifies an external data source and client-defined columns to arrange queried data from data fields of the external data source. DBMS generates an ingest schema for the queried data of the external data source such that the ingest schema specifies ingest data fields from the data fields of the external data source to be mapped to the one or more client-defined columns. Additionally, the DBMS identifies surplus data in the external data source, which has not been specified by the ingest schema and is different from the queried data defined by the client-requested query schema. The process not only ingests the ingest data fields from the external data source but also the surplus data from the external data source. In response to receiving a query according to a different query schema, DBMS generates a result set that based on the surplus data of the external data source.

Abstract: Techniques are described for generating a map of operations to ingest external data. In an embodiment, DBMS receives a query schema that specifies an external data source and client-defined columns to arrange queried data from data fields of the external data source. DBMS generates an ingest schema for the queried data of the external data source such that the ingest schema specifies ingest data fields from the data fields of the external data source to be mapped to the one or more client-defined columns. Additionally, the DBMS identifies surplus data in the external data source, which has not been specified by the ingest schema and is different from the queried data defined by the client-requested query schema. The process not only ingests the ingest data fields from the external data source but also the surplus data from the external data source. In response to receiving a query according to a different query schema, DBMS generates a result set that based on the surplus data of the external data source.

Abstract: According to one or more embodiments, multiple related data streams are compressed jointly, such that substantive similarities between the multiple related streams are leveraged to reduce the overall size of the resulting compressed data. Specifically, given the compressed version of a primary data stream, a secondary data stream may be highly compressed by utilizing portions of the compressed version of the primary data stream to represent similar portions of the secondary data stream. A compression application is configured to receive (either concurrently or separately) information identifying multiple related input streams. For each input stream provided to the compression application, the application outputs one output stream. The computing system writes the output streams of the compression application to storage. Embodiments allow the size of the compressed version of a secondary data stream to be greatly reduced compared to the size of the stream compressed without reference to a primary data stream.

Abstract: Herein are spatially scalable techniques for anisotropic compression of shared entropy between alternate representations of same data. In an embodiment, a computer compresses an uncompressed independent column into a compressed independent column. Based on the compressed independent column, an uncompressed dependent column is compressed into a compressed dependent column. The compressed independent column and the compressed dependent column are stored in a same file. In an embodiment, a computer stores, in metadata, an encrypted private key for decrypting an encrypted column. The encrypted column and the metadata are stored in a file. A request to read the encrypted column is received. Based on a public key and the file, the encrypted private key is decrypted into a decrypted private key. The public key is contained in the request and/or the file. The request is executed by decrypting, based on the decrypted private key and the file, the encrypted column.

Abstract: According to one or more embodiments, multiple related data streams are compressed jointly, such that substantive similarities between the multiple related streams are leveraged to reduce the overall size of the resulting compressed data. Specifically, given the compressed version of a primary data stream, a secondary data stream may be highly compressed by utilizing portions of the compressed version of the primary data stream to represent similar portions of the secondary data stream. A compression application is configured to receive (either concurrently or separately) information identifying multiple related input streams. For each input stream provided to the compression application, the application outputs one output stream. The computing system writes the output streams of the compression application to storage. Embodiments allow the size of the compressed version of a secondary data stream to be greatly reduced compared to the size of the stream compressed without reference to a primary data stream.

Abstract: Methods and systems for processing data packets enable a communication device having limited memory to participate in network protocols. One communicating device formats packets in accordance with the specifications of one or two or more communication protocols, with one or more additional constraints, and transmits it to an other communication device. The other communication device performs a specified processing of the received packet to generate a reply packet in response to the received packet, which conforms to the one of two or more communication protocols, while also satisfying one or more additional formatting constraints.

Abstract: A network (100) includes a first communication device (110) and a second communication device (120). The first communication device (110) receives a first packet containing information for the first communication device (110) and determines whether adequate resources exist for handling the packet. When inadequate resources are determined to exist, the first communication device (110) generates a second packet, stores the information in the second packet, and transmits the second packet. The second communication device (120) receives the second packet, generates a third packet, transfers the information to the third packet, and transmits the third packet to the first communication device (110). The first communication device (110) receives the third packet and processes the information in the third packet.

Abstract: Methods and systems for transmitting data packets enable a communication device having limited memory to transmit data files in a packet-based network. The communication device partitions the sequence number field transmitted with a data packet into multiple portions. The communication device then generates a sequence number corresponding to the portions, where at least one portion identifies a particular segment of the data file. The communication device then transmits a data packet including a segment of a data file and the sequence number to a receiving device. When the communication device receives the acknowledgement packet from the receiving device, the communication device may determine which segment of the data file to transmit next.

Abstract: A system predicts one or more fields of a packet that includes multiple fields. Each of the fields stores a value. The system receives one or more of the fields of the packet and determines the value of at least one of the received fields (710). The system predicts a value of at least one other one of the fields of the packet based on the value of the received field before the other field is received (715). The system then processes the packet based on the received fields and the predicted field(s) (720).

Abstract: A system for predicting one or more fields of a packet that includes multiple fields, the packet belonging to a set of packets broken up into one or more packets by the process of fragmentation or segmentation. Each of the fields stores a value. The system receives one or more of the fields of the packet and determines the value of at least one of the received fields. The system predicts a value of at least one other one of the fields of the packet based on the value of the received field before the other field is received. The system then processes the packet based on the received fields and the predicted field(s).