Abstract: A method for compressing columnar data may include generating, for a data column included in a data chunk, a dictionary enumerating, in a sorted order, a first set of unique values included in the first data column. A compression technique for generated a compressed representation of the data column having a fewest quantity of bytes may be identified based at least on the dictionary. The compression technique including a dictionary compression applying the dictionary and/or another compression technique. A compressed data chunk may be generated by applying the compression technique to compress the data column included in the data chunk. The compressed data chunk may be stored at a database in a variable-size persistent page whose size is allocated based on the size of the compressed representation of the data column. Related systems and articles of manufacture are also provided.

Abstract: One embodiment provides for a method of transmitting data between multiple compute nodes of a distributed compute system, the method comprising creating a global view of communication operations to be performed between the multiple compute nodes of the distributed compute system, the global view created using information specific to a machine learning model associated with the distributed compute system; using the global view to determine a communication cost of the communication operations; and automatically determining a number of network endpoints for use in transmitting the data between the multiple compute nodes of the distributed compute system.

Abstract: An operation method of a server for updating firmware includes: generating a first delta file including a plurality of blocks based on a plurality of update areas included in a first version firmware; generating a second delta file by repositioning the plurality of blocks included in the first delta file such that a plurality of unit blocks are generated by grouping control blocks, difference blocks, and extra blocks, each of which corresponds to the plurality of update areas, respectively; generating a plurality of swap blocks based on extra blocks among the plurality of blocks; and generating a third delta file by adding the generated plurality of swap blocks to the second delta file.

Abstract: A method and apparatus for compression units in an index block is provided. A particular leaf block of a database index for a table is generated. The particular leaf block corresponds to a particular range of key values. A compression scheme is determined for a plurality of index entries with key values within the particular range. A compression unit that includes a plurality of compressed index entries is generated by applying the compression scheme to the plurality of index entries. Compression metadata describing the compression scheme is generated. The compression unit and the compression metadata is stored in the particular leaf block. In response to a request, at least a portion of the compression unit is decompressed to access one or more index entries.

Abstract: Database systems may dynamically management concurrency levels for performing queries. A query may be received at a database system and a memory usage for the query may be predicted. A determination may be made as to whether available memory is enough to satisfy the predicted memory usage for the query. If the available memory is enough to satisfy the predicted memory usage for the query, then an increase in a concurrency level for performing queries at the database system may be made. The query may be allowed to execute concurrently with other queries according to the increased concurrency level.

Abstract: The invention relates to a method for packaging media and data for transmission over a broadband communications network link for display at a terminal destination. Features of the method included creating desired media segments for transmission to the terminal destination and for creating targeting information, for example in whole or in part in the form of metadata, for transmission to the terminal destination. Other features include packaging the created media segments and the created targeting information in a single package and unpacking the single package for display of the targeted media at the terminal destination.

Abstract: Systems and processes for operating an intelligent automated assistant are provided. An intelligent automated assistant receives a user input and generates a set of one or more token sequences based on the user input. The set of one or more token sequences are interpreted to generate a plurality of candidate interpretations each including a corresponding action and metadata associated with the action. A top candidate interpretation is selected from the plurality of candidate interpretations, and the corresponding action is performed based on the associated metadata.

Abstract: The present invention relates to a method and a system for maintaining consistency of data between data-sets stored in a master database of a master computing node and corresponding data-sets stored in a replication database of at least one replication computing node. Each time an updated version of a stored data set is received, the master computing node is configured for updating a corresponding data-set stored in the master database and transmitting replication data relating to the updated data-set version to the at least one replication computing node for replication. The replication data comprising a bit-vector storing data values extracted from data fields of the data-set classified as volatile data fields. The data values stored in the bit-vector representing the changes in the data-set to be replicated in the at least one replication node.

Abstract: A method provides for classifying data fields of a dataset. A classifier configured for determining confidence values for a plurality of data classes for the data fields may be applied. Using the confidence values, data class candidates may be identified. Data fields may be determined for which a plurality of data class candidates is identifiable. Using previous user-selected data class assignments, a probability may be determined for the data class candidates that the respective data class candidate is a data class to which the respective data field is to be assigned. The data fields may be classified using the probabilities to select for the data fields a data class from the data class candidates. The dataset may be provided with metadata identifying for the data fields the data classes to which the respective data fields are assigned.

Abstract: A system is provided for electronic data compression by automated time-dependent compression algorithm. In particular, the system may track instances in which a particular dataset is used, copied, or accessed over time. For certain datasets (e.g., datasets that have not been accessed for a threshold amount of time), the system may use a time-based compression algorithm that progressively removes the least significant bits of such datasets as time passes. The compression of the datasets may continue until the system detects that further compression would cause the dataset to be unreadable or unrecoverable. In this way, the system may minimize the computing resources allocated to storing datasets that are not frequently accessed.

Abstract: Systems and methods are disclosed for scalable bucket merging in a data intake and query system. Various components of a bucket manager can be used to monitor recently-created buckets of data in common storage that are associated with a particular tenant and a particular index, apply a comprehensive bucket merge policy to determine groups of buckets that qualify for merging, merge those group of buckets into merged buckets to be stored in the common storage, and update any information associated with the merged buckets and pre-merged buckets. These components may be shared across multiple tenants, and some of these components may be dynamically scalable based on need. This approach may also provide many additional benefits, including improved search performance from merged buckets, efficient resource utilization associated with discriminate merging, and redundancy in case of component failure.

Abstract: The invention discloses an aggregation optimized processing method for time-series data, characterized by comprising the following steps: writing a time-series data record into a database, forming a time-series database file, wherein the time-series database file comprises a data file and an index file, the data file comprises multiple data blocks, the index file comprises index blocks, and each index block correspond to one data block; by scanning an index file according to a start time period and a stop time period, extracting all index blocks of the time series that need to be aggregated that meet the time period conditions, and then sorting the index blocks according to the data block offset recorded in the index block; and by scanning the data file according to a data block offset order recorded in sorted index blocks, performing specified reading and calculating on each data block, and aggregating calculation results.

Abstract: Methods for creating emails with complex graphics and customized web fonts that are ensured to render consistently across multiple platforms and email client applications and systems that ensure consistent rendering of created emails with complex graphics and customized web fonts by automatically transforming, completely transparently, the contents of each created email to a form that renders properly across the platforms and email clients are disclosed.

Abstract: A columnar storage method and a query method and system for time series data. The storage method includes: dividing a column of time series data into a plurality of pages, wherein each page stores a part of data points of the column of time series data and the sum of the data points stored in all the pages is all the data points in the column of time series data (S1); and setting two parts, i.e., a page header and a page body, for each page, storing summary index information of all the data points in the page in the page header of the page and storing data value information of all the data points in the page in the page body of the page (S2).

Abstract: A system and method for highly efficient encoding of data that includes extended functionality for asymmetric encoding/decoding and network policy enforcement. In the case of asymmetric encoding/decoding the original data is encoded by an encoder according to a codebook and sent to a decoder, but the output of the decoder depends on data manipulation rules applied at the decoding stage to transform the decoded data into a different data set from the original data. In the case of network policy enforcement, a behavior appendix into the codebook, such that the encoder and/or decoder at each node of the network comply with network behavioral rules, limits, and policies during encoding and decoding.

Abstract: A method for automatically identifying applications that circumvent permissions. The method includes logging network traffic transmitted by one or more computing devices while the one or more computing devices execute one or more applications, identifying, based on analyzing the logged network traffic and permissions granted to the one or more applications, those of the one or more applications that caused permission-protected data to be transmitted without having permission to access that data as circumventing permissions, and generating a report indicating the applications that were identified as circumventing permissions.

Abstract: In some examples, a system computes respective values for corresponding data value indicators added to and removed from a deduplication data store in which duplicated data values have been eliminated, where each respective data value indicator of the data value indicators represents presence of a unique data value in the deduplication data store. The system updates an estimator based on the respective values, to reflect an addition of a first data value indicator to the deduplication data store and a removal of a second data value indicator from the deduplication data store. The system computes, using the updated estimator, a parameter relating to data deduplication at the deduplication data store.

Abstract: A computer system implements a file system migration by determining a destination namespace of a destination file system based at least in part on a namespace of a source file system. At least a portion of the destination namespace includes an organization that is altered as compared to a corresponding portion of the source namespace.

Abstract: Embodiments are disclosed for a method for private transfer learning. The method includes generating a machine learning model comprising a training application programming interface (API) and an inferencing API. The method further includes encrypting the machine learning model using a predetermined encryption mechanism. The method additionally includes copying the encrypted machine learning model to a trusted execution environment. The method also includes executing the machine learning model in the trusted execution environment using the inferencing API.

Abstract: The present disclosure involves systems, software, and computer implemented methods for compressed columnar data search using fingerprints. One example method includes compressing columnar data that includes dividing the columnar data into multiple data blocks and generating a fingerprint for each data block, storing the compressed columnar data and the generated fingerprints in an in-memory database, receiving a query for the columnar data, for each in-memory data block stored in the in-memory database, determining whether the in-memory data block satisfies the query and in response to a determination that the in-memory data block does not satisfy the query, pruning the in-memory data block from the multiple data blocks to generate an unpruned set of data blocks, decompressing the unpruned set of data blocks, and performing a query search on the decompressed unpruned set of data blocks for the received query.

Abstract: A primary storage system reads a plurality of journal data, and performs collective compression that is compression of data that is at least a part of the plurality of pieces of journal data in the plurality of journals and is larger than a size of one journal data. The collectively compressed data, which is a plurality of pieces of journal data subjected to collective compression, is a transfer target from the primary storage system to the secondary storage system. The journal is journal data and metadata including a write order of the journal data and associated with the journal data. The journal data is a copy of data written in the primary volume. The secondary storage system acquires a plurality of pieces of journal data by decompressing one or more pieces of collectively compressed data, and writes the plurality of pieces of journal data to the secondary volume.

Abstract: Systems and method for providing tier selection for data based on a weighted flash fragmentation factor. A weighted flash fragmentation factor is determined indicating a severity of fragmentation in a non-volatile storage based on a logical block address range in a logical-to-physical mapping table for data from a host device to be stored in the tiered data storage system. The factor is shared with the host device to determine a tier selection. The data is stored according to the tier selection based on the factor.

Abstract: A system and method for generating a predicted vehicle trajectory includes a generative adversarial network configured to receive a trajectory vector of a target vehicle and generate a set of latent state vectors using the received trajectory vector and an artificial neural network. The latent state vectors each comprise a high-level sub-vector, ZH. The GAN enforces ZH to be correlated to an annotation coding representing semantic categories of vehicle trajectories. The GAN selects a subset, from the set of latent state vectors, using farthest point sampling and generates a predicted vehicle trajectory based on the selected subset of latent state vectors.

Abstract: Efficient aggregation of time series data is disclosed, including: obtaining a first entry value corresponding to an item, wherein the first entry value comprises a first recorded data point that is associated with a first time interval; generating a compressed block based at least in part on compressing the first entry value with at least a second entry value; storing the compressed block in a document corresponding to the item; determining that the item matches an aggregation search query; decompressing the compressed block from the document corresponding to the item to obtain the first entry value and the second entry value; and generating an aggregation result in response to the aggregation search query based on at least a portion of the first entry value and the second entry value.

Abstract: Systems, apparatuses, and methods related to a computing tile are described. The computing tile may perform operations on received data to extract some of the received data. The computing tile may perform operations without intervening commands. The computing tile may perform operations on data streamed through the computing tile to extract relevant data from data received by the computing tile. In an example, the computing tile is configured to receive a command to initiate an operation to reduce a size of a block of data from a first size to a second size. The computing tile can then receive a block of data from a memory device coupled to the apparatus. The computing tile can then perform an operation on the block of data to extract predetermined data from the block of data to reduce a size of the block of data from a first size to a second size.

Abstract: A system and method for standardizing user interface elements are presented. A first application is identified having a higher use metric than a second application, the first application including one or more user interface elements that have one or more respective parameters. The second application has one or more user interface elements that are similar to the user interface elements of the first application and has one or more respective parameters that are different than the respective parameters of the user interface elements of the first application. A determination of similarity is made between the user interface elements applications based upon at least one predetermined criterion. Based on the determination, one or more parameters of the user interface elements of the second application are modified to match one or more parameters of the first application.

Abstract: A method of encoding input data includes dividing the input data into a plurality of data packets, an input packet of the plurality of data packets including a plurality of digits in a first base system, base-converting the input packet from the first base system to generate a base-converted packet including a plurality of converted digits in a second base system, the second base system having a base value lower than that of the first base system, and incrementing the converted digits to generate a coded packet for transmission through a communication channel.

Abstract: A method for data reduction may comprise computing (i) a first sketch of a first segment and (ii) a second sketch of a second segment. The first sketch and the second sketch may each comprise a set of features that are representative of or unique to the corresponding first and second segments. The method also comprise processing the first sketch and the second sketch to generate a similarity metric indicative of whether the second segment is similar to the first segment. The method may further comprise (1) performing a differencing operation on the second segment relative to the first segment when the similarity metric is greater than or equal to a similarity threshold, or (2) storing the first segment and the second segment in a database without performing the differencing operation when the similarity metric is less than the similarity threshold.

Abstract: A network node generates a reduced size textual network log by including a set of numerical values for a log entry within a textual network log for a network, the log entry constituting an instance of a recognizable pattern within the textual network log; and then outputs the reduced size textual network log to a network controller for configuring the network.

Abstract: Methods and apparatuses are provided for editing and generating a short video based upon a long video. The method includes: obtaining a plurality of short source video clips as candidate video clips; obtaining attributes of each short source video clip; obtaining a plurality of target base videos according to a target genre, processing the plurality of target base videos by splitting each target base video into a plurality of short target base video clips, and obtaining attributes of each short target base video clip; selecting short target video clips from the plurality of short source video clips, based on distribution of the attributes obtained for the plurality of the short source video clips and the plurality of short target base video clips; and editing and combining the short target video clips selected from the plurality of short source video clips, to obtain a target video.

Abstract: A database includes a plurality of physical entity records of insurance claims, and each physical entity record includes physical data values for at least one numeric attribute and partition-specifying values concerning values for one or more nominal attributes from one or more insurance claim records. A method for detecting anomalies in the database includes retrieving and partitioning the plurality of physical entity records from the database, training an unsupervised anomaly detection algorithm on the plurality of physical entity records to obtain a trained anomaly detection model for each partition, calculating an anomaly score for each physical entity record using the trained anomaly detection model associated with each partition, and updating the plurality of physical entity records in the database by adding the associated anomaly score. The method is used to determine if a user-provided physical entity record is fraudulent using the anomaly score.

Abstract: An apparatus to facilitate memory data compression is disclosed. The apparatus includes a memory and having a plurality of banks to store main data and metadata associated with the main data and a memory management unit (MMU) coupled to the plurality of banks to perform a hash function to compute indices into virtual address locations in memory for the main data and the metadata and adjust the metadata virtual address locations to store each adjusted metadata virtual address location in a bank storing the associated main data.

Abstract: Log messages are compressed, searched, and decompressed. A dictionary is used to store non-numeric expressions found in log messages. Both numeric and non-numeric expressions found in log messages are represented by placeholders in a string of log “type” information. Another dictionary is used to store the log type information. A compressed log message contains a key to the log-type dictionary and a sequence of values that are keys to the non-numeric dictionary and/or numeric values. Searching may be performed by parsing a search query into subqueries that target the dictionaries and/or content of the compressed log messages. A dictionary may reference segments that contain a number of log messages, so that all log message need not be considered for some searches.

Abstract: A hardware-implemented file reader includes an interface, multiple hardware-implemented column readers and a hardware-implemented record reconstructor. The interface is configured to access a file including multiple records. The records store values in accordance with a nested structure that supports optional values and repeated values. The file is stored in a columnar format having multiple columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column to the nested structure of the records. Each column reader is configured to be assigned to a respective selected column, and to read and decompress both the values and the structure information from at least a portion of the selected column. The record reconstructor is configured to reconstruct one or more of the records from at least portions of the columns that are read by the column readers, and to output the reconstructed records.

Abstract: A computer-implemented method for deduplicating records includes generating a block comprising of a group of records. The method also includes creating one or more record pairs from the block, and calculating one or more features based on one or more fields within the one or more record pairs. The method further includes inputting the one or more features into a machine language trained model to predict a probability score. The probability score indicates whether two records are duplicates. The method also includes storing the probability score as links between two vertices in a graph, and displaying one or more duplicate records by querying the graph.

Abstract: An apparatus and method for providing ML processing for one or more ML applications operating on one or more Internet of Things (IoT) devices includes receiving a ML request from an IoT device. The ML request can be generated by a ML application operating on the IoT device and include input data collected by the first ML application. A ML model to perform ML processing of the input data included in the ML request is identified and provided to an ML core for ML processing along with the input data included in the first ML request. The ML core produces ML processing output data based on ML processing by the ML core of input data included in the ML request using the ML model. The ML processing output data can be transmitted to the IoT device.

Abstract: Herein are spatially scalable techniques for anisotropic compression of shared entropy between alternate representations of same data. In an embodiment, a computer compresses an uncompressed independent column into a compressed independent column. Based on the compressed independent column, an uncompressed dependent column is compressed into a compressed dependent column. The compressed independent column and the compressed dependent column are stored in a same file. In an embodiment, a computer stores, in metadata, an encrypted private key for decrypting an encrypted column. The encrypted column and the metadata are stored in a file. A request to read the encrypted column is received. Based on a public key and the file, the encrypted private key is decrypted into a decrypted private key. The public key is contained in the request and/or the file. The request is executed by decrypting, based on the decrypted private key and the file, the encrypted column.

Abstract: Techniques and solutions are described for compressing data and facilitating access to compressed data. Compression can be applied to proper data subsets of a data set, such as to columns of a table. Using various methods, the proper data subsets can be evaluated to be included in a group of proper data subsets to be compressed using a first compression technique, where unselected proper data subsets are not compressed using the first compression technique. Data in the data set can be reordered based on a reordering sequence for the proper data subsets. Reordering data in the data set can improve compression when at least a portion of the proper data subsets are compressed. A data structure is provided that facilitates accessing specified data stored in a compressed format.

Abstract: Embodiments of the present disclosure provide to a method and a device for acquiring a data model in a knowledge graph, an apparatus and a storage medium. The method includes: receiving a knowledge entry describing a relationship between an entity and an object; determining a plurality of candidate object types of the object according to at least one of the entity, the relationship and the object; determining an object type for generating a data model that matches the knowledge entry from the plurality of candidate object types based on a preset rule; and generating the data model based at least on the object type.

Abstract: Various embodiments are generally directed to techniques for crowdsourcing metadata, such as to create, maintain, utilize, and/or communicate metadata in a datastore. Some embodiments are particularly directed to collecting, managing, and/or distributing metadata via a user interface application (UIA), or an extension thereof. Many embodiments may facilitate collaborative creation of a repository for metadata, such as notes, characteristics, and/or statistics corresponding to data viewed with the user interface application. For example, the comments regarding websites may be collected from a first user via a first instantiation of the web browser extension and distributed to a second user via a second instantiation of the web browser extension. In several embodiments, the metadata may be collected and managed with instantiations of the web browser extension (herein after “WBE instants”) and a data manager may create, maintain, utilize, and/or communicate metadata in a datastore.

Abstract: Fusing online and mapped terrain estimates by using weighted grid cells that scales the values returned from online terrain and mapped terrain is disclosed. Previously mapped terrain data and online terrain data are fused and a grid having cells of a predetermined size is overlaid on the terrain map. Each cell may include terrain data based on weighted mapped terrain data and weighted online terrain data, where the weighting values for the mapped terrain data and for the online terrain data may be different. A fused terrain estimate may be a result of a weighted mean for each cell smoothed to reduce noise.

Abstract: A method for compressing data in a local cache of a web server is described. A local cache compression engine accesses values in the local cache and determines a cardinality of the values of the local cache. The local cache compression engine determines a compression rate of a compression algorithm based on the cardinality of the values of the local cache. The compression algorithm is applied to the cache based on the compression rate to generate a compressed local cache.

Abstract: According to various embodiments, methods and systems are provided to accelerate artificial intelligence (AI) model training with advanced interconnect communication technologies and systematic zero-value compression over a distributed training system. According to an exemplary method, during each iteration of a Scatter-Reduce process performed on a cluster of processors arranged in a logical ring to train a neural network model, a processor receives a compressed data block from a prior processor in the logical ring, performs an operation on the received compressed data block and a compressed data block generated on the processor to obtain a calculated data block, and sends the calculated data block to a following processor in the logical ring. A compressed data block calculated from corresponding data blocks from the processors can be identified on each processor and distributed to each other processor and decompressed therein for use in the AI model training.

Abstract: A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and (3) organizing the data blocks within the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: A system and method, including determining, by a processor, a data type for each column of a database table; determining, by the processor and based on the determined data type for each column of the database table, an indication of a size of the database table; calculating, by the processor and based on the determined indication of the size of the database table, a start nbit size for a nbit compression process to be used on the database table; specifying, by the processor, the calculated start nbit size for the nbit compression process; and compressing the database table by executing the nbit data compression process using the specified start nbit size.

Abstract: Data to be stored in a data block for a columnar database table may be compressed according to a multi-level compression scheme. Data to be stored in the data block may be received. The data may be compressed according a column-specific compression technique to produce compressed data. The compressed data may then be compressed according to a second compression technique different than the column-specific compression technique to produce multi-level compressed data. The multi-level compressed data may be stored in the data block. When reading from the data block, multi-level compressed data may be decompressed according to the column-specific compression technique and the default compression technique applied to the data.

Abstract: A method, computer program product, and computing system for determining whether storage space usage on a storage system has reached a predefined threshold of a total storage capacity of the storage system. An out-of-space mode may be initiated on the storage system in response to determining that the storage space usage has reached the predefined threshold of the total storage capacity of the storage system. An amount of free storage capacity for exiting the out-of-space mode may be determined based upon, at least in part, a predefined minimum free storage capacity, a predefined maximum free storage capacity, and the total storage capacity of the storage system.

Abstract: The present disclosure relates to determining the shareability of values of node profiles. Record objects and electronic activities of a system of record corresponding to a data source provider may be accessed. Each record object may correspond to a record object type and have one or more object field-value pairs. Node profiles may be maintained. Values of fields corresponding to a predetermined type of field including fewer than a predetermined threshold number of data source providers may be identified. A restriction tag used to restrict populating other node profiles may be generated. Provision of the value with a second data source provider may be restricted.

Abstract: A system for (pre)configuration for IoT device setup is provided. In some implementations, the system performs operations comprising receiving, by an intermediary device (e.g., smartphone) and from a server (e.g., cloud computing system), a configuration profile for setup of an IoT device. The configuration profile can include an identifier (e.g., SSID) of a first wireless network (e.g., Wi-Fi and/or WLAN network). The operations can further comprise establishing, by the intermediary device, a connection to a second wireless network provided by the IoT device. Thereafter, the operations can further comprise providing, by the intermediary device and over the second wireless network, the configuration profile to the IoT device, wherein providing the configuration profile to the IoT device causes the Internet of Things device to establish (e.g., automatically) a connection to the first wireless network. Related systems, methods, and articles of manufacture are also described.

Abstract: A data compression method includes: obtaining a to-be-compressed object; searching a recommendation record for a recommended compression coding rule that meets a compression rate condition, the recommendation record being configured to record a compression coding rule of a historical compressed object and corresponding compression rate information, and the historical compressed object being of a same type as the to-be-compressed object; and if the recommended compression coding rule that meets the compression rate condition is found, compressing the to-be-compressed object by using the recommended compression coding rule; and if the recommended compression coding rule that meets the compression rate condition is not found, starting a regular compression coding process to obtain estimated compression rates of a plurality of compression coding rules for the to-be-compressed object, selecting a target compression coding rule based on at least the estimated compression rates, and compressing the to-be-compressed