Abstract: A network node generates a reduced size textual network log by including a set of numerical values for a log entry within a textual network log for a network, the log entry constituting an instance of a recognizable pattern within the textual network log; and then outputs the reduced size textual network log to a network controller for configuring the network.

Abstract: Methods and apparatuses are provided for editing and generating a short video based upon a long video. The method includes: obtaining a plurality of short source video clips as candidate video clips; obtaining attributes of each short source video clip; obtaining a plurality of target base videos according to a target genre, processing the plurality of target base videos by splitting each target base video into a plurality of short target base video clips, and obtaining attributes of each short target base video clip; selecting short target video clips from the plurality of short source video clips, based on distribution of the attributes obtained for the plurality of the short source video clips and the plurality of short target base video clips; and editing and combining the short target video clips selected from the plurality of short source video clips, to obtain a target video.

Abstract: A database includes a plurality of physical entity records of insurance claims, and each physical entity record includes physical data values for at least one numeric attribute and partition-specifying values concerning values for one or more nominal attributes from one or more insurance claim records. A method for detecting anomalies in the database includes retrieving and partitioning the plurality of physical entity records from the database, training an unsupervised anomaly detection algorithm on the plurality of physical entity records to obtain a trained anomaly detection model for each partition, calculating an anomaly score for each physical entity record using the trained anomaly detection model associated with each partition, and updating the plurality of physical entity records in the database by adding the associated anomaly score. The method is used to determine if a user-provided physical entity record is fraudulent using the anomaly score.

Abstract: An apparatus to facilitate memory data compression is disclosed. The apparatus includes a memory and having a plurality of banks to store main data and metadata associated with the main data and a memory management unit (MMU) coupled to the plurality of banks to perform a hash function to compute indices into virtual address locations in memory for the main data and the metadata and adjust the metadata virtual address locations to store each adjusted metadata virtual address location in a bank storing the associated main data.

Abstract: Log messages are compressed, searched, and decompressed. A dictionary is used to store non-numeric expressions found in log messages. Both numeric and non-numeric expressions found in log messages are represented by placeholders in a string of log “type” information. Another dictionary is used to store the log type information. A compressed log message contains a key to the log-type dictionary and a sequence of values that are keys to the non-numeric dictionary and/or numeric values. Searching may be performed by parsing a search query into subqueries that target the dictionaries and/or content of the compressed log messages. A dictionary may reference segments that contain a number of log messages, so that all log message need not be considered for some searches.

Abstract: A hardware-implemented file reader includes an interface, multiple hardware-implemented column readers and a hardware-implemented record reconstructor. The interface is configured to access a file including multiple records. The records store values in accordance with a nested structure that supports optional values and repeated values. The file is stored in a columnar format having multiple columns, each column storing (i) compressed values and (ii) corresponding compressed structure information that associates the values in the column to the nested structure of the records. Each column reader is configured to be assigned to a respective selected column, and to read and decompress both the values and the structure information from at least a portion of the selected column. The record reconstructor is configured to reconstruct one or more of the records from at least portions of the columns that are read by the column readers, and to output the reconstructed records.

Abstract: A computer-implemented method for deduplicating records includes generating a block comprising of a group of records. The method also includes creating one or more record pairs from the block, and calculating one or more features based on one or more fields within the one or more record pairs. The method further includes inputting the one or more features into a machine language trained model to predict a probability score. The probability score indicates whether two records are duplicates. The method also includes storing the probability score as links between two vertices in a graph, and displaying one or more duplicate records by querying the graph.

Abstract: An apparatus and method for providing ML processing for one or more ML applications operating on one or more Internet of Things (IoT) devices includes receiving a ML request from an IoT device. The ML request can be generated by a ML application operating on the IoT device and include input data collected by the first ML application. A ML model to perform ML processing of the input data included in the ML request is identified and provided to an ML core for ML processing along with the input data included in the first ML request. The ML core produces ML processing output data based on ML processing by the ML core of input data included in the ML request using the ML model. The ML processing output data can be transmitted to the IoT device.

Abstract: Herein are spatially scalable techniques for anisotropic compression of shared entropy between alternate representations of same data. In an embodiment, a computer compresses an uncompressed independent column into a compressed independent column. Based on the compressed independent column, an uncompressed dependent column is compressed into a compressed dependent column. The compressed independent column and the compressed dependent column are stored in a same file. In an embodiment, a computer stores, in metadata, an encrypted private key for decrypting an encrypted column. The encrypted column and the metadata are stored in a file. A request to read the encrypted column is received. Based on a public key and the file, the encrypted private key is decrypted into a decrypted private key. The public key is contained in the request and/or the file. The request is executed by decrypting, based on the decrypted private key and the file, the encrypted column.

Abstract: Techniques and solutions are described for compressing data and facilitating access to compressed data. Compression can be applied to proper data subsets of a data set, such as to columns of a table. Using various methods, the proper data subsets can be evaluated to be included in a group of proper data subsets to be compressed using a first compression technique, where unselected proper data subsets are not compressed using the first compression technique. Data in the data set can be reordered based on a reordering sequence for the proper data subsets. Reordering data in the data set can improve compression when at least a portion of the proper data subsets are compressed. A data structure is provided that facilitates accessing specified data stored in a compressed format.

Abstract: Embodiments of the present disclosure provide to a method and a device for acquiring a data model in a knowledge graph, an apparatus and a storage medium. The method includes: receiving a knowledge entry describing a relationship between an entity and an object; determining a plurality of candidate object types of the object according to at least one of the entity, the relationship and the object; determining an object type for generating a data model that matches the knowledge entry from the plurality of candidate object types based on a preset rule; and generating the data model based at least on the object type.

Abstract: Various embodiments are generally directed to techniques for crowdsourcing metadata, such as to create, maintain, utilize, and/or communicate metadata in a datastore. Some embodiments are particularly directed to collecting, managing, and/or distributing metadata via a user interface application (UIA), or an extension thereof. Many embodiments may facilitate collaborative creation of a repository for metadata, such as notes, characteristics, and/or statistics corresponding to data viewed with the user interface application. For example, the comments regarding websites may be collected from a first user via a first instantiation of the web browser extension and distributed to a second user via a second instantiation of the web browser extension. In several embodiments, the metadata may be collected and managed with instantiations of the web browser extension (herein after “WBE instants”) and a data manager may create, maintain, utilize, and/or communicate metadata in a datastore.

Abstract: Fusing online and mapped terrain estimates by using weighted grid cells that scales the values returned from online terrain and mapped terrain is disclosed. Previously mapped terrain data and online terrain data are fused and a grid having cells of a predetermined size is overlaid on the terrain map. Each cell may include terrain data based on weighted mapped terrain data and weighted online terrain data, where the weighting values for the mapped terrain data and for the online terrain data may be different. A fused terrain estimate may be a result of a weighted mean for each cell smoothed to reduce noise.

Abstract: A method for compressing data in a local cache of a web server is described. A local cache compression engine accesses values in the local cache and determines a cardinality of the values of the local cache. The local cache compression engine determines a compression rate of a compression algorithm based on the cardinality of the values of the local cache. The compression algorithm is applied to the cache based on the compression rate to generate a compressed local cache.

Abstract: According to various embodiments, methods and systems are provided to accelerate artificial intelligence (AI) model training with advanced interconnect communication technologies and systematic zero-value compression over a distributed training system. According to an exemplary method, during each iteration of a Scatter-Reduce process performed on a cluster of processors arranged in a logical ring to train a neural network model, a processor receives a compressed data block from a prior processor in the logical ring, performs an operation on the received compressed data block and a compressed data block generated on the processor to obtain a calculated data block, and sends the calculated data block to a following processor in the logical ring. A compressed data block calculated from corresponding data blocks from the processors can be identified on each processor and distributed to each other processor and decompressed therein for use in the AI model training.

Abstract: A method performed by a block-storage server, of storing data is described. The method includes (1) receiving, from a remote file server, data blocks to be written to persistent block storage managed by the block-storage server; (2) receiving, from the remote file server, metadata describing a placement of the data blocks in a filesystem managed by the remote file server; and (3) organizing the data blocks within the persistent block storage based, at least in part, on the received metadata. An apparatus, system, and computer program product for performing a similar method are also provided.

Abstract: A system and method, including determining, by a processor, a data type for each column of a database table; determining, by the processor and based on the determined data type for each column of the database table, an indication of a size of the database table; calculating, by the processor and based on the determined indication of the size of the database table, a start nbit size for a nbit compression process to be used on the database table; specifying, by the processor, the calculated start nbit size for the nbit compression process; and compressing the database table by executing the nbit data compression process using the specified start nbit size.

Abstract: Data to be stored in a data block for a columnar database table may be compressed according to a multi-level compression scheme. Data to be stored in the data block may be received. The data may be compressed according a column-specific compression technique to produce compressed data. The compressed data may then be compressed according to a second compression technique different than the column-specific compression technique to produce multi-level compressed data. The multi-level compressed data may be stored in the data block. When reading from the data block, multi-level compressed data may be decompressed according to the column-specific compression technique and the default compression technique applied to the data.

Abstract: A method, computer program product, and computing system for determining whether storage space usage on a storage system has reached a predefined threshold of a total storage capacity of the storage system. An out-of-space mode may be initiated on the storage system in response to determining that the storage space usage has reached the predefined threshold of the total storage capacity of the storage system. An amount of free storage capacity for exiting the out-of-space mode may be determined based upon, at least in part, a predefined minimum free storage capacity, a predefined maximum free storage capacity, and the total storage capacity of the storage system.

Abstract: The present disclosure relates to determining the shareability of values of node profiles. Record objects and electronic activities of a system of record corresponding to a data source provider may be accessed. Each record object may correspond to a record object type and have one or more object field-value pairs. Node profiles may be maintained. Values of fields corresponding to a predetermined type of field including fewer than a predetermined threshold number of data source providers may be identified. A restriction tag used to restrict populating other node profiles may be generated. Provision of the value with a second data source provider may be restricted.

Abstract: A system for (pre)configuration for IoT device setup is provided. In some implementations, the system performs operations comprising receiving, by an intermediary device (e.g., smartphone) and from a server (e.g., cloud computing system), a configuration profile for setup of an IoT device. The configuration profile can include an identifier (e.g., SSID) of a first wireless network (e.g., Wi-Fi and/or WLAN network). The operations can further comprise establishing, by the intermediary device, a connection to a second wireless network provided by the IoT device. Thereafter, the operations can further comprise providing, by the intermediary device and over the second wireless network, the configuration profile to the IoT device, wherein providing the configuration profile to the IoT device causes the Internet of Things device to establish (e.g., automatically) a connection to the first wireless network. Related systems, methods, and articles of manufacture are also described.

Abstract: A data compression method includes: obtaining a to-be-compressed object; searching a recommendation record for a recommended compression coding rule that meets a compression rate condition, the recommendation record being configured to record a compression coding rule of a historical compressed object and corresponding compression rate information, and the historical compressed object being of a same type as the to-be-compressed object; and if the recommended compression coding rule that meets the compression rate condition is found, compressing the to-be-compressed object by using the recommended compression coding rule; and if the recommended compression coding rule that meets the compression rate condition is not found, starting a regular compression coding process to obtain estimated compression rates of a plurality of compression coding rules for the to-be-compressed object, selecting a target compression coding rule based on at least the estimated compression rates, and compressing the to-be-compressed

Abstract: According to embodiments of the present disclosure, a method and an apparatus for responding to a question, and a storage medium are provided. The method includes: determining a question characteristic representation corresponding to a question for an object; determining a comment characteristic representation corresponding to a first comment for the object; generating a first target characteristic representation by utilizing the question characteristic representation and the comment characteristic representation; and determining an answer for the question based on the first target characteristic representation.

Abstract: Embodiments of the present disclosure provide a method and device for deduplication. Specifically, the method may comprise obtaining a property of a file stream, the property of a file stream including a file type or a magic number identifying a format of a protocol or a file. The method further includes in response to receiving an I/O request for a data block of the file stream, assigning a deduplication level to the I/O request based on the property of the file stream. Moreover, the method further includes deduplicating the data block of the file stream based on the deduplication level assigned to the I/O request. In addition, a corresponding device and computer program product are provided.

Abstract: Hybrid tables can be used in different use-case scenarios. Hybrid tables provide a flexible mechanism to support files and data in different formats while providing access to the different types of data as part of one table. This flexibility can allow the use of hybrid tables in data lake or other similar environments.

Abstract: The present disclosure describes a service which provides primary in-line deduplication. A streaming application program interface (API) may allow for streaming records into a storage system with high throughput and low latency. As part of this process, the API allows user to add identifiers as a field used for data deduplication. The deduplication service keeps a moving window of the identifiers in memory and does in-line deduplication by quickly determining whether data is a duplicate. Keeping only deduplication keys in memory reduces the cost of running the service. Moreover, the real-time nature of the moving window approach allows for storing deduplication information alongside the data and accessing it immediately on read. In this regard, read after write consistency is supported, and costs are reduced.

Abstract: A data storage system includes accelerated storages that store data encoded using a first data integrity scheme. The data storage system also includes a global enumerator that obtains a request to use a second data integrity scheme to store the data; in response to obtaining the request: performs a recoding process for the data including: reading data chunks, corresponding to the data, from the storages; generating new data chunks based on: the read data chunks and the second data integrity scheme; storing the new data chunks in the accelerated storages in an arrangement specified by the second data integrity scheme; and deleting the data chunks after storing the new data chunks.

Abstract: Technologies for predictive caching include a computing device to receive sensor data generated by one or more sensors of the computing device and determine a device context of the computing device based on the sensor data. Based on the device context, the computing device determines a file to cache that has similar characteristics to another file recently accessed by a user of the computing device. The computing device includes a file cache with a first partition to store files identified to have similar characteristics to files recently accessed by a user and a second partition to store files identified based on access patterns of the user. The computing device stores the determined file to the first partition.

Abstract: Systems and methods include detection of deletion of a record of a database table due to a database transaction, creation of a log entry including one or more key field values of the record of the database table, an identifier of a user associated with the database transaction, and an indicator of a delete operation, determination of whether the identifier identifies an archiving user, and, in a case that the identifier identifies an archiving user, change the indicator to an indicator of an archive operation to create a changed log entry.

Abstract: A technique for performing data reduction applies deduplication principles when performing data compression, providing a form of enhanced compression. The technique obtains a chunk of data that contains multiple extents and applies deduplication actions to identify duplicate extents within the chunk. The technique marks duplicate extents in metadata. Such duplicate extents need not be compressed using conventional data compression, saving computational resources and considerable time.

Abstract: A system and method for highly efficient encoding of data that includes extended functionality for asymmetric encoding/decoding and network policy enforcement. In the case of asymmetric encoding/decoding the original data is encoded by an encoder according to a codebook and sent to a decoder, but the output of the decoder depends on data manipulation rules applied at the decoding stage to transform the decoded data, into a different data set from the original data. In the case of network pokey enforcement, a behavior appendix into the codebook, such that the encoder and/or decoder at each node of the network comply with network behavioral rules, limits, and policies during encoding and decoding.

Abstract: A data store system may include a storage device configured to store a plurality of data store tables. The data store system a further include a processor in communication with the storage device. The processor may receive a request to encode a column of a data store table from the plurality of data store tables. The processor may further generate a bit value representation of each value in the column of the data store table. The processor may further generate an index. The index may include an index value representative of each bit position of the bit value representations. The processor may further reorder bits of each bit value representation according to a predetermined pattern. The processor may further encode each reordered bit value representation according to an encoding technique. The processor may further store each encoded reordered bit value representations and the index. A method and computer-readable medium are also disclosed.

Abstract: A data compression method includes obtaining N to-be-compressed data blocks and N pieces of protection information (PI), where the N to-be-compressed data blocks are in a one-to-one correspondence with the N pieces of PI, and N is a positive integer greater than or equal to 2, compressing the N to-be-compressed data blocks to obtain a compressed data block, and compressing the N pieces of PI to obtain compressed PI.

Abstract: Techniques for -memory ingestion for highly available distributed time-series databases are described. A method of in-memory ingestion may include obtaining, by a host of a time series database, time series data from one or more electronic devices, the time series database including a plurality of portions of the time series database spread across a plurality of hosts, the plurality of portions of the time series database including at least one hot portion and a plurality of cold portions, storing the time series data in a volatile storage location associated with the hot portion on the host, detecting an event to store the time series data associated with the hot portion to a non-volatile storage location on the host, and storing the time series data associated with the hot portion to the non-volatile storage location.

Abstract: The present invention relates to a method and a system for maintaining consistency of data between data-sets stored in a master database of a master computing node and corresponding data-sets stored in a replication database of at least one replication computing node. Each time an updated version of a stored data set is received, the master computing node is configured for updating a corresponding data-set stored in the master database and transmitting replication data relating to the updated data-set version to the at least one replication computing node for replication. The replication data comprising a bit-vector storing data values extracted from data fields of the data-set classified as volatile data fields. The data values stored in the bit-vector representing the changes in the data-set to be replicated in the at least one replication node.

Abstract: A technique for managing data storage includes generating digests of chunks of data. The digests are arranged to provide similar results for similar chunks but to weight contributions from different regions of chunks differently based on their entropy. Chunks that differ in low-entropy regions tend to produce more similar digests than do chunks that differ in high-entropy regions. The technique further includes identifying similar chunks based on similarity of their digests and preforming data reduction on one chunk by reference to another, storing any differences between the chunks separately.

Abstract: Methods, computer program products, computer systems, and the like are disclosed that provide for improved write performance in erasure-coded storage systems in an efficient and effective manner. These can include identifying a data segment, identifying metadata, persisting the data segment to a storage system, storing the metadata in the journal, and persisting the metadata to the storage system. In such embodiments, the metadata is associated with the data segment by virtue of the metadata comprising a fingerprint of the data segment. Further, in such embodiments, the persisting the data segment to the storage system is performed without storing the data segment in a journal, and the storage system is an erasure-coded storage system.

Abstract: Input data can be losslessly reduced by using a data structure that organizes prime data elements based on their contents. Alternatively, the data structure can organize prime data elements based on the contents of a name that is derived from the prime data elements. Specifically, video data can be losslessly reduced by (1) using the data structure to identify a set of prime data elements, and (2) using the set of prime data elements to losslessly reduce intra-frames. The input data can be dynamically partitioned based on the memory usage of components of the data structure. Parcels can be created based on the partitions to facilitate archiving and movement of the data. The losslessly reduced data can be stored using a set of distilled files and a set of prime data element files.

Abstract: In one embodiment, an apparatus comprises a network interface and a processor. The processor is to: receive, via the network interface, a plurality of data streams to be routed over a network, wherein the plurality of data streams correspond to sensor data captured by a plurality of sensors; identify, from the plurality of data streams, a set of related data streams that are contextually related; identify a convergence function to be performed on the set of related data streams, wherein the convergence function is for transforming the set of related data streams into a converged data stream that is smaller in size than the set of related data streams; perform the convergence function to transform the set of related data streams into the converged data stream; and route, via the network interface, the converged data stream to one or more corresponding destinations over the network.

Abstract: A method and apparatus of a device that compresses an object stored in memory is described. In an exemplary embodiment, the device receives an indication that the object is to be compressed. The device further selects one of a plurality of compression algorithms based on at least a characteristic of the object. In addition, the device compresses the object in-memory using the selected compression algorithm.

Abstract: Technology is described herein that uses an object-encoding system to convert an object image into a combined encoding. The object image depicts a reference object, while the combined encoding represents an environment image. The environment image, in turn, depicts an estimate of an environment that has produced the illumination effects exhibited by the reference object. The combined encoding includes: a first part that represents image content in the environment image within a high range of intensities values; and a second part that represents image content within a low range of intensity values. Also described herein is a training system that trains the object-encoding system based on combined encodings produced by a separately-trained environment-encoding system. Also described herein are various applications of the object-encoding system and environment-encoding system.

Abstract: A method is used in managing data reduction in storage systems using machine learning. A value representing a data reduction assessment for a first data block in a storage system is calculated using a hash of the data block. The value is used to train a machine learning system to assess data reduction associated with a second data block in the storage system without performing the data reduction on the second data block, where assessing data reduction associated with the second data block indicates a probability as to whether the second data block can be reduced.

Abstract: A regenerated container file is detected, and a file in the regenerated container file is determined that is different from any file in an existing container file related to the regenerated container file. To store the regenerated container file, the different file is sent to the data storage for storing.

Abstract: According to embodiments, point cloud data transmission method may include encoding point cloud data, encapsulating a bitstream that includes the encoded point cloud data into a file, and transmitting the file, the bitstream is stored either in a single track or in multiple tracks of the file, the file further includes signaling data, wherein the signaling data include at least one parameter set and spatial region information, and the encoded point cloud data include geometry data and attribute data.

Abstract: Executing a big data analytics pipeline in a storage system that includes compute resources and shared storage resources, including: receiving, from a data producer, a dataset; storing, within the storage system, the dataset; allocating processing resources to an analytics application; and executing the analytics application on the processing resources, including ingesting the dataset from the storage system.

Abstract: A technology is disclosed that makes it possible even for an analyst, who has poor knowledge relating to field data, to select and use analysis data in analysis. A data catalog automatic generation system that generates a catalog tag to be used to select analysis data from collected field data is configured such that, based on a set classification rule input, a relationship between an objective variable as an analysis perspective relating to field data and an explanatory variable or a causal relationship between a plurality of the explanatory variables is extracted, and based on a result of the extraction, a catalog tag of the objective variable and a catalog tag of the explanatory function are specified and attached.

Abstract: An information processing device obtains a first index value which indicates a popularity degree of a blog containing one or a plurality of articles, obtains, for each of the plurality of articles contained in the blog, a second index value which indicates an accessibility to the article, and determines whether or not to compress the article contained in the blog based on the first index value and the second index value.

Abstract: An information processing device obtains an increasing tendency of a storage capacity utilized by a blog containing at least one article, sets, to the blog, a threshold for determining whether or not to compress at least a part of the article contained in the blog in accordance with the increasing tendency, determines whether or not the blog is to be compressed based on a total data amount of the at least one article contained in the blog and on the threshold, and determines whether or not to compress each of the at least one article contained in the blog in accordance with a degree of accessibility.

Abstract: In a streaming media method, a set of one or more server devices can provide media playlists that skip media segments. For example, a client device can receive a full media playlist from a server device, and the full media playlist can indicate that the server device supports providing updated playlists that are condensed by skipping media segments. The client device can then request an updated playlist that skips media segments and use that updated playlist to request and receive steaming content.

Abstract: At least a portion of data of a tree data structure is serialized into a first set of flat data. At least a portion of a structure of the tree data structure is serialized to reproduce at least the portion of the structure in a second set of flat data. It is determined to access a desired data of the tree data structure from an archive. The second set of flat data is read to reconstitute at least the portion of a structure of the tree data structure. The reconstituted structure of the tree data structure is used to determine a data offset associated with the first set of flat data, wherein the data offset corresponds to the desired data. The desired data is accessed using the data offset associated with the first set of flat data.