Abstract: A method of storing remote sensing big data in a Hbase database. The method includes: obtaining a remote sensing image, and pre-cropping the remote sensing image based on a Google S2 algorithm; selecting a grid level based on a resolution of the remote sensing image; demarcating a coverage scope of the remote sensing image; calculating grids covered within the coverage scope; cropping the remote sensing image according to the grids, to yield a plurality of image blocks, and serializing the plurality of image blocks; establishing an Hbase table; establishing a table storage model containing RowKey, and calculating the RowKey according to attributes of an image block; and integrating data structure of multi-source remote sensing image data according to attribute expression of a zoned remote sensing image in a plurality of dimensions, and storing serialized image blocks in an Hbase database according to the RowKey.

Abstract: Aspects of the present invention disclose a method for ordering rows of a table based on access statistics. The method includes one or more processors adding a respective tracking mechanism to the attribute of each respective instance of the set of rows of the table of the database. The method further includes adding a tracking mechanism to the respective instances of the set of rows. The method further includes gathering access data of the respective tracking mechanism of the attribute of each respective instance of the set of rows. The method further includes determining each respective instance of an attribute value that corresponds to the access data of the respective tracking mechanism of each respective instance of the set of rows. The method further includes ordering each respective instance of the set of rows of the table based at least in part on the respective instances of the attribute values.

Abstract: A data-replication system replicates a source database to a target database by first intelligently identifying and prioritizing a set of hot queries that have each in the past been submitted to the source database at a rate exceeding a threshold submission rate. Any previously replicated content contained in a database table accessed by a hot query is copied to a set of memory-resident blocks. The blocks are then updated with incremental changes made to the table's contents since the most-recent previous replication. Each updated block is copied to the target database in order of the priority of the block's corresponding hot query. When all blocks that contain data of a particular table have been copied into the target database, that table becomes available to users of the target database despite the fact that the entire database has not yet been fully replicated.

Abstract: Innovative techniques are disclosed for performing a combined sort and aggregation operation involving eagerly performing aggregation while sorting. The techniques described herein enable an aggregation and a group-by operation to be performed using an amount of system memory that is far less than the system memory required to store all the data records being processed while minimizing the need to spill data to disk. This combined sort and aggregation operation has better performance than conventional techniques since system memory is used more efficiently. In certain embodiments, a combined sort and aggregation operation is disclosed that enables the efficient sorting and aggregation of data records when the desired aggregation function is composable (such as SUM, COUNT, MIN and MAX aggregate operations).

Abstract: A system and method for statistical fingerprinting of structured datasets begins by dividing the structured database into groups of data subsets. These subsets are created based on the structure of the data; for example, data delineated by columns and rows may be broken into subsets by designating each column as a subset. A fingerprint is derived from each subset, and then the fingerprint for each subset is combined in order to create an overall fingerprint for the dataset. By applying this process to a “wild file” of unknown provenance, and comparing the result to a data owner's files, it may be determined if data in the wild file was wrongfully acquired from the data owner.

Abstract: Embodiments can provide a computer implemented method, in a data processing system comprising a processor and a memory comprising instructions which are executed by the processor to cause the processor to implement an enhanced corpus management system, the method comprising: identifying one or more functional domain categories; ingesting one or more incoming documents to form an open-domain corpus; for each functional domain category, identifying one or more representative documents to establish a seed sub-corpus; calculating a degree of fit score between each of the one or more incoming documents and the one or more established functional domain category seed sub-corpora; and assigning one or more of the incoming documents to one or more of the functional domain categories based upon the degree of fit score to create an enhanced corpus.

Abstract: Techniques for non-power-of-two partitioning of a data set as well as generation and selection of partition schemes for the data set. In an embodiment, one or more iterations of a partition scheme is for a non-power-of-two number of partitions. Extended hash partitioning may be used to partition a data set into a non-power-of-two number of partitions by determining the partition identifier of each tuple of the data set using the extended hash partitioning algorithm. In an embodiment, multiple partition schemes are generated for multiple data sets, based on properties of the data sets and/or availability of computing resources for the partition operation or the subsequent operation to the partition operation. The generated partition schemes may use non-power-of-two partitioning for one or more iterations of a generated partition scheme. The most optimal partition scheme may be selected from the generated partition schemes based on optimization policies.

Abstract: It is possible execute processing large-scale data and improve the processing efficiency while suppressing the complexity of a hardware circuit. A data processing device includes a processor and a FPGA connected to the processor. The processor is configured to acquire a query plan including target identification information identifying data to be processed and a processing detail for the data to be processed, generate, based on the query plan, a plurality of FPGA commands to process a plurality of row group data items constituting the data identified by the target identification information and to be processed, and transmit the FPGA commands to the FPGA. The FPGA is configured to execute processing on the row group data items based on the transmitted FPGA commands and return results of executing the processing to the processor.

Abstract: A search apparatus includes a first control unit that, in a case of displaying a search result of content information searched with a selected search word, performs control such that recommended words corresponding to the search word are displayed; a second control unit that, in a case of displaying a search result of content information searched by adding a selected recommended word, which is selected from the recommended words, as a search word, performs control such that the selected recommended word is displayed in a search word display region at which the search words are displayed, and the recommended words corresponding to the selected search word and the added search word are displayed; a reception unit that receives a stored search word, out of the selected search word and the added search word, the stored search word being displayed in a storage state indicating that the stored search word is not used for search of the content information but is capable of being reselected; and a third control unit

Abstract: The disclosed embodiments include a method performed by a data system. The method includes automatically learning relationship(s) among datasets based on one or more of a user query or an observation of a data flow through the data system. The method further includes generating an optimized data structure based on the learned relationships among the datasets. The data system then modifies a query plan to obtain query results that satisfy a query by reading the optimized data structure in lieu of reading the datasets.

Abstract: Methods and systems for generating an electronic database, the electronic database comprising database elements being organized in a single table, the single table comprising at least four columns, the at least four columns including: a first column for storing identification numbers of the database elements, a second column for storing a numbers of a parent element for database elements being dependent from other database elements, a third column for storing database elements values, a fourth column for storing a code of the in-use datatypes; and at least five rows, the at least five rows including a first row representing a root element, a second row representing a datatype, a third representing a term, a fourth row representing the term's attribute and a fifth row representing data.

Abstract: A service implements techniques for modifying a data table stored in a key-value or non-relational database via a table-oriented schema. In some aspects, a request to modify a table-oriented schema of a table of data may be received. In response to the request, an association between a customer identifier of elements of the table and an internal identifier of elements of the table may be stored that updates a prior association between a customer identifier and an internal identifier. The table-oriented schema of the table may be modified using one or more keys that are generated using the internal identifier.

Abstract: The present disclosure relates to systems, methods, and non-transitory computer readable storage media for implementing a scalable, secure, efficient, and adaptable distributed digital ledger transaction network. Indeed, the disclosed systems can reduce storage and processing requirements, improve security of implementing computing devices and underlying digital assets, accommodate a wide variety of different digital programs (or “smart contracts”), and scale to accommodate billions of users and associated digital transactions. For example, the disclosed systems can utilize a host of features that improve storage, account/address management, digital transaction execution, consensus, and synchronization processes. The disclosed systems can also utilize a new programming language that improves efficiency and security of the distributed digital ledger transaction network.

Abstract: A series of graphical user interfaces allows a user to define rules for parsing unstructured and structured log data and generate an aggregate rule to identify events across multiple lines of one or more log files. A first graphical user interface allows a user to select a line from a log file under analysis, search for matching lines in the log file, define rules for parsing the log file, and tokenize attributes of the selected lines. A second user interface allows the user to aggregate defined rules so that messages may be correlated to identify events that occur across multiple lines.

Abstract: In one embodiment, a system is provided. The system includes a first client. The system also includes an analysis server coupled to the first client. The system further includes a first customer database of information coupled to the analysis server. The first customer database is to embody forecast data and to receive essentially real-time updates to the forecast data. The first customer database supports an OLAP cube associated with the analysis server.

Abstract: Methods and systems for selective archival of data include, in some aspects, determining, via an electronic processing circuit, a plurality of accessors for a data format, displaying indicators for at least a portion of the plurality of accessors on an electronic display, receiving input defining a selection of a subset of the plurality of accessors from an input device; and outputting the selected subset of accessors with data encoded in the data format to an archive.

Abstract: Database environments may choose to schedule complex analytics processing to be performed by specialized processing environments by caching source datasets or other data needed for the analytics and then outputting results back to customer datasets. It is complex to schedule user database operations, such as running dataflows, recipes, scripts, rules, or the like that may rely on output from the analytics, if the user database operations are on one schedule, while the analytics is on another schedule. User/source datasets may become out of sync and one or both environments may operate on stale data. One way to resolve this problem is to define triggers that, for example, monitor for changes to datasets (or other items of interest) by analytics or other activity and automatically run dataflows, recipes, or the like that are related to the changed datasets (or other items of interest).

Abstract: The subject technology receives first metadata corresponding to a set of micro-partitions. The subject technology stores a first data structure and a second data structure in storage as a first file and a second file, first data structure including the first metadata and a second data structure including second metadata, the first metadata corresponding to a set of micro-partitions, the second metadata for a grouping of the first metadata, the second data structure including information associating the second metadata to the first metadata. The subject technology stores third metadata for a table, the third metadata comprising: cumulative table metadata comprising global information about a plurality of micro-partitions of the table, the cumulative table metadata being stored in a metadata micro-partition associated with the table.

Abstract: A method executes at a computing device that includes a display, one or more processors, and memory. The device displays a user interface that includes a data pane and a calculation pane. The data pane comprises data rows and data columns. Each of the data columns corresponds to a data field from a data source. The device receives a first user input to specify a grouping on a first data field. The device receives a second user input to specify an aggregation function on a second data field. In response to the first and second user inputs, for each distinct value of the first data field, the device aggregates corresponding values of the second data field according to the aggregation function. A plurality of first data rows are displayed in the calculation pane. The device saves the rows of data displayed in the calculation pane as a new data source.

Abstract: A de-serialized data store and transaction system includes a content addressable memory mapped to an n-tuple-based multimodal data structure, the n-tuple-based multimodal data structure supporting heterogeneous nested data types including a streamable abstract syntax tree and a self-describing operational interface comprising commands transmitted via a plurality of nested s-expressions.