Abstract: A distributed, transactional database uses timestamps, such as logical clock values, for entry versioning and transaction management in the database. To write to the database, a service requests a timestamp to be inserted into the database with a new version of data. During a backup procedure, a cleanup process is paused, issuing new timestamps is paused, and a backup timestamp is generated, which results in an effective backup copy. During a restore of a backup, a snapshot of the database is loaded and any entries older than the backup timestamp are deleted, which ensures that a consistent restore has occurred.

Abstract: According to an embodiment, a server device includes one or more processors and a first storage. The processors receive first data of transmission data from a transmitting device. The processors further receive second data of the transmission data, transmitted from the transmitting device in response to a transmission request. The processors deliver the first data and the second data to a receiving device. The first storage stores the first data and the second data. The processors delete at least one of the first data and the second data from the first storage according to a predetermined condition, and store data including at least a portion of the deleted data in a second storage.

Abstract: A tree data structure for a file is maintained. The tree data structure is used to capture different versions of the file at different moments of time. An access pattern associated with the file stored on a storage system is determined based at least in part on time-series data associated with the file using the tree data structure that captures different versions of the file at different moments in time. The determined access pattern indicates a period for which the file is likely to be accessed. The file is migrated from a secondary storage tier to a primary storage tier.

Abstract: Offline building of a projected data subset may be performed. A request to create a data set that is a projected subset of data from a source data set may be received. A data store separate from the data store storing the source data set may store a copy of the source data set that is used to replicate items to the projected subset of data according to a schema for the projected data subset. Updates made to the source data set may also be replicated to the projected data subset according to the schema. Conflicts between replicated items and replicated updates to the projected data set may be resolved by comparing a version identifier for the replicated update and replicated item to determine what to store in the projected data subset.

Abstract: The described technology is generally directed towards bulk-mode replication of data objects to a destination replication site. When replicating a group of objects, and there is enough object data to fill one or more chunk data structures, bulk-mode data replication is performed, with the destination site replicating the objects into chunk data structures and performing erasure coding to protect the data. This avoids the need for inefficient preliminary data protection scheme at the replication destination. When there are multiple objects to be replicated but not enough object data to fill a chunk data structure, bulk-mode data replication transfers the objects to the destination site for replicating as individual objects. Standard object-level replication can be used for single object replication. The technology thus facilitates high-performance data transfer to destination replication sites and efficient handling of data at the replication destination site.

Abstract: The subject technology receives a query plan, the query plan comprising a set of query operations, the set of query operations including at least one aggregation. The subject technology analyzes the at least one aggregation to generate a modified query plan, the modified query plan including at least a top aggregation operator, an intermediate aggregation operator, and a bottom aggregation operator. The subject technology performs, with respect to the intermediate aggregation operator, at least one operation comprising: the subject technology receives an input intermediate data type; the subject technology performs an internalize operation on the input intermediate data type to generate an internal state; the subject technology performs an accumulate operation on the internal state to generate intermediate data; and the subject technology performs an externalize operation on the intermediate data to generate an output data type.

Abstract: A mapping and query service for mapping between object-oriented programming objects and deep key-value data stores. The service to implement a store operation for a mapping and query service that supports the storage of a set of one or more objects having classes and fields written in source code of an object-oriented programming language in a deep key-value data store.

Abstract: A virtualized computing system includes: a host cluster including hosts executing a virtualization layer on hardware platforms thereof, the virtualization layer configured to support execution of virtual machines (VMs), the VMs including a pod VM, the pod VM including a container engine configured to support execution of containers in the pod VM, the pod VM including a first virtual disk attached thereto; and an orchestration control plane integrated with the virtualization layer, the orchestration control plane including a master server in communication with a pod VM controller, the pod VM controller configured to execute in the virtualization layer external to the VMs and cooperate with a pod VM agent in the pod VM, the pod VM agent generating root directories for the containers in the pod VM, each of the root directories comprising a union a read/write ephemeral layer stored on the first virtual disk and a read-only layer.

Abstract: Techniques described and suggested herein include distributed deletion request processing and verification. For example, incident to migration of original data from a first data store to a second data store, verifications and confirmations related to removing the original data from the first data store may be performed so as to ensure the integrity of the original data represented on the second data store prior to removing the actual original data on the first data store. In some embodiments, the verifications and confirmations performed in connection with a deletion request may be apportioned to multiple entities, each of which may not fully trust the others. As a result, in some embodiments, a given deletion request may only be fulfilled if all of the entities involved in the verification process individually provide authorization to execute the deletion request.

Abstract: Systems and methods for implementing client-side load balancing for remote application servers. An example method comprises receiving, by a publishing server, an application server status information reflecting computing resource utilization by each application server of a plurality of application servers; receiving a request initiated by a client computing device to access a published application executable by at least a subset of the plurality of application servers; and responsive to determining that a load metric reflecting performance of the publishing server exceeds a threshold value, transmitting an instruction to the client computing device to perform client-side load balancing using a snapshot of the application server status information to select an application server of the subset of the plurality of application servers.

Abstract: File access statistics associated with a content file are maintained by a file tier agent of a secondary storage system. The content file is accessible via a primary storage system. A target storage tier corresponding to the file access statistics associated with the content file is determined according to one or more policies. At least a portion of data of the content file is migrated to the determined storage tier while accessibility of the content file via the primary storage system is maintained.

Abstract: A cloud-based data governance system includes a processing unit, a network adapter, and memory for storing data and code. The network adapter establishes a connection with a remote data storage system associated with a remote file system over a wide-area network (WAN). The code includes an event collection interface, a data governance service, and an enforcement service. The event collection interface is configured to capture an event from the remote data storage system. The event is indicative of a file system operation executed on a data object of the remote file system. The data governance service is configured to receive the event from the event collection interface and to process the event to determine whether the file system operation conflicts with a governance policy of the data governance system. The enforcement service executes a set of remediation actions if the file system operation does conflict with the governance policy.

Abstract: A method of synchronizing a plurality of end points is disclosed. A spoke is connected to a hub. The connecting of the spoke to the hub includes performing inbound synchronization of all of a set of data items or a subset of the set of data items with a replica. One or more additional spokes are connected to the hub. The connecting of the one or more additional spokes to the hub includes performing inbound synchronization of all of an additional set of data items or a subset of the additional set of data items with an additional replica. A subset of a set of data items stored in the replica or the additional replica is selected for outbound synchronization with any of a plurality of spokes except a spoke from which the subset of data items originated.

Abstract: Embodiments are disclosed for a method that may include accessing, from a first data source, events including interactions between a mobile device and one or more network devices on a network at a locale, and receiving external data from a second data source that excludes the network devices. The method may further include determining, based on the interactions, one or more geographic positions and one or more corresponding time intervals of the mobile device at the geographic positions, and correlating the geographic positions and the time intervals with the external data to obtain a metric.

Abstract: A database system provides automatic synchronization from one or more databases to a table. The synchronized portion of each source database table is periodically imported into a corresponding portion of a destination database table. For each synchronized field in the table with data from multiple source databases, one of the source databases is set as a primary source, which determines the data type and field configuration of the field. Data from secondary sources are cast to the data type established by the primary source and represented using the primary source's field configuration. The source database table may also include data that is not synchronized with the destination database table. Similarly, the destination database table may be enriched with data that is not included in the source database table.

Abstract: Methods and apparatus for a relational modeler and renderer for non-relational data are disclosed. In response to an indication of non-relational data sources, a relational data model based on one or more model inclusion criteria is generated to represent at least some contents of the non-relational data sources. In response to a model population request from a client, data obtained from the one or more non-relational data sources for one or more tables of the model is stored in cache storage allocated on behalf of the client. In response to a read-only operation request directed to the model, results of the read-only operation are obtained from the cache storage and provided to the requester.

Abstract: Disclosed herein are methods, systems, and processes for migration between storage tiers. Such a method, for example, can include extracting one or more characteristics of a replication workload, determining one or more storage costs of each storage tier of a plurality of storage tiers (where the one or more storage costs are determined for the replication workload and the one or more storage costs are determined based, at least in part, on the one or more characteristics), identifying one or more storage tiers of the plurality of storage tiers (where the identifying is based, at least in part, on the one or more storage costs), and migrating at least a portion of the replication workload from a target storage unit in an initial storage tier to a storage unit in the one or more storage tiers.

Abstract: Systems and methods for providing for a personal data storage, retrieval, and sharing system are described. One method includes importing known user content during an initial content upload phase. The method further includes storing at least a portion of the known user content in a user information database. The method further includes identifying the first document and the second document as including duplicate data or being duplicate copies of one another. The method further includes identifying the second document as a master copy. The method further includes extracting data from the second document. The method further includes extracting additional data associated with the user from the account holder database. The method further includes aggregating the extracted data and the extracted additional data. The method further includes updating user information associated with a user account of the user based on the aggregated data.

Abstract: A database is configured to analyze user queries to dynamically partition the database according to a partition scheme. User queries can be rewritten based on the partition scheme so that, in response to queries, partitions including relevant data are read while partitions including irrelevant data can be skipped, reducing latency. Files can be named according to the partition scheme and stored on respective partitions so that low partition management can be implemented by underlying systems. Blocks within files can be sorted and statistics can be determined. The statistics can be used to find and read relevant blocks and skip irrelevant blocks.

Abstract: Techniques are provided for selective re-indexing of a database. A pre-defined set of document properties are associated with one or more documents stored in a primary database. A schema defines which of the document properties are stored in a search database. When a search query is executed against a search database, a document satisfying the search query is returned from the search database along with the corresponding schema version number. If the schema version number associated with the document stored in the storage database is different from a target version number stored in a schema version lookup table, the data in the search database is presumed to be stale and a re-indexing of the search database is requested to bring the database current.

Abstract: Generally described, one or more aspects of the present application relate to a public snapshot service for creating and managing block-level snapshots. For example, the public snapshot service can create a snapshot based on (i) a specification of a parent snapshot to be used as a basis for a given snapshot, and (ii) a specification of the data blocks to be backed up in the given snapshot. The data blocks in the snapshot may be stored in a specific storage path structure that facilitates computation of a block difference with respect to the parent snapshot.

Abstract: Embodiments of the present disclosure relate to a method for managing backup data, an electronic device, and a computer program product. This method comprises: determining a number of times of space recycling operations that have been executed on a backup data block; determining, based on the number of times, a current popularity of the backup data block, the current popularity at least indicating a probability that the backup data block will be recycled in a to-be-executed space recycling operation; and moving, based on a determination that a former storage area where the backup data block is located does not correspond to the current popularity, the backup data block to a target storage area corresponding to the popularity. In this way, data with similar popularity can be managed in a more centralized manner, thereby reducing data rewriting caused by subsequent space recycling.

Abstract: In one example, the present disclosure describes a device, computer-readable medium, and method for instantiating intelligent service delivery parameters within protected hardware. For instance, in one example, a method includes instantiating within a user endpoint device an end-to-end slice of a communications network, wherein the slice is configured to carry data over the communications network subject to a predefined combination of network service metrics, transmitting user data from the user endpoint device to the communications network via the slice when the user endpoint device is able to establish a radio connection to the communications network, and storing the user data locally in the user endpoint device when the user endpoint device is unable to establish a radio connection to the communications network.

Abstract: Aspects of the disclosure relate to management of databases in different server environments. In particular, various aspects of this disclosure relate to correction, synchronization, and/or migration of databases between different database servers. A feed file that is rejected from loading in a database associated with a source server may prioritized in a destination server. A feed file hierarchy of the rejected feed file may be determined and the destination server may process loading of the rejected feed file to a database based on the determine feed file hierarchy. Any corrections applied at the destination server may also be applied at the source server.

Abstract: Various embodiments herein each include at least one of systems, methods, and software for generating, storing, and using non-unique secondary indexes of semi-structured data in database tables. One method embodiment includes creating and storing a non-unique secondary index (NUSI) for a database table based on column data stored in rows of the database table. The NUSI in some such embodiments identifying an offset to specific data included in the column data of at least some rows of the database table where the specific data is located. Other embodiments include building a NUSI index that includes values embedded therein.

Abstract: A system includes reception of a first instruction at a first system to effect a first change to a hierarchy data model, storage of a first record including first values specifying the first change in a local memory of the first system, reception of a second instruction to effect a second change to the hierarchy data model, storage of a second record including second values specifying the second change in the local memory of the first computer system, reception of an instruction to save the changed hierarchy data model, and, in response to the instruction, transmit the first record and the second record to a second system. The first record and the second record are received and merged to generate a third record including third values specifying a third change to the hierarchy data model, and a query language statement is generated to effect the third change to the hierarchy data model based on the third record.

Abstract: Systems and methods for performing file-level restore operations for block-level data volumes are described. In some embodiments, the systems and methods restore data from a block-level data volume contained in secondary storage by receiving a request to restore one or more files from the block-level data volume, mounting a virtual disk to the block-level data volume, accessing one or more mount paths established by the virtual disk between the data agent and the block-level data volume, and browsing data from one or more files within the block-level data volume via the established one or more mount paths provided by the virtual disk.

Abstract: Compiling source code objects to improve efficiency of compiling is described herein. The compiling includes determining, by a compiler, an object type of a to-be-compiled object in source code. A counter is set for the to-be-compiled object. When the object type of the to-be-compiled object is an object type that can be operated by only one thread at one moment, the compiler sets a counter counting rule for the counter of the to-be-compiled object.

Abstract: One example method includes discovering an application instance on a host, reporting the existence of the application instance, discovering application components of the application instance, and mapping the application components to information concerning an underlying filesystem and information concerning an underlying physical drive. The example method additionally includes freezing the application instance in response to a first instruction, and then thawing the application instance in response to a second instruction.

Abstract: According to one embodiment, a method, computer system, and computer program product for data synchronization between a source database system and a target database system, the source database system including a source dataset and the target database system including a target dataset that is configured to include a copy of the source dataset. The embodiment may include configuring the target database system to detect data changes in the source dataset. The embodiment may include applying the detected data changes on the target dataset.

Abstract: Various aspects of the subject technology relate to systems, methods, and machine-readable media for game state synchronization. The method includes recording game state changes in a distributed queue. The method also includes updating a global state based on the recorded game state changes in the distributed queue. The method also includes recording a snapshot of the global state, the snapshot comprising an offset from a latest game state change recorded in the distributed queue. The method also includes updating at least one in-sync service of a plurality of in-sync services based on the game state changes and the snapshot of the global state.

Abstract: Systems and methods to optimize utilization of a tablespace for export from a native database recovery environment are described. The system receives a database from a source host, operating in a native database recovery environment, at a backup host operating in a foreign database recovery environment. The foreign database recovery environment utilizes foreign snapshot files and foreign incremental files for storing the database. The system receives a tablespace identifier that identifies a tablespace and a point-in-time that identifies file information for export from the backup host to the source host. The backup host initiates a job to generate script information; create directories; materialize the file information; utilize an auxiliary database to generate tablespace metadata information; and communicate the tablespace metadata information and the script information and the file information, via the directories, and over a network, to the source host, to enable the source host to recover the tablespace.

Abstract: The invention relates to computer-implemented method for updating a target table with changes introduced into a source table. New data records are stored into partitions of the source table. An incremental update module copies the new data records from the source table into the target table, thereby assigning a new target partition ID contained within a first value domain to each copied data record. A view specifies a set of visible partition IDs as a union of the first value domain and of a current target partition ID set and allows the execution of database queries selectively on target table partitions having assigned a target partition ID that is element of the visible partition ID set. A batch update module performs an atomic batch update operation that comprises copying partitions of the source table into a respective new partition in the target table.

Abstract: File access statistics associated with a content file are maintained by a file tier agent of a secondary storage system. The content file is accessible via a primary storage system. A target storage tier corresponding to the file access statistics associated with the content file is determined according to one or more policies. At least a portion of data of the content file is migrated to the determined storage tier while accessibility of the content file via the primary storage system is maintained.

Abstract: An apparatus includes at least one processing device, with the at least one processing device being configured, in conjunction with synchronous replication of at least one logical storage volume between first and second storage systems arranged in an active-active configuration, to create a first snapshot of the logical storage volume in the first storage system, to create a second snapshot of the logical storage volume in the second storage system, to create a third snapshot of the logical storage volume in the first storage system, to initiate an asynchronous replication cycle to transfer differential data between the first and the third snapshots in the first storage system to the second storage system, and to utilize the second snapshot and the transferred differential data to create an additional snapshot of the logical storage volume in the second storage system that is synchronized with the third snapshot in the first storage system.

Abstract: Methods and systems for backing up and restoring different point in time versions of a virtual machine, a real machine, an application, a database, or a set of electronic files using a plurality of independently managed snapshot chains are described. The different point in time versions of the data being backed-up may be stored using two or more snapshot chains corresponding with two or more data partitions of the data being backed-up. Over time, additional full image snapshots may be acquired from an external server or generated locally by a storage appliance to limit the snapshot chain lengths and to limit the aggregate block chain lengths for the snapshot chains. Acquisition and generation of the additional full image snapshots may be staggered across different data partitions to limit computational and storage costs per snapshot.

Abstract: A key-value storage device includes a nonvolatile memory device and a memory controller. The nonvolatile memory device stores a value, a key which is referenced to identify the value, and key age data which are changed based on an erase operation of the value, and the memory controller that receives an erase command directing erasing of the value corresponding to the key from a host, generates hash data, a size of which is smaller than a size of the key, in response to the erase command, and transmits a complete message to the host. The memory controller accesses the key and the key age data stored in the nonvolatile memory device based on the hash data and erases the value based on the accessed key and the accessed key age data during an idle time after the transmission of the complete message.

Abstract: In one aspect, data replication processes include setting a minimum snap set creation interval, a maximum snap set creation interval, and a recovery time threshold (RTT). An aspect further includes creating snap sets during a synchronous replication process based on the snap set creation intervals and the RTT. The snap set creation further includes monitoring an amount of data changes since the last snap set creation, monitoring throughput statistics between a source and target system, and estimating an amount of time to replicate the data changes to the target system based on the data changes and throughput statistics. Upon determining the time to replicate the data changes reaches the RTT and the time since the last snap set creation passes the minimum interval, the snap set creation includes creating a next snap set, storing the next snap set at the source, and replicating the next snap set at the target.

Abstract: Generally described, one or more aspects of the present application relate to a public snapshot service for creating and managing block-level snapshots. Snapshots can be configured to enable writing at the sub-block level. Subsequent modifications made to blocks storage volume that forms the basis of the snapshots can be backed up as a sub-block level operations.

Abstract: Generally described, one or more aspects of the present application relate to a public snapshot service for creating and managing block-level snapshots. For example, the public snapshot service can create a snapshot based on (i) a specification of a parent snapshot to be used as a basis for a given snapshot, and (ii) a specification of the data blocks to be backed up in the given snapshot. The data blocks in the snapshot may be stored in a specific storage path structure that facilitates computation of a block difference with respect to the parent snapshot.

Abstract: Embodiments of the present disclosure provide a method, device and computer program product for migrating data. The method comprises initiating a first migration job for a first client and a second migration job for a second client. The method also comprises setting a first cache file for the first client and a second cache file for the second client, and a cache file records a hash value of a migrated file of a client. Moreover, the method further comprises executing the first migration job and the second migration job in parallel, wherein the first migration job is executed using the first cache file and the second migration job is executed using the second cache file.

Abstract: Example methods and systems are disclosed to provide autonomous vehicle sensor security. An example method may include generating, by a first autonomous vehicle, a first map instance of a physical environment using first environmental information generated by a first sensor of a first autonomous vehicle. A second map instance from at least one of a second autonomous vehicle located in the physical environment is received. The first map instance may be correlated with the second map instance. In response to a discrepancy between the first map instance and the second map instance, a secure sensor may be activated to generate a third map instance. In response to the third map instance verifying that the discrepancy accurately describes the physical environment, the first environmental information including the discrepancy is used to navigate the first autonomous vehicle.

Abstract: Embodiments are disclosed for a method that may include accessing, from a first data source, events including interactions between a mobile device and one or more network devices on a network at a locale, and receiving external data from a second data source that excludes the network devices. The method may further include determining, based on the interactions, one or more geographic positions and one or more corresponding time intervals of the mobile device at the geographic positions, and correlating the geographic positions and the time intervals with the external data to obtain a metric.

Abstract: A system for an in-memory row storage architecture can be provided. In some implementations, the system performs operations comprising processing a database transaction affecting at least a first row in an in-memory row store and at least a second row in a persistent page store, logging changes to the second row within a page store transaction log as part of the processing and prior to committing the database transaction, logging a final aggregated result of the first row as part of committing the database transaction within a row store transaction log that is separate and distinct from the page store transaction log, and altering at least a portion of the in-memory row store based on accessing the row store transaction log. Related systems, methods, and articles of manufacture are also described.

Abstract: A method for synchronizing a local file system (LFS) and a remote (cloud) file system (RFS) includes the step of storing events indicative of differences between the LFS and the RFS, where the events are associated with file system objects of the LFS and RFS. The method further includes the steps of prioritizing the events, generating file system operations for the events based at least in part on the prioritization of the events, and performing at least some of the file system operations to synchronize the LFS and RFS. Prioritizing the events includes defining a plurality of service classes (e.g., based on type of event, event attribute, etc.), assigning a priority to each of the service classes, and assigning each event to one of said service classes. Systems are also described. The invention allocates limited synchronization resources based on a predefined policy to synchronize important file system objects sooner.

Abstract: Techniques for archiving data in cloud/object storage using local metadata staging are provided. In one set of embodiments, a computer system residing at an on-premises site comprising on-premises storage can receive a snapshot of a dataset to be archived. The computer system can package data in the snapshot into one or more fixed-size data chunks and upload the one or more fixed-size data chunks to cloud/object storage. Further, concurrently with the packaging and the uploading, the computer system can stage metadata for the snapshot in the on-premises storage. Then, upon uploading all of the data of the snapshot, the computer system can upload the metadata staged in the on-premises storage to the cloud/object storage.

Abstract: A history preserving data pipeline computer system and method. In one aspect, the history preserving data pipeline system provides immutable and versioned datasets. Because datasets are immutable and versioned, the system makes it possible to determine the data in a dataset at a point in time in the past, even if that data is no longer in the current version of the dataset.

Abstract: The present disclosure describes systems and methods for aggregation and management of cloud storage among a plurality of providers via file fragmenting to provide increased reliability and security. In one implementation, fragments or blocks may be distributed among a plurality of cloud storage providers, such that no provider retains a complete copy of a file. Accordingly, even if an individual service is compromised, a malicious actor cannot access the data. In another implementation, file fragmenting may be performed in a non-standard method such that file headers and metadata are divided across separate fragments, obfuscating the original file metadata.

Abstract: A system and method for database migration is provided. In an example embodiment, the database migration technique may include execution modules having a first connection to a source database and a second connection to a target database, and orchestration modules adapted to identify first and second subsets of statements from a single storage table, wherein the first and second subsets of statements are executed by the execution modules during an uptime period of the source database.

Abstract: Methods, non-transitory computer readable media, and computing devices that group objects with different service level objectives for an application includes receiving a request including a service level data to provision a volume. One or more aggregates for the received service level is identified. a resource pool including the identified one or more aggregates is generated. The volume including the generated resource pool with the identified one or more aggregates for the received service level is provisioned.