Abstract: In one embodiment, an apparatus comprises a source system comprising a processing device coupled to memory. The processing device is configured to obtain an IO operation corresponding to an address of the source system. The IO operation comprises first user data. The processing device is further configured to store metadata associated with the IO operation in a first journal barrier of a replication journal of the source system and to close the first journal barrier. The processing device is further configured to determine that the first user data associated with the IO operation is missing from the first journal barrier and to obtain second user data from the address. The processing device is further configured to identify an interval from the first journal barrier to a second journal barrier and to provide the first journal barrier and the interval to a destination system.

Abstract: Systems, methods, and machine-storage mediums for optimizing snapshot image processing are described. The system receives a first read request to read data from optimized snapshot information including snapshot information and cached snapshot information. The first read request includes a first offset identifying a first storage location and a first length. The snapshot information includes a full snapshot and at least one incremental snapshot. The system identifies a first portion of the data is stored in the snapshot information responsive to identifying the first portion of the data is not stored in the cache snapshot information. The system identifies a second portion of data is stored in the optimized snapshot information, reads the first portion of data and the second portion of data from the optimized snapshot information, and communicates the data, including the first and second portions of the data, to the job.

Abstract: A digital data processor includes an instruction memory storing instructions each specifying a data processing operation and at least one data operand field, an instruction decoder coupled to the instruction memory for sequentially recalling instructions from the instruction memory and determining the data processing operation and the at least one data operand, and at least one operational unit coupled to a data register file and to the instruction decoder to perform a data processing operation upon at least one operand corresponding to an instruction decoded by the instruction decoder and storing results of the data processing operation. The at least one operational unit is configured to perform a table write in response to a look up table write instruction by writing at least one data element from a source data register to a specified location in a specified number of at least one table.

Abstract: Methods, systems, and devices for redundancy-based error detection in a memory device are described. A memory device may read multiple copies of a codeword from memory and generate for each codeword copy an error detection bit that indicates whether the memory device detected an error in that codeword. Additionally, the memory device may compare the codeword copies and generate one or more match bits that indicate whether corresponding portions of the codewords match. Using a combination of the error detection bits and the match bits, the memory device may determine the error status of each codeword.

Abstract: A file management device is a file management device that is connected to a plurality of manufacturing machines or a plurality of robots through a communication network and includes: an input unit to which a user inputs a file deletion command including a file name of files to be deleted; and a file deletion unit that causes the plurality of manufacturing machines or the plurality of robots to delete the files to be deleted with the same file name stored in the plurality of manufacturing machines or the plurality of robots in response to the file deletion command input through the input unit.

Abstract: A data disaster recovery method performed by a disaster recovery site includes: selecting a first backup copy when taking over a service of a production site; receiving service data through a virtual machine; obtaining changed block tracking CBT) information based on a CBT technology and the received service data, where the CBT information includes incremental information generated after the disaster recovery site receives the first backup copy; and sending, by the disaster recovery site, a first message to the production site, where the first message includes the CBT information.

Abstract: In general, one or more embodiments of the invention relates to systems and methods for detecting ransomware attacks earlier and closer to the time of attack. The ransomware attack can be detect by determining a change rate of data blocks between snapshots. The ransomware attack can also be detected by determining the pattern of changes in the blocks deviates from a normal pattern. By making these determinations, a quick identification of possible ransomware attacks can be made and other methods of mitigating the attack can be deployed when they are may still be useful to mitigate potential damage to a user's data.

Abstract: A method is performed by a device of a group of devices in a distributed data replication system. The method includes storing an index of objects in the distributed data replication system, the index being replicated while the objects are stored locally by the plurality of devices in the distributed data replication system. The method also includes conducting a scan of at least a portion of the index and identifying a redundant replica(s) of the at least one of the objects based on the scan of the index. The method further includes de-duplicating the redundant replica(s), and updating the index to reflect the status of the redundant replica.

Abstract: A three-dimensional stacked integrated circuit (3D SIC) having a non-volatile memory die, a volatile memory die, a logic die, and a thermal management component. The non-volatile memory die, the volatile memory die, the logic die, and the thermal management component are stacked. The thermal management component can be stacked in between the non-volatile memory die and the logic die, stacked in between the volatile memory die and the logic die, or both.

Abstract: A method includes creating a deduplicated universal share (US) of data objects, which in turn includes receiving a US of the data objects, deduplicating the US, wherein deduplicating the US includes: hashing segments of the US to generate respective US segment fingerprints; comparing US segment fingerprints to fingerprints for respective segments held in deduplication storage in order to identify segments in the deduplication storage that equate to the US segments, respectively, of the US; storing identifiers that directly or indirectly identify locations, respectively, of the segments, respectively, in the deduplication storage that equate to the US segments, respectively, of the US. After creating the deduplicated universal share, a deduplicated backup of the US is created without reassembling the US from segments held in the deduplication storage, the creating the deduplicated backup including: creating a list that comprises copies of the stored identifiers, and storing the list.

Abstract: In one set of embodiments, a hypervisor of a host system can determine that a delta between local and remote memory access latencies for each of a subset of NUMA nodes of the host system is less than a threshold. In response, the hypervisor can enable page sharing across the subset of NUMA nodes, where enabling page sharing comprises associating the subset of NUMA nodes with a single page sharing table, and where the single page sharing table holds entries identifying host physical memory pages of the host system that are shared by virtual machines (VMs) placed on the subset of NUMA nodes.

Abstract: In a dual node system, each node can store metadata updates in a volatile memory metadata log. Metadata pages can be used to access data stored on a volume. A node designated as preferred or affined for a volume can be expected to receive I/Os directed to the volume. A preferred node for a volume can record, in its volatile memory metadata log, the metadata changes for the metadata pages used to access data stored on the volume. A non-preferred or non-affined node for the volume can infrequently receive I/Os directed to the volume. A non-preferred node for a volume can record, in its volatile memory metadata log, pointers or references to the metadata changes for the metadata pages used to access data stored on the volume, where the pointers or references map to persisted copies of the meta changes as stored in a persisted metadata log.

Abstract: Technologies for providing efficient migration of services include a server device. The server device includes compute engine circuitry to execute a set of services on behalf of a terminal device and migration accelerator circuitry. The migration accelerator circuitry is to determine whether execution of the services is to be migrated from an edge station in which the present server device is located to a second edge station in which a second server device is located, determine a prioritization of the services executed by the server device, and send, in response to a determination that the services are to be migrated and as a function of the determined prioritization, data utilized by each service to the second server device of the second edge station to migrate the services. Other embodiments are also described and claimed.

Abstract: Systems, methods, and machine-readable media for creating, deleting, and restoring volume snapshots in a remote data store are disclosed. A storage volume and a storage operating system are implemented in a software container. Through a user interface, a user may create a snapshot of the volume to a cloud storage. A user may also delete individual snapshots from the cloud storage. Further, deletion of a most recent snapshot may occur by awaiting deletion (though marking as deleted to the user) until a next snapshot is received. Snapshots in the cloud storage are manipulatable even after destruction of the source volume (by destruction of the container, for example). A controller outside the container is used by implementing the same API as the controller in the container had. Full restores of snapshots in the cloud are also possible even when the original container and volume have been destroyed.

Abstract: Various processes for efficiently and effectively managing huge pages include a process for optimizing memory deduplication of huge pages, optimizing the promotion of one or more base pages to one or more huge pages and optimizing memory compaction of a memory space associated with a huge page.

Abstract: Methods, systems, and devices for performing memory command verification are described. A system may include a memory device and a memory controller, which may be external (e.g., a host device). The memory device may receive, from the memory controller, a command indicating a type of operation and an address. The memory device may decode the command and execute an operation (e.g., the operation corresponding to the decoded command) at an execution location on the memory device. The system (e.g., the memory device or the memory controller) may determine whether the executed operation and execution location match the type of operation and address indicated in the command, and the system may thereby determine an error associated with the decoding, the execution, or both of the command.

Abstract: A Virtual Tape Emulation (VTE) application is able to be updated without disrupting IO operations by hosts on a set of virtual tape drives managed by the VTE application. When an old VTE application is to be replaced with a new VTE application, the new VTE application is provided with the same configuration information as the old VTE application. Each time a host unmounts a virtual tape from a virtual tape drive on the old VTE application, device management for that virtual tape drive is migrated from the old VTE application to the new VTE application. Specifically, path and channel information for the virtual tape drive is transferred to the new VTE application, and applied by the new VTE application to a respective device on the new VTE application. Once device management for all virtual tape drives has been transferred, the old VTE application is terminated.

Abstract: An apparatus comprises at least one processing device that includes a processor and a memory coupled to the processor. The at least one processing device is configured to receive storage access protocol commands directed by one or more host devices to storage devices of a storage system over a storage area network, to generate statistics relating to the received storage access protocol commands, to process the generated statistics in a machine learning system trained to recognize anomalous access patterns to the storage devices over the storage area network, and to generate an alert indicative of an access anomaly based at least in part on the processing of the generated statistics in the machine learning system. A multi-path input-output (MPIO) driver of the one or more host devices may be provided with the alert and configured to initiate one or more remediation actions responsive to the alert.

Abstract: A storage system comprises nodes and drives, and includes at least one mounting area to install the drives. A node manages a parity group constituted of the plurality of drives; generates, in a case where a target drive that is installed in a first mounting area and that belongs to a first parity group is to be moved from the first mounting area to a second mounting area, difference information regarding a storage area where data was written into the target drive during a period in which the target drive is moved from the first mounting area to the second mounting area; and restore data written into the storage area by using data stored in other drives than the target drive that belongs to the first parity group based on the difference information, and write the data into the target drive that has been moved to the second mounting area.

Abstract: Systems and methods for cognitive encryption of data are disclosed. The methods may include maintaining a plurality of data storage systems in communication with an external metadata management system, operating the metadata management system to store metadata corresponding to data residing on the plurality of data storage systems, identifying a candidate data set residing on at least one of the plurality of data storage systems on which at least one security action should be performed using information included in the metadata management system, and in response to identifying the candidate data set, identifying the at least one security action.

Abstract: A request to perform a coordinated action is received. The coordinated action comprises a first plurality of operations. In response to reception of the request to perform the coordinated action, a second plurality of operations is performed in preparation for performing the coordinated action. A response to the request to perform the coordinated action is sent. The response to the request to perform the coordinated action comprises an indication that the first device will perform the coordinated action. An indication that the coordinated action is to be performed is received. In response to reception of the indication that the coordinated action is to be performed, the first plurality of operations is performed.

Abstract: The present disclosure relates to a data snapshot method and apparatus, a computer device and a storage medium. The method includes: acquiring the capacity of to-be-written data after snapshot, and comparing the capacity of the to-be-written data with a preset capacity; when the capacity of the to-be-written data is greater than or equal to the preset capacity, writing the to-be-written data into a snapshot volume in a Redirect On Write manner; when the capacity of the to-be-written data is less than the preset capacity, writing the to-be-written data into a solid state drive in a Redirect On Write manner; and when a background write-back thread detects that there is data writing into the solid state drive, writing corresponding data in a source volume into the snapshot volume in a Copy On Write manner, and writing the to-be-written data in the solid state drive into the source volume.

Abstract: This application provides a data backup method. The method includes: obtaining, by a first node, an identifier of a backup execution node from a storage device; and backing up data of the first node in the storage device responsive to determining that an ID of the first node is the same as the ID of the backup execution node. The embodiments of this application can improve reliability of the data stored by the first node, and prevent a plurality of nodes from redundantly backing up duplicate data in the storage device. The techniques disclosed herein reduce resource consumption.

Abstract: Embodiments include methods, systems and computer program products for managing a flash memory device. Aspects include monitoring a percentage of memory of the flash memory device that is in a ready to use state. Aspects also include operating the flash memory device in a first operating mode based on a determination that the percentage is greater than a first threshold value. Aspects further include operating the flash memory device in a second operating mode based on a determination that the percentage has fallen below the first threshold value. Aspects include operating the flash memory device in a third operating mode until the percentage exceeds the first threshold value based on a determination that the percentage has fallen below a second threshold value, which is lower than the first threshold value. The erasing of ready to erase memory block stripes is only performed during the third operating mode.

Abstract: A synthetic full backup of a source volume is retrieved, the synthetic full having been created from merging a backup chain including an incremental backup into a full backup. The full backup corresponds to a state of the source volume at a first time. The synthetic full corresponds to a state of the source volume at a second time, after the first time. A bitmap embedded into a snapshot of the source volume taken at the second time is retrieved. The bitmap identifies data from the source volume copied to secondary storage for the incremental backup. The backup chain is regenerated by copying from the snapshot the data identified by the bitmap to a regenerated incremental backup.

Abstract: Methods and systems for backing up and restoring sets of electronic files using sets of pseudo-virtual disks are described. The sets of electronic files may be sourced from or be stored using one or more different data sources including one or more real machines and/or one or more virtual machines. A first snapshot of the sets of electronic files may be aggregated from the different data sources and stored using a first pseudo-virtual disk. A second snapshot of the sets of electronic files may be aggregated from the different data sources subsequent to the generation of the first pseudo-virtual disk and stored using the first pseudo-virtual disk or a second pseudo-virtual disk different from the first pseudo-virtual disk.

Abstract: Disclosed are a Physical to Virtual (P2V) migration method and apparatus, and a storage medium. The method includes that: disk data of a Physical Machine (PM) is transmitted to a server, wherein the disk data is used by the server to create and start a Virtual Machine (VM) based on the disk data of the PM; a first dirty data generated by running of a program on the PM in a transmission process of the disk data of the PM is acquired; the disk data for creating the VM is updated based on the acquired first dirty data; and based on the updated disk data for the VM, the VM is controlled to resume running the program based on a progress of the program.

Abstract: A method and apparatus for expanding cloud volume, and a device and a readable medium are provided. The method includes: determining whether there is an request of a host in a cloud volume and confirming a snapshot started during cloud backup (S1); in response to there being no IO request, expanding a snapshot volume of the snapshot started during the cloud backup and the cloud volume (S2); expanding a bitmap of the snapshot started during the cloud backup, and setting a bitmap obtained by expanding as a first preset value (S3); after completion of a cloud backup task, expanding a snapshot volume of an another snapshot that is not started in the cloud backup (S4); and expanding a bitmap of the another snapshot that is not started in the the present cloud backup, and setting a bitmap obtained by expanding as a second preset value (S5).

Abstract: Methods and systems for using a hierarchical consistency group (CG) in a storage system are provided. A parent CG is associated with at least a first child CG having a plurality of storage volumes. An atomic application programming interface (API) provisions the parent CG and the first child CG by allocating storage and storing policies for the parent CG and the first CG. A storage service selected from a backup service, a replication service and a cloning service for the parent CG and the first CG is executed based on the stored policies.

Abstract: Aspects of a storage device are thermal management of a non-volatile storage device are provided. In various embodiments, a storage device includes corresponding memory locations on two or more dies. Corresponding memory locations on each die form an addressable group. A controller in thermal communication with each of the dies may detect an excess temperature on one of the dies while performing sequential host writes. Upon such detection, the controller may disable all writes to the detected die while continuing to perform writes to the memory locations of the other dies without throttling the other dies. The controller may then reactivate writes to the detected die when the temperature drops below a threshold.

Abstract: Disclosed herein are systems and method for determining a backup schedule on a computer system. In one exemplary aspect, a method may comprise collecting user behavior data on the computer system. The method may comprise analyzing the user behavior data to determine an optimal time of a backup session to create backup copies of modified data stored on a volume of the computer system and determining an optimal duration of the backup session based on the analyzed user behavior. The method may comprise determining a portion of the modified data that can be saved during the backup session within the optimal duration at the optimal time of backup, and performing the backup session comprising the portion.

Abstract: A memory controller is configured with a plurality of processors to be operated in parallel so that overhead of firmware may be reduced. The memory controller includes a first processor and a second processor. The first processor is configured to generate a command corresponding to a request received from a host and to translate a logical address included in the request into a physical address of a memory device. The second processor is configured to operate on data to be output to the memory device or on data received from the memory device. If the request is received from the host when the second processor is in an idle state, the first processor may control the second processor to release the idle state of the second processor and to perform an operation of logging command information corresponding to the request.

Abstract: A system and methods for application recovery across storage systems is provided. In some examples, the method includes replicating, from a source storage system to a target storage system, a volume including an application configuration for an application utilizing the volume, wherein the application is executing within a source application host. The method also includes detecting that the application executed in the source application host is unavailable. The method also includes retrieving, from the volume on the target storage system, the application configuration for the application. The method also includes instantiating the application on a target application host using the application configuration retrieved from the volume on the target storage system, wherein the application on the target application host is configured to direct data requests to the volume on the target storage system.

Abstract: Disclosed are techniques that provide for deduplication in an efficient and effective manner. For example, such methods, computer program products, and computer systems can include generating new feature information for one or more portions of a new backup image, generating first container range information by performing a container range calculation using the new feature information, generating existing feature information for one or more portions of an existing backup image, generating second container range information by performing the container range calculation using the existing feature information, determining a container range affinity between the first container range information and the second container range information, identifying at least one portion of the one or more portions of the existing backup image using a result of the determining, and prefetching the one or more fingerprints corresponding to the at least one portion of the one or more portions of the existing backup image.

Abstract: Systems, apparatus and methods are provided for low temperature management of a storage system. An apparatus may include a temperature sensor to generate a temperature reading, a timer configured with a time interval, a backup battery, one or more non-volatile memory (NVM) devices and a storage controller. The storage controller may be configured to: maintain a standby mode for low temperature management until a host electronic system has been turned off, start the timer and check the temperature reading when the host electronic system is turned off, determine that the temperature reading is below a temperature threshold, set the time interval based on the temperature reading, receive an interrupt from the timer when the timer counts to the time Interval, and perform low-temperature management operations for data stored in the one or more NVM devices using power supplied by the backup battery.

Abstract: According to one aspect of the present disclosure, a system includes an active network element having circuitry for executing a primary application and a transmission control protocol (TCP) module, multiple standby network elements having circuitry for executing a secondary copy of the primary application and a secondary TCP module, and a network connection coupled to one or more of the active and standby network elements, wherein the active network element and standby network elements are coupled to transfer data and acknowledgments via their respective TCP modules, and wherein the standby network elements are reconfigurable to communicate via the network connection to a peer regardless of the failure of one or two of the network elements.

Abstract: A primary storage array calculates signatures of chunks of production device data that are sent to a target device on a secondary storage array. The chunk signatures are sent to a signature device on the secondary storage array, where the chunk signatures are stored within the same LBA range on the signature device as their corresponding chunks are stored on the target device. Snaps of the target and signature device are created and associated as a snap pair. Later, the primary storage array calculates signatures of changed chunks of production device data that are sent to the target device. The changed chunk signatures are sent to the signature device. New snaps of the target and signature device are created and associated as a new snap pair. Chunk data is validated by calculating signatures of the chunks from the target device and comparing those signatures with the chunk signatures from the signature device.

Abstract: The Multi-Client Backup Deduplication Apparatuses, Methods and Systems (“MCBD”) transforms backup request, restore request, virtual machine initialization request inputs via MCBD components into backup response, restore response, virtual machine initialization response outputs. A backup request identifying a source volume is obtained. The source volume's master file table is retrieved and a file entry is selected. File data runs associated with the selected entry are determined. The file is reassembled in a buffer and split into file chunks Upon determining that a file chunk is not indexed in a chunk index, the file chunk is stored on a target volume and indexed in the chunk index. File chunk slice datastructures that map source volume offset location of file data on the source volume to the corresponding file chunk offset location of that file data in the file chunk on the target volume are generated and stored in a manifest file.

Abstract: Determining effective space utilization in a storage system, including: identifying an amount of data stored within the storage system that is associated with a user-visible entity; identifying an amount of data stored within the storage system that is associated with all snapshots of the user-visible entity; and reporting, in dependence upon the an amount of data stored within the storage system that is associated with the user-visible entity and the amount of data stored within the storage system that is associated with all snapshots of the user-visible entity, a total capacity utilization associated with the user-visible entity.

Abstract: Techniques are provided for block allocation for persistent memory during aggregate transition. In a high availability pair including first and second nodes, the first node makes a determination that control of a first aggregate is to transition from the first node to the second node. A portion of available free storage space is allocated from a first persistent memory of the first node as allocated pages within the first persistent memory. Metadata information for the allocated pages is updated with an identifier of the first aggregate to create updated metadata information reserving the allocated pages for the first aggregate. The updated metadata information is mirrored to the second node, so that the second node also reserves those pages. Control of the first aggregate is transitioned to the second node. As a result, the nodes do not attempt allocating the same free pages to different aggregates during a transition.

Abstract: A distributed storage system includes one or a plurality of storage units including a plurality of physical storage devices, and a plurality of computers connected to the one or plurality of storage units via a communication network. When receiving a write request for a logical volume, the computer writes write data corresponding to the write request and redundant data for making the write data redundant in a plurality of physical storage devices of the storage unit in a distributed manner, and collectively controls writing of a journal of write data for managing a write history of the write data and a journal of redundant data for managing a write history of the redundant data.

Abstract: Disclosed methods and systems leverage resources in a storage management system to restore a selected backup to a production site. The backup is partitioned into blocks with associated signatures. The production site may have blocks that have not changed from when the backup occurred, so those blocks do not need to be restored. Block signatures from the production site are compared with block signatures from the incremental backup to identify blocks that need to be restored. Efficiency may be achieved by synchronizing the replacement blocks from more easily accessible location where available before synchronizing from less accessible locations. In some embodiments, a user may specify the location of the site with the replacement blocks.

Abstract: A technique improves storage efficiency of an object store configured to maintain numerous snapshots for long-term storage in an archival storage system by efficiently determining data that is exclusively owned by an expiring snapshot to allow deletion of the expiring snapshot from the object store. The technique involves managing index data structures to enable efficient garbage collection across a very large number of data objects. When a snapshot expires, the technique obviates the need to scan the numerous snapshot data objects to determine which index structures are no longer needed and can be reclaimed (garbage collected). The technique is directed to management of underlying storage based on different sets of policies. When certain snapshots expire and are ready for deletion, the technique is directed to finding those data blocks that are no longer referenced (used) by any valid snapshots.

Abstract: A memory controlling device configured to connect to a memory module including a resistance switching memory cell array which is partitioned into a plurality of partitions including a first partition and a second partition is provided. A first controlling module accesses the memory module. A second controlling module determines whether there is a conflict for the first partition to which a read request targets when an incoming request is the read request, instructs the first controlling module to read target data of the read request from the memory module when a write to the second partition is in progress, and suspends the read request when a write to the first partition is in progress.

Abstract: Described is a system for synchronizing expiration times for incremental backup data stored on a cloud-based object storage. More particularly, the system may provide a layer of intelligence when updating the expiration times associated with backup data to ensure that a full recovery of the client data to a point-in-time of any incremental backup may be performed. To provide such a capability, the system may maintain specialized metadata identifying expiration times for objects and a list of objects required to perform a full recovery to a point-in-time for each of the performed backups (e.g. full or incremental). The system may access this metadata to identify which objects stored by a previous backup are still referenced by a subsequent backup. Based on the identified objects, the system may synchronize object expiration times to ensure objects are not prematurely deleted from the object storage.

Abstract: An illustrative live synchronization feature uses file system block-level backup copies, snapshot techniques, change tracking, and volume-level granularity to ensure the integrity of destination volumes. Two protection mechanisms guard the destination data and ensure consistency from one live sync restore to the next. First, an inter-job software snapshot captures the destination volume image after each restore. The snapshot is created at the very end of each live sync restore and is reverted at the beginning of the next live sync restore. A second and more granular protection mechanism uses intra-job block monitoring to detect, and later to reverse, changes that the snapshots cannot capture. This second mechanism acts as a mini-block-level restore nested inside another block-level restore. This dual approach ensures that each incremental live sync restore finds the destination volume with a guaranteed pristine image identical to where the preceding live sync restore left it.

Abstract: Systems, methods, and computer readable storage mediums for replicating data to the cloud. A first dataset is replicated from a first storage subsystem to the cloud, with only the changes made to the first dataset since a previous replication event being replicated to the cloud. Additionally, the data may be deduplicated and compressed before being replicated from the first storage subsystem to the cloud. In some cases, the first dataset may be replicated to the cloud in a stream format which is not directly usable and then, at a later point in time, loaded onto a second storage subsystem from the cloud.

Abstract: System and method for storage data in SSD may be provided. The method may include receiving data writing feature information sent by a file system during an initialization process. The method may include determining, based on the data writing feature information, a size of metadata storage space corresponding to the metadata. The method may further include determining, based on the size of metadata storage space, a target storage region for storing the metadata in the SSD.

Abstract: According to some embodiments, methods and systems may be associated with a cloud computing environment. A workload prediction framework may receive observed workload information associated with a database in the cloud computing environment (e.g., a DataBase as a Service (“DBaaS”)). Based on the observed workload information, a Statement Arrival Rate (“SAR”) prediction may be generated. In addition, a host variable assignment prediction may be generated based on the observed workload information. The workload prediction framework may then use the SAR prediction and the host variable assignment prediction to automatically create a workload prediction for the database. A physical database design advisor (e.g., a table partitioning advisor) may receive the workload prediction and, responsive to the workload prediction, automatically generate a recommended physical layout for the database (e.g., using a cost model, the current physical layout, and an objective function).

Abstract: Aspects of the present disclosure relate to watermarks and watermarking techniques for data streaming pipelines. Time stamp and offset timeline data is shared by computing instances along the pipeline to enable improved watermarking of the data stream through the pipeline. The improved watermarks enable better determination of completeness for the data stream and improve materialization of the results. The watermarking techniques can include periodically publishing watermark data by processing units of a vertex, fetching a merged watermark for a vertex by a vertex, and/or watching a data storage for the watermark data for events. Consensus algorithms can be used to maintain consensus among vertices for the watermark data.