Abstract: Systems and methods for managing data, such as metadata or index databases. In one exemplary method, a notification that an existing file has been modified or that a new file has been created is received by an indexing software component, which then, in response to the notification performs an indexing operation, where the notification is either not based solely on time or user input or the notification includes an identifier that identifies the file. Other methods in data processing systems and machine readable media are also described.

Abstract: Systems and methods for managing electronic data are disclosed. Various data management operations can be performed based on a metabase formed from metadata. Such metadata can be identified from an index of data interactions generated by a journaling module, and obtained from their associated data objects stored in one or more storage devices. In various embodiments, such processing of the index and storing of the metadata can facilitate, for example, enhanced data management operations, enhanced data identification operations, enhanced storage operations, data classification for organizing and storing the metadata, cataloging of metadata for the stored metadata, and/or user interfaces for managing data. In various embodiments, the metabase can be configured in different ways. For example, the metabase can be stored separately from the data objects so as to allow obtaining of information about the data objects without accessing the data objects or a data structure used by a file system.

Abstract: A method and system for monitoring and maintaining the consistency of replicated databases in a shared-nothing database cluster architecture is presented. The method involves the continuous monitoring of the database transaction logs that are maintained by the autonomous database managers that manage the individual database replicas in the cluster. In the event that data discrepancies are detected via the comparison of the transaction logs of the individual database replicas, remedial action is carried out according to configured rules in the main cluster controller. These remedial actions may include, but are not limited to, making a database node inactive within the cluster, or automatically repairing any data discrepancies that are detected among the database replicas in the cluster.

Abstract: Transactional record management methods and systems enabling multiple independent servers (such as database servers) using shared storage to initiate transactions in parallel without inter server communication and without locking the records used by the transaction. The in-flight transactions can be included in a shared transaction log without a final determination of whether the transaction committed. The log updates can be broadcast to each of the servers, which each parse the log, using the same rules of analysis, and therefore each compute server can independently and asynchronously come to the same conclusion as to which transactions aborted and which transactions committed.

Abstract: Techniques for making light-weight checkpoints in logs of streams of transactions and for extending the logs from the checkpoints. The state saved in the light weight checkpoint need only include the state of all transactions that are active at a prior point in the log and are still active at the checkpoint. A log is extended from the checkpoint by processing new transactions in the transaction stream beginning at the prior point to produce the extending log. When the checkpoint is reached, the state saved at the checkpoint is used to continue to produce the extending log. Techniques are further disclosed for selecting points in the stream of transactions at which the checkpoints may be made and for determining the distance between checkpoints. As disclosed, the log in which the checkpoints are used is a logical log made from a physical log produced by a relational database system. The logical log may be used for replication and for data mining.

Abstract: Data written in the primary logical volume of the first storage device are transmitted to the third storage device via the second storage device, the data being written in the same location as the primary logical volume within the secondary logical volume in the third storage device; when transmission of the data stops among the first to the third storage devices, the respective second storage device and the third storage device manage locations in the secondary logical volume where the data held thereby are to be written; and, when transmission of the data resumes among the first to the third storage devices, the locations in the secondary logical volume managed by the respective second and the third storage devices are aggregated, the data to be written in the respective aggregated location in the secondary logical volume being transmitted from the first storage device to the third storage device via the second storage device.

Abstract: Techniques for end-to-end tracing of database operations in source code for database applications are provided. Source code on the client side is instrumented to measure time spent on selected database operation statements (e.g., queries). The selected database operation statements can be uniquely tagged or identified. Tracing is enabled on the database application on the server side so that a log is produced, where the log includes performance (e.g., time spent) information on database operation statements that are executed. The performance information in the log is correlated to the selected database operation statements so that, among other things, the time spent on the client side and the time spent on the server side can be presented for analysis.

Abstract: A data management system or “DMS” provides a wide range of data services to data sources associated with a set of application host servers. The data management system typically comprises one or more regions, with each region having one or more clusters. A given cluster has one or more nodes that share storage. To facilitate the data service, a host driver embedded in an application server connects an application and its data to a cluster. The host driver provides a method and apparatus for capturing real-time data transactions in the form of an event journal that is provided to the data management system. The driver functions to translate traditional file/database/block I/O into a continuous, application-aware, output data stream. Using the streams generated in this manner, the DMS offers a wide range of data services that include, by way of example only: data protection (and recovery), disaster recovery (data distribution and data replication), data copy, and data query and access.

Abstract: The invention provides a method of managing data updates in DOS-based data storage device having an OTP memory die that includes a code region having a first memory capacity and a code region access resolution and a data region having a second memory capacity and a data region access resolution. The second memory capacity is larger than the first memory capacity and the code region access resolution is finer than the data region access resolution. The method includes chronologically writing a log entry in the code region indicating the change in FAT and root directory for each change in user data written in the data region.

Abstract: A system for logging database transactions comprises a memory (comprising a direct access storage device) operable to store a database, a database log, and a plurality of log buffers associated with the database log. Each log buffer is operable to store at least one log record associated with a database transaction. The system further comprises a processor operable to log a first record of a first database transaction using the database log and write the database log to the storage device. The processor is further operable to log a second record of a second database transaction in a first of the plurality of log buffers substantially concurrently to the database log being written to the storage device.

Abstract: Aspects of the subject matter described herein relate to checkpoints for a file system. In aspects, updates to the file system are organized into checkpoint buckets. When a checkpoint is desired, subsequent updates are directed to another checkpoint bucket. After global tables have been updated for updates in the current checkpoint bucket, a logical copy of the global tables is created. This logical copy is stored as part of the checkpoint data. To assist in recovery, a checkpoint manager may wait until all updates of the current checkpoint bucket have been written to storage before writing final checkpoint data to storage. This final checkpoint data may refer to the logical copy of the global tables and include a validation code to verify that the checkpoint data is correct.

Abstract: A plurality of second groups respectively including one or more second volumes are configured in correspondence with each of the first groups of a remote copy source in a remote copy destination, journals are acquired from the first storage apparatus periodically and in the order the journals were created for each of the configured second groups, and the acquired journals are reflected in the corresponding second volume in the corresponding second group. In addition, the latest time stamp of each of the second groups containing the journals retained in the second volume in an unreflected state is compared, the time difference of the latest and oldest time stamps is detected, and prescribed control processing is executed for acquiring the journals regarding the second group with the oldest time stamp in preference to the journals regarding other second groups when the time difference exceeds a preset threshold value.

Abstract: Aspects of the subject matter described herein relate to data change ordering in multi-log based replication. In aspects, local seeds are maintained for subtransactions involved in a transaction, where each subtransaction may occur on a different node that hosts one or more database fragments involved in the transaction. When a subtransaction communicates with another subtransaction in a transaction, the subtransaction sends its local seed to the other subtransaction. The receiving subtransaction compares its local seed with the received seed and updates its local seed if the received seed is logically after its local seed. A subtransaction uses a local seed to generate sequence identifiers for changes made by the subtransaction. These identifiers allow data changes of a transaction that are made on multiple nodes to be partially ordered relative to other changes made during the transaction.

Abstract: In certain embodiments, a tiered storage system is disclosed that provides for failover protection during data backup operations. The system can provide for an index, or catalog, for identifying and enabling restoration of backup data located on a storage device. The system further maintains a set of transaction logs generated by media agent modules that identify metadata with respect to individual data chunks of a backup file on the storage device. A copy of the catalog and transaction logs can be stored at a location accessible by each of the media agent modules. In this manner, in case of a failure of one media agent module during backup, the transaction logs and existing catalog can be used by a second media agent module to resume the backup operation without requiring a restart of the backup process.

Abstract: Various technologies and techniques are disclosed for improving configuration of database audits. Multiple audits can be created for a database. Any one of the audits can be turned off without impacting an operation of any remaining audits. Database audits can be ported from one database server instance to another. Database audit metadata is stored in a database separately from a database server instance to which the database is attached. The database can be detached from the database server instance and attached to a second database server instance, with the database audit metadata moving along with the database automatically. Any audits associated with the database audit metadata are automatically activated for the second database server instance. Database audits can be configured using a hierarchical model.

Abstract: Architecture for replicating a database through log shipping. A replication service replicates data using one or more new internal application program interfaces (APIs), a replay configuration API which abstracts configuration information for each destination (or target) instance. APIs for log file management as part of the replication process are provided for opening communications for log shipping, tracking success of log shipping and closing out log shipping communications. Log shipping APIs are also provided for log truncation control. A ship control object is provided as a base class which includes an API for other components of the replication service to operate on log files when the log files are placed into a log directory. The ship control API is utilized by the replication service and for interaction with storage system log files.

Abstract: A minimized journaling mechanism stores minimized journal data in a format that allows for display and outputting the journal data in human-readable form. When a change to a record occurs, instead of writing only the changed bytes, all of the bytes in each field that changed are written to the journal, along with all of the bytes in each field selected to be always journaled. A default object is created with default data in all of the fields. When the journal entry needs to be output in human-readable form, the default object is read, and the minimized journal entry is then overlaid on the default object. The result is an object that contains default data in all non-selected fields that were not changed, with the journal data in all fields and that did change and in all fields that were selected to always be journaled.

Abstract: Technologies are described herein for high-performance and space efficient journaling mechanisms. A journal queue can buffer incoming I/O operations. The queue can be read by a journal write module. The journal write module can form metadata headers to store into the journal along with each I/O operation. Compound metadata headers may be formed for multiple I/O operations and the combination of multiple I/O operations may be efficiently stored to the journal as one journal write. Multiple journal entries may be written into the journal in parallel while maintaining the sequential ordering of the journal. A queue depth threshold can be established for determining if journal entries should be combined into compound entries or not. Multiple concurrent readers can support continuous data protection, and various data replication features. Each reader can be represented as a reader pointer and a journal writer can invalidate reader pointers when related data is overwritten.

Abstract: An administration server in a database management system retrieves log files in a plurality of formats from a plurality of clients through helper programs running on the clients. The plurality of clients can include Web servers, application servers, and database servers. The log files can be generated by software modules on the clients. An administration engine converts log entries in the log files into a unified format for display. The converted log entries can be stored in a log database. Upon a user request, the administration server presents the log entries to the user in a log viewer. The log viewer can display log entries originated from heterogeneous software modules in a unified view.

Abstract: A method, system, and computer program product for providing continuous data protection for write data supplied from a host incorporating a processor device to data storage in accordance with write input/output (I/O) of an owning application is provided. The write data is identified as one of a synchronous candidate and an asynchronous candidate. The write data identified as the one of the synchronous candidate and the asynchronous candidate is transferred to at least one sidefile. The write data transferred to the at least one sidefile is secured in a nonvolatile storage journal. The nonvolatile storage journal is preset with a timeout limit applicable to write data identified as the synchronous candidate. If the timeout limit is exceeded, a loss mitigation operation to safeguard the write data identified as the synchronous candidate is performed.

Abstract: A method and apparatus for efficiently logging changes to a file system are disclosed. A file system filter driver intercepts I/O requests directed to a file system, and analyzes the I/O requests to determine whether the requests are write or modify requests directed to a file included in a list of files to be monitored for changes. If the I/O request is directed to such a file, the file system filter driver logs the I/O request to a b-tree structured change log file. Furthermore, in one embodiment, the file system filter driver is configured to log file system changes to multiple log files to coincide with snapshot operations, such that the file system changes to be included in each snapshot are included in separate log files.

Abstract: A method and system for acquiring a quiesceing set of information associated with a virtual machine. A virtual machine is cloned. The cloned virtual machine has an associated persistent storage device. The state of the persistent storage device is transformed into a quiesced state of the cloned virtual machine by utilizing a shut-down process. The shut-down process is executed on the cloned virtual machine to quiesce the cloned virtual machine and the quiesceing set of information of the cloned virtual machine is automatically reduced to information stored on the persistent storage device.

Abstract: Continuous data protection is performed as two parallel processes: copying a data block from the storage device into the backup storage device (creating initial backup) and copying the data block to be written to the data storage into the incremental backup. When a write command is directed to a data storage block, it's intercepted and redirected to the backup storage, and data, which is to be written in accord to the write request, is written to the incremental backup on the backup storage. If write command is also directed to a data storage block identified for backup that has not yet been backed up, the identified data storage block is copied from the storage device to the intermediate storage device, the write command is executed on the identified data storage block from the storage device, and the data storage block is copied from the intermediate storage device to the backup storage device. In case of an error accessing a block on the storage device, the block is marked as invalid.

Abstract: A method and system for generating and restoring a backup image are disclosed. A server receives a request, which includes a backup path, to generate a backup image. The server generates a backup image offset map that indicates a relative location where a file is stored in the backup image before file data has been written to the backup image. The server stores the backup image offset map in a portion of the backup image on a storage device. The backup image offset map comprises an array having a number of elements equal to a number of inodes on a file system that hosts a backup path, an element of the array representing an inode on the file system, and indicates the relative location where a file that is associated with a number of the inode is stored in the backup image.

Abstract: Disclosed herein are systems and methods for managing the distribution and synchronization of data across multiple locations. The invention uses a static, rule-based mechanism that results in fast, successful data updates in a consistent and controlled manner and without the requirement for manual reconciliation. In one embodiment the data relates to financial transactions.

Abstract: Various embodiments include one or more of systems, methods, software, and data structures that allow dual access to concurrent data in a database management system including uncommitted updated data. Some embodiments include a dual snapshot of data in a multi-version concurrency control style database, but the embodiments described herein are equally applicable to virtually any other type of database and database management system.

Abstract: One aspect of the invention is a method for backing up data. An example of the method includes querying a backup infrastructure to ascertain backup capabilities of the backup infrastructure. This example also may include broadcasting the capabilities of the backup infrastructure, wherein the capabilities are broadcasted by the backup infrastructure. This example also includes determining if the backup infrastructure supports a desired backup type. If the backup infrastructure supports a desired backup type, this example may also include performing a backup of data using the desired backup type.

Abstract: A method and system for efficient space management for single-instance-storage volumes is provided. A backup module storing data within a collection of containers according to access locality and retention time of the data, wherein the retention time defines an amount of time the data is stored within the collection of containers before deletion of the data, and the access locality comprises an order in which the data is to be accessed is further provided. A compaction module compacting the stored data by selecting at least two containers among the collection of containers, wherein the selection is performed using a predetermined criteria that includes access locality and retention time of the data is also provided. The compaction module distributes the data among the at least two containers. The compaction criteria creates an imbalance among the containers to create more empty, full, or nearly full containers.

Abstract: Creating a synthetic database backup using binary logs. A full backup of a database may be stored, followed by one or more incremental backups. The one or more incremental backups may include first binary log files. A first synthetic differential backup may be created using the first binary log files. Creating the first synthetic differential backup may include translating the first binary log files into a first plurality of SQL statements. The first plurality of SQL statements may be executable to recreate the changes in the database since the full backup. Translating each of the first binary log files into the first plurality of SQL statements may be performed before performing a recovery process to decrease recovery time.

Abstract: A storage system capable of identifying all volumes to be audited is provided. Operation logs 3005 concerning the operations of volumes are associated with a time and stored in storage apparatuses 1400 and 1500, and a management computer 1100 collects information about these operation logs 3005, restores the status history of the operation logs 3005, sets, as an audit range, the operation logs 3005 including volumes with an audit period and audit target data stored therein, extracts the operation logs 3005 belonging to the audit range from the restored status history of the operation logs 3005, and transmits the extracted operation logs 3005 to a computer (requesting computer) 1000. As a result, all the audit target volumes can be identified.

Abstract: The present invention generally relates to log message processing such that events can be detected and alarms can be generated. For example, log messages are generated by a variety of network platforms (e.g., Windows servers, Linux servers, UNIX servers, databases, workstations, etc.). Often, relatively large numbers of logs are generated from these platforms in different formats. A log manager described herein collects such log data using various protocols (e.g., Syslog, SNMP, SMTP, etc.) to determine events. That is, the log manager may communicate with the network platforms using appropriate protocols to collect log messages therefrom. The log manager may then determine events (e.g., unauthorized access, logins, etc.) from the log data and transfer the events to an event manager. The event manager may analyze the events and determine whether alarms should be generated therefrom.

Abstract: A method, apparatus, and system of a sequencing technique to account for a clock error in a storage area network are disclosed. In one embodiment, a system of a backup server includes a processing module to examine a data timestamped with a sequence of characters denoting a time according to a clock source, an analysis module to determine that the data has been timestamped at an earlier time than an other data previously received, a substitution module to provide the data an incremental sequence number placed with the data using an algorithm until a new data is received that includes a future timestamp with a later timestamp than the timestamp of the other data, and a storage module to store the data.

Abstract: A method and apparatus for agent-less auditing of a server, wherein the apparatus processes data for auditing a server. One or more portions of audit information (e.g., transaction log, trace log, or both) are coupled from a target server to an information server. The one or more portions of the audit information are processed at the information server to create audit data. The audit data is stored in an audit data repository.

Abstract: The present invention provides a data logging system and a data logging method for heterogeneous data. The record file and the index file according to the present invention are constructed in such a way that the data from the log record corresponding to a given target visual data line are accessible within essentially constant time independent of the position of the visual data line within the data view, the size of the data view, the size of the log record, and the size of the record file. In the present invention, the interpreted data for each log record, occupying zero or at least one of the visual data lines, are represented within the data view continuously side by side. The computer readable medium having recorded thereon a program allowing a computer to execute the data logging method according to the present invention and the computer program product for the same are also provided.

Abstract: A mechanism is provided for archiving and restoring transaction files of a database system without requiring the copying of the transaction files. A server system executes a host application that implements the database system comprising a set of database files and a set of transaction files that contains updates to the set of database files, the files being stored in a storage system. A backup engine interacts with the host application to archive a transaction file having an original filename in an original directory by creating a corresponding archive filename to the transaction file in an archive directory. The archive transaction files may be used later to perform a copy-less restore process of the database system that does not require copying of any transaction files. The copy-less restore process restores any altered original filenames by replacing, in the original directory, an altered original filename with its corresponding archive filename.

Abstract: Embodiments of the invention provide systems and methods for maintaining audit records for a database or other resource. According to one embodiment, a method for maintaining audit records for a database can comprise detecting an operation involving at least one record of the database. An audit record can be generated for the operation and the audit record can be stored in an audit table in the database. Insert operations and select operations for the audit table can be supported but other operations for the audit table can be prohibited. Additionally or alternatively, creation of more than one audit table having a same name and schema can be prevented.

Abstract: Disclosed is a method to synchronize the state of an application and an application's objects with data stored on the storage system. The storage system provides API's to create special data, called a marker journal, and stores it on a journal volume. The marker contains application information, e.g. file name, operation on the file, timestamp, etc. Since the journal volume contains markers as well as any changed data in the chronological order, IO activities to the storage system and application activities can be synchronized.

Abstract: Described are a method and computer program product for capturing file operations applied to a data set. One or more file operations having a corresponding time sequence which modify said data set are recorded. A request for a copy of the data set in a consistent state is received. It is determined at which point in said corresponding time sequence said data set is in a consistent state as a result of applying a portion of said file operations. The point in said corresponding time sequence at which said data set is in a consistent state is marked. A system is also described for capturing file operations as applied to a data set. The system includes a mirroring driver and a replication service. Also described is a data structure stored in a memory including a plurality of records corresponding to a time sequence of recorded file operations, and one or more consistency point markers inserted into said plurality of records.

Abstract: A transaction safe file system uses two sets of file allocation tables and bitmap images to perform file modifications on one of the sets while the other set remains a last known good set. After a modification is complete, a pointer is changed to the newly modified set, and the newly modified set becomes the last known good set. The sets are then synchronized. The file allocation table is used to define cluster chains while the bitmap image is used to determine if a cluster is free or not. In some operations, only the bitmap image may need to be manipulated. The file system may be used in a transaction safe mode as well as a non-transaction safe mode.

Abstract: Data written in the primary logical volume of the first storage device are transmitted to the third storage device via the second storage device, the data being written in the same location as the primary logical volume within the secondary logical volume in the third storage device; when transmission of the data stops among the first to the third storage devices, the respective second storage device and the third storage device manage locations in the secondary logical volume where the data held thereby are to be written; and, when transmission of the data resumes among the first to the third storage devices, the locations in the secondary logical volume managed by the respective second and the third storage devices are aggregated, the data to be written in the respective aggregated location in the secondary logical volume being transmitted from the first storage device to the third storage device via the second storage device.

Abstract: Architecture that eliminates the need for on-disk full backups of data retaining only changes that have occurred, in a separate table. Thus, the architecture provides for incremental recovery of incremental changes in a relational database (e.g., SQL). The architecture provides improved recovery time and recovery point objectives. By using the incremental capture of changed data (e.g., in an XML format), the capability is provided to capture schema changes, query the incremental change data and efficiently restore user data to an earlier point-in-time state. Changes (e.g., insert, update and delete operations) are tracked (e.g., continuously) by a set of triggers and the incrementally captured changed rows are inserted in a data capture table (a differential change “delta” table) in a human-readable format (e.g., XML). Rollback is also provided.

Abstract: A system and method for checkpoint tuning in a computer environment comprising a processor coupled to a statistical database and at least one database or table manager and data log files. The data log files store data relating to the operations of the database or table manager. The processor executes a module that captures data comprising the number of operations, the time taken by the checkpoint process, folded journal entries, long running transactions, and the ratio of WRITE:READ operations. The processor then stores the captured data and processes the collected data, generating statistics therefrom. The statistical data is stored in the statistical database. The system selects checkpoint intervals based on the data statistics; and implements the checkpoint intervals.

Abstract: A first log reader publishes first messages to a plurality of queues. The messages comprise changes for transactions extracted from a log by the first log reader. In response to one of the queues becoming unavailable, a second log reader is launched to read and extract from the log, and to publish second messages comprising changes for transactions extracted from the log by the second log reader to the unavailable queue as a catch-up queue. In response to the catch-up queue becoming available and the second log reader reaching the end of the log, the publishing of the second messages for the catch-up queue is transferred from the second log reader to the first log reader.

Abstract: In some embodiments, a method includes receiving input that includes a criteria for filtering a set of records to define a subset of records within the set of records. The criteria is defined by an end design user. The method includes updating a business class group with the criteria, wherein the business class group is associated with an action for access the subset of records through the business class group. The method includes outputting a business application to be executed by at least one end user to access the subset of records through the business class group.

Abstract: Assigning session identifications to log entries and generating anonymous log entries are provided. In order to balance users' privacy concerns with the need for analysis of the log entries to provide high quality search results, non-user-specific data fields, such as a user's location (e.g., city, state, and latitude/longitude) and connection speed, are inserted into the log entries, and user-specific data fields, such as the IP address and cookie identifications, are deleted from the log entries. In addition or alternatively, prior to anonymization of the log entries, session identifications are assigned to identified groups of log entries. The groups are identified based on factors such as the user's identification, the IP address, the time of search, and differences between the search terms used in the search queries.

Abstract: One embodiment is a method that builds a target database with transaction logs from an online source database. The transaction logs include both existing data in the source database and updates that occur to the source database while the target database is being built.

Abstract: System, method, computer program product embodiments and combinations and sub-combinations thereof for data replication in a database system environment are described. In an aspect, the data replication includes grouping, in-memory, a plurality of transactions to be replicated as a single transaction from a source database system to a target database system. A plurality of net row changes is compiled for the plurality of transactions, and data inconsistency detection and resolution within a command application order are performed. Further included is bulk application of the plurality of net row changes to the target database system.

Abstract: In one embodiment, systems and methods are provided for nonlinear journaling. In one embodiment, groups of data designated for storage in a data storage unit are journaled into persistent storage. In one embodiment, the journal data is recorded nonlinearly. In one embodiment, a linked data structure records data and data descriptors in persistent storage.

Abstract: A method, article of manufacture, and apparatus for tracking a plurality of objects being stored are disclosed. In an embodiment, this comprises computing the hash value of the name of each object being stored, concatenating the hashed values into a hint, and storing the hint in an index. The size of the hash values is determined by dividing the number of objects into the space available for storing the concatenation, and the range of the hash values may be determined by choosing a prime number smaller than the number that can be represented by the space available for storing the concatenation. Either the hint or the longest pathname containing the objects can be stored, and this may be selected based on the application.

Abstract: A backup system that utilizes contextual and semantic concepts is described. The backup system provides for the ability to create a version changes log for listing and tracking all the changes in the different versions of the file. The version changes log creates a contextual description around the changes, deletions and additions. The semantic concept log is created from the version changes log to create a log of all of the semantic concepts associated with each change. A visualization builder then creates visualizations that can be used by the user to search for changes, deletions and additions whether in a text file or an image file.