Abstract: Techniques for improving database searches are described herein. In an embodiment, a server computer system stores one or more first datasets in a first data repository and one or more second datasets in a second data repository. The server computer receives a request to perform an analysis on a particular dataset. The server computer determines that the particular dataset is stored in the first data repository and the second data repository. Based, at least in part, on an attribute of the request, the server computer selects the second data repository and responds to the request with data from the particular dataset stored in the second data repository.

Abstract: A computer-implemented method includes determining a plurality of bulk sets for querying database records. The method also includes assigning a plurality of keysets to the plurality of bulk sets, with each keyset comprising a unique set of dimension attribute values from the database records. The method also includes calculating a predicted load score of each bulk set. The method also includes performing a transfer of a keyset from a first bulk set to a second bulk set when the transfer reduces a difference between predicted load scores of the first bulk set and the second bulk set. The method also includes, after the transfer, executing bulk queries using the plurality of bulk sets.

Abstract: Approaches for preserving customizations made to a data management system. Metadata that identifies a mapping between data fields of one or more data source schemas to columns of one or more data tier schemas is maintained. Each of the one or more data tier schemas stores data retrieved from the one or more data source schemas. Values from the metadata are read to dynamically generate software processes. The software processes may update one or more existing ETL processes to reflect an updated state of the one or more data tier schemas and may generate any new ETL processes required to retrieve data stored in the one or more data source schemas and load the retrieved data into the one or more data tier schemas. Customizations made to portions the data management system are preserved in an automated fashion when new versions of system components are deployed.

Abstract: A method and system for analyzing a corpus of data artifacts is disclosed. The method comprises obtaining, by a computer, a semantic representation of the data artifacts, where the semantic representation indicates (1) entities identified in the data artifacts, and (2) semantic relationships among the entities as indicated by the data artifacts. The method further comprises clustering the data artifacts into clusters of semantically related data artifacts based on the semantic representation and inferring additional semantic relationships between pairs of the entities. The inferring comprises applying, on a cluster-by-cluster basis, a multi-tiered network of inference engines to a portion of the semantic representation corresponding to the cluster, where the multi-tiered network of inference engines includes a domain-independent inference tier and a domain-specific inference tier.

Abstract: Certain aspects and features of the present disclosure relate to systems and methods that generate machine-learning models to predict whether user devices are likely to meet defined objectives. For example, a machine-learning model can be generated to predict whether or not a user device is likely to access a resource. In some implementations, a semi-supervised model can be used to determine to what extent user devices are predicted to satisfy the defined objective(s). For example, a resource-affinity parameter can be generated as a result of inputting various data points into a semi-supervised model. The various data points can be access from a plurality of data sources, and can represent one or more activities or attributes associated with a user. The value of the resource-affinity parameter can be evaluated to determine the extent to which the user is likely to meet an objective.

Abstract: Aspects described herein provide solutions to problems posed by a user. Input that includes a specified subject may be received from a user. A specified descriptor for the specified subject may be obtained from a hierarchical taxonomy associated with the specified subject. An information repository may be searched based on the specific descriptor, and entries stored at the information repository that are associated with the specific descriptor may be indicated in a list of results. The specific descriptor may be iteratively generalized to obtain generalized descriptors, and the information repository may be searched based on the generalized descriptors. The generalized descriptors may also be specialized to obtain specialized descriptors, and the information repository may be searched based on the specialized descriptors. The list of results may include entries stored at the information repository that are respectively associated with the generalized descriptors and the specialized descriptors.

Abstract: Methods and systems of aligning annotation of fields of documents are provided. Training information that includes first measurement information pertaining to features of each of a plurality of fields associated with training clusters for documents of a document type is accessed. A first training cluster is annotated with a first name and the second training cluster is annotated with a second name. An electronic classification model is generated based on the training information. Second measurement information for features of fields associated with new clusters of a new document is accessed. Each of the new clusters is automatically annotated based on the second measurement information using the classification model. For example, a first new cluster that has fields of the first field type is annotated with the first name and a second new cluster that has fields of the second field type is annotated with the second name.

Abstract: A method, system and computer program product are provided for reorganizing a database table according to a set of queries. More specifically, the table is reorganized such that random I/O is reduced by more tightly grouping together rows in the table associated with each of the referencing queries. This enables more associated rows from a given table relevant to a query to be read into memory for a given I/O operation.

Abstract: A system receives a source and a metadata layer that describes the source. The source may comprise source records with fields containing source data, and the metadata layer may include metadata comprising at least one of a field data type, a field data length, a field description, or a record length. The processor may further validate the metadata layer against the source and write results to a log. The processor may further be configured for transforming the source records into transformed records for a load ready file. The processor may further balance a number of records in the source against a number of transformed records in the load ready file to generate a transformation failure rate.

Abstract: Certain examples described herein provide a data processing system and method adapted for event processing. These examples provide for distribution of data processing operations between server computing devices. In one case, a plurality of processing stages are implemented using computing instances on the server computing devices. In this case, the computing instances are assigned to the server computing devices in order to perform at least one data processing operation in parallel. Certain examples described herein then provide for the distribution of data between computing instances such that parallelism is maintained for data processing operations. In certain cases, a composite key is used. In this case, a composite key value is computed for a set of data fields associated with a data item to be processed. This key value is computed based on a data processing operation to be performed.

Abstract: A method, executed by a computer, for de-duplicating data includes receiving a dataset, pivoting the dataset along a set of columns that have a common domain to provide a pivoted dataset, de-duplicating the pivoted dataset to provide a de-duplicated dataset, and using the de-duplicated dataset. De-duplicating the pivoted dataset may include computing similarity scores for records that have different primary keys and merging records that have a similarity score that exceeds a selected threshold value. The method may include determining the set of columns having a common domain by referencing a business catalog and/or conducting a data classification operation on some or all of the columns of the dataset. The method may also include pivoting the dataset along another set of columns that have a different common domain. A computer system and computer program product corresponding to the method are also disclosed herein.

Abstract: Compiling test cases. The method may include uploading at least one extract, transform, load job (ETL job) and a plurality of test cases. The method may also include identifying a plurality of design features associated with the at least one uploaded ETL job. The method may further include identifying at least one feature change within the plurality of identified design features. The method may also include determining a plurality of related test cases within the plurality of uploaded test cases corresponding to the at least one identified feature change. The method may further include creating a test bucket based on a grouping of the determined plurality of related test cases. The method may also include adding at least one of a plurality of corresponding ETL jobs and a plurality of corresponding ETL job stages associated with the at least one identified feature change to the created test bucket.

Abstract: An audit scoring system may include authorizing a user device to configure an audit for one or more form identification numbers. The system may receive values for each disposition associated with form data. The system may also assign values to each disposition such that a query can be performed against the audit using a form identification number associated with the form data.

Abstract: A method for metering accelerator usage in a computing system includes receiving, from an accelerator, a set of metering rules, the metering rules including a unit of accelerator usage and a tariff per unit of accelerator usage. The unit of accelerator usage corresponds with an amount of work performed by the accelerator in executing data processing operations. The method may further include receiving on a first bus from a partition of the computing system a command block, the command block having data processing operations executable by the accelerator. The method may additionally include calculating, based on the command block and the set of metering rules, an accelerator usage value for the partition. The method may then include providing, on a second bus, the accelerator usage value.

Abstract: Embodiments of the present invention disclose a method, computer program product, and system for text author anonymization for de-identification of written text. This may be used to replace written text after learning text author writing characteristics, such frequently written terms, grammar patterns, grammar errors, and writing tone. Portions of the written text which may identify the text author may be replaced, for example the text author writing characteristics, social status, geographical location, and specific organization.

Abstract: Embodiments of the present invention disclose a method, computer program product, and system for text author anonymization for de-identification of written text. This may be used to replace written text after learning text author writing characteristics, such frequently written terms, grammar patterns, grammar errors, and writing tone. Portions of the written text which may identify the text author may be replaced, for example the text author writing characteristics, social status, geographical location, and specific organization.

Abstract: An example technique for defining a user function can include receiving a user defined function that gathers data from a data source external to a database and returns the data to the database. The example technique for defining the user function can also include creating an instance of a programming class that extends from an application programming interface (API). The instance of the programming class including the user defined function. The example technique for defining the user function can also include executing the user defined function provided by the instance of the programming class.

Abstract: A method for testing an extract, transform, load job (ETL job) is provided. The method may include receiving an uploaded ETL job. The method may also include executing a functionality test of the uploaded ETL job using a test bucket. The method may further include determining the executed functionality test produces a failure result. The method may also include creating at least one test sub-job corresponding to at least one stage within on the uploaded ETL job. The method may further include generating a plurality of test data for each of the at least one created test sub-job. The method may also include executing the at least one created test sub-job using the plurality of generated test data. The method may further include creating a report based on a plurality of results associated with the at least one executed test sub-job.

Abstract: A method, system, and computer program product derive data schema for application to a data set. One or more processors generate a directed acyclic weighted graph that encodes data types and semantic types used by a data set. One or more processors assign estimated frequencies for each component of the directed acyclic weighted graph, where the estimated frequencies predict a likelihood of a particular data schema element being used by any data set. One or more processors traverse through paths in the directed acyclic weighted graph with a predetermined portion of the data set to determine a data schema that correctly defines data from the data set and identifies any errors in the data set, and then apply the data schema to the data set to generate clean data that is properly formatted.

Abstract: Systems and methods that enable transaction oriented functions for a multidimensional database are provided. The systems and methods store a plurality of transactions in a multidimensional database, generate a first dimension including respective balances resulting from each of the transactions, and generate a second dimension identifying respective rules corresponding to each of the transactions.

Abstract: A method, system, and processor-readable storage medium are directed towards generating a report derived from data, such as event data, stored on a plurality of distributed nodes. In one embodiment the analysis is generated using a “divide and conquer” algorithm, such that each distributed node analyzes locally stored event data while an aggregating node combines these analysis results to generate the report. In one embodiment, each distributed node also transmits a list of event data references associated with the analysis result to the aggregating node. The aggregating node may then generate a global ordered list of data references based on the list of event data references received from each distributed node. Subsequently, in response to a user selection of a range of global event data, the report may dynamically retrieve event data from one or more distributed nodes for display according to the global order.

Abstract: Metadata area legends are stored at a first location, such as a dedicated partition of a persistent data storage device (PDSD). The metadata area legends have a number of descriptors that describe a number of reserved metadata areas that lie at a second location that is logically separate from the first location, such as a regular partition of the PDSD. Requests to delete the metadata area legends, as well as requests to add new metadata area legends, can be verified to prevent the metadata area legends from being accidentally or maliciously changed.

Abstract: Methods, devices, and systems (for outputting a case summary) receive an electronic medical record (EMR) for the medical patient, extract medical data from the EMR, provide a list of medical problems relevant to the EMR, identifying relations between the medical problems and the medical data using a question-answering (QA) system, and output the clinical summary for the EMR. The clinical summary comprises the list of medical problems, the medical data, and the relations.

Abstract: Disclosed herein are methods, systems, and computer program products directed to a guidance engine. The guidance engine is configured to query a knowledge base for guidance with respect to a property of a software application. The guidance engine receives a responsive query from the knowledge base that is based on the property. The responsive query informs a user of the guidance engine how to address a vulnerability within the software application by performing a transform with respect to a property of the software application.

Abstract: A computer system architecture and method for providing compliance with data regulations, by: (a) collecting a data input stream with a data collection terminal; (b) using a compliance device driver resident in the data collection terminal to: (1) select data corresponding to pre-identified data compliance fields, and (2) apply a compliance markup language parser to generate pseudonymized data; and (c) using an automated compliance network appliance and an automated compliance server to: (1) transmit the pseudonymized data into immutable audit ledger, wherein the immutable audit ledger is assembled and verified by blockchain, and (2) transmit the data input stream into a data lake; (d) hosting access portals for accessing data: (1) stored in the data lake, and (2) stored in the immutable audit ledger.

Abstract: A present invention embodiment relates to inserting data to a data table, where the data table has multiple data records and involves at least one organizing attribute. The embodiment defines a set of data blocks having records of the data table and assigns to each data block of the set of data blocks a predefined maximum number of records of the data table or a predefined maximum amount of storage. Data records for each data block of the set of data blocks may be stored on a storage system. Further data records may be sorted based on values of the organizing attribute in the buffered data records. The sorted data records may be written on the storage system in their order. Attribute value information may be determined for the further data blocks for supporting query processing against the data table.

Abstract: Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

Abstract: A computer-implemented method includes inserting a bloom filter creation stage after an inner data source identification stage, wherein a join operation is to be performed to join an outer data source with the inner data source. The method inserts a bloom filter search stage after an outer data source identification stage, wherein each row of data from the outer data source is searched against a bloom filter for the inner data source during the bloom filter search stage. The method initializes a read on the inner data source. Subsequent to determining the bloom filter creation stage is complete, the method initializes a read on the outer data source. The method performs the join operation at a join stage.

Abstract: Methods, systems, and computer-readable media for implementing sensitive data usage detection using static analysis are disclosed. A specification of one or more operations exposed by a service in a service-oriented system is obtained from a repository. The names of the one or more operations are determined in the specification. The names of one or more parameters of the one or more operations are determined in the specification. The names of the one or more operations and the names of the one or more parameters are checked against a dictionary of sensitive terms. One or more sensitive operations are determined among the one or more operations. One or more consumers of the one or more sensitive operations are determined.

Abstract: Method of organizing and storing data in an information infrastructure by either processing data throughput or a method for processing data throughput or data mining in a distributed computing system works using a tagging routine. The system and method operates on sensitive content (e.g., trade secrets) or select content (e.g., critical content) and applies configurable filters (expanding, contracting, hierarchical/orthogonal classifier filters) to separate and store sensitive/select data in designated, distributed data stores or to supplement data collections. Sensitive/select data stores may have security clearance requirements. Data output is controlled to provide one or more of storage in predetermined data stores, data mining, coping, transfer to other storage sites, supplemental data searches, and presentation subject to data inquiry.

Abstract: A computer-implemented method includes inserting a bloom filter creation stage after an inner data source identification stage, wherein a join operation is to be performed to join an outer data source with the inner data source. The method inserts a bloom filter search stage after an outer data source identification stage, wherein each row of data from the outer data source is searched against a bloom filter for the inner data source during the bloom filter search stage. The method initializes a read on the inner data source. Subsequent to determining the bloom filter creation stage is complete, the method initializes a read on the outer data source. The method performs the join operation at a join stage.

Abstract: Embodiments include a method, system, and computer program product for acquiring a data repository, the data repository being associated with a log configured to receive metadata. Then, a content of the log with respect to at least one manipulation of the data repository is modified by adding first metadata of the metadata. Further, signatures that identify control statements configured to govern the at least one manipulation of the data repository are collected by adding second metadata of the metadata into the log.

Abstract: In one embodiment, a method includes obtaining a first data set from a first data source and a second data set from a second data source, the first data set including a first plurality of entities and the second data set including a second plurality of entities. The method also includes identifying a verified relationship between a first entity from the first plurality of entities and a second entity from the second plurality of entities and determining that a third entity from the first plurality of entities has a first same-as relationship with a fourth entity from the second plurality of entities based on one or more of the verified relationship or relationships between the first plurality of entities and the second plurality of entities. The method further includes generating first output data including the first same-as relationship.

Abstract: The subject matter described herein relates to methods, systems, and computer readable media for testing network equipment devices using connection-oriented protocols. In some examples, a method for testing a network equipment device under test (DUT) includes executing, by a network equipment test device, a test script to test the network equipment DUT using a connection-oriented protocol. The method includes inserting, during a connection establishment process of the connection-oriented protocol for a network flow to the network equipment DUT, a flow-identifying sequence number into a sequence number field of a first message of the network flow. The flow-identifying sequence number is a number uniquely identifying the network flow from other network flows. The method includes receiving a second message from the network equipment DUT and determining that the second message belongs to the network flow by extracting the flow-identifying sequence number from the second message.

Abstract: Systems, methods, and apparatuses can provide staged migration from one or more user accounts from a source system to a destination system using an application local to the source systems. The automated staged data migration can provide coordinated staged data migration of large amounts of data in a single automated process. The system can provide automated and remotely controlled staged migrations of data that may perform source system type detection and work with multiple types of systems.

Abstract: Metadata is managed for different distributed federated workspaces. A distributed metadata infrastructure is provided for supporting at least two different distributed federated workspaces. The distributed metadata infrastructure monitors the at least two different distributed federated workspaces to identify common metadata between the at least two different distributed federated workspaces. The distributed metadata infrastructure presents a consistent view of the common metadata.

Abstract: Elements of a database object are removed. The database object is stored as a plurality of different object portions, where each object portion is associated with one or more versions of transaction identifiers stored separately from the database object. An oldest transaction identifier is determined for a transaction for which data portions of the database object remains visible. Each object portion is examined and object portions with a threshold amount of data to remove are determined based on a comparison of the transaction identifiers for those object portions and the oldest transaction identifier. Data from the database object are removed in response to a sufficient quantity of data is to be removed from object portions containing the threshold amount of data.

Abstract: Embodiments of the invention relate to systems and methods for providing an anonymization engine. One embodiment of the present invention relates to a method comprising receiving a message directed at a recipient computer located outside a secure area by a privacy computer located within a secure area. The privacy computer may identify private information using a plurality of privacy rules and anonymize the message according to the plurality of privacy rules. Another embodiment may be directed to a method comprising receiving a request for sensitive data from a requesting computer. An anonymization computer may determine a sensitive data record associated with the request and may anonymize the sensitive data record by performing at least two of: removing unnecessary sensitive data entries from the sensitive data record, masking the sensitive data entries to maintain format, separating the sensitive data entries into associated data groupings, and de-contexting the data.

Abstract: Certain aspects and features of the present disclosure relate to systems and methods that generate machine-learning models to predict whether user devices are likely to meet defined objectives. For example, a machine-learning model can be generated to predict whether or not a user device is likely to access a resource. In some implementations, a semi-supervised model can be used to determine to what extent user devices are predicted to satisfy the defined objective(s). For example, a resource-affinity parameter can be generated as a result of inputting various data points into a semi-supervised model. The various data points can be access from a plurality of data sources, and can represent one or more activities or attributes associated with a user. The value of the resource-affinity parameter can be evaluated to determine the extent to which the user is likely to meet an objective.

Abstract: A cloud-based ETL system provides error detection, error correction and reporting of data integration flows hosted by cloud services. Categories of errors are identified using one or more checks at different points of a data integration flow and one or more actions selected based at least in part on the error category. A determination can be made whether the error category is fault tolerant and one or more actions can be selected based at least in part on the error fault tolerance to correct the error, restart a flow, or generate a notification assisting a user to correct the error.

Abstract: Techniques relating to managing “bad” or “imperfect” data being imported into a database system are described herein. A lifecycle technology solution helps receive data from a variety of different data sources of a variety of known and/or unknown formats, standardize it, fit it to a known taxonomy through model-assisted classification, store it to a database in a manner that is consistent with the taxonomy, and allow it to be queried for a variety of different usages. Auto-classification, enrichment, clustering model and model stacks, and/or other disclosed techniques, may be used in these and/or other regards.

Abstract: Techniques are disclosed for an energy grid data platform through which information associated with an electrical grid can be accessed by one or more entities. In some embodiments, an energy grid data platform includes one or more modeling engines configured to receive data associated with an electrical grid and generate data models describing various aspects of the electrical grid. For example an electrical grid model describing the state of the electrical grid at one or more points of connection may be based at least in part on data received from sensors at the edge of the electrical grid. As another example, a physical grid model describing the physical arrangement and logical relationships between physical objects associated with the electrical grid can be generated based at least in part on received imagery data.

Abstract: An example embodiment may include a security enforcement point device disposed within a managed network and a security decision point device disposed within a computational instance of a remote network management platform. The security decision point device may be configured to: receive a message by way of the managed network; parse the message to identify observable indicators of one or more of the security threats, where the observable indicators include at least one of a network addresses, a hyperlink, or a representation of an attached file; remotely query a security threat database for the observable indicators; receive, from the security threat database, an indication that the observable indicators are associated with a particular security threat, and transmit, to the security enforcement point device, a command to update its associated security policy such that the particular security threat is mitigated.

Abstract: Compiling test cases. The method may include uploading at least one extract, transform, load job (ETL job) and a plurality of test cases. The method may also include identifying a plurality of design features associated with the at least one uploaded ETL job. The method may further include identifying at least one feature change within the plurality of identified design features. The method may also include determining a plurality of related test cases within the plurality of uploaded test cases corresponding to the at least one identified feature change. The method may further include creating a test bucket based on a grouping of the determined plurality of related test cases. The method may also include adding at least one of a plurality of corresponding ETL jobs and a plurality of corresponding ETL job stages associated with the at least one identified feature change to the created test bucket.

Abstract: A communication apparatus capable of performing facsimile communication includes at least one memory that stores a set of instructions and at least one processor. The processor executes the instructions to receive facsimile data, store the received facsimile data in a storage, generate a preview image based on the stored facsimile data, and transmit, according to a request for a preview image from an external device, the generated preview image to the external device. In addition, the stored facsimile data is deleted according to an instruction, but deletion of the facsimile data in the storage is prohibited while the external device is displaying the preview image of the facsimile data if the instruction is received from another external device.

Abstract: Disclosed herein are system, method, and computer program product embodiments for utilizing join cardinality to improve the performance of query processing. An embodiment operates by a database engine receiving a query including a join operation on a first and a second table using a join predicate, and a tag for the join operation indicating a cardinality relationship between the first and the second table when satisfying the join predicate. Then, the database engine generates a query processing plan for performing the query. For the join operation, the database engine selects one of the first or the second tables to remove from the join operation based on a type of the join operation and the tag. Based on the selecting, the database engine may remove may the join operation and the selected table from the query processing plan.

Abstract: Concepts and technologies are disclosed herein for workflow-based resource management. A process can execute a resource management application. The process can obtain load data that describes a resource load associated with resources. The processor can analyze the load data to detect a change in resource load associated with one of the resources and identify workflows that includes the resource. The workflows can include a description of how the resources are utilized during execution of a process by the resources. The processor can identify a relevant workflow that is associated with the process and issue commands to resources associated with the relevant workflow to proactively adjust capacity of the resources associated with the relevant workflow.

Abstract: A communication protocol and system is disclosed for network communications between a data service residing on a client that provides network communications between one or more mobile applications on a source and a network based on a process number. The shared data service communicates with a data service plug-in on the server side associated with the process number, in order to handle requests from the mobile applications that access the network through the data service. Predetermined network connection, priority, and additional rules can be used to control what plug-in can be reached through what type of network connection.

Abstract: Resources for data lineage discovery, data lineage analysis, role-based security, notification. The resources may include or involve machine readable memory that is configured to store a technical data element (“TDE”); a receiver that is configured to receive a query for data lineage information corresponding to a business element identifier; and a processor configured to: register a logical association between the business element identifier and the TDE; and formulate the data lineage information of the TDE associated with the business element identifier. The receiver may be configured to receive a criterion that is required to access one or more technical data elements (“TDEs”) associated with the business element identifier. The receiver may be configured to receive an election to receive a notification of a change of data lineage. The processor may be configured to toggle between a first data lineage graph and a second data lineage graph.

Abstract: Methods and system are disclosed that provision contextual information associated with business data. In one aspect, a framework middleware receives a request via an application to establish connection. Upon processing the request, the application may establish connection with multiple databases associated with multiple business management systems via an adapter framework associated with the framework middleware. Based on the established connection, the business data from the multiple databases maybe retrieved via a real-time provisioning adapter framework associated with the framework middleware. Multiple routines in the framework middleware may be executed to trigger a system landscape transformation model to transform the retrieved business data. Based on parameters associated with the transformed business data, associated contextual information maybe determined. The determined contextual information may be provisioned by rendering the contextual information on a user interface.