Abstract: Embodiments of the present disclosure provide a data gathering and query method for collecting ongoing updates to large, unstructured or semi-structured databases. The method comprises gathering a plurality of events defined in a database syntax that is not structured and aggregating the plurality of events into one or more part files. Each of the one or more part files store a subset of the plurality of events in a columnar format, and each of the one or more part files comprises a header file that includes metadata corresponding to a subset of the plurality of events stored in the part file and is separate from the subset of events stored in the part file. The method further comprises uploading the one or more part files to a cloud storage repository configured to store the one or more part files so that they can be queried by a query server based on the header files.

Abstract: Embodiments of the present disclosure relate to generating a high level security policy for a data repository without knowledge of the access control, entitlement, and other models of the data repository. A set of abstractions that define a security policy language may be generated based on data in a data repository collection. The set of abstractions may define a security policy language, which may be provided to a security administrator who can define a security policy with the security policy language. The security policy may be translated into a common physical language to generate a common physical policy. The processing device may then translate the common physical policy into a set of commands for each of one or more data repositories that the data repository collection is comprised of.

Abstract: A log message classifier employs machine learning for identifying a corresponding parser for interpreting the incoming log message and for retraining a classification logic model processing the incoming log messages. Voluminous log messages generate a large amount of data, typically in a text form. Data fields are parseable from the message by a parser that knows a format of the message. The classification logic is trained by a set of messages having a known format for defining groups of messages recognizable by a corresponding parser. The classification logic is defined by a random forest that outputs a corresponding group and confidence value for each incoming message. Groups may be split to define new groups based on a recurring matching tail (latter portion) of the incoming messages. A trend of decreased confidence scores triggers a periodic retraining of the random forest, and may also generate an alert to operators.

Abstract: Embodiments of the present disclosure relate to utilizing an existing login process of a data repository to enable the data repository to delegate MFA functionality to an external MFA system. When a purported user attempts to log in to the data repository, a delegation module within the login process may insert a record into a table associated with the login process. A program executing on a security device external to the data repository may periodically poll the table for new records and upon detecting the new record, may call the external MFA system to verify the login attempt. The external MFA system may indicate to the program whether the login attempt was verified and the program may update the table with the indication. Upon detecting the indication, the delegation module may complete or terminate the login attempt based on the indication.

Abstract: Embodiments of the present disclosure relate to generating a high level security policy for a data repository without knowledge of the access control, entitlement, and other models of the data repository. A set of abstractions that define a security policy language may be generated based on data in a data repository collection. The set of abstractions may define a security policy language, which may be provided to a security administrator who can define a security policy with the security policy language. The security policy may be translated into a common physical language to generate a common physical policy. The processing device may then translate the common physical policy into a set of commands for each of one or more data repositories that the data repository collection is comprised of.

Abstract: Classification for data intake operations in an enterprise ensures that sensitive data is not disseminated inappropriately, but incurs substantial time, effort and expense. A method of classifying data in a large set of data repositories captures a set of raw rules resulting from inputs indicative of evaluations and conclusions of data classification operations, typically by logging data classification operations, and identifies patterns in the set of raw rules by consolidating duplicative conditions and eliminating inconsequential conditions. External conditions and observations may be referenced for applying a context to the rules based on a usage or domain of the data, and data sets of disparate entities may be examined for anonymizing the data and combining with other sets of anonymized data.

Abstract: Embodiments of the present disclosure provide a data gathering and query method for collecting ongoing updates to large, unstructured or semi-structured databases. The method comprises gathering a plurality of events defined in a database syntax that is not structured and aggregating the plurality of events into one or more part files. Each of the one or more part files store a subset of the plurality of events in a columnar format, and each of the one or more part files comprises a header file that includes metadata corresponding to a subset of the plurality of events stored in the part file and is separate from the subset of events stored in the part file. The method further comprises uploading the one or more part files to a cloud storage repository configured to store the one or more part files so that they can be queried by a query server based on the header files.

Abstract: A data store classification approach identifies metadata and contextual aspects of data that extend beyond the mere content or label of the data to examine organizational, locational, and proximity features that tend to suggest whether a data item may or may not be sensitive. These aspects place the data in a context around which inferences of sensitivity may be derived by a machine learning representation or similar configuration. Features and corresponding attributes of the data items are derived and associated with the data by a model. The model defines an enriched data representation of the data in conjunction with the attributes that indicate a sensitive data item. The attributes and data items can be evaluated as to whether or not a data item is a sensitive or private data item so that relevant decisions about privacy and security may be made.

Abstract: Classification for data intake operations in an enterprise ensures that sensitive data is not disseminated inappropriately, but incurs substantial time, effort and expense. A method of classifying data in a large set of data repositories captures a set of raw rules resulting from inputs indicative of evaluations and conclusions of data classification operations, typically by logging data classification operations, and identifies patterns in the set of raw rules by consolidating duplicative conditions and eliminating inconsequential conditions. External conditions and observations may be referenced for applying a context to the rules based on a usage or domain of the data, and data sets of disparate entities may be examined for anonymizing the data and combining with other sets of anonymized data.

Abstract: A data gathering and query method for collecting ongoing updates to large, unstructured databases performing data collection from multiple sites. A large repository of unstructured or semi-structured data according to a JSON syntax receives periodic updates from an enterprise site for gathered event data. A bigdata cloud store receives the additions as columnar parts that arrange the data in a columnar form for storing similarly named fields consecutively. The enterprise site generating the event data arranges the block files containing the columnar data, and header files containing metadata. Incremental time and/or size triggers the periodic part upload, and a query server in network communication with the cloud store integrates the incoming additions by receiving the header files and updating a catalog of collections in the cloud store, without downloading the larger block files containing the actual columnar data. Query requests operate on the cloud store without moving the block files.

Abstract: A data gathering and query method for collecting ongoing updates to large, unstructured databases performing data collection from multiple sites. A large repository of unstructured or semi-structured data according to a JSON syntax receives periodic updates from an enterprise site for gathered event data. A bigdata cloud store receives the additions as columnar parts that arrange the data in a columnar form for storing similarly named fields consecutively. The enterprise site generating the event data arranges the block files containing the columnar data, and header files containing metadata. Incremental time and/or size triggers the periodic part upload, and a query server in network communication with the cloud store integrates the incoming additions by receiving the header files and updating a catalog of collections in the cloud store, without downloading the larger block files containing the actual columnar data. Query requests operate on the cloud store without moving the block files.

Abstract: A query server identifies data collections of interest in a cloud store, and categorizes the collections based on an intended usage. Depending on the intended usage, the categorized data may be cataloged, indexed, or undergo a full intake into a column store. In a database of large data collections, some collections may experience sparse or indefinite usage. Cataloging or indexing position the collections for subsequent query access, but defers the computational burden. The full intake performs a columnar shredding of the collection for facilitating eminent and regular query access. Upon invocation of query activity, an instantiation of virtual machines provided by the cloud store vendor implements query logic, such that the VMs launch in conjunction with the cloud store having the collections. Collections therefore incur processing based on their expected usage—full intake for high query traffic collections, and reduced cataloging for maintaining accessibility of collections of indefinite query interest.

Abstract: Data storage for unstructured data such as JSON data stored as collections of documents transforms the JSON data into a columnar form of storing unstructured data by grouping similar fields together for facilitating retrieval of the individual fields from a range of documents. Groups of fields are stored in individual files for each field. Compound data such as arrays and subdocuments are also broken down into files for each atomic field. In other words, a compound document structure that defines a hierarchy or “tree” of fields is flattened such that each “leaf” of the tree is stored in a separate file.

Abstract: Mechanisms are provided for propagating source identification information from an application front-end system in an application layer to a data layer inspection system associated with a back-end system. An incoming user request is received, at the data layer inspection system, from a gateway system associated with the application front-end system. One or more outgoing statements targeting a back-end system are received at the data layer inspection system. The data layer inspection system accesses a mapping data structure based on the one or more outgoing statements to thereby correlate the one or more outgoing statements with the incoming user request. The data layer inspection system retrieves source identification information associated with the incoming user request based on the correlation of the one or more outgoing statements with the incoming user request. The data layer inspection system performs a data layer inspection operation based on the source identification information.

Abstract: A log message classifier employs machine learning for identifying a corresponding parser for interpreting the incoming log message and for retraining a classification logic model processing the incoming log messages. Voluminous log messages generate a large amount of data, typically in a text form. Data fields are parseable from the message by a parser that knows a format of the message. The classification logic is trained by a set of messages having a known format for defining groups of messages recognizable by a corresponding parser. The classification logic is defined by a random forest that outputs a corresponding group and confidence value for each incoming message. Groups may be split to define new groups based on a recurring matching tail (latter portion) of the incoming messages. A trend of decreased confidence scores triggers a periodic retraining of the random forest, and may also generate an alert to operators.

Abstract: A query server identifies data collections of interest in a cloud store, and categorizes the collections based on an intended usage. Depending on the intended usage, the categorized data may be cataloged, indexed, or undergo a full intake into a column store. In a database of large data collections, some collections may experience sparse or indefinite usage. Cataloging or indexing position the collections for subsequent query access, but defers the computational burden. The full intake performs a columnar shredding of the collection for facilitating eminent and regular query access. Upon invocation of query activity, an instantiation of virtual machines provided by the cloud store vendor implements query logic, such that the VMs launch in conjunction with the cloud store having the collections. Collections therefore incur processing based on their expected usage-full intake for high query traffic collections, and reduced cataloging for maintaining accessibility of collections of indefinite query interest.

Abstract: A log message classifier employs machine learning for identifying a corresponding parser for interpreting the incoming log message and for retraining a classification logic model processing the incoming log messages. Voluminous log messages generate a large amount of data, typically in a text form. Data fields are parseable from the message by a parser that knows a format of the message. The classification logic is trained by a set of messages having a known format for defining groups of messages recognizable by a corresponding parser. The classification logic is defined by a random forest that outputs a corresponding group and confidence value for each incoming message. Groups may be split to define new groups based on a recurring matching tail (latter portion) of the incoming messages. A trend of decreased confidence scores triggers a periodic retraining of the random forest, and may also generate an alert to operators.

Abstract: A query server performs method of generating a query result using an aggregation pipeline by identifying, based on a query, a sequence of operations to be applied to documents from an unstructured database, in which a portion of the operations are dependent on other operations in the sequence of operations. The pipeline determines, from the operations, lightweight and heavyweight operations, in which the heavyweight operations generate a materialized result have a substantial impact on processing resources. The pipeline defers the lightweight operations until a materialized result is needed, for performing with a corresponding heavyweight operation, in which the materialized result includes either creation of a new document or movement of substantial data from a document. Lightweight operations are grouped with heavyweight operations such that multiple operations can be collapsed into a single operation that act upon the data together thus avoiding the number of materializations.

Abstract: A system and method of storing data in an unstructured or semi-structured database, such as a JSON database, includes defining a columnar store, or hybrid column portion having a set of values for each field, such that each set stores a sequence of values from commonly named fields of each document together, and defining a hybrid store, or hybrid row portion having, for each document in the columnar store, a hybrid entry storing fields from the same document together, such that each entry in the hybrid store includes fields of a corresponding document represented in the columnar store. The hybrid arrangement provides that both the columnar store and the hybrid store are derived from the same set of documents, and each of the columnar store and the hybrid store are configured to fulfill a query request for determining whether to access the columnar store or the hybrid store.

Abstract: A query engine for an unstructured database satisfies window based queries and analytics by defining a window of documents, and performing analytics on the window using a default value for omitted field. A tabular index containing only values needed for analytics and document ordering defines each window. The tabular index includes all fields from each document that are required to satisfy the query, retrieved on a single pass by the query engine so that multiple fetches to the same document are avoided. Since each document in the window need not contain all the same fields as the other documents, an adapter includes logic for defining a default or placeholder value for a field called for in an analytic computation but nonexistent in a particular document. By retrieving only the computationally relevant fields, and by performing the retrieval only once on each document, the I/O overhead is greatly reduced.