AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR PROVIDING AI-EXPLAINABILITY FOR PROCESSING DATA THROUGH MULTIPLE LAYERS

Abstract

Systems and methods for providing explainability for processing data through multiple layers are provided. An input layer is configured to receive an evidence data set comprising a plurality of evidence items, apply evidence processing models to the evidence data set to generate evidence understanding data, and generate input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer. A presentation layer is configured to receive data (the evidence understanding data and/or data generated based on the evidence understanding data), apply one or more presentation generation models to the received data to generate presentation data, and generate presentation-layer explainability data for presentation to the user, wherein the presentation-layer explainability data represents information about the processing of the received data set by the presentation layer.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No. 63/217,119 filed Jun. 30, 2021; U.S. Provisional Application No. 63/217,123 filed Jun. 30, 2021; U.S. Provisional Application No. 63/217,127 filed Jun. 30, 2021; U.S. Provisional Application No. 63/217,131 filed Jun. 30, 2021; and U.S. Provisional Application No. 63/217,134, filed Jun. 30, 2021, the entire contents of each of which are incorporated herein by reference.

FIELD

This related generally to AI explainability, and more specifically to AI-augmented auditing platform including techniques for providing AI-explainability for processing data through multiple layers.

BACKGROUND

Known techniques for processing data and generating adjudications based on the processed data include enterprise resource planning (ERP) systems with data-warehouse and/or data mart capabilities, which allow drill-down and rollup of digital data. These systems may provide direct linkage of aggregated digital data or data processed by simple mathematical formulas only. Furthermore, known techniques include content description frameworks for multimedia content including text, images, and video (see, e.g., “Multimedia Content Description Framework” U.S. Pat. No. 6,564,263 (L. D. Bergman, M.Y. Y. Kim, C.-S. Li, R. Mohan, J. R. Smith), which forms the basis for the multimedia metadata standard MPEG-7. Furthermore, known techniques include business rules engines that allow forward and backward chaining of business rules as part of reasoning processes. Furthermore, known techniques include conducting testing with tick-marks, pulldown menus, and manual entries of explanations on workpaper, or with circling or highlighting areas of interest. Known automation approaches use ad hoc approaches for identifying results needing follow-up, or are based on single-layer taxonomy/vocabulary for defining outputs.

SUMMARY

As explained above, known techniques for processing data and generating adjudications include enterprise resource planning (ERP) systems and may provide direct linkage of aggregated data or data processed by simple mathematical formulas only. Furthermore, known techniques include content description frameworks for multimedia content including text, images, and video. Furthermore, known techniques include business rules engines and include conducting testing manual entries of explanations. Known automation approaches use ad hoc approaches for identifying results needing follow-up, or are based on single-layer taxonomy/vocabulary for defining outputs.

However, known business rules engines do not retain a provenance of the deduction processes that they apply—e.g., they do not retain a log of rules that have been applied by the engine. Moreover, known techniques do not provide for traceability (including full traceability) and provenance of interpretation, reasoning, deliberation, and adjudication through various layers, including from raw evidence through final observation, conclusion, and recommendation. Furthermore, known techniques do not provide flexibility in presenting generated recommendations and/or remediations, adaptability in light of evolving needs, nor minimization and localization of necessary changes within systems. Furthermore, known techniques do not provide robust and flexible frameworks for applying different taxonomy/vocabulary/logic at each of a plurality of layers, generating output data and explainability outputs at each of the plurality of layers, and allowing localized modification of any of the layers without modification of other layers.

Accordingly, there is a need for improved systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers, wherein said improved systems address one or more of the above-identified shortcomings.

Disclosed herein are systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers. In some embodiments, the techniques disclosed herein may be applied to tracking the reasoning and adjudication processes by which evidence data and information are processed through a plurality of layers during an audit process. The fine-grained explainability of the reasoning process is critical for the human auditor to understand and believe in the conclusion and recommendation. The decoupling of the tracking at each stage of the deliberation enable maximal flexibility while minimize the necessary modification when frequent changes of the required summary, conclusion, and recommendation are required.

In some embodiments, a first system for providing explainability for processing data through multiple layers is provided, the first system comprising one or more processors configured to cause the first system to provide: an input layer configured to: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate an input-layer output for presentation to a user, wherein the input layer output represents the processing of the evidence data set by the input layer; a presentation layer configured to: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate a presentation output for presentation to the user, wherein the presentation output comprises the presentation data.

In some embodiments of the first system, the one or more processors are configured to provide: one or more intermediate layers configured to: receive the evidence understanding data generated by the input layer; apply one or more intermediate-layer processing models to the evidence understanding data to generate the data received by the presentation layer; provide the data received by the presentation layer to the presentation layer; and generate an intermediate-layer output for presentation to the user, wherein the intermediate-layer output represents the processing of the evidence understanding data by the one or more intermediate layers.

In some embodiments of the first system, the input layer and the presentation layer are each configured to apply a respective ontology.

In some embodiments of the first system, the one or more processors are configured to: receive a user input comprising an instruction to modify the input layer; modify the input layer in accordance with the user input without modifying the presentation layer.

In some embodiments of the first system, the one or more processors are configured to: receive a user input comprising an instruction to modify the presentation layer; and modify the presentation layer in accordance with the user input without modifying the input layer.

In some embodiments of the first system, the one or more processors are configured to initialize the presentation layer by applying one or more machine learning models to classify output data from one or more prior analyses performed by the system.

In some embodiments of the first system, the one or more processors are configured to: receive utilization data representing a manner in which the presentation output is utilized by one or more users; and automatically modify the presentation layer in accordance with the utilization data.

In some embodiments, a non-transitory computer-readable storage medium storing instructions for providing explainability for processing data through multiple layers is provided, wherein the instructions are configured to be executed by a system comprising one or more processors to cause the system to provide: an input layer configured to: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate an input-layer output for presentation to a user, wherein the input layer output represents the processing of the evidence data set by the input layer; a presentation layer configured to: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate a presentation output for presentation to the user, wherein the presentation output comprises the presentation data.

In some embodiments, a method for providing explainability for processing data through multiple layers is provided, wherein the method is performed by a system comprising one or more processors, the method comprising: by an input layer of the system: receiving an evidence data set comprising a plurality of evidence items; applying one or more evidence processing models to the evidence data set to generate evidence understanding data; and generating an input-layer output for presentation to a user, wherein the input layer output represents the processing of the evidence data set by the input layer; by a presentation layer of the system: receiving data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; applying one or more presentation generation models to the received data to generate presentation data; and generating a presentation output for presentation to the user, wherein the presentation output comprises the presentation data.

In some embodiments, a second system for providing explainability for processing data through multiple data-processing layers is provided, the second system comprising one or more processors configured to cause the second system to cause the second system to: at an input layer: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer; at a presentation layer: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer; cause display of the presentation data; and cause display of one or more of: the input-layer explainability data and the presentation-layer explainability data.

In some embodiments, any one or more of the features, characteristics, or aspects of any one or more of the above systems, methods, or non-transitory computer-readable storage media may be combined, in whole or in part, with one another and/or with any one or more of the features, characteristics, or aspects (in whole or in part) of any other embodiment or disclosure herein.

BRIEF DESCRIPTION OF THE FIGURES

Various embodiments are described with reference to the accompanying figures, in which:

FIG. 1 shows one example of data processing of revenue account data, in accordance with some embodiments.

FIG. 2 shows a capture of a sales order, in accordance with some embodiments.

FIG. 3 shows vouching of a sales order against the purchase order, a shipment against a Bill of Lading, and a payment against a variety of payment details, in accordance with some embodiments.

FIG. 4 shows a schematic diagram of a system for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers, in accordance with some embodiments.

FIG. 5 shows a diagram of a different classes of data, in accordance with some embodiments.

FIG. 6 shows an ontology for explanation of evidence evaluation, in accordance with some embodiments.

FIG. 7 shows a schematic diagram of data processing for financial data, in accordance with some embodiments.

FIG. 8 illustrates an example of a computer, according to some embodiments.

DETAILED DESCRIPTION

As explained above, known techniques for processing data and generating adjudications include enterprise resource planning (ERP) systems and may provide direct linkage of aggregated data or data processed by simple mathematical formulas only. Furthermore, known techniques include content description frameworks for multimedia content including text, images, and video. Furthermore, known techniques include business rules engines and include conducting testing manual entries of explanations. Known automation approaches use ad hoc approaches for identifying results needing follow-up, or are based on single-layer taxonomy/vocabulary for defining outputs.

However, known business rules engines do not retain a provenance of the deduction processes that they apply—e.g., they do not retain a log of rules that have been applied by the engine. Moreover, known techniques do not provide for traceability (including full traceability) and provenance of interpretation, reasoning, deliberation, and adjudication through various layers, including from raw evidence through final observation, conclusion, and recommendation. Furthermore, known techniques do not provide flexibility in presenting generated recommendations and/or remediations, adaptability in light of evolving needs, nor minimization and localization of necessary changes within systems. Furthermore, known techniques do not provide robust and flexible frameworks for applying different taxonomy/vocabulary/logic at each of a plurality of layers, generating output data and explainability outputs at each of the plurality of layers, and allowing localized modification of any of the layers without modification of other layers.

Accordingly, there is a need for improved systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers, wherein said improved systems address one or more of the above-identified shortcomings.

Disclosed herein are systems and methods for tracking the reasoning and adjudication processes by which data is processed through a plurality of interconnected layers. In some embodiments, the techniques disclosed herein may be applied to tracking the reasoning and adjudication processes by which evidence data and information are processed through a plurality of layers during an audit process. The fine-grained explainability of the reasoning process is critical for the human auditor to understand and believe in the conclusion and recommendation. The decoupling of the tracking at each stage of the deliberation enable maximal flexibility while minimize the necessary modification when frequent changes of the required summary, conclusion, and recommendation are required.

In some embodiments, the systems and methods disclosed herein provide full explainability for digitizing evidence that is not yet in digital form, for example by applying one or more document-understanding techniques to ingested documents for which the included data is not yet in digital form. In some embodiments, documents may be processed via one or more document understanding operations comprising one or more of structural, semantic, and/or linguistic analysis of the documents. In some embodiments, the systems may apply one or more natural-language understanding (NLU) techniques to extract information from received documents and to generate structured data based on said ingested documents, such that said structured data can be further processed by one or more data processing models, including by being processed through a plurality of layers and by being used to generate explainable outputs at one or more of the plurality of layers, as described herein. (Any data processing operation referenced herein may include application of one or more models trained by machine-learning.)

In some embodiments, the systems and methods disclosed herein provide full explainability for interpretation of processes and results during cross-validation between and/or among evidence.

In some embodiments, the systems and methods disclosed herein provide full explainability for reporting results from one or more final adjudications (e.g., assessments, classifications, quantifications, characterizations, and/or scores made on the basis of evidence provided) based on input data regarding available evidence. For example, adjudications based on available evidence may include adjudications as to whether one or more accounting policies and/or auditing standards are satisfied.

In some embodiments, the systems and methods disclosed herein provide full explainability for describing patterns, trends, and/or insights gained from evaluations of integrity of evidence, including evaluations made based on an individual piece of evidence and/or evaluations made based on multiple pieces of evidence.

In some embodiments, the systems and methods disclosed herein provide full explainability for describing final recommendations (e.g., outputs provided to one or more systems or users wherein the output indicates a recommended action) based on input data regarding available evidence, including recommendations generated based on an individual piece of evidence and/or based on multiple pieces of evidence.

In some embodiments, a system for providing explainability for data processing may define a plurality of layers through which data may be processed. Each layer may be provided by one or more processors, including embodiments in which layers are provided by separate processors and in which layers are provided by the same processors. Each layer may receive input data to the layer, apply one or more data processing operations to the received input data to generate output data, and may generate explainability data to be presented to a user to explain the input data, processing operations, and/or output data of the layer. The layers may be communicatively interconnected with one another such that output data from one layer may form all or part of the input data for another layer.

The plurality of layers may include an input layer which is communicatively coupled with one or more data sources for providing raw evidence data as input to the input layer. The input layer may apply one or more data processing operations to the received evidence data to generate output data of the input layer. The input layer may then provide the generated output data to one or more subsequent layers to which it is communicatively connected, such that the generated output data may be used by the one or more subsequent layers as input data. The input layer may additionally generate an explainability output, such as a user-facing visualization, report, or other human-readable information that may be displayed, transmitted, and/or presented to one or more users. The explainability output for the input layer may indicate information regarding the manner in which the input layer received evidence data, processed said evidence data, generated output data based on said evidence data, and/or provided said generated output data to one or more subsequent layers. In some embodiments, the input layer may be configured for providing fine-grained precision interpretation of evidence data.

The plurality of layers may include one or more intermediate layers that may be directly and/or indirectly communicatively coupled with and downstream of the input layer. The intermediate layers may be disposed in series and/or in parallel with one another. An intermediate layer may receive, as input data, the output data that is generated by the input layer. Alternatively or additionally, the intermediate layer may receive, as input data, output data that was generated by another intermediate layer. An intermediate layer may apply one or more data processing operations to the received data to generate output data of the intermediate layer. An intermediate layer may then provide the generated output data to one or more subsequent layers to which it is communicatively connected, such that the generated output data may be used by the one or more subsequent layers (e.g., another intermediate layer and/or a presentation layer, as described below) as input data. An intermediate layer may additionally generate an explainability output for the intermediate layer, such as a user-facing visualization, report, or other human-readable information that may be displayed, transmitted, and/or presented to one or more users. The explainability output for the intermediate layer may indicate information regarding the manner in which the intermediate layer received data, processed said received data, generated output data based on said received data, and/or provided said generated output data to one or more subsequent layers. In some embodiments, an intermediate layer may be configured for providing visibility and traceability of intermediate reasoning and adjudication of multiple related pieces of evidence.

The plurality of layers may include a presentation layer that may be directly and/or indirectly communicatively coupled with and downstream of one or more of the intermediate layers and/or the input layer. The presentation layer may receive, as input data, the output data that is generated by the input layer and/or by one or more of the input layers. The presentation layer may apply one or more data processing operations to the received data to generate output data of the presentation layer, wherein the output data of the presentation layer may include actionable insights, observations, interpretations, recommendations, and/or remediations. The output data of the presentation layer may be displayed, transmitted, and/or presented to one or more users. The output data of the presentation layer may be stored, transmitted to one or more other systems, and/or used to selectively trigger one or more automated actions by the system. The presentation layer may additionally generate an explainability output for the presentation layer, such as a user-facing visualization, report, or other human-readable information that may be displayed, transmitted, and/or presented to one or more users. The explainability output for the presentation layer may indicate information regarding the manner in which the presentation layer received data, processed said received data, generated output data based on said received data, and/or presented or provided said generated output data to one or more users and/or one or more other systems. In some embodiments, the presentation layer may be configured for presenting actionable insight, observation, interpretation, recommendation, and remediation.

The system may be configured such that a respective ontology (e.g., taxonomy and/or vocabulary) is established for each layer. The system may store data representing the respective ontology for each layer. In some embodiments, the system may store definition ontology data for a respective layer to enable explanations within the environment are standardized. In some embodiments, the ontology for a respective layer may be codified into reason code; codification of ontological data into reason code may simplify the tracing of deduction and reasoning processes.

In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define a manner in which the layer is interconnected (e.g., configured to receive data from and/or provide data to) one or more other layers of the system.

In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define deductive reasoning processes able to be applied by the layer, including one or more deterministic reasoning frameworks and/or one or more probabilistic reasoning frameworks. In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define inductive reasoning processes able to be applied by the layer, including one or more machine learning models and/or one or more deep learning models. In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define abductive reasoning processes able to be applied by the layer. In some embodiments, data stored for a respective layer (including, e.g., the ontology for a respective layer) may define general-purpose logic processes able to be applied by the layer.

In some embodiments, the system may be configured to be able to be modified/reconfigured in accordance with instructions received from a user, in accordance with performance data regarding the performance of the system, and/or in accordance with data received from one or more other systems. In some embodiments, the system may be configured such that modification of codification may be made with localized impact. The system may be configured such that information defining the ontology, data processing operations, explainability output generation, and/or layer interconnectedness of any one layer is able to be modified without modifying any of the other layers in the system. In one example, modification of the presentation layer may be achieved by modifying logic rules that connect the presentation layer to one or more upstream layers (e.g., intermediate layer(s) and/or input layer) without modifying the upstream layers themselves. In another example, modification of the input layer may be achieved by modifying logic rules that connect the input layer one or more downstream layers (e.g., intermediate layer(s) and/or presentation layer) without modifying the downstream layers themselves. In another example, modification of an intermediate layer may be achieved by modifying logic rules that connect the intermediate layer to one or more upstream or downstream layers (e.g., input layer, other intermediate layer(s), and/or presentation layer) without modifying the upstream or downstream layers themselves.

In some embodiments, the presentation layer may be configured in accordance with results of one or more supervised or unsupervised machine learning models, such as one or more clustering models and/or classification models. The presentation layer may be initialized, in some embodiments, based at least in part on historical data (e.g., data taken from one or more prior projects and/or prior analyses). The system may apply a supervised or unsupervised learning model to cluster/classify output data from one or more prior analyses, for example by applying the model(s) to data representing commentary, observations, summaries, conclusions, and/or recommendations from one or more prior analyses. The resulting output data from applying the model(s) to said data may be used to initialize the presentation layer.

In some embodiments, the system may be configured to continuously/iteratively learn via utilization of one or more of the layers. For example, the system may be configured to monitor user engagement with output data and/or with explainability data generated by one or more of the layers; user utilization of output data and/or of explainability data generated by one or more of the layers; and/or explicit user feedback regarding presentation layer output data and/or explainability data generate by one or more of the layers. For example, the system may track which portions of output data and/or explainability data are selected by the user to cause the system to “drill down” to display additional information. On the basis of the monitored engagement, utilization, and/or feedback data, the system may automatically update configurations of one or more of the layers (e.g., to adjust future explainability outputs and/or to adjust future recommendations). Updates made on this basis may include updates as described elsewhere herein, for example updates to individual layers and/or updates to the manner in which layers are interconnected with one another. In this way, the system may be configured to apply a feedback loop such that output data and explainability outputs generated by the system may be continuously/iteratively improved over time.

Some embodiments of the systems and methods described herein may be applied in the audit of financial data, such as data regarding revenue and receivable for an entity. While the figures below may describe the systems and methods disclosed herein as they relate to an audit of financial data, a person of ordinary skill in the art will understand, in light of the disclosure herein, that the systems and methods described herein may be applicable to any financial statement line item (FSLI)—including expense and payable, property, plant and equipment (PPE) (including lease accounting), cash and cash equivalent, JE, etc.—and may also be applicable to automatically processing any kind of data and providing explainability therefor, including processing and providing explainability for data that is not financial data.

Herein, the disclosed embodiments demonstrate how the systems and methods disclosed herein may be applied to an audit of financial data.

FIG. 1 shows one example of data processing, in accordance with some embodiments. As shown in FIG. 1, revenue account data may be processed to extract data corresponding to each of a plurality of individual transactions therein. As further shown in FIG. 1, data corresponding to an individual transaction may then be further processed to extract data corresponding to various aspects of the individual transaction, such as sales order creation data, shipping preparation data, invoice data, and payment receipt data.

Account receivable and revenue accounts within the chart of accounts capture the revenue generated through the order to cash process, as shown in FIG. 1. The order to cash process includes the creation of sales order, prepare shipping (if the order involves a shipment), invoice the customer, and receive the payment when the customer pay. This process may be repeated for all the transactions that are recorded in the revenue account within the general ledger. As shown, a single revenue account may include data representing a plurality of individual transactions. The data for any individual transaction may include data representing one or more stages of a process associated with the transaction. In some embodiments, any one or more of the data processing operations described herein may be performed on a per-transaction basis and/or on the basis of a set of transactions. Information may be visualized or otherwise presented (e.g., via a presentation layer) to a user for a single transaction, multiple transactions, a single set of transactions, and/or multiple sets of transactions.

During the order to cash process, various information systems may participate in the process. As shown in FIG. 2, sales order data may be captured in an order management system (which may be part of the ERP system) and may trigger a warehouse management system (e.g., comprising one or more processors) to generate data usable for preparing the shipment according to the delivery date. When a product is shipped, an inventory management system (e.g., comprising one or more processors) may record the reduction of the inventory, and an order management system (e.g., comprising one or more processors) may generate invoice data and may transmit said invoice data to the customer (e.g., based on the delivery term). While invoicing the customer, data regarding this transaction may be posted in a data store associated with the revenue account (credit) and/or in a data store associated with the account receivable account (debit). When payment is received, data regarding the payment may be generated and may be recorded in the data store associated with the account receivable (credit) and/or in a data store associated with cash account (debit) within general ledger.

An audit of revenue account, as a result, may include one or more data processing operations that validate data values associated with the account by tracing the transaction through the system in conjunction with corroborating evidence to ensure that each transaction has been posted correctly according to one or more criteria, for example criteria associated with accounting standards such as accounting policy ASC 606 (IFRS 15).

FIG. 2 shows how a transaction may be traces end-to-end to detect any discrepancies.

As shown in FIG. 3, a sales order may be vouched against a purchase order (e.g., by comparing sales order data against purchase order data), a shipment may be vouched against the bill of lading (e.g., by comparing shipment data against bill of lading data), and a payment may be vouched against a variety of payment details such as bank statement, credit card processor settlement report, ACH daily report, etc. (e.g., by comparing payment data against data associated with one or more of the variety of payment details).

FIG. 4 shows a schematic diagram of a system for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers, in accordance with some embodiments. As shown in FIG. 4, a plurality of pieces of evidence data 402 (comprising 4102a, 402b, and 402c) (e.g., documents, files, structured data, unstructured data, partially structured data, ERP representations, endogenous data, exogenous data, etc.) may be received by the system and may be subject to one or more data processing techniques (e.g., document understanding techniques, AI models, etc.) to generate evidence understanding data 404. In some embodiments, evidence understanding data 404 may comprise data extracted from evidence data 402 and/or may comprise normalized or otherwise processed data resulting from applying one or more data processing techniques to evidence data 402. In some embodiments, system 400 may process the input evidence data 402 via an input layer of the system comprising one or more processors, applying one or more data processing operations to evidence data 402 to generate the evidence understanding data 404. In some embodiments, the input layer may also generate explainability output data (not shown) that may be provided (e.g., displayed, visualized, etc.) for a user to indicate information regarding the evidence data 402 received, the manner of processing of said received data 402, and/or the evidence understanding data generated 404.

As further shown in FIG. 4, evidence understanding data 404 may then be processed via one or more reasoning/adjudication logic layers in order to generate one or more instances of intermediate results data. Each reasoning/adjudication logic layer may comprise one or more processors configured to apply one or more data processing operations to process input data and to generate output data and explainability output data. In FIG. 4, reasoning/adjudication logic layer 406 may comprise one or more processors configured to accept evidence understanding data 404 as input data, to process said input data, and to generate intermediate results data 408 as output data. In some embodiments, an intermediate layer may also generate an explainability output that may be provided (e.g., displayed, visualized, etc.) for a user to indicate information regarding the input data for the intermediate layer, the manner of processing said input data, and/or the intermediate results data generated. For example, reasoning/adjudication logic layer 406 may also produce explainability output data (not shown) that may be provided (e.g., displayed, visualized, etc.) for a user to indicate information regarding the data 404 received, the manner of processing of said received data 404, and/or the generated output data 408.

In some embodiments, system 400 may include a plurality of intermediate layers arranged in series and/or in parallel, and system 400 may apply one or more data processing operations to the data processed through the intermediate layers. In some embodiments, multiple intermediate layers may receive the same input data. In some embodiments, one intermediate layer may forward received input data to another intermediate layer in the same format that it was received. In some embodiments, an upstream intermediate layer (e.g., layer 406) may generate output data (e.g., 408) that may itself be used as input data for a downstream intermediate layer (e.g., layer 410) to produce downstream intermediate output data (e.g., 412).

As further shown in FIG. 4, intermediate results data (e.g., intermediate results data from one or more of the intermediate layers) such as results data 408 and/or 412 may be processed via one or more reasoning/adjudication logic layers in order to generate recommendation and remediation data. System 400 may process the intermediate results data (e.g., the intermediate results data from the final intermediate layer) via one or more presentation layers of the system, applying one or more data processing operations to the intermediate results data to generate recommendation and remediation data. For example, in system 400, reasoning/adjudication logic layer 414 may receive intermediate results data 412 as input data and may process said received input data via one or more data processing operations in order to generate recommendation and remediation data 416. In some embodiments, a presentation layer may also generate an explainability output that may be provided (e.g., displayed, visualized, etc.) for a user to indicate information regarding the input data for the presentation layer, the manner of processing said input data, and/or the recommendation & remediation data generated. As shown in FIG. 4, the recommendation and remediation data itself may also be provided (e.g., displayed, visualized, etc.) to a human user, such as a human user of user system 418.

FIG. 4 shows an overall architecture of a multilayer reasoning process and a manner in which output data and/or explanation data may be generated at each of a plurality of layers. The leftmost layer may interprets raw evidence to generate understanding data 404, and may generate associated explainability data that may be stored and/or presented to a user. The rightmost layer 414 may generate output data representing an overall observation, conclusion, and/or recommendation for a user such as an auditor, and may also generate associated explainability data. Intermediate layers capture the explanation of the intermediate results. The logic applied by one or more layers of system 400 may include deductive, inductive, abductive, and/or general-purpose logic to take input from one layer and produce out data for presentation to a user, storage, and/or use by another layer.

In some embodiments, a multilayer data processing such as system 400 may be configured to process data through one or more an input layers, a plurality of intermediate layers, and one or more presentation layers.

An input layer may be configured to receive input data from one or more external data sources, such as one or more databases, data stores, data repositories, live data feeds, or the like. The data may be received in structured, unstructured, and/or partially-structured (e.g., semi-structured) format. The data may be received by the input layer on a scheduled basis, in response to a user input, in response to one or more trigger conditions being met, and/or in response to the data being manually sent.

The input layer may comprise one or more processors configured to apply one or more data processing operations to some or all of the received input data to thereby generate output data. The output data may comprise, for example, data extracted from raw input data and/or normalized data generated based on input data. The generated output data may be stored locally, transmitted to one or more other system components, and/or forwarded to one or more downstream layers in the multilayer system for further processing.

In addition to the output data, the one or more processors of the input layer may additionally generate explainability output data, which may include metadata indicating a manner in which the output data was generated, for example including an indication of the input data, an indication of the data processing operation(s) applied, an indicating of a time and/or place of data processing, and/or an indication of one or more configurations associated with said data processing. The generated explainability output data may be stored locally or remotely and/or transmitted to one or more other system components.

An intermediate layer may be configured to receive input data from an input layer and/or from another intermediate layer that is upstream of the intermediate layer. In some embodiments, an intermediate layer may also receive input data from one or more external data sources, in a same or similar manner as described above with respect to an input layer. The data may be received in structured, unstructured, and/or partially-structured (e.g., semi-structured) format. The data may be received by the intermediate layer on a scheduled basis, in response to a user input, in response to one or more trigger conditions being met, in response to the data being manually sent, and/or in response to the data being generated by an upstream layer including an input layer and/or another intermediate layer.

The intermediate layer may comprise one or more processors configured to apply one or more data processing operations to some or all of the received input data to thereby generate output data. The output data may comprise, for example, one or more adjudications, scores, calculations, classifications, identified events, and/or identified content generated based on the input data. The generated output data may be stored locally, transmitted to one or more other system components, and/or forwarded to one or more downstream layers in the multilayer system for further processing.

In addition to the output data, the one or more processors of the intermediate layer may additionally generate explainability output data, which may include metadata indicating a manner in which the output data was generated, for example including an indication of the input data, an indication of the data processing operation(s) applied, an indicating of a time and/or place of data processing, and/or an indication of one or more configurations associated with said data processing. The generated explainability output data may be stored locally or remotely and/or transmitted to one or more other system components.

A presentation layer may be configured to receive input data from an input layer and/or from an intermediate layer upstream of the presentation layer. In some embodiments, a presentation layer may also receive input data from one or more external data sources, in a same or similar manner as described above with respect to an input layer. The data may be received in structured, unstructured, and/or partially-structured (e.g., semi-structured) format. The data may be received by the presentation layer on a scheduled basis, in response to a user input, in response to one or more trigger conditions being met, in response to the data being manually sent, and/or in response to the data being generated by an upstream layer including an input layer and/or an intermediate layer.

The presentation layer may comprise one or more processors configured to apply one or more data processing operations to some or all of the received input data to thereby generate output data. The output data may comprise, for example, one or more adjudications, scores, calculations, classifications, identified events, identified content, alerts, user interface objects, and/or visualizations generated based on the input data. In some embodiments, output data from a presentation layer may comprise an indication of classification, such as a risk classification. In some embodiments, output data from a presentation layer may indicate a risk classification for a transaction represented by the original input data received by the system. In some embodiments, output data from a presentation layer may indicate an irregularity, anomaly, and/or inconsistency for data that was received by and processed by the system. In some embodiments, output data from a presentation layer may indicate whether one or more criteria (e.g., vouching criteria, tracing criteria, related-party transaction criteria, data integrity criteria, process integrity criteria, and/or policy integrity criteria) are satisfied by the data that was received by and processed by the system. In some embodiments, output data from a presentation layer may comprise a confidence score associated with one or more indications (e.g., labels or classifications) of the output data. The generated output data may be stored locally, transmitted to one or more other system components, and/or presented (e.g., displayed on a display of a user system) to a user.

In addition to the output data, the one or more processors of the presentation layer may additionally generate explainability output data, which may include metadata indicating a manner in which the output data was generated, for example including an indication of the input data, an indication of the data processing operation(s) applied, an indicating of a time and/or place of data processing, and/or an indication of one or more configurations associated with said data processing. The generated explainability output data may be stored locally or remotely and/or transmitted to one or more other system components.

In some embodiments, different layers in a multilayer data processing system may leverage one or more of the same processors for data processing operations performed at different layers; in some embodiments, one or more separate processors may be used for data processing operations performed at different layers.

In some embodiments, a multilayer data processing system may only present (e.g., display) output data generated by a presentation layer to a user. In some embodiments, the system may additionally present some or all output data generated by one or more upstream layers to the user. In some embodiments, the system may present some or all of the original input data received by the system.

In some embodiments, in addition to output data, the system may present some or all of the explainability data generated by one or more of the layers (including input layers, intermediate layers, and/or presentation layers). In some embodiments, explainability data may be presented (e.g., displayed, for example as part of a visualization) to a user upon being generated, upon presentation to the user of output data associated with the explainability data, and/or in response to one or more requests from the user requesting the display of explainability data. In some embodiments, a user interface may initially display output data generated by one or more presentation layers, and the interface may afford the user the opportunity to execute one or more user inputs requesting display of explainability data associated with the displayed output data. For example, a user execute an input by clicking on a visualization, selecting one or more affordances, and/or entering one or more input strings into one or more fields. The user input may indicate a request for explainability data to be presented, for example by indicating (e.g., clicking on) a piece of output data for which associated explainability data should be requested. The user input may indicate a request for explainability data to be presented, for example by indicating a type of explainability data, an explainability visualization type, and/or one or more layer of the system for which to display explainability data. In some embodiments, a user may click on output data in order to “drill down” into associated explainability data, for example causing the system to display explainability data for the layer that generated the output data that was clicked, and/or causing the system to display output data for one or more upstream layers that was used as input to generate the output data that was clicked. In some embodiment, a user may execute a plurality of successive drill-down inputs, clicking on newly-displayed output data and/or newly-displayed explainability data to drill down further into the data that was processed through the multilayer system. By execute a plurality of successive drill-down inputs, a user may move backwards from the final output data (initially displayed to the user) back through all of the layers of the system, such that the system displays all intermediate data and the original input data that was used to arrive at the selected output data and/or such that the system displays explainability data for all system layers leveraged to arrive at the output data.

In some embodiments, the one or more processors of a layer of the system may be configured to apply one or more data processing operations that may include deductive reasoning data processing operations, inductive reasoning data processing operations, and/or abductive reasoning data processing operations.

In some embodiments, input data originally provided to an input layer of the system may include data indicating time information, and the data-processing operations performed by one or more layers of the system may include generating time-dependent output data. Time dependent information can be presented to the input either sequentially (e.g., for information that belong to the same point in time) or concurrently (e.g., for information that belongs to more than one point in time).

In some embodiments, (alternatively to or in addition to time-dependency as described above) input data originally provided to an input layer of the system may data indicating location information, and the data-processing operations performed by one or more layers of the system may include generating location-dependent output data. In some embodiments, location may be expressed by a two-dimensional description (x, y), which may for example represent a latitude and longitude; or location may be expressed by a three-dimensional description (x, y, z), where the third dimension indicates a vertical position, altitude, as measured from sea-level, and/or height off of the ground.

In some embodiments, when the system generates output data that is time-dependent and/or location-dependent, a user may execute one or more inputs to request display of output data (and/or associated explainability data) for period of time and/or for a location that is specified by the input. In some embodiments, the system may responsively update the displayed output data in order to display information relevant to the user-specified time and/or location. In some embodiments, the system may display output data that has already been generated; in some embodiments, the system may activate one or more of the data processing layers in order to process data to generate output data for the user-specified time and/or location.

In some embodiments, the one or more processors of a layer of the system may be configured to determine which of a plurality of data processing operations should be applied. The one or more processors may automatically select one or more data processing operations to apply based on system settings, user input, and/or based on input data generated by one or more upstream layers of the system. In some embodiments, the input data (e.g., data format, data type, data content, etc.) may dictate which one or more data processing operations are applied by a layer (and/or may dictate whether the layer applies any data processing operations or alternatively simply forwards the input data to another layer).

In some embodiments, the system may be configurable according to one or more user inputs, which may be provided to the system by an end user and/or administrator. In some embodiments, the system may be configurable according to one or more user inputs that specify a manner in which one or more layers processes data. For example some embodiments, a user input may indicate a type or identity of analysis, a type or identity of test, a type or identity of model, one or more weight values, and/or one or more threshold values, and the system may configure a specified layer in accordance with the user input. In some embodiments, reconfiguring a layer of the multilayer system may be done without reconfiguring any other layers of the system, including upstream layers that provide input data to the reconfigured layer and including downstream layers that accept the output data of the reconfigured layer. In some embodiments, a layer may be reconfigured (e.g., by adjusting data processing operations applied by the layer) without changing the format of data accepted as input by the layer and without changing the format of data generated as output by the layer. In this manner, a layer may be adjusted and may continue to interact with upstream and downstream layers in the same manner as before the adjustment was made.

In some embodiments, a layer of the system may be reconfigurable to change the amount, type, and/or content of explainability data generated by a layer. In some embodiments, the explainability data generated by a layer may be changed without changing the output data itself that is generated by the layer, such that downstream layers that receive the output data from that layer are not affected by the reconfiguration.

In some embodiments, a layer may be reconfigured in accordance with a change to an ontology that is leveraged by a data processing operation applied by the layer. For example, if an additional class is identified, then the additional class may be added to an ontology used by a classifier of the layer. In some embodiments, the addition of the new class cause the layer to generate output data in a same format (e.g., using a same language and/or a same set of symbols) as before the addition, such that downstream layers do not need to be adjusted to interpret output data that is generated in accordance with the new class. In some embodiments, the layer to which an additional class is added may be configured such that it applies one or more translation operations, normalization operations, and/or other data processing operations to convert data generated in accordance with the new class into a format that can be interpreted by downstream layers without requiring adjustments to the downstream layers. (In some embodiments, a downstream layer may be adjusted in order to ensure that the downstream layer can process all possible upstream output data when an upstream layer is reconfigured to generate output data comprising one or more new classes, new symbols, or the like.)

In some embodiments, the system may be configurable according to one or more user inputs that specify a manner in which one or more layers are interconnected with one or more other layers. For example, a user input may indicate: (a) from which upstream layers and/or other data sources a certain layer should accept input; and/or (b) to which downstream layers a certain layer should provide output data for further processing by the downstream layers. In some embodiments, a user input may specify whether a given layer should (or should not) forward any received data (without processing) to one or more subsequent layers. In some embodiments, a user input may indicate whether receipt of input data from a given source and/or provision of output data to a given source is optional or required. In some embodiments, a user may configure the interconnections between the layers by selecting one or more layers to be linked downstream or upstream to a given layer from a menu, and/or by using a drag-and-drop user interface to rearrange layers and/or to place, rearrange, and/or remove connecting links between visualizations representing respective layers.

FIG. 5 shows data layer schema 500, which may comprise a plurality of data layers that may be processed through layers of a multilayer data-processing system such as system 400 shown in FIG. 4. As shown, schema 500 may include raw data 502, information data 504, integrity observation data 506, insight data 508, actionable insight data 510, and recommendation data 512. Each of the different data layers in schema 500 may be stored in one or more databases, together or separately from one another, optionally in association with metadata including explainability metadata indicating the manner in which data in the corresponding was derived (and/or the underlying data from which it was derived).

In some embodiments of a multilayer data-processing schema such as schema 500, raw evidence data may be ingested and may be processed through one or more normalization operations with respect to context (such as customer/vendor/product master data) for being evaluated as evidence. Data and process integrity validation (see Disclosure on Composable Data-Process-Policy integrity framework for FSLI audit) data processing operations may be applied in order to perform validation and produce integrity observation data. This observation data may include report mismatches of various fields, such as customer name, address, shipping term, payment term, line items, unit price, and/or total amount. The interpretation of evidence evaluation data that belongs to the same transaction may then combined through deductive, inductive, and/or abductive reasoning data processing operations. Further aggregation data processing operations may be applied based on customer, location, and/or product, and said data processing operations may generate insight data. Insight data, such as data indicating one or more distinct out-of-ordinary behaviors or events, may be subject to one or more data processing operations to generate actionable insight data, which itself may be processed by one or more data processing operations to generate recommendation data indicating one or more potential solutions for such behaviors and/or events.

FIG. 5 shows a diagram of a different classes of data that may be input into, processed by, generated by, and/or output by a system (e,g., system 400) for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers, in accordance with some embodiments.

As shown in FIG. 5, a system may be figured to receive raw data 502 (e.g., evidence data) including, for example, sales order data, purchase order data, invoice data, inventory ledger data, and credit memo data. In some embodiments, raw data 502 may be the original data provided to a multilayer data-processing system.

As further shown in FIG. 5, the system may be configured to generate information data 504 based at least in part on all or part of the raw data 502 received. Information data generated 504 may include, for example, normalized data (e.g., normalized sales order data, normalized purchase order data, etc.). The system may further be configured to receive information data, including for example customer information, vendor information, and/or product information. Raw data 502 may be processed by an input data layer to generate information data 504. The process of processing raw data 502 to produce information data 504 may share any one or more characteristics in common with the processing of evidence data 402 to produce evidence understanding data 404 as described above with reference to FIG. 4. In some embodiments, raw data may 502 share any one or more characteristics in common with evidence 402 and information data 504 may share any one or more characteristics in common with evidence understanding data 404. In some embodiments, processing raw data 502 to produce information data 504 may also include generating explainability data indicating the manner in which information data 504 was produced. In some embodiments, customer master data, vendor master data, and/or product master data may include data that is included in raw data 502 and/or in information data 504.

As further shown in FIG. 5, the system may be configured to generate integrity observation data 506 based on all or part of the information data 504 (and/or based on other upstream data). Integrity observation data 506 may include, for example, data regarding one or more inventory relief mismatches and/or regarding one or more credit memo/refund mismatches. The process of processing information data 504 to produce integrity observation data 506 may share any one or more characteristics in common with the processing of data by one or more intermediate layers of system 400 as described above with respect to FIG. 4. In some embodiments, the system may generate explainability data indicating the manner in which integrity observation data 506 was produced.

As further shown in FIG. 5, the system may be configured to generate insight data 508 based on all or part of integrity observation data 506 (and/or based on other upstream data). Insight data may include, for example, data indicating one or more irregular inventory clusters and/or indicating one or more irregular refund clusters. The process of processing integrity observation data 506 to produce insight data 508 may share any one or more characteristics in common with the processing of data by one or more intermediate layers of system 400 as described above with respect to FIG. 4. In some embodiments, the system may generate explainability data indicating the manner in which insight data 508 was produced.

As further shown in FIG. 5, the system may be configured to generate actionable insight data 510 based on all or part of insight data 508 (and/or based on other upstream data). Actionable insight data 510 may include, for example, data indicating irregular invent and refund activities. The process of processing insight data 508 to produce actionable insight data 510 may share any one or more characteristics in common with the processing of data by one or more intermediate layers of system 400 as described above with respect to FIG. 4. In some embodiments, the system may generate explainability data indicating the manner in which actionable insight data 510 was produced.

As further shown in FIG. 5, the system may be configured to generate recommendation data 512 based on all or part of actionable insight data 510 (and/or based on other upstream data). Recommendation data 512 may include, for example, data indicating a specific recommended course of action, for example a recommendation to monitor a specific office (e.g., due to irregular inventory and refund activities in that office). The process of processing actionable insight data 510 to produce recommendation data 512 may share any one or more characteristics in common with the processing of data by a presentation layer such as layer 414 of system 400 as described above with respect to FIG. 4. In some embodiments, the system may generate explainability data indicating the manner in which recommendation data 512 was produced.

In some embodiments, a system for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers may process each of the different classes of data described with respect to FIG. 5 via a different layer of the system, such that each class of data is processed by one or more data processing operations that are configured to operate on data of that type.

FIG. 6 shows the ontology for the explanation of evidence evaluation. In terms of the order process, the potential observation from evidence evaluation could include shipped before order placed, invoiced before order placed, invoiced but not shipped, shipped but not invoiced. In terms of Payment, Refund and Collection process, the observation could include un-vouched payment, unusual refund, refund without inventory return, Refund for shipping discrepancy, refund for product damage In terms of the accounting process, the observation could include A/R reversal, JE reversal/adjustment, mismatch between invoice and AR and between payment and payment journal.

In some embodiments, a system for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers, such as those described herein, may identify one or more of the anomalous classifications shown in the bottom portion of FIG. 6.

FIG. 7 shows a schematic diagram of data processing for financial data, in accordance with some embodiments. In some embodiments, the data processing depicted in FIG. 7 may be performed by a system for tracking reasoning and adjudication processes for processing data through a plurality of interconnected layers, such as those described herein.

In some embodiments of a system architecture 700, evidence data such as purchase order data 708, bill of lading data 706, and bank statement data 704 may be processed through a plurality of data processing operations (e.g., document-understanding models, AI models, machine learning models, etc.) to generate one or more instances of intermediate data (e.g., data regarding vouching, invoices, revenue, contracts, and/or prices). The intermediate data may additionally be generated on the basis of ERP data. In some embodiments, the intermediate data may be used to generate recommendation data 716, such as a recommendation to perform a revenue audit.

FIG. 7 shows logic from input layer to data integrity evaluation and process integrity evaluation. The output from both data and process integrity may feed into a semi-final layer in policy integrity and then the final observation and adjudication whether revenue from a specific transaction can be included for revenue recognition.

In the arrangement shown in FIG. 7, purchase order data 708, bill of lading data 706, and bank statement data 704 may be received by data integrity system 710, which may include one or more processors configured to perform one or more data integrity data processing operations including, for example, purchase order vouching, bill-of-lading vouching, and/or payment vouching. Data integrity system 710 may additionally receive ERP data from ERP data source 702, and may use said ERP data to apply the one or more data integrity data processing operations. Data integrity system 710 may validate one or more assertions (e.g., existence, completeness, and/or accuracy) by comparing external evidence (e.g., document data) to ERP data. The one or more data integrity data processing operations may include any of those performed by data integrity system 120 as described in U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR APPLYING A COMPOSABLE ASSURANCE INTEGRITY FRAMEWORK,” filed Jun. 30, 2022, Attorney Docket No. 13574-20070.00.

ERP data from ERP data source 702 may additionally be received by process integrity system 712, which may include one or more processors configured to perform one or more process integrity data processing operations. Process integrity system 712 may validate assertions (e.g., accuracy, cutoff, and/or classification) by tracing transactions based on ERP data from data source 702, for example by following a process (e.g., an order-to-cash transaction process, or a procure-to-pay transaction process) end to end. The one or more process integrity data processing operations may include any of those performed by process integrity system 110 as described in U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR APPLYING A COMPOSABLE ASSURANCE INTEGRITY FRAMEWORK,” filed Jun. 30, 2022, Attorney Docket No. 13574-20070.00.

The output data generated by data integrity system 710 and/or process integrity system 712 may be received by policy integrity system 714, which may include one or more processors configured to perform one or more policy integrity data processing operations. Policy integrity system 714 may validates and adjudicate whether the input data (e.g., representing an overall transaction) is consistent with one or more predefined criteria, for example including accounting standards criteria and/or policy standards criteria. The one or more policy integrity data processing operations may include any of those performed by process integrity system 110 as described in U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR APPLYING A COMPOSABLE ASSURANCE INTEGRITY FRAMEWORK,” filed Jun. 30, 2022, Attorney Docket No. 13574-20070.00. The output data generated by policy integrity system 714 may include, or may be used to in turn generate, recommendation data 716.

FIG. 7 demonstrates an example of an architecture that may conform with the multilayer explainability techniques described herein, for example by allowing the decomposition of verification of whether the revenue from a transaction can be recognized according to the accounting principles and auditing standards (e.g., by system 714) into whether the revenue actually exists (e.g., by data integrity verification system 710) and whether there is any discrepancy during the execution of an associated business process (e.g., by systems 710 and 712).

Below are steps for, in some embodiments, establishing summary/recommendation data to be presented to a human user:

• • 1. Collect historical summary/observation for the specific audit area (such as revenue & receivable audit)
  • 2. Provide supervised classification or unsupervised clustering using appropriate ML models to establish the appropriate ontology and/or classification tree for the observation & recommendation. This becomes the initial set of the observation & recommendation
  • 3. Monitor the usage statistics of the observation and recommendation when the observation/recommendation is used by the human user and collect additional observations/recommendation
  • 4. Continuous learning and evolving the observation & recommendation based on the usage statistics and the augmentation/curation by human users

Below are steps for, in some embodiments, tracing a reasoning process. For a specific order to cash transaction, we could potentially go through the following evaluation:

• • 1. Data integrity
    • a. PO was vouched successfully in terms of the existence of PO#, customer name, total amount, and date
    • b. BoL was missing—so there is no independent validation of the shipment
    • c. Payment was vouched partially with respect to the payment journal and account receivable as separate entries of payment journal related to the same invoice was not vouched
  • 2. Process integrity
    • a. Sales order was cross validated with invoice, after identifying potential typo when entering the PO# into the sales order
    • b. Cross validation between invoice and inventory, AR rollforward, AR remove extended, and other modules all indicate consistency
  • 3. Policy integrity
    • a. Transfer of control was evaluated only based on the shipping term specified by the PO and Sales order to be FOB, and hence the date of title transfer was the date when the products leaving the shipping dock
    • b. Collectability based on the previous payment behavior indicating no delinquent payment, but with a trend of paying closer to the due date.
    • c. Recalculating of recognizable revenue in terms of unit price x quantity seems to indicate consistency
  • 4. Final recommendation
    • a. The revenue recognized is consistent with the accounting policy and will be included in the final recognizable revenue

FIG. 8 illustrates an example of a computer, according to some embodiments. Computer 800 can be a component of a system for providing an AI-augmented auditing platform including techniques for providing AI-explainability for processing data through multiple layers. In some embodiments, computer 800 may execute any one or more of the methods described herein.

Computer 800 can be a host computer connected to a network. Computer 800 can be a client computer or a server. As shown in FIG. 8, computer 800 can be any suitable type of microprocessor-based device, such as a personal computer, workstation, server, or handheld computing device, such as a phone or tablet. The computer can include, for example, one or more of processor 810, input device 820, output device 830, storage 840, and communication device 860. Input device 820 and output device 830 can correspond to those described above and can either be connectable or integrated with the computer.

Input device 820 can be any suitable device that provides input, such as a touch screen or monitor, keyboard, mouse, or voice-recognition device. Output device 830 can be any suitable device that provides an output, such as a touch screen, monitor, printer, disk drive, or speaker.

Storage 840 can be any suitable device that provides storage, such as an electrical, magnetic, or optical memory, including a random access memory (RAM), cache, hard drive, CD-ROM drive, tape drive, or removable storage disk. Communication device 860 can include any suitable device capable of transmitting and receiving signals over a network, such as a network interface chip or card. The components of the computer can be connected in any suitable manner, such as via a physical bus or wirelessly. Storage 840 can be a non-transitory computer-readable storage medium comprising one or more programs, which, when executed by one or more processors, such as processor 810, cause the one or more processors to execute methods described herein.

Software 850, which can be stored in storage 840 and executed by processor 810, can include, for example, the programming that embodies the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and/or devices as described above). In some embodiments, software 850 can include a combination of servers such as application servers and database servers.

Software 850 can also be stored and/or transported within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of this disclosure, a computer-readable storage medium can be any medium, such as storage 840, that can contain or store programming for use by or in connection with an instruction execution system, apparatus, or device.

Software 850 can also be propagated within any transport medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of this disclosure, a transport medium can be any medium that can communicate, propagate, or transport programming for use by or in connection with an instruction execution system, apparatus, or device. The transport-readable medium can include but is not limited to, an electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation medium.

Computer 800 may be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communications protocol and can be secured by any suitable security protocol. The network can comprise network links of any suitable arrangement that can implement the transmission and reception of network signals, such as wireless network connections, T1 or T3 lines, cable networks, DSL, or telephone lines.

Computer 800 can implement any operating system suitable for operating on the network. Software 850 can be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, application software embodying the functionality of the present disclosure can be deployed in different configurations, such as in a client/server arrangement or through a Web browser as a Web-based application or Web service, for example.

Following is a list of embodiments:

• • Embodiment 1. A system for providing explainability for processing data through multiple data-processing layers, the system comprising one or more processors configured to cause the system to cause the system to:
    • at an input layer:
      • receive an evidence data set comprising a plurality of evidence items;
      • apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and
      • generate input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer;
    • at a presentation layer:
      • receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data;
      • apply one or more presentation generation models to the received data to generate presentation data; and
      • generate presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer;
    • cause display of the presentation data; and
    • cause display of one or more of: the input-layer explainability data and the presentation-layer explainability data.
  • Embodiment 2. The system of embodiment 1, wherein the one or more processors are configured to cause the system to:
    • at one or more intermediate layers:
      • receive the evidence understanding data generated by the input layer;
      • apply one or more intermediate-layer processing models to the evidence understanding data to generate the data received by the presentation layer;
      • provide the data received by the presentation layer to the presentation layer; and
      • generate intermediate-layer explainability data, wherein the intermediate-layer explainability data represents information about the processing of the evidence understanding data by the one or more intermediate layers.
  • Embodiment 3. The system of any one of embodiments 1-2, wherein the input layer and the presentation layer are each configured to apply a respective ontology.
  • Embodiment 4. The system of any one of embodiments 1-3, wherein the one or more processors are configured to cause the system to:
    • receive a user input comprising an instruction to modify the input layer; and
    • modify the input layer in accordance with the user input without modifying the presentation layer.
  • Embodiment 5. The system of any one of embodiments 1-4, wherein the one or more processors are configured to:
    • receive a user input comprising an instruction to modify the presentation layer; and
    • modify the presentation layer in accordance with the user input without modifying the input layer.
  • Embodiment 6. The system of embodiment 5, wherein modifying the presentation layer comprises modifying the one or more presentation generation models while maintaining an input data format for the one or more presentation generation models and maintain an output data format for the one or more presentation generation models.
  • Embodiment 7. The system of any one of embodiments 5-6, wherein modifying the presentation layer comprises modifying one or more connections of the presentation layer to one or more other layers of the system.
  • Embodiment 8. The system of any one of embodiments 1-7, wherein the one or more processors are configured to initialize the presentation layer by applying one or more machine learning models to classify output data from one or more prior analyses performed by the system.
  • Embodiment 9. The system of any one of embodiments 1-8, wherein the one or more processors are configured to cause the system to:
    • receive utilization data representing a manner in which the presentation output is utilized by one or more users; and
    • automatically modify the presentation layer in accordance with the utilization data.
  • Embodiment 10. The system of any one of embodiments 1-9, wherein the one or more processors are configured to cause the system to:
    • receive a user input comprising a selection of a portion of the displayed presentation data,
    • wherein causing display of one or more of the input-layer explainability data and the presentation-layer explainability data is performed in accordance with the user input.
  • Embodiment 11. The system of any one of embodiments 1-10, wherein the one or more processors are configured to cause the system to, at the presentation layer, select, based on the received data, the one or more presentation generation models from a superset of presentation generation models.
  • Embodiment 12. A non-transitory computer-readable storage medium storing instructions for providing explainability for processing data through multiple data-processing layers, wherein the instructions are configured to be executed by a system comprising one or more processors to cause the system to:
    • at an input layer:
      • receive an evidence data set comprising a plurality of evidence items;
      • apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and
      • generate input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer;
    • at a presentation layer:
      • receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data;
      • apply one or more presentation generation models to the received data to generate presentation data; and
      • generate presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer;
    • cause display of the presentation data; and
    • cause display of one or more of: the input-layer explainability data and the presentation-layer explainability data.
  • Embodiment 13. A method for providing explainability for processing data through multiple data-processing layers, wherein the method is performed by a system comprising one or more processors, the method comprising:
    • at an input layer:
      • receiving an evidence data set comprising a plurality of evidence items;
      • applying one or more evidence processing models to the evidence data set to generate evidence understanding data; and
      • generating input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer;
    • at a presentation layer:
      • receiving data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data;
      • applying one or more presentation generation models to the received data to generate presentation data; and
      • generating presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer;
    • causing display of the presentation data; and
    • causing display of one or more of: the input-layer explainability data and the presentation-layer explainability data.

This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR AUTOMATED ASSESSMENT OF VOUCHING EVIDENCE”, filed Jun. 30, 2022, Attorney Docket no. 13574-20068.00.

This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR AUTOMATED ADJUDICATION OF COMMERCIAL SUBSTANCE, RELATED PARTIES, AND COLLECTABILITY”, filed Jun. 30, 2022, Attorney Docket no. 13574-20069.00.

This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR APPLYING A COMPOSABLE ASSURANCE INTEGRITY FRAMEWORK”, filed Jun. 30, 2022, Attorney Docket no. 13574-20070.00.

This application incorporates by reference the entire contents of the U.S. Patent Application titled “AI-AUGMENTED AUDITING PLATFORM INCLUDING TECHNIQUES FOR AUTOMATED DOCUMENT PROCESSING”, filed Jun. 30, 2022, Attorney Docket no. 13574-20071.00.

Claims

1. A system for providing explainability for processing data through multiple data-processing layers, the system comprising one or more processors configured to cause the system to cause the system to:

at an input layer: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer;

at a presentation layer: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer;

cause display of the presentation data; and

cause display of one or more of: the input-layer explainability data and the presentation-layer explainability data.

2. The system of claim 1, wherein the one or more processors are configured to cause the system to:

at one or more intermediate layers: receive the evidence understanding data generated by the input layer; apply one or more intermediate-layer processing models to the evidence understanding data to generate the data received by the presentation layer; provide the data received by the presentation layer to the presentation layer; and generate intermediate-layer explainability data, wherein the intermediate-layer explainability data represents information about the processing of the evidence understanding data by the one or more intermediate layers.

3. The system of claim 1, wherein the input layer and the presentation layer are each configured to apply a respective ontology.

4. The system of claim 1, wherein the one or more processors are configured to cause the system to:

receive a user input comprising an instruction to modify the input layer; and

modify the input layer in accordance with the user input without modifying the presentation layer.

5. The system of claim 1, wherein the one or more processors are configured to:

receive a user input comprising an instruction to modify the presentation layer; and

modify the presentation layer in accordance with the user input without modifying the input layer.

6. The system of claim 5, wherein modifying the presentation layer comprises modifying the one or more presentation generation models while maintaining an input data format for the one or more presentation generation models and maintain an output data format for the one or more presentation generation models.

7. The system of claim 5, wherein modifying the presentation layer comprises modifying one or more connections of the presentation layer to one or more other layers of the system.

8. The system of claim 1, wherein the one or more processors are configured to initialize the presentation layer by applying one or more machine learning models to classify output data from one or more prior analyses performed by the system.

9. The system of claim 1, wherein the one or more processors are configured to cause the system to:

receive utilization data representing a manner in which the presentation output is utilized by one or more users; and

automatically modify the presentation layer in accordance with the utilization data.

10. The system of claim 1, wherein the one or more processors are configured to cause the system to:

receive a user input comprising a selection of a portion of the displayed presentation data,

wherein causing display of one or more of the input-layer explainability data and the presentation-layer explainability data is performed in accordance with the user input.

11. The system of claim 1, wherein the one or more processors are configured to cause the system to, at the presentation layer, select, based on the received data, the one or more presentation generation models from a superset of presentation generation models.

12. A non-transitory computer-readable storage medium storing instructions for providing explainability for processing data through multiple data-processing layers, wherein the instructions are configured to be executed by a system comprising one or more processors to cause the system to:

at an input layer: receive an evidence data set comprising a plurality of evidence items; apply one or more evidence processing models to the evidence data set to generate evidence understanding data; and generate input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer;

at a presentation layer: receive data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; apply one or more presentation generation models to the received data to generate presentation data; and generate presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer;

cause display of the presentation data; and

cause display of one or more of: the input-layer explainability data and the presentation-layer explainability data.

13. A method for providing explainability for processing data through multiple data-processing layers, wherein the method is performed by a system comprising one or more processors, the method comprising:

at an input layer: receiving an evidence data set comprising a plurality of evidence items; applying one or more evidence processing models to the evidence data set to generate evidence understanding data; and generating input-layer explainability data, wherein the input-layer explainability data represents information about the processing of the evidence data set by the input layer;

at a presentation layer: receiving data, wherein the received data includes one of: the evidence understanding data, and data generated based on the evidence understanding data; applying one or more presentation generation models to the received data to generate presentation data; and generating presentation-layer explainability data, wherein the presentation-layer explainability data represents information about the processing of the received data by the input layer;

causing display of the presentation data; and

causing display of one or more of: the input-layer explainability data and the presentation-layer explainability data.