Abstract: The present disclosure relates to systems and methods for generating user-specific outputs using modular AI agents and hybrid data representation engines. At least one processor may receive user data inputs and task configuration data from multiple sources, where the task configuration data specifies a user-specific task. The processor may process the received data to generate raw input data and transmit it, along with the task configuration data, to a control plane configured to determine processing parameters for modular AI agents. The modular AI agents may include a feature selection agent to extract user interaction features, a numerical transformation agent to convert numerical values into text formats, and a summarization agent to generate a data representation with task-specific information. A hybrid data representation engine may generate user-specific outputs, which may be evaluated using a large language model, stored in a persistent storage layer, and provided via APIs to downstream systems.

Abstract: A computer-implemented method includes obtaining at least one metric from a framework processing large-scale data, generating at least one prompt from the at least one metric, providing the at least one prompt to a selected large language model (LLM), the LLM being trained with data related to the framework, converting a recommendation from the LLM into a first configuration for the framework, forming a second configuration by applying at least one guiderail to the first configuration, applying the second configuration to the framework for processing a first data processing job, evaluating a cause of failure of the first data processing job, automatically adjusting, based on the cause of failure, the second configuration to form a third configuration, and apply the third configuration to the framework for processing a second data processing job, the second data processing job being a resubmitted first data processing job.