Abstract: Simultaneous Weighted Preference Optimization (SWEPO) is a method for enhancing machine learning model alignment by addressing alignment biases. This approach involves calculating mean reward scores for multiple responses to a query, computing deviations, and assigning weights based on these deviations. The method partitions responses into positive and negative sets, generating a weighted contrastive loss function to optimize model parameters. This process prioritizes responses with significant deviations, improving model performance by focusing on the most informative examples. The system can be implemented on a single or distributed computing architecture, facilitating efficient training and inference processes.

Abstract: Described herein are techniques for evaluating large language model outputs through autonomous generation of context-aware evaluation criteria without relying on static human-defined standards. The approach enables dynamic generation of evaluation criteria tailored to specific instructions and responses, while incorporating context-specific knowledge crucial for accurate assessment. A framework implements both absolute evaluation against reference answers and relative comparison between multiple responses. Knowledge distillation techniques create efficient smaller models capable of criteria generation and evaluation with performance comparable to larger models. The technique demonstrates significant improvements in evaluation accuracy across diverse tasks while reducing computational costs through optimized model architectures. Additionally, the approach enhances preference-based learning through dynamically generated evaluation criteria, improving model alignment with human judgment.