Smallest open source model to outperform GPT-4o-mini as an evaluator
SAN FRANCISCO, Dec. 19, 2024 /PRNewswire/ — Patronus AI today announced the release of GLIDER, its groundbreaking 3.8B parameter model designed as a fast, flexible, and explainable judge for language models. The new open-source model is the smallest model to outperform GPT-4o-mini used as an evaluator, offering institutions a fast and cost-effective solution for evaluations without sacrificing quality.
Traditional proprietary LLMs like GPT-4 are widely used to evaluate the performance and accuracy of other language models, but they come with their own challenges—high costs, limited scalability, and a lack of transparency. Developers often end up relying on opaque outputs without understanding why something was scored the way it was.
GLIDER delivers the first small, explainable ‘LLM-as-a-judge’ solution, providing real-time evaluations with transparent reasoning and actionable insights. Instead of just assigning a score, GLIDER explains the “why” behind it, enabling developers to make informed decisions with confidence. For every evaluation, GLIDER outputs a list of detailed reasons behind the score, highlighting the most critical phrases from the input that influenced the result. This gives developers both a high-level understanding of the model’s performance and a deeper view into its failure points.
“Our mission is to make AI evaluation accessible to everyone,” said Anand Kannappan, CEO and Co-founder of Patronus AI. “This new 3.8B parameter model represents a major step forward in democratizing high-performance evaluations. By combining speed, versatility, and explainability with an open-source approach, we’re enabling organizations to deploy powerful guardrail systems without sacrificing cost-efficiency or privacy. It’s a significant contribution to the AI community, proving that smaller models can drive big innovations.”
The new judge model is a lightweight yet powerful evaluation tool, purpose-built to address the needs of organizations seeking robust and versatile assessment capabilities. Key features include:
- Explainability: Generates high-quality reasoning chains and text highlighting for visualization, improving decision transparency and benchmark scores.
- Broad Applicability: Trained on 183 real-world evaluation criteria across 685 domains, ensuring broad applicability.
- Versatile Judgments: Evaluates not only model outputs but also user inputs, contexts, metadata, and more.
- Low Latency: Served at a latency of 1 second on the Patronus platform for real-time applications.
- Flexible Scoring Systems: Supports binary (0-1), 3-point, and 5-point Likert-based rubric scales for tailored evaluations and preference evaluations.
- Factuality and Creativity: Excels in tasks requiring factual accuracy and subjective human-like metrics such as coherence and fluency, making it ideal for creative and business applications alike.
The new model addresses a critical demand for fast, reliable guardrail systems without compromising privacy or quality. With open weights derived from open-source models, this model supports on-premises deployment for diverse evaluation use cases like LLM guardrails and subjective text analysis. By offering high performance in a small package, Patronus AI’s GLIDER democratizes access to advanced evaluation capabilities and promotes community-driven innovation.
“Our new model challenges the assumption that only large-scale models (30B+ parameters) can deliver robust and explainable evaluations,” said Rebecca Qian, CTO and Co-founder. “By demonstrating that smaller models can achieve similar results, we’re setting a new benchmark for the community. Its explainability features not only enhance model decisions but also improve overall performance, paving the way for broader adoption in guardrailing, subjective analysis, and workflow evaluations requiring human-like judgment.”
About Patronus AI
Patronus AI is the first automated evaluation and security platform that helps companies use large language models (LLMs) confidently. The company was founded by machine learning experts Anand Kannappan and Rebecca Qian, formerly of Meta AI and Meta Reality Labs. For more information, please visit https://www.patronus.ai/.
SOURCE Patronus AI
