Open Financial LLM Leaderboard
Specialized zero-shot financial evaluation framework for language models.
The Open Financial LLM Leaderboard (OFLL) provides a transparent, zero-shot benchmarking environment tailored exclusively for models handling complex financial reports, market forecasting, and sentiment metrics.
Overview
General NLP benchmarks do not measure stock prediction capability or credit risk scoring accurately. OFLL covers seven financial categories (Information Extraction, Textual Analysis, QA, Text Generation, Risk, Forecasting, Decision-Making) using real-world datasets.
Problem
- Traditional AI benchmarks evaluate general conversation but fail on high-dimensional financial logic.
- Evaluating model accuracy on specialized financial texts requires standardized zero-shot datasets.
Motivation
Developed under the Fintech Open Source Foundation (FINOS) and SecureFinAI Lab at Columbia University to provide an authoritative model index for financial institutions.
Solution
A zero-shot evaluation runner that scores financial domains and saves structured results JSON files directly to a Hugging Face space.
Implementation
Gradio interface, Python evaluation scripts, and structured JSON results parsers.
Engineering Decisions
Requiring explicit model SHA hash verification to prevent data-contamination and lookup cheating.
Evaluating models zero-shot to measure true context comprehension rather than prompt-engineered memorization.
Challenges & Lessons
Processing highly unstructured financial filing formats (Form 10-K/10-Q) into clean question-answer tokens.
Leaderboard Ingestion Pipeline
Flagship system
Hugging Face Model Submit ↓ (Validation / SHA Check) Evaluation Queue (eval-queue) ↓ (Zero-shot Runner) Task Evaluators (IE, TA, QA, TG, RM, FO, DM) ↓ (Scoring: Accuracy, MCC, ROUGE) Structured JSON Results ↓ Gradio Leaderboard Interface (OFLL)
Zero-shot prevents test leakage.
Results are publicly auditable.
Components
Functional building blocks
The flagship documentation uses the same component taxonomy across every system.
Zero-Shot Evaluator
Runs financial tasks against model weights.
Result Parser
Aggregates scores (Accuracy, F1, MCC) into JSON templates.
Gradio Frontend
Visualizes rankings, filters, and categories.
Features
Core capabilities
These are product features in the engineering sense, not marketing copy.
7 specialized task categories
✓Zero-shot evaluation mode
✓Hugging Face model integration
✓Update Config
Initialize target evaluation parameters.
from src.about import Task
task = Task("StockForecasting", "F1", "direction", category="Forecasting")Example output
A governed action should return enough evidence to review without reconstructing the entire runtime path.
Registered task: StockForecasting Metric set to: F1
Documentation
- OFLL contribution guidelines
Papers & Consultation Briefs
Benchmarks
- Pipeline execution (7B model): <45 mins
- Result parsing latency: <2.3s
Roadmap
The roadmap separates what is already shipped from the next layer of work so the public record stays explicit.
Shipped
Gradio UI table · 7 task categories · FINOS open source release
Next
Automatic model execution pipeline · Multi-modal document parsing support
Related Systems
Connected systems
Systems in the ecosystem that depend on or complement this flagship project.