S&P AI Benchmarks by KenshoFinance FundamentalsLong-Document QA
Sign upSign in

S&P AI Benchmarks by Kensho

A series of benchmarks that evaluate AI systems including Large Language Models (LLMs) for business and finance use cases.

S&P AI Benchmarks by Kensho consists of two evaluation sets informed by S&P Global’s world-class data and industry expertise. These benchmarks are designed to assess the ability of LLMs to solve real-world business and finance questions and were developed in collaboration with experts across S&P Global to ensure accuracy and reliability.

Everyone is welcome to sign up and participate, from academic labs and large corporations to independent model developers. The public-facing leaderboards are designed to encourage innovation and collaborative understanding.

Why We Created S&P AI Benchmarks

Although today’s LLMs generally demonstrate strong performance on question-answering (QA) and code generation tasks, it remains difficult for models to reason about quantities and numbers. This poses issues for using LLMs for real-world applications in business and finance, as these fields can require transparent and precise reasoning capabilities, along with a wide breadth of technical knowledge.

Training models
Establishing trust

Existing benchmarks for these domains include tasks such as sentiment analysis, text classification, or named-entity extraction. With S&P AI Benchmarks, we’ve created rigorous and challenging tasks that are rooted in realistic use cases for business professionals. Our goal is to build trustworthy, objective evaluation sets to encourage the development of better models for business and finance.

To learn more, read our latest research papers.

“Bizbench: A Quantitative Reasoning Benchmark for Business and Finance” (ACL 2024)
“DocFinQA: A Long-Context Financial Reasoning Dataset” (ACL 2024)

Ready to find your place on the leaderboards?

Frequently Asked Questions

Where can I find the evaluation set/questions?
Once you click “submit” and sign into your Kensho account, you will have access to the full list of questions.
When can I expect to receive the results for my submission?
Please allow 2-4 business days for a Kensho team member to review and approve your submission. You will receive an email notification with details about your scores.
How many times can I submit?
You can submit the same model to each leaderboard once per business day.
Is there a participation fee?
This is a free benchmarking leaderboard, but participants bear the cost of running the model.
What am I expected to submit?
In our submission process, you are only required to provide the outputs of your model; there is no need to share the actual model itself.
Do I need to give my LLM extra information to accurately run the tests?
Your LLM simply needs to generate a response to each question in our evaluation set; no pre-training is necessary. However, you and your team are free to prompt however you wish. You’re not restricted to asking the model the questions verbatim.
What measures are in place to ensure the evaluation process is fair and free from bias?
The evaluation is based on numerical and string matching, so the model will either generate the right or wrong numerical answer to the question. As a result, the evaluation does not require any subjective judgment calls.
Who and how would someone use this? Can you share a couple example use cases?
A researcher would use this benchmark to track improvements of their model during pre-training or fine tuning. A technology consultant can use it to prove the value-add of their LLM-powered application services. A product manager or risk manager would use this as an independent third party benchmark to verify their AI product's performance and reliability. A product or business leader would use this tool as a standardized metric to compare LLMs and inform purchasing decisions.
How will you manage my submission?
Because users submit their model’s output, not the model itself, we do not see or collect your model code or model weights. We use submitted outputs to calculate your score, which will be populated on the leaderboard. Any user who wishes to have their scores removed from the leaderboard can request so at any time.
Will the ranking give me feedback on why my LLM did or didn’t rank well?
After completing the benchmark, you will receive a score for each task. We don’t provide further or specific feedback on which questions were correct or incorrect.
Do my results have to be displayed on the leaderboard?
By default, your model’s score will be displayed on the leaderboard. If you prefer not to have your results shown, you can reach out to our team at benchmarks@kensho.com to opt out. By allowing your score to remain public, you are benefitting the broader community, as we can all collectively learn about model performance—ultimately fueling improvements over time.