Benchmark Participation Agreement
This Benchmark Participation Agreement (hereinafter referred to as “Agreement”) is made by and between Kensho Technologies, LLC (hereinafter referred to as “KENSHO”) and the entity submitting its Large Language Model outputs for evaluation (hereinafter referred to as “Benchmark Participant”).
CAREFULLY READ THE FOLLOWING TERMS AND CONDITIONS. THIS AGREEMENT CONTAINS IMPORTANT INFORMATION ABOUT RIGHTS AND OBLIGATIONS, AS WELL AS LIMITATIONS AND EXCLUSIONS. BY CLICKING ON THE ACCEPT BUTTON, BENCHMARK PARTICIPANT IS CONSENTING TO BE BOUND BY AND IS A PARTY TO THIS AGREEMENT AND BENCHMARK PARTICIPANT’S REPRESENTATIVE WHO IS ACCEPTING THIS AGREEMENT AND SUBMITTING ITS LARGE LANGUAGE MODEL OUTPUTS FOR EVALUATION REPRESENTS THEY ARE DULY AUTHORIZED TO COMMIT BENCHMARK PARTICIPANT TO THE TERMS AND CONDITIONS OF THIS AGREEMENT.
1. Scope and Purpose.
This Agreement sets forth the terms and conditions under which Benchmark Participant is requesting, and being granted, participation in an early access benchmark project under the S&P AI Benchmarks by Kensho program (the “Beta Project”). The Beta Project is intended to provide comparative analysis of finance-specific Large Language Models (LLMs) for use by teams such as data science and machine learning teams at financial institutions and financial service technology providers, and the opportunity to test the Benchmark Participant LLMs’ competence on understanding domain knowledge, extracting key financial elements, and performing quantitative reasoning.
Benchmark Participant will be provided with instructions for submitting Benchmark Participant’s LLM outputs for analysis. Upon completion of the submission process, KENSHO will perform an analysis using its proprietary AI machine learning evaluation tool, and results may be published on a publicly available benchmark results website (the “Dashboard”).
KENSHO reserves the right, in its sole discretion, to discontinue or suspend the Beta Project at any time and without prior notice.
2. Benchmark Data and Results.
Benchmark Participant will submit benchmark data relating to its own LLM in the form of answers to the provided benchmarking questions (the “Benchmark Data”). Benchmark Participant will submit Benchmark Data that follows KENSHO benchmark eligibility criteria as dictated by the user guide provided, and will not provide knowingly false information.
Benchmark Data will be analyzed and evaluated by financial and data science professionals, and KENSHO’s AI machine learning evaluation tool, to produce score(s) (the “Results”). Analysis and Results are based on question-answering tasks focused on finance and business. These tasks evaluate LLMs financial background knowledge, ability to parse financial documents, and capacity to perform quantitative reasoning.
Benchmark Participant hereby irrevocably grants to KENSHO all rights and permissions in or relating to its Benchmark Data as KENSHO determines necessary to perform the Beta Project and produce the Results, to enforce this Agreement or exercise KENSHO’s rights, and to update and make improvements to the S&P AI Benchmarks by Kensho program.
Benchmark Participant grants express permission to publish Results on the Dashboard, and to identify Benchmark Participant and its LLM by name, unless Benchmark Participant chooses to list Results with their entity name being anonymous by selecting this option when submitting Benchmark Data.
Benchmark Participant acknowledges that KENSHO in its sole discretion may decide not to publish Results and may remove Results previously published at any time. Benchmark Participant may request that its published Results be removed from the Dashboard by providing a written request to benchmarks@kensho.com. Following receipt of removal request KENSHO will remove Results from the Dashboard within thirty (30) days.
KENSHO may generate reports and articles describing Beta Project for publication in professional and research journals, and present results of the Beta Project at professional conferences.
Benchmark Participant may not use the Beta Project or any of its components or Results as the basis for developing a competitive benchmarking solution (or contract with a third party to do so).
3. Fees. The parties will bear their own costs and expenses associated with this Agreement.
4. Feedback. Benchmark Participant may provide suggestions, enhancement requests, and recommendations (individually and collectively, “Feedback”) regarding the Beta Project. KENSHO may contact Benchmark Participant and Benchmark Participant agrees to make available a reasonable amount of time to discuss the Beta Project with KENSHO if so requested.
5. Intellectual Property.
5.1 KENSHO and its affiliates maintain all rights, title and interest in and to all its patents, inventions, copyrights, trademarks, domain names, trade secrets, know-how and any other intellectual property and/or proprietary rights in and to the Beta Project, AI machine learning evaluation tool, and all content available through and generated by the Beta Project.
5.2 Benchmark Participant grants to KENSHO a fully paid, royalty-free, irrevocable, worldwide, non- exclusive and fully sub-licensable right and license to use, reproduce, and display Feedback solely for the purposes of improving KENSHO’s products and services. KENSHO agrees not to use Feedback in promotional materials in a way that directly identifies Benchmark Participant or its employees without obtaining prior written permission.
5.3 Benchmark Participant shall not use or display any of the name or logos of KENSHO or its affiliates (as between Benchmark Participant and KENSHO, “Kensho Marks”) without KENSHO’s prior written consent to each such use. Benchmark Participant agrees not to use, register, or take other action with respect to the Kensho Marks unless expressly agreed in writing or to add to, delete from or modify the Kensho Marks or misrepresent the relationship between KENSHO and Benchmark Participant.
6. Termination.
Either party may terminate this Agreement at any time, for any or no reason, and at either party’s convenience by providing written notice to the other. Notice of termination by Benchmark Participant to KENSHO will be made according to Section 12. Notices.
Termination of this Agreement shall not limit KENSHO’s rights granted herein as related to any Feedback, Benchmark Data, or posted Results provided before such termination. In addition, this paragraph and sections 7 through 13, and all obligations thereunder, shall survive any termination of this Agreement and will remain in effect until fulfilled.
7. Confidential Information. Benchmark Participant agrees that it will not, without the express prior written consent of KENSHO, disclose confidential, proprietary and/or trade secret information (irrespective of the form or communication and in whatever form maintained, whether documentary, computerized or otherwise), related to KENSHO or its affiliates or the Beta Project which is furnished to the Benchmark Participant by or on behalf of KENSHO or its affiliates before, on or after the date hereof, including (i) the features and functionality of the Beta Project, the benchmark evaluation questions, and any know how, trade secrets, code, flowcharts, diagrams, manuals, schematics, development tools, specifications, design documents, marketing information, financial information, business plans or reports made available to Benchmark Participant and (ii) any analyses, compilations, studies, or other documents that contain or otherwise reflect such information (the “Confidential Information”) or any part thereof to any third party, except to the extent that such Confidential Information (a) is or becomes generally available to the public through any means other than as a result of any act or omission by Benchmark Participant; (b) is rightfully received by Benchmark Participant from a third party that is not subject to any obligation of confidentiality with respect thereto and without limitation as to its use; or (c) is independently developed by Benchmark Participant without any reliance on any Confidential Information. At the termination of this Agreement or at any time by request of KENSHO, Benchmark Participant will promptly return to KENSHO or destroy (such destruction if requested to be certified by Benchmark Participant in writing) all Confidential Information in its possession and further agrees that it will not duplicate, translate, modify, copy, print, disassemble, decompile or otherwise tamper with the Beta Project or any Confidential Information.
8. Representations and Warranties, Disclaimers.
THE BETA PROJECT IS PROVIDED “AS IS”. KENSHO MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED, REGARDING THE BETA PROJECT OR RESULTS, INCLUDING ANY REPRESENTATION THAT THE BETA PROJECT OR RESULTS WILL BE ACCURATE OR ERROR-FREE. TO THE FULLEST EXTENT PERMITTED UNDER APPLICABLE LAW, KENSHO DISCLAIMS ANY IMPLIED OR STATUTORY WARRANTY, INCLUDING ANY IMPLIED WARRANTY OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE IN RESPECT OF THE BETA PROJECT OR RESULTS. FOR THE AVOIDANCE OF DOUBT, THE BETA PROJECT AND RESULTS MAY CONTAIN DEFECTS WHICH MAY BE MATERIAL, AND IS NOT EXPECTED TO OPERATE AT THE LEVEL OF PERFORMANCE OR COMPATIBILITY OF A FINAL, GENERALLY AVAILABLE PRODUCT OFFERING. THE BETA PROJECT MAY BE WITHDRAWN AT ANY TIME. ACCORDINGLY, PARTICIPATION IN THE BETA PROJECT AND USE OF RESULTS IS ENTIRELY AT BENCHMARK PARTICIPANT’S OWN RISK. IN NO EVENT SHALL KENSHO OR ITS AFFILIATES BE LIABLE FOR ANY DAMAGE WHATSOEVER ARISING OUT OF THE USE OF OR INABILITY TO USE THE BETA PROJECT OR RESULTS, EVEN IF KENSHO OR ITS AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Benchmark Participant covenants that it will not assert against KENSHO or its affiliates or any of their directors, officers, employees, contractors, or agents any claim or action arising from its participation in the Beta Project or use or exploitation of the Results. Benchmark Participant represents and warrants that it will have or has executed all necessary licenses and agreements, sufficient to enable it to provide Benchmark Data and participate in the Beta Project, and that such does not violate any laws, including privacy laws.
9. Limitation of Liability.
KENSHO and Benchmark Participant’s entire liability for all claims in the aggregate arising under this Agreement will not exceed one thousand U.S. Dollars ($1,000.00). This limit applies regardless of why a party claims damages from the other, including default, fundamental breach, negligence, misrepresentation, or other contract or tort claim.
Neither party will be liable for any consequential, indirect or special damages arising hereunder. Neither KENSHO nor its affiliates shall in any way be liable to Benchmark Participant nor any client of Benchmark Participant for any inaccuracies, errors or omissions, regardless of cause (except in event of willful misconduct by KENSHO), or for any damages (whether direct or indirect) howsoever arising under or in connection with the Benchmark Project or this Agreement. This limitation does not limit or exclude liability for death or for personal injury or arising from fraud.
10. Governing Law and Venue. This Agreement will be construed and enforced in accordance with the laws of the State of New York without regard to its conflict of laws principles. The parties expressly disclaim the application of the United Nations Convention on the International Sale of Goods to this Agreement. Benchmark Participant and KENSHO agree to resolve disputes arising from or related to this Agreement and use of the Beta Project in the state or federal courts in the Borough of Manhattan, New York, New York. Benchmark Participant and Kensho agree to submit to the jurisdiction of, and agree that venue is proper, in these courts in any such legal action or proceeding.
11. Publicity. Except as otherwise expressly stated in this Agreement, each party agrees not to use or refer to this Agreement or its terms in any advertising, publicity, or other marketing activities without the express written approval of the other party.
12. Notices. All legal notices to KENSHO under this Agreement shall be in writing to Kensho Technologies, LLC, 44 Brattle Street, Third Floor, Cambridge, MA 02138, Attn: Legal Department-Notices and delivered by hand or sent by reputable courier service or registered or certified mail, return receipt requested.
13. Miscellaneous. This Agreement does not create a partnership, agency relationship, or joint venture between the parties. Any assignment of this Agreement by Benchmark Participant in whole or in part without KENSHO’s prior written consent will be null and void, except an assignment to a successor that is not a competitor of KENSHO made in connection with a merger or sale of all or substantially all of Benchmark Participant’s assets or stock. If any provision of this Agreement is unenforceable, that provision will be modified to render it enforceable to the extent possible to affect the parties’ intention and the remaining provisions will not be affected. Failure of KENSHO to enforce a right under this Agreement shall not act as a waiver of that right or the ability to later assert that right relative to the particular situation involved. This Agreement is the entire agreement between Benchmark Participant and KENSHO with respect to the subject matter hereof and supersede all prior representations, agreements and understandings, written or oral. This Agreement may only be altered, amended, or modified by duly executed written instrument. This Agreement shall be binding upon the parties and their respective administrators, successors, and assigns.