DrivenData Benchmark Rules

---

GUIDELINES

These guidelines are intended to summarize a few main features of the benchmark rules. Full details are provided in the benchmark rules below.

One account per participant

You cannot sign up to DrivenData from multiple accounts and therefore you cannot submit from multiple accounts.

Submission limits

The rate of allowed submissions is restricted to a fixed value on a per-benchmark basis. Any attempt to circumvent this limit will result in submission removal and account ban.

External Data Rule: External Data with Rights

External data is permitted in this Benchmark provided Participants have all rights, licenses and permissions to use the external data as contemplated in the Benchmark Rules, and that such data complies with any external data restrictions stated on the Benchmark Website. Any use of external data must be pursuant to and in accordance with a valid license that permits use of the external data in this Benchmark without the need for additional approvals from any third party.

Closed-source code and models

Closed-source code and models are allowed.

Cheating

Participating using more than one DrivenData account is deemed cheating and, if discovered, will result in removal from the Benchmark and may result in banning or deactivation of affected DrivenData accounts.

Questions?

The benchmark rules are in place to promote fair benchmarking and useful solutions. If you are ever unsure whether your solution meets the benchmark rules, ask the organizers in the benchmark forum or send an email to info@drivendata.org.

---

FULL RULES

NO PURCHASE NECESSARY TO PARTICIPATE IN THIS BENCHMARK.

BY PARTICIPATING IN THIS BENCHMARK, PARTICIPANT AGREES TO THESE BENCHMARK RULES, WHICH ARE A CONTRACT, SO READ THEM CAREFULLY BEFORE PARTICIPATING.

Definitions

These are the complete, official rules (the "Benchmark Rules") for the benchmark (the "Benchmark") described on the website listed above or linking to these Benchmark Rules (the "Benchmark Website"). The Benchmark Rules incorporate by reference the contents of the Benchmark Website listed above.

By downloading a dataset linked from the Benchmark Website, submitting an entry to this Benchmark (a "Submission"), or joining a Team in this Benchmark, you are agreeing to be bound by these Benchmark Rules which constitute a binding agreement between you and DrivenData and, if applicable, any rules and restrictions that may be imposed by the third-party sponsor of this benchmark (the "Benchmark Sponsor").

"Data" means the data or datasets linked from the Benchmark Website for the purpose of use by Participants in the Benchmark. For the avoidance of doubt, Data includes any prototype or executable code provided to Participants by DrivenData or via the Benchmark Website.

"Submission" means any code, model artifacts, documentation, or other materials submitted to the Benchmark Website for evaluation.

Benchmark host and period

The Benchmark is hosted by DrivenData, LLC ("DrivenData"). The Benchmark will run according to the dates listed on the Benchmark Website (the "Benchmark Period").

Participation

The Benchmark is open to all participants, subject to applicable laws and any restrictions listed on the Benchmark Website. You are solely responsible for complying with your employer's policies regarding participation. Participants (including each member of a Team) must not be under any legal or contractual obligation that would prohibit their participation in this Benchmark as described in these Benchmark Rules. If participating in this Benchmark would result in a violation by Participant of any law applicable to them or any agreement to which they are a party or employer or company policies and procedures, such Participant is ineligible. If you are participating within the scope of your employment, as an employee, contractor, or agent of a legal entity, you warrant that such party has full knowledge of your actions and has consented thereto.

BY PARTICIPATING IN THE BENCHMARK YOU ACCEPT THE CONDITIONS STATED IN THESE BENCHMARK RULES INCLUDING THE RULES ON THE BENCHMARK WEBSITE AND WARRANT THAT YOU ARE ELIGIBLE TO PARTICIPATE IN THE BENCHMARK. IF YOU DO NOT ACCEPT ALL OF THESE BENCHMARK RULES, THEN PLEASE DO NOT REGISTER FOR THE BENCHMARK. WE RECOMMEND THAT YOU PRINT OUT A COPY OF THESE BENCHMARK RULES FOR YOUR FUTURE REFERENCE.

Registration and teams

To participate, register through the Benchmark Website within the Benchmark Period. You may participate individually or as part of a team. Each participant must have a single, unique DrivenData account and may not participate on more than one team.

Each registered individual or Team is referred to as a Participant. You may only participate using a single, unique DrivenData account registered at http://www.drivendata.org. Participating using more than one DrivenData account per individual is a breach of these Benchmark Rules and DrivenData reserves the right to remove any individual (or Team including an individual) who is found to breach these Benchmark Rules.

Multiple individuals may collaborate as a Team. You may not participate on more than one Team. Each Team member must be a single individual operating a separate DrivenData account. You must register individually for the Benchmark before joining a Team. You must confirm your Team membership to make it official by responding to the Team notification message which will be sent to your Account.

Submissions

Each Submission must be uploaded to the Benchmark Website in the manner and format specified on the Benchmark Website. All Submissions must be received during the Benchmark Period.

Submissions must comply with all applicable laws and regulations and must not include content that infringes third-party intellectual property rights.

USER SUBMISSIONS MUST IN THEIR ENTIRETY COMPLY WITH ALL APPLICABLE FEDERAL, STATE, LOCAL, AND INTERNATIONAL LAWS AND REGULATIONS. WITHOUT LIMITING THE FOREGOING, IN ORDER TO BE ELIGIBLE, YOU WARRANT THAT YOUR SUBMISSION FOR THIS BENCHMARK MEETS THE FOLLOWING SUBMISSION CRITERIA, ALL AS DETERMINED IN THE SOLE DISCRETION OF BENCHMARK SPONSOR AND/OR DRIVENDATA:

• Does not include or anticipate the inclusion of any content that is in violation of or infringes third party intellectual property rights including, but not limited to copyrights, including music copyrights, patents, trade secrets, trademarks, and rights of publicity or privacy.
• Is free and clear of all liens, claims, encumbrances or demands of any third parties.
• Does not include or anticipate the inclusion of any unsuitable or offensive content, including nudity, sexually explicit, disparaging, libelous or other inappropriate content.
• Is suitable for a general audience.
• Does not contain any claims that are not and cannot be substantiated or that would be false and/or misleading to a reasonable consumer.

Participants retain all intellectual property rights in their Submissions. By submitting, you grant DrivenData a limited, non-exclusive license to run, evaluate, and display your Submission for the administration of the Benchmark and for public benchmarking results (e.g., leaderboard display).

Evaluation and results

The evaluation metric used for scoring and ranking Submissions will be displayed on the Benchmark Website. Scores displayed on the public leaderboard while the benchmark is running may or may not be the same as the final scores on the private leaderboard, depending on how samples from the Data are used for evaluation.

Data use and code sharing

DATA. Participants must use the Data only as permitted by these Benchmark Rules and any associated data use rules specified on the Benchmark Website. Participants are responsible for ensuring that any external data, code, or models they use comply with the applicable licenses and terms.

Unless otherwise permitted by the terms of the Benchmark Website, Participants must use the Data solely for the purpose and duration of the Benchmark, including but not limited to reading and learning from the Data, analyzing the Data, modifying the Data and generally preparing your Submission and any underlying models and participating in forum discussions on the Benchmark Website. If the Data includes information from individuals, Participants explicitly agree to make no attempt to utilize the Data to identify specific individuals, and will immediately delete relevant Data and notify DrivenData in the event of inadvertent discovery of the identity of any person represented in the Data during the Benchmark. Participants agree to use suitable measures to prevent persons who have not formally agreed to these Benchmark Rules from gaining access to the Data and agree not to transmit, duplicate, publish, redistribute or otherwise provide or make available the Data to any party not participating in the Benchmark. Participants agree to notify DrivenData immediately upon learning of any possible unauthorized transmission or unauthorized access of the Data and agree to work with DrivenData to rectify any unauthorized transmission. Participants agree that participation in the Benchmark shall not be construed as having or being granted a license (expressly, by implication, estoppel, or otherwise) under, or any right of ownership in, any of the Data.

EXTERNAL DATA. Participants’ use of external data is subject to the External Data Rule specified in the Benchmark Guidelines and any rules on external data expressly stated on the Benchmark Website (note: if no External Data Rule is specified above, the External Data Rule is deemed to be No External Data).

• NO EXTERNAL DATA. Unless otherwise expressly stated on the Benchmark Website, Participants must not use data other than the Data to develop and test their models and Submissions.

• OPEN EXTERNAL DATA. Participants may use external data other than the Data, including pre-trained models, to develop and test their solutions, provided that they comply with any external data restrictions stated on the Benchmark Website and the external data is freely and publicly available to all Participants under a permissive open license that does not prohibit commercial use (e.g., CC0 or CC-BY).

• EXTERNAL DATA WITH RIGHTS. Participants may use external data other than the Data, including pre-trained models, to develop and test their solutions, provided that the Participant has all rights, licenses and permissions to use the external data as contemplated in these Benchmark Rules. Participant represents and warrants that any use of any external data in connection with participation in this Benchmark is pursuant to and in accordance with a valid license that permit use of the external data in this Benchmark without the need for additional approvals from any third party, and agrees to provide DrivenData with proof (to DrivenData’s satisfaction) of such license upon request at any time during the Benchmark or evaluation process. Failure to satisfy any of these requirements in accordance with DrivenData’s instructions and timing, or DrivenData’s reasonable belief, as determined in DrivenData’s sole discretion, that a Participant does not have all necessary consents and licenses (notwithstanding any certification by Participant) will result in removal from the Benchmark.

Furthermore, unless otherwise expressly stated on the Benchmark Website:

• Participants may annotate provided training data as long as they are included with solutions to enable reproduction and do not overfit to the test set;
• Participants may not add any manual annotations to the provided test data. Eligible solutions need to be able to run on test samples automatically using the test data as provided.
• Participants may use test data for unsupervised learning. Participants may not access, reconstruct, or disclose the test data in any form. This includes (but is not limited to) logging, exporting, copying, or otherwise attempting to view or exfiltrate the test data. Any attempt to access or disclose the test data will result in immediate submission removal and account ban.

Benchmark Sponsor reserves the right in its sole discretion to remove any Participant who Benchmark Sponsor discovers has undertaken or attempted to undertake the use of data other than the Data, or who uses the Data other than as permitted according to the Benchmark Website and in these Benchmark Rules, in the course of the Benchmark.

SNOMED terms of use

SNOMED Clinical Terms® (SNOMED CT®) are used by permission of the SNOMED International. All rights reserved. SNOMED CT® was originally created by the College of American Pathologists. "SNOMED", "SNOMED CT" and "SNOMED Clinical Terms" are registered trademarks of SNOMED International (www.snomed.org). SNOMED CT has been created by combining SNOMED RT and a computer based nomenclature and classification known as Clinical Terms Version 3, formerly known as Read Codes Version 3, which was created on behalf of the UK Department of Health and is Crown copyright.

Use of SNOMED CT in the SNOMED CT Entity Linking Benchmark is governed by the conditions of the following SNOMED CT license issued by SNOMED International:

1. The meaning of the terms "Affiliate", or "Data Analysis System", "Data Creation System", "Derivative", "End User", "Extension", "Member", "Non-Member Territory", "SNOMED CT" and "SNOMED CT Content" are as defined in the SNOMED International Affiliate License Agreement.
2. Information about Affiliate Licensing is available at https://www.snomed.org/snomed-ct/get-snomed. Individuals or organizations wishing to register as SNOMED International Affiliates can register at mlds.ihtsdotools.org, subject to acceptance of the Affiliate License Agreement.
3. The current list of SNOMED International Member Territories can be viewed at snomed.org/members. Countries not included in that list are "Non-Member Territories".
4. Benchmark participants who do not hold an SNOMED International Affiliate License may access SNOMED CT® as part of, for the purpose of participating in, and only during the SNOMED CT Entity Linking Benchmark.

Participant warranties and obligations

By registering, you agree that (a) your Account is complete, correct and accurate and (b) your registration may be rejected or terminated and all Submissions submitted by you and/or your Team may be removed if any of the information in your Account is (or DrivenData has reasonable grounds to believe it is) incomplete, incorrect or inaccurate. You are solely responsible for your Account. All registration information is deemed collected in the United States.

Participation is subject to all applicable federal, state, provincial, territorial and local laws, rules and regulations. You agree to comply with all applicable laws and regulations at all times during your participation in the Benchmark. Void where prohibited or restricted by law. You are responsible for checking applicable laws and regulations in your jurisdiction before participating in the Benchmark to make sure that your participation is legal. You are responsible for abiding by your employer's policies regarding participation in the Benchmark. DrivenData disclaims any and all liability or responsibility for disputes arising between you and your employer related to this Benchmark.

Each Participant is solely responsible for all equipment, including but not necessarily limited to a computer and internet connection necessary to access the Benchmark Website and to develop and upload any Submission, and any telephone, data, hosting or other service fees associated with such access, as well as all costs incurred by or on behalf of the Participant in participating in the Benchmark.

By entering a Submission, you represent and warrant that all information you enter on the Benchmark Website is true and complete to the best of your knowledge, that you have the right and authority to make the Submission (including any underlying code and model) on your own behalf or on behalf of the persons and entities that you specify within the Submission, and that your Submission:

• complies with all applicable federal, state, provincial, territorial, local and international laws, statutes, ordinances, rules and regulations;
• is your own original work, or is used by permission, in which case full and proper credit and identity is given and the third party contributions are clearly identified within your Submission;
• does not violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights of privacy, publicity or other intellectual property or other rights of any person or entity;
• does not contain malicious code, such as viruses, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information; and
• does not and will not violate any applicable law, statute, ordinance, rule or regulation.

A breach of any of these warranties will result in the corresponding Submission being invalid.

Cheating

Participating using more than one DrivenData account is deemed cheating and, if discovered, will result in removal from the Benchmark and any other affected Benchmarks and may result in banning or deactivation of affected DrivenData accounts.

DrivenData reserves the right to request information associated with our investigation of suspected cheating. Failure to respond to these requests (including failure to furnish the requested information) within ten (10) days is grounds for removal.

Delivery and receipt of entries

DrivenData is not responsible for (a) late, lost, stolen, damaged, garbled, incomplete, incorrect or misdirected Submissions or other communications, (b) errors, omissions, interruptions, deletions, defects, or delays in operations or transmission of information, in each case whether arising by way of technical or other failures or malfunctions of computer hardware, software, communications devices, or transmission lines, or (c) data corruption, theft, destruction, unauthorized access to or alteration of Submission materials, loss or otherwise. DrivenData is not responsible for electronic communications or emails which are undeliverable as a result of any form of active or passive filtering of any kind, or insufficient space in any email account to receive email messages. DrivenData disclaims any liability for damage to any computer system resulting from participation in, or accessing or downloading information in connection with, the Benchmark.

Reservation of rights

DrivenData, in its sole discretion, reserves the right to remove any Submission or Participant tampering with the submission process, the operation of the Benchmark Website, the benchmarking process, or who is otherwise in violation of the Benchmark Rules, including attempts to access or disclose test data. DrivenData reserves the right at any time to remove a Submission from the Benchmark where, acting in good faith, it believes there are reasonable grounds to warrant removal. For example, DrivenData's determination that the Submission does not provide the functionality described or required, or the Submission appears to be purposely designed to circumvent these Benchmark Rules or the spirit of the Benchmark would be grounds for removal. DrivenData reserves the right to cancel, terminate or modify the Benchmark if it is not capable of completion as planned for any reason, including infection by computer virus, bugs, tampering, unauthorized intervention or technical failures of any sort. If DrivenData determines (at any time and in their sole discretion) that any Participant is engaging in behavior that DrivenData deems obnoxious, inappropriate, threatening, illegal or that is intended to annoy, abuse, or harass any other person or entity, DrivenData reserves the right to remove such Participant (and Participant's Team, if applicable).

General release

By participating in the Benchmark, you release and discharge any Benchmark Sponsor and DrivenData from any liability whatsoever in connection with the Benchmark, including, without limitation, legal claims, costs, injuries, losses or damages, demands or actions of any kind (including, without limitation: personal injuries; death; damage to, loss or destruction of property; rights of publicity or privacy; and defamation or portrayal in a false light). Any Benchmark Sponsor and DrivenData will not be responsible for typographical, printing or other inadvertent errors in these Benchmark Rules or in other materials relating to the Benchmark. Additionally, you hereby agree to indemnify any Benchmark Sponsor and DrivenData from any and all losses, damages, costs, expenses, rights, claims, demands and actions (including attorney's fees and expenses for litigation and settlement), which may be brought against any one or more of them by anyone claiming to have suffered loss or damage as a result of your participation in the Benchmark or for any breach of any representation or warranty herein.

Limitations of liability

By participating in the Benchmark, each Participant agrees that: (1) any and all disputes, claims, and causes of action arising out of or in connection with the Benchmark shall be resolved individually without resort to any form of class action; (2) any claims, judgments and awards shall be limited to actual out-of-pocket costs incurred, including costs associated with participating in the Benchmark, but in no event attorney's fees; and (3) under no circumstances will a Participant be permitted to obtain any award for, and each Participant hereby waives all rights to claim, punitive, incidental or consequential damages and any and all rights to have damages multiplied or otherwise increased and any other damages, other than damages for actual out-of-pocket expenses.

DrivenData, LLC, any Benchmark Sponsor, and the employees, officers, directors, shareholders, agents, representatives, affiliates, subsidiaries, and advertising, promotional and legal advisors thereof (collectively, the "Released Parties") are not responsible and shall not be liable for: (i) late, lost, delayed, damaged, misdirected, incomplete, illegible, or unintelligible entries; (ii) telephone, electronic, hardware, or software program, network, Internet, or computer malfunctions, failures, or difficulties; (iii) errors in transmission; (iv) any condition caused by events beyond the control of the Released Parties that may cause the Benchmark to be disrupted or corrupted; (v) any injuries, losses, or damages of any kind resulting from participation in the Benchmark (however (but only if required by law in your jurisdiction), this release and hold harmless commitment does not apply to cases of bodily injury or loss of life to the extent that any death or personal injury is caused by the negligence of DrivenData, any Benchmark Sponsor, or other third parties, where liability to the injured party cannot be excluded by law); or (vi) any printing or typographical errors in any materials associated with the Benchmark.

Disclaimer of Warranties

You understand that we cannot and do not guarantee or warrant that the Data or other files available for downloading from the internet or the Benchmark Website will be free of viruses or other destructive code. You are responsible for implementing sufficient procedures and checkpoints to satisfy your particular requirements for anti-virus protection and accuracy of data input and output, and for maintaining a means external to our site for any reconstruction of any lost data. RELEASED PARTIES WILL NOT BE LIABLE FOR ANY LOSS OR DAMAGE CAUSED BY A DISTRIBUTED DENIAL-OF-SERVICE ATTACK, VIRUSES OR OTHER TECHNOLOGICALLY HARMFUL MATERIAL THAT MAY INFECT YOUR COMPUTER EQUIPMENT, COMPUTER PROGRAMS, DATA OR OTHER PROPRIETARY MATERIAL DUE TO YOUR USE OF THE WEBSITE OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE OR TO YOUR DOWNLOADING OF ANY MATERIAL POSTED ON IT, OR ON ANY WEBSITE LINKED TO IT.

YOUR USE OF THE BENCHMARK WEBSITE, ITS CONTENT AND ANY SERVICES OR ITEMS OBTAINED THROUGH THE BENCHMARK WEBSITE ARE PROVIDED ON AN "AS IS" AND "AS AVAILABLE" BASIS, WITHOUT ANY WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, BEYOND ANY NON-EXCLUDABLE GUARANTEES (DEFINED BELOW). NEITHER THE RELEASED PARTIES NOR ANY PERSON ASSOCIATED WITH RELEASED PARTIES MAKES ANY WARRANTY OR REPRESENTATION WITH RESPECT TO THE COMPLETENESS, SECURITY, RELIABILITY, QUALITY, ACCURACY OR AVAILABILITY OF THE BENCHMARK WEBSITE. WITHOUT LIMITING THE FOREGOING, NEITHER RELEASED PARTIES NOR ANYONE ASSOCIATED WITH THE RELEASED PARTIES REPRESENTS OR WARRANTS THAT THE BENCHMARK WEBSITE, ITS CONTENT OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE WILL BE ACCURATE, RELIABLE, ERROR-FREE OR UNINTERRUPTED, THAT DEFECTS WILL BE CORRECTED, THAT OUR SITE OR THE SERVER THAT MAKES IT AVAILABLE ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS OR THAT THE BENCHMARK WEBSITE OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE WILL OTHERWISE MEET YOUR NEEDS OR EXPECTATIONS.

NOTHING IN THESE BENCHMARK RULES LIMITS, EXCLUDES, OR MODIFIES OR PURPORTS TO LIMIT, EXCLUDE, OR MODIFY ANY STATUTORY CONSUMER GUARANTEE OR ANY IMPLIED CONDITION OR WARRANTY, THE EXCLUSION OF WHICH FROM THESE BENCHMARK RULES WOULD CONTRAVENE ANY STATUTE OR CAUSE ANY PART OF THESE TERMS TO BE VOID ("NON-EXCLUDABLE GUARANTEES"). SUBJECT TO THE LIMITATIONS IN THE PRECEDING SENTENCE AND TO THE MAXIMUM EXTENT PERMITTED BY ANY MANDATORY PROVISIONS OF APPLICABLE LAW, THE RELEASED PARTIES HEREBY DISCLAIM ALL WARRANTIES OF ANY KIND, WHETHER EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF MERCHANTABILITY, NON-INFRINGEMENT AND FITNESS FOR PARTICULAR PURPOSE.

THE FOREGOING DOES NOT AFFECT ANY WARRANTIES WHICH CANNOT BE EXCLUDED OR LIMITED UNDER APPLICABLE LAW.

Governing Law and Jurisdiction

DrivenData shall be the sole interpreter of these Benchmark Rules.

All matters relating to the Benchmark Rules or the Benchmark and any dispute or claim arising therefrom or related thereto (in each case, including non-contractual disputes or claims), shall be governed by and construed in accordance with the internal laws of the Commonwealth of Massachusetts without giving effect to any choice or conflict of law provision or rule (whether of the Commonwealth of Massachusetts or any other jurisdiction).

Any legal suit, action or proceeding arising out of, or related to, the Benchmark or the Benchmark Website shall be instituted exclusively in the federal courts of the United States or the courts of the Commonwealth of Massachusetts although we retain the right to bring any suit, action or proceeding against you for breach of these Benchmark Rules in your country of residence or any other relevant country. You waive any and all objections to the exercise of jurisdiction over you by such courts and to venue in such courts.

The invalidity or unenforceability of any provision of these Benchmark Rules shall not affect the validity or enforceability of any other provision. In the event that any provision is determined to be invalid or otherwise unenforceable or illegal, these Benchmark Rules shall otherwise remain in effect and be construed in accordance with their terms as if the invalid or illegal provision was not contained herein.