Contribute to Tamil Bench
தமிழ் பெஞ்சைக்கு பங்களியுங்கள்

What the benchmark has cost so far
இதுவரை செலவானதன் மொத்தம்

Every OpenRouter charge this project can account for, summed from the per-answer usage.cost field across all 32 answer sheets that recorded one. No salaries, no servers, no paid datasets — just API calls.

Of that, $1.5517 is the runs standing on the leaderboard and $0.0561 is off-board work — n=5 smoke tests, superseded re-runs and partial sheets that were still billed but never reached the board. A further 24 paid-model sheets covering 3,024 API calls record no charge at all; they were certainly billed, but the sheets do not say by how much, so this figure is a floor, not a closed total. 14 free-tier :free sheets are excluded because they genuinely cost nothing.

8 of 18 board models reported a charge, but only 4 of those were actually billed -- the other 4 recorded $0.00 (3 of them were free-tier `:free` endpoints, which are $0.00 by definition). Machines marked *free* cost nothing to run; the bill is made up of the commercial endpoints.

Board coverage: 8 of 18 models on the board reported a charge; 29 qualifying sheets predate usage accounting and record none. Across the whole results/ directory the accounted total is $1.6078, with 24 further paid-model sheets unaccounted. Check the raw sheets · விடைத்தாள்களைப் பார்க்கவும்.

Per-model spend
மாதிரி வாரியான செலவு

Contribute money · UPI
UPI மூலம் பங்களிப்பு

Scan with any UPI app (GPay, PhonePe, Paytm, BHIM) and enter whatever amount you want. Every rupee goes toward API calls that keep new models on the board. There is no minimum and no subscription — a one-time amount is fine.

or type the ID · அல்லது ID-ஐத் தட்டச்சு செய்யவும்

logic@ybl

A one-tap UPI link: upi://pay?pa=logic@ybl&pn=Cashless…

Contribute code
கோடு பங்களிப்பு

The whole benchmark is three Python scripts and a folder of answer sheets, MIT licensed. CONTRIBUTING.md is the full guide; this is the short version of what is actually useful.

• Add a task runner. Each test is one run_<task>() function in bench.py plus a subcommand in main(). This is the main way in.
• Fix or extend scoring. IndicQA uses exact match and token F1; MILU and IndicXNLI use accuracy over valid rows. A better Tamil-aware normaliser or a confidence interval that reflects the real sampling is a genuine improvement.
• Handle the awkward cases. Models that answer in Tamil script, in English, in transliteration, or in a mix of all three. A model that fails on format rather than knowledge is a measurement bug, and the fix belongs here.
• Report a broken run. A dead endpoint, a sheet with fewer rows than expected, a score that looks wrong. Open an issue — no run needed to do this.

Small, reviewable pull requests are easier to land than large ones. If you are unsure whether something is wanted, open the issue first — that costs you a paragraph and saves you a wasted afternoon.

Add a new test
புதிய சோதனை சேர்க்கவும்

This is the contribution with the most leverage: every test added widens what the board can detect. A task qualifies if it is Tamil-specific, has gold labels that are not guessable from the prompt alone, and is not already covered by MILU, IndicQA or IndicXNLI.

1. Propose it first. Open a new-test issue with the dataset source, licence, language, size, and what a good score means. Datasets that cannot be redistributed belong in data/, which is gitignored — do not commit a copy.
2. Write the runner. A run_<task>() function in bench.py that samples with SEED = 42, calls the model at temperature 0, and writes results/<task>_<org>_<model-slug>_n<N>.jsonl (/ becomes _ in the model id). Register the subcommand in main().
3. Gate it. Add a FULL_MIN entry in build_site.py so a smoke-test sheet can never reach the leaderboard. MILU needs 150 rows, IndicQA 90, IndicXNLI 150; pick a floor at least as large as your real task and say why in the PR.
4. Label it. Add entries to TASK_LABELS (English) and TASK_LABELS_TA (Tamil), and register the page in generate_test_pages() so it gets its own sortable scoreboard.
5. Show it works. Run at least two models end to end and commit the two sheets. Scores computed from sheets nobody can re-derive are not evidence.

Three rules are not negotiable, because breaking any of them makes the board quietly wrong rather than obviously broken: sampling stays at seed 42 so models stay comparable; failed API calls are never scored as 0% — they are counted and reported as n_errors; and result sheets are append-only evidence, never edited to fix a number.

Request a run for any model
ஏதேனும் மாதிரிக்கான ஓட்டைக் கோரவும்

Name any OpenRouter model id (or a free/open endpoint you want compared) and which exams to run. Runs cost roughly a few dollars each for frontier models and nothing for :free endpoints; a request does not commit you to anything. If nobody funds it, the run simply goes to the back of the queue.