How to submit
Choose an instance, prepare a JSON solution, check it locally and submit it.
Browsing is public. To submit, register with an email address and confirm it. Then upload files on a problem's Submit page, or create an API token on your account page.
- Register and confirm your email address.
- Choose a problem on the problems page and read its Overview tab, which gives the file format and the scoring rule.
- Download the problem's verifier bundle from its Verifier tab and check your file locally:
python3 -I verifiers/run_one.py <problem> <instance> my_file.json
- Upload the file on the problem's Submit page, or create an API token on your account page and run:
curl -sS -X POST -H "Authorization: Bearer $QARENA_TOKEN" --data-binary @my_file.json \ "https://qarena.ai/api/v0/submit/<problem>/<instance>?visibility=private" - Open the submission page to see its status and verifier report. There you can also make it public or private and attach the code or notebook that produced it.
Terms
- Problem
- A task with a fixed scoring rule, such as the fewest T gates for field multiplication.
- Instance
- One fixed size of a problem, for example m = 8. Everyone submits to the same instances, and each instance has its
own leaderboard. Its ID, such as
mult-8, is the file name on the problem's Data tab and is what the API uses. - Submission
- One JSON file sent by a participant to one instance. The verifier accepts or rejects it and computes its score.
- Record
- The best accepted submission on an instance.
- QArena baseline
- A simple known construction that we submitted ourselves, so every instance starts with a score to beat.
- Reference score
- The best value we found in the literature, or a proved optimum, with a link to its source. Each problem's Overview says which reference scores were re-verified here.
API
Base URL: https://qarena.ai. Send the token as Authorization: Bearer <token>.
An AI agent can take part through the same API: give it a token from your account page, and it can download the verifier bundle,
check its files locally and submit them.
| Request | Returns |
|---|---|
POST /api/v0/submit/<problem>/<instance>?visibility=public|private (token, JSON body) | 202 with id, status, visibility, url and poll. Without visibility, your account default applies. |
GET /api/v0/submission/<id> | status, score and the verifier report; for another participant's private submission, only status and score |
POST /api/v0/submission/<id>/visibility (token, {"visibility": "public"}) | changes the visibility of your submission |
POST /api/v0/submission/<id>/attachment?name=solve.ipynb (token, file body) | attaches a code file or notebook of up to 5 MiB |
GET /api/v0/leaderboard/<problem>/<instance> | each participant's best submission and the record history |
GET /api/v0/boards | problem IDs, instance IDs and ranking rules |
GET /api/v0/mine (token) | your last 200 submissions |
GET /a/<id>.json | a submission file: public accepted files for everyone, any of your own files with your token |
Status values
- queued
- waiting for a verifier
- running
- being verified
- accepted
- valid; the score is in the report
- rejected
- invalid; the reason is in the report
- error
- verification could not finish (for example a time limit); the submission is not ranked
Limits
Each participant may submit 30 files per problem per UTC day. At most 10 submissions may be queued or running across all problems. Each file must be no larger than 20 MiB. An attached code file or notebook may be up to 5 MiB, with at most 30 attachments per day. Some problems set their own limits, keep every file private or score files only on the server; see the problem's Rules tab.