- Test users
- 980 states from 720 new users
- Scored questions
- 2,156 of 2,506
- Families
- 7
- Splits
- Data is divided into training, validation, development and test sets.
Ranking
Swipe horizontally to compare all scores.
| Rank | Model | Family-macro accuracy ↑ | TEST-A ↑ | TEST-B ↑ | Brier ↓ | ECE ↓ |
|---|---|---|---|---|---|---|
| 1 | Pev-27BFine-tuned (LoRA on Qwen3.8-27B) | 0.915 | 0.913 | 0.917 | 0.125 | 0.065 |
| 2 | gpt-6-astraAPI model | 0.873 | 0.876 | 0.871 | 0.180 | 0.074 |
| 3 | Kev-27BThird-party decision model | 0.823 | 0.825 | 0.821 | 0.218 | 0.074 |
| 4 | JevThird-party decision model (TypeSafe) | 0.788 | 0.779 | 0.797 | 0.266 | 0.123 |
| 5 | Qwen3.8-27BOpen-weight base | 0.762 | 0.759 | 0.766 | 0.264 | 0.067 |
| 6 | Qwen3.5-4BOpen-weight base | 0.652 | 0.627 | 0.677 | 0.389 | 0.074 |
Research
Method and breakdowns
The research page explains how the scores are produced and breaks them down.
Submit
Add a model
Send frozen TEST predictions, or a model or endpoint we can run, to research@envloop.ai. We score submissions with the same verifier and add them with their provenance. Do not train on TEST.