Relantic, a company in Columbus, Ohio, posted RealityBench to arXiv on October 9, 2026, and revised it on October 14. It is a benchmark, a fixed test scored the same way for every model, that checks whether vision-language models (AI models that answer questions about images) can review the photos crews take to prove work was done. Seven of its eleven scenes come from rooftop solar installation and four from vehicle repair intake. Across 26 models, the best read the tape measure at a solar module frame within 1/8 inch on 36% of photos.
What it does
Each item is one photo and one prompt. The model first judges acceptability, meaning whether the photo meets the stated requirement well enough to serve as proof of work. It then answers a documentation question about what the photo shows, such as the number on the tape or whether a lag bolt is present.
The photos are rendered in Blender, a 3D modeling program, through a simulated phone camera, and 30% of frames have missed focus or hand shake. Because every scene is built, the true answer is always known. The solar scenes follow an install in order:
- A close-up of one L-foot showing its flashing and lag bolt.
- The low point of the wire under the modules.
- A tape measure standing from the roof to the module frame.
- A digital angle gauge on the module.
- Equipment nameplates, the skyline from the ridge, and the panel’s circuit directory card.
That list matches the photo checklist a solar or electrical subcontractor’s QA reviewer works through, and the closeout photos a GC’s project engineer collects.
Evidence so far
The evidence is the authors’ paper and their leaderboard on Hugging Face. Relantic built the scenes, wrote the scoring rules, ran the models at list API prices, and supplied the human raters (four authors and two colleagues). Results from an independent rerun are not yet public. The paper states that the benchmark measures third-party models and no Relantic product.
| Model | Run | Acceptability F1 | Documentation score | Cost for all 1378 items (USD) |
|---|---|---|---|---|
| GPT-6 Astra | Paper | 0.61 | 0.83 | 30.05 |
| Gemini 4 Argon | October | 0.60 | 0.86 | 4.96 |
| GPT-6.1 Sol | October | 0.59 | 0.82 | 6.22 |
| Claude Opus 5.5 | Paper | 0.58 | 0.83 | 16.36 |
| GLM-5.3 Flash | Paper | 0.57 | 0.75 | 1.30 |
| Gemini 3.8 Flash | Paper | 0.54 | 0.84 | 7.16 |
| Gemma 4 31B | Paper | 0.52 | 0.79 | 0.70 |
| GPT-6 Luna | Paper | 0.43 | 0.62 | 0.69 |
| Single human rater | Paper | 0.59 | ||
| Always answer "adequate" | Paper | 0.33 |
The acceptability column uses F1, a score from 0 to 1 that penalizes both bad photos a model accepts and good photos it rejects. Answering “adequate” every time scores 0.33. The best models land near 0.6, about where a single non-author rater scored (0.59) on a separate set of rated frames. Raters disagree with each other, too. An individual rater matched the others’ majority on 76% to 92% of frames.
Documentation splits by skill. The authors report that eight of ten fields asking a model to read text, such as nameplates and directory cards, scored 0.90 to 1.00 averaged across models. Questions about small parts, spatial relations and measurements scored lower.
| Question about a solar install photo | Mean of 26 models (%) | Best model (%) |
|---|---|---|
| Is the lag bolt present | 67% | 82% |
| Does any wire touch the roof | 57% | 74% |
| Is the tape leaning | 55% | 82% |
| True module-to-roof gap within 1/4 in | 23% | 48% |
| Tape reading at the module frame within 1/8 in | 11% | 36% |
On every acceptable tape frame the tape stood within 5 degrees of upright, yet the models called it tilted 75% of the time. In that task, 97% of the models’ wrong rejections came from misreading the scene, while the models applied the requirement consistently to what they reported seeing.
The authors also tested real photos, including 133 tape readings on 84 photos of a tape held against everyday objects, which they released. GPT-6 Astra read 44% of those within 1/8 inch and Claude Opus 5.5 read 35%. Model rankings carried over from rendered to real photos on all three questions tested, with Spearman rank correlations (a measure of how closely two rankings agree, where 1 means identical) of 0.63 to 0.79.
The paper priced one human pass over the 1,378 photos at $11 to $165, using $10 an hour as a low bound and $35.91 an hour from BLS wage data for construction and building inspectors as the high end. Claude Opus 5.5 cost $16.36 for the same set.
How to measure it on your projects
Anyway, your own install photos are the better test. If you pilot a model as a first-pass photo reviewer, track the photo retake rate: the share of required photos in a job’s closeout packet that the reviewer sends back for a reshoot.
Capture the baseline before the pilot:
- Pull the photo packets from three to five recent jobs and count required photos, photos sent back, and reviewer minutes per packet.
- Run the same packets through the model and record its verdict on each photo.
- Count photos the model accepted that your reviewer rejected, and the reverse, as separate numbers.
Leave tape readings and gauge values with the reviewer during the pilot, since measurement scored lowest. A first step on one job is to take one closeout packet from a recent solar or electrical job and compare the model’s accept and reject calls with your reviewer’s, photo by photo. ConSynth-X is another test built on synthetic jobsite images.
Availability and cost
The rendered benchmark is a free download on Hugging Face after you agree to share contact information. The frames, ground truth and model results carry a CC BY 4.0 license and the scoring code an MIT license, and the scripts run against any OpenAI-compatible endpoint. Across the 28 priced models on the leaderboard, one run of the full set cost from $0.56 to $45.99 at list prices. The simulated scenes themselves, which can generate new tasks, are available by contacting Relantic.