Top AI model matches building drawings to photos 83.9% of the time in the ARCH-B benchmark

Every question has a short instruction, one or more reference images, and four answer images with one correct choice, so a model that guesses scores 25%. The 354 questions fall into 11 task types. Most ask for the image that matches the references, in either direction between drawings and photos. Two task types ask for the one image that belongs to a different building.

“Choose the photograph that corresponds to the building shown in this/these floorplan/s.”

A typical ARCH-B instruction

The images come from real-estate listings on Zillow and Realtor.com and from project pages on ArchDaily and Dezeen, collected daily from November 2024 to October 2025. That gave the authors 3.9 million images, 58,000 of them floor plans, grouped by the listing or project they came from. The grouping is what makes an answer checkable, because every image in a group shows the same building.

The wrong answers are deliberately similar to the right one. For each question, the authors picked the three images from other buildings that an image model rated most similar to the correct one, so each decoy is a similar-looking building or drawing. They then cut the pool down in three steps:

1. Generate 2,200 candidate questions, 200 per task type.
2. Run them through four screening models (Claude Sonnet 4, Gemini 2.5 Pro, Pixtral Large and GPT-4.1) and keep the 1,013 that two or fewer of the four answered correctly.
3. Have two researchers review every remaining question by hand and drop any with duplicate images, mislabeled drawing types, or no single answer a person could defend.

In a contractor’s office, the likely user is a VDC manager or technology lead comparing vendors whose products read drawings. The released runner lets you score a given model on the same 354 questions. The source images are listing photos and published design work, and the set contains no construction document sets or progress photos. The authors name construction documents and images from different stages of construction as possible additions in future versions.

The results come from the authors’ own runs of the 25 models and a crowdsourced human study, all reported in the paper. A replication by another group is not yet public.

The more useful finding is in the task table. Averaged across all 25 models, accuracy was 58.78% when the model had photos and had to pick the floor plan, and 44.14% when it had floor plans and had to pick the photo. People scored about the same in both directions, 34.74% and 36.41%.

The authors read this as models finding it easier to check a plan’s layout against views they can see than to picture how a plan will look once built.

The best model on each task beat the human average by 40.38 to 68.00 percentage points.

That human baseline needs a caveat the authors state themselves. Participants were English-speaking US adults on the Prolific research platform who each answered 10 questions from one task type (they were paid $2 per session whatever their score), and the authors call them a non-expert baseline. They suggest future comparisons with architecture students and practitioners. A superintendent or architect who reads elevations every day would be a different reference point.

The authors also checked whether screening with four models had made the benchmark hard only for those four. They grouped questions by how many screening models got them right and scored the models outside the screening set on each group.

Accuracy rose from 28.65% to 59.02% as screening difficulty fell, which the authors take as evidence that the hard questions are hard in general. Per question, average model accuracy was 53.36%, with a range from 4% to 96%. No question was answered correctly by every model, and none was missed by all of them.

ARCH-B is free. The question set, a JSON schema and the evaluation runner are posted as ancillary files with the arXiv paper, and the README and runner are under the MIT License. The question file links to each image at its original public address with a SHA-256 hash, a fingerprint the runner uses to confirm the image has not changed, and the authors do not redistribute the images themselves.

The runner calls each model provider’s API, the paid connection a program uses to send questions to the model, and the README notes that provider use can incur charges. The authors also host a version of the human-study quiz at archbench.codecolab.org, where anyone can try sample questions.