Artificial intelligence news.
Straight from the original announcements.

Property Finder found 96.8% of its AI evaluation labels came from one model

October 27, 2026, 17:53By the AI News Bites Editorial TeamSource: AWS
Property Finder found 96.8% of its AI evaluation labels came from one model
Image: AWS

Property Finder's engineers found that 96.8 percent of the labels used to evaluate their vision AI had been copied from a single model's predictions, not checked by human experts. According to AWS, fixing that bias let the company compare models fairly on Amazon Bedrock and reach higher accuracy at 86 percent lower cost.

Key points
  • Property Finder found 96.8 percent of its evaluation labels were inherited from one model's predictions, not human-verified.
  • The team hand-annotated 356 UAE images with an independent expert and moved to multi-label classification.
  • Prompts were refined over versions v0 to v27, scored mainly with Micro F1 on a 256-image development set.
  • Proprietary model A scored a Micro F1 of 0.55 on that set, while Llama 4 Maverick 17B Instruct scored 0.75 with the v21 prompt.
  • According to AWS, the work achieved higher accuracy at 86 percent lower cost on Amazon Bedrock.

A biased answer key hid better options

Property Finder, a digital real estate application, processes tens of thousands of property images every day across the Middle East and North Africa. Its system scores 11 quality attributes per image, including watermark detection, room classification, furnishing status, photo quality and construction indicators. According to AWS, thousands of new listings arrive with an average of 15 images each, and all need quality assessment before going live.

The team was looking for a more cost-effective option when it hit a problem: measured against its existing labels, every alternative looked worse. The cause was the labels themselves. Of the labeled samples, 96.8 percent came straight from the existing vision language model, not from independent human verification. Any alternative was therefore judged on how closely it imitated the incumbent, not on whether it was correct.

The labels also contained errors. Watermark text had character-level mistakes marked as correct, including hallucinated brand names and missing characters. Photo classification forced a single label onto images showing several room types. In the remaining 3.2 percent of human-labeled samples, annotators had added categories outside the specification, such as Office.

Rebuilding ground truth with independent experts

Property Finder's team hand-annotated 356 UAE images, selected from unlabeled production listings across representative industry segments. A separate set of Egypt images served as a cross-regional check on 230 images, but AWS does not report those results. The first 256 UAE images formed the development set for tuning, and the final model comparison uses all 356, including 100 images annotated after the development runs.

A strategy and analytics expert with knowledge of listing quality standards annotated every image, with no access to earlier model outputs. The AWS Generative AI Innovation Center team ran clarification sessions to settle ambiguous cases and keep labeling consistent. AWS notes that 356 images is roughly enough to detect moderate differences between models, while rare classes have fewer examples, so smaller differences on those attributes may go unnoticed.

The team also moved from single-label to multi-label classification. A hallway photo that shows a bedroom through a doorway is genuinely both, and multi-label scoring stops penalizing models for reasonable secondary predictions.

  • Micro F1 was the primary metric, because it weights each prediction equally.
  • Macro F1 was reported alongside it, since Micro F1 is dominated by frequent classes and can hide weak results on rare attributes.
  • Per-label precision and recall helped find specific failure modes.
  • Per-sample intersection over union measured how closely predicted labels overlapped the expert's.
  • Exact-match accuracy was used for the nine single-label attributes, such as rotation or watermark detection.

Prompt optimization as an evolutionary loop

With a trustworthy answer key, the team tuned prompts in a loop that AWS compares to reinforcement learning: each prompt version acts as an agent, the Micro F1 score is the reward, and error analysis guides the next change. The approach resembles GEPA (Genetic-Pareto reflective prompt evolution), in which prompt components are mutated, evaluated and selected. The loop ran from version v0 to v27. AI assistants reviewed failed images and proposed changes, and MIPROv2, an automated optimizer from the DSPy library, supported the photo classification task in v27.

The first prompt used generic instructions to classify an image into categories such as Living Room, Kitchen, Bedroom and Bathroom. Proprietary model A, Property Finder's production baseline, scored a Micro F1 of 0.55 on the 256-image development set using v22, a prompt tuned for its model family. AWS says scores before v21 used the earlier single-label setup and are not directly comparable.

Several fixes came from looking at individual failures. Early models labeled rooms seen through doorways as the secondary room type, so the team told them to classify by the room filling most of the frame. Regional rules for decorative Arabic text overlays, which are not watermarks, removed a whole category of false positives. The v21 prompt then introduced multi-label output and became the shared starting point for every model. With it, Llama 4 Maverick 17B Instruct from Meta scored a Micro F1 of 0.75 on the development set.

What it means for teams evaluating models

The lesson AWS draws is that the hardest part was not finding a better model but discovering that the evaluation method was the blocker. Teams that treat their production model's outputs as ground truth may be rejecting cheaper or more accurate alternatives before testing them.

AWS also suggests a hybrid approach for future dataset expansion: models from different families surface candidate labels and disagreements, and human experts adjudicate every case, so no model output is used directly as a label. The source excerpt ends before the full final comparison, so the complete results are in the original AWS post.

Related news

AWS shows how to route coding agent requests with NeMo Switchyard
For developersNovember 13, 2026, 23:30

AWS shows how to route coding agent requests with NeMo Switchyard

AWS has published a sample deployment that routes coding agent requests between models on Amazon Bedrock using NVIDIA NeMo Switchyard and LiteLLM. The approach measures…

AWS details four ways to speed up Amazon Bedrock AgentCore sessions
For developersNovember 13, 2026, 23:02

AWS details four ways to speed up Amazon Bedrock AgentCore sessions

AWS has outlined four architecture patterns for reducing session startup delays in Amazon Bedrock AgentCore runtime, including snapshot-based initialization, build hooks…

AWS adds three-tier Data Capture to SageMaker HyperPod
Products and toolsNovember 13, 2026, 19:01

AWS adds three-tier Data Capture to SageMaker HyperPod

Amazon SageMaker HyperPod now includes Data Capture, a built-in capability that records live inference inputs and outputs to Amazon S3. AWS says the feature is designed…