Google Research's Agile Deliberation helps users define subjective image categories

The problem with fuzzy categories

Machine learning models learn from examples labeled by humans, and that process assumes the labeler already has a clear and stable definition of the category. According to Google Research, this holds for objective categories such as a stop sign or a golden retriever, but not for subjective ones such as "healthy food", "inspiring artwork" or content moderation tasks.

In those domains, boundaries can shift with cultural context, personal philosophies or evolving operational policies. The authors, Otilia Stretcu and Leijie Wang, also note that people rarely begin with a complete specification. They start with an intuitive idea and discover the real boundaries and exceptions only when they see real-world examples. Traditional workflows do not account for this, and force users to give rigid labels while their own mental model is still changing.

How the two-stage workflow works

Agile Deliberation splits the task into concept scoping and concept iteration.

In the scoping stage, the user breaks a broad concept into a hierarchy of in-scope and out-of-scope definitions. For "Healthy Food", positive sub-categories could be "Salads", "Grain Bowls" and "Lean Proteins", while exclusions could be "Processed Foods" or "Raw, Unprepared Ingredients". This gives the model an initial structured baseline.

In the iteration stage, the framework uses vision-language models (VLMs) to find and show semantically borderline images that challenge the current text-based definitions. The approach draws on active learning, but instead of choosing examples only to reduce machine uncertainty, it picks them to push the user to reflect. One example from the post is a fruit smoothie: is it a healthy beverage because it contains raw fruit, or an excluded processed food because of its sugar content?

The user's feedback goes into an automatic prompt optimization loop, so the VLM's classification boundaries move along with the refined definitions. The team has open-sourced the code and algorithms on GitHub.

Live study design

Because subjective concepts have no static ground truth, the team said offline evaluation on pre-labeled datasets cannot capture how a person's mental model shifts. It built a functional prototype and ran 18 live sessions of 1.5 hours each. Participants used the system and alternative baselines in randomized order to prevent learning bias.

Two domains were tested: Healthy Food, and Pay-to-Play, a content moderation challenge about identifying clickbait images in online entertainment or gaming that promise misleading financial or digital rewards. Agile Deliberation was compared with three approaches:

• Zero-shot classifiers: users write one prompt and cannot see edge cases.
• Modeling collaborator: an AI agent decomposes the concept on its own, leaving the user out of the loop.
• Manual deliberation: users inspect spreadsheets, guess search queries and rewrite prompts by hand.

Results and what they suggest

According to Google Research, participants who went through the structured process showed significantly higher labeling consistency over time and built classifiers that substantially outperformed all standard baselines. The gains were most pronounced on the most challenging subjective images, where traditional classifiers tend to fail because they rely on default, generic AI assumptions rather than the user's intent. The company also says the framework drastically reduced user fatigue and anxiety by replacing manual prompt engineering with direct feedback on borderline images.

Expert participants described the system as a diagnostic tool that helped uncover "blind spots" in their initial definitions. For teams building classifiers for moderation or curation, the findings point to clarifying the human definition first as a way to get models that match intent. The source text is cut off before the full performance figures, so the size of the gains is not reported here.