Artificial intelligence news.
Straight from the original announcements.

Salesforce Builds Interactive Testbed for Real-Time Visual AI Assistants

November 12, 2026, 17:00By the AI News Bites Editorial TeamSource: Salesforce
Salesforce Builds Interactive Testbed for Real-Time Visual AI Assistants
Image: Salesforce

Salesforce AI Research has built Interactive Visual Gym, a simulated environment for testing whether real-time visual AI assistants can change the outcome of physical tasks. The company says the system lets simulated users react to advice, exposing failures that prerecorded video evaluations can miss.

Key points
  • Interactive Visual Gym lets simulated users react to visual AI guidance.
  • IVG-Bench includes 315 scenarios across three environments.
  • The strongest tested assistant completed 53.4% of tasks within the time limit.
  • Simulation scores matched people’s assistant preferences 78.5% of the time.
  • Salesforce used the simulator to improve an open-source model’s task performance.

Why static tests are not enough

Visual AI assistants are moving beyond chat windows into real-time guidance for physical work. They can watch live video and instruct people through multistep tasks, but Salesforce argues that a correct warning is not necessarily useful unless it arrives early enough to affect what the user does next.

Many existing evaluations replay prerecorded video and compare an assistant’s response with an answer key. This can show whether a warning was correct, but the recording cannot react to the warning. It therefore cannot establish whether the user trusted the advice, became confused, or avoided an incorrect action.

Salesforce places its work alongside recent developments in real-time interaction. The company says Thinking Machines previewed models in May that continuously take in audio, video, and text. In June, Meta researchers released a wearable video dataset for proactive procedural assistance, and in September Google introduced Gemini 3.8 Live, combining real-time visual understanding, spoken conversation, and background task work.

How Interactive Visual Gym works

Interactive Visual Gym, or IVG, recreates everyday physical tasks such as cooking and managing smartphone settings. Each task pairs a simulated user with a visual AI assistant. The user’s actions and the simulated environment change in response to the assistant’s guidance.

In one kitchen example, a simulated cook assembles a burger in a specified order. If the cook selects the wrong ingredient, the assistant must identify the mistake quickly enough for the cook to respond. The system uses natural-language instructions, such as “go to the pan with the meat,” rather than the joint velocities and degree coordinates used by classical robotics simulators.

IVG can schedule mistakes, plan changes, questions, and reminder requests at precise points while keeping the user’s dialogue consistent with its behavior. Simulated users also vary in skill, trust in AI, preferred execution speed, and tolerance for interruption. This allows the same task to be tested with users who need detailed guidance and others who prefer fewer interruptions.

Benchmark finds timing and persistence problems

Using IVG, Salesforce AI Research created IVG-Bench, consisting of 315 scenarios across three environments. The team evaluated 15 assistants, each running on a different model. According to Salesforce, the strongest assistant, running on Gemini 4 Argon, completed 53.4% of tasks within the time limit when averaged across environments. Open-weight models generally trailed the leading frontier models.

The evaluation measures both task success and interaction quality. Salesforce says an assistant can help users complete tasks more often while still giving clumsy advice. Dynamic testing also reveals problems that static tests can overlook, including latency that makes correct advice stale and repeated instructions after a user has acknowledged them. For some open-weight models, guidance deteriorated as tasks continued.

In tests where people controlled the simulated cook and ranked assistants, Salesforce says its automated interaction-quality score matched people’s preferences 78.5% of the time. The company also found that simulated users behaved similarly to real people, although simulated users spoke and replied far more often.

A testbed for deployment decisions

Salesforce says IVG can also be used to improve visual assistants. In an early proof of concept, the team trained an open-source model in the simulator to prioritize task completion and error prevention. The model’s success rate improved, and the assistant became better at delivering advice at the right time and intervening before mistakes occurred.

For leaders considering visual AI assistants for frontline teams, Salesforce recommends evaluating whether the system measurably improves task completion, whether its guidance arrives while it can still change the outcome, and whether it remains effective deep into a long task or after a user goes off script. IVG is designed to provide that feedback loop at the scale of a dataset, without requiring thousands of live training runs with people of different skill levels.

Related news

JetBrains asks 12 AI models for 1,200 programming hot takes
ResearchNovember 15, 2026, 18:02

JetBrains asks 12 AI models for 1,200 programming hot takes

Twelve AI models overwhelmingly attacked Python, Rust and TypeScript in a JetBrains experiment that collected 1,200 opinions about programming languages. The models also…

Salesforce says policy update doubled Agentforce Memory recall
Products and toolsNovember 4, 2026, 19:00

Salesforce says policy update doubled Agentforce Memory recall

Salesforce AI Research says it has roughly doubled the memory quality of Agentforce by changing the rules that decide what an AI agent keeps about a customer. According…

UW study finds AI models score some accents lower in speech evaluations
ResearchNovember 4, 2026, 17:49

UW study finds AI models score some accents lower in speech evaluations

A University of Washington-led study found that leading AI models rated Chinese and Nigerian English accents lower than American and British accents when evaluating…