KS&R Report Finds Synthetic Data Succeeds in Simple Test, Struggles in Complex One

KS&R has released an independent evaluation of four synthetic data providers, finding that all matched a real data baseline in a simple study but fell outside it in an initial complex study. The report uses the consultancy's ECHO Index to assess how faithfully synthetic data reproduces human responses.
- KS&R evaluated production models from four synthetic data providers.
- All four matched the baseline in the simple study.
- All four initial submissions fell outside the baseline in the complex study.
- The report is based on two datasets, not a ranking of providers.
- KS&R plans to extend the ECHO Index to digital twins and synthetic personas in Q1 2027.
Four providers tested against real data
The State of Synthetic evaluates production models from LagrangeDATA, Livepanel, Simulacra and Lewsearch. Each provider generated a complete synthetic copy of two public datasets, one simple and one complex.
KS&R scored every submission against a fit-for-purpose baseline generated from the real data alone. The evaluation focused on sample boosting, where a model generates new respondents for questions that real people have already answered.
According to KS&R, all four providers landed within the baseline for the simple study. In the complex study, however, all four initial submissions fell far outside it.
The report examines where models failed
KS&R says the full report shows where each model broke down and how targeted changes moved submissions closer to the baseline. It also describes cases where a provider's subject-matter expertise made a measurable difference on particular questions.
The findings apply to specific submissions on two datasets. KS&R says they are not a ranking of the firms and do not amount to a verdict on synthetic data as a whole.
The consultancy created the ECHO Index, described as a patent-pending evaluation methodology, to give buyers a neutral way to assess synthetic data. The firm says it has no synthetic products to defend, while providers often validate their own models and present a single score as evidence.
“Synthetic data is viable when it is governed. Don't buy the promise. Buy the evidence,” Ben Cortese, PhD, Senior Vice President, Decision Sciences & Innovation at KS&R, said in the announcement.
A framework for governed synthetic studies
The report arrives as insights teams face pressure to study hard-to-reach audiences, work faster and operate with tight budgets. KS&R says synthetic data is being sold faster than governance can be developed, making independent evaluation a central concern for buyers.
The report gives research leaders guidance on running a governed synthetic study, including how to budget for iteration and which questions to ask providers before purchasing. KS&R recommends that buyers have synthetic data evaluated by a neutral third party before relying on it.
The executive summary and full report are available free from KS&R. Providers can have their models scored and listed in the ECHO Registry, while research and insights leaders can work with KS&R to assess where synthetic data fits, test data they have already purchased or check a model before committing to it.
KS&R is continuing the evaluation and plans to extend the ECHO Index to digital twins and synthetic personas, with completion planned for Q1 2027. For users of synthetic data, the results support a more conditional approach: success on a simple task does not establish performance on a complex one, so testing and governance remain part of the buying process.


