Snowflake prepares more autonomous data pipelines and AI agents
Snowflake has unveiled several data management updates designed to support AI agents and applications. The announcements cover everything from automated compute management and Spark workflows to mirroring transaction data.
- Adaptive Compute automatically adjusts computing resources to the workload.
- Snowpark Connect runs existing PySpark, Java and Scala code on Snowflake.
- The write-audit-publish workflow validates data changes on an isolated branch.
- Postgres data mirroring continuously replicates Postgres data for analytics and AI workloads.
- Snowflake says hybrid tables can deliver up to eight times greater throughput in certain cases.
More automated data processing and Spark support
A central element of the November 3, 2026 announcement is Adaptive Compute, which Snowflake says automatically adjusts computing resources to the workload. The service is generally available, so users no longer need to manually tune warehouses or clusters. Spark workloads can also run on it.
Snowpark Connect for Apache Spark is now generally available. The solution runs existing PySpark, Java and Scala code directly on Snowflake, without the need to manage external clusters. The Next-Gen File Reader, available in private preview, applies Spark's native read semantics to JSON, CSV, XML and Parquet files.
Snowpark Connect also supports Apache Iceberg tables in environments managed by Horizon, AWS Glue and Unity Catalog. Generally available incremental reads process only changes between Iceberg snapshots. Iceberg Time Travel and tags provide rollback capabilities and shareable checkpoints.
Data quality checks and declarative data pipelines
Snowflake is using the so-called write-audit-publish workflow to help validate changes. The system creates an isolated branch where modifications can be validated, then published or discarded. The branches can be queried from Snowflake and other engines, making it possible to assess the expected downstream impact.
The Spark Monitoring UI, available in public preview, provides unified debugging for sessions and batch jobs down to individual query profiles. The Code Bundles for Apache Spark enables jobs to be submitted through a REST API or SQL, allowing Spark jobs to be integrated into existing schedulers such as Airflow or Snowflake tasks. Snowflake CoCo's Spark migration capabilities support converting PySpark, Spark Scala and Spark Java notebooks and jobs.
Custom incrementalization for Dynamic Tables is generally available, providing full control over incremental logic and scheduling. Dynamic Tables can also be used in dbt, while Dynamic Iceberg Tables are designed to update open-format data products. In private preview, the latter can also be updated directly in an external Iceberg catalog.
Transaction data for analytics and applications
Snowflake says AI agents and applications need up-to-date operational data to work accurately. Postgres data mirroring is generally available and continuously replicates Postgres tables to Snowflake for analytics and AI workloads. Snowflake data mirroring sends data in the opposite direction, and this feature will soon be available in public preview.
Snowflake says hybrid tables can deliver up to eight times greater throughput for certain workloads. In such cases, a workload that previously required eight XS warehouses could be handled by a single unit, potentially cutting costs by more than 80 percent. P50 latency is three times lower, while P99 latency is approximately two times lower for the affected operational workloads.
According to the company, the updates allow AI agents to perform routine tasks such as checking process status, identifying the causes of errors and suggesting fixes. Proposed code changes will continue to go through the team's existing review process via CoCo's GitHub integration. Users can therefore build on fresh, validated data within the same governed platform.
Snowflake: Fuel Every Agent and App With Data That Keeps Pace
