Artificial intelligence news.
Straight from the original announcements.

Databricks beta streams Unity Catalog data directly to GPUs

November 9, 2026, 16:00By the AI News Bites Editorial TeamSource: Databricks
Databricks beta streams Unity Catalog data directly to GPUs
Image: Databricks

Databricks has released UCDataset in beta on AI Runtime, a native PyTorch data loader that streams Unity Catalog tables directly to GPUs. The company says it can cut training time by 10x while removing the need for exported data copies and custom sharding code.

Key points
  • UCDataset is in beta on Databricks AI Runtime.
  • It streams Unity Catalog tables directly to GPUs for PyTorch training.
  • Databricks reports a 10x training time improvement in a simulated workload.
  • The beta reads Delta tables, including tables with deletion vectors.
  • Unity Catalog permissions apply to training jobs using the live table.

Direct path from Unity Catalog to GPUs

UCDataset lets teams use a Unity Catalog table as a data source for distributed PyTorch training. Users specify the table, the columns required by the model and a row filter, then UCDataset streams training ready batches to the GPUs in the job.

According to Databricks, the loader reads the table's files directly from cloud storage. It splits matching rows evenly across GPUs and data loader workers, so teams do not need to write sampler or sharding code. UCDataset also produces whole batches, allowing it to be wrapped in a standard PyTorch DataLoader with batch_size=None.

The package is available on PyPI and can be installed with %pip install databricks-uc-dataset. The beta reads Delta tables registered in Unity Catalog, including tables with deletion vectors.

Databricks reports 10x faster benchmark

Databricks says UCDataset addresses a common training bottleneck: moving data to GPUs quickly enough to keep them busy. Teams often export Unity Catalog tables to Parquet before a run, but that creates a second copy, can become stale when new data arrives and falls outside Unity Catalog lineage.

In a simulated production workload, Databricks streamed a 300 GB feature table across 8 NVIDIA H100 GPUs to train a recommendation model. The company reports an average 10x improvement in training time compared with loading data directly from Parquet.

UCDataset can also apply filters and deletion vectors as data is read, meaning rows that do not match a filter do not reach the training code. Databricks says newly appended rows are picked up automatically on the next run.

Governance remains attached to the live table

Training jobs read data as the identity that runs them, and Unity Catalog permissions apply to the table, according to Databricks. This avoids the need to secure a separately exported copy of the data. In the beta, however, UCDataset cannot read tables that have row filters or column masks.

Databricks provides a cookbook example for a click through rate model using a table called main.recsys.ctr_features. The example trains on 16 GPUs across two nodes with a standard PyTorch distributed data parallel loop. To use more GPUs, users change num_accelerators in the workload configuration while leaving the training code unchanged.

For unstructured data in Unity Catalog volumes, Databricks directs users to UCVolumeDataset, which streams files with local caching. The company says it is also expanding UCDataset to support multimodal training through the FILE type, including images, audio and documents. For users with structured Delta tables, the beta offers a way to connect existing governance and live data directly to PyTorch training without an export step.

Related news

Eclipse Aerospace brings two decades of flight data into Databricks
Products and toolsNovember 12, 2026, 20:20

Eclipse Aerospace brings two decades of flight data into Databricks

Eclipse Aerospace has modernized the ingestion of nearly 20 years of proprietary flight data with Databricks, giving service centers and engineers faster access to…

Products and tools
Products and toolsNovember 12, 2026, 19:26

Databricks page says Genie Migrate is generally available

Databricks says Genie Migrate is now generally available for migration to the Databricks Lakehouse. The company page carrying the announcement is currently unavailable…

Databricks says Lakehouse//RT cuts the cost of real-time analytics
Products and toolsNovember 12, 2026, 17:53

Databricks says Lakehouse//RT cuts the cost of real-time analytics

Databricks says Lakehouse//RT is designed to absorb the rising query volumes generated by AI agents without a proportional increase in infrastructure spending. Customers…