Spark Declarative Pipelines provides an easier way to define and execute data pipelines for both batch and streaming ETL workloads across any Apache Spark-supported data source, including cloud ...

Since its launch in 2013, Databricks has relied on its ecosystem of partners, such as Fivetran, Rudderstack, and dbt, to provide tools for data preparation and loading. But now, at its annual Data + ...

Apache Spark 4.3 brings stateful streaming with millisecond-scale latency, history-preserving Auto CDC, and faster data and ...

The name of Databricks' annual conference has gone from "Spark Summit" to "Spark + AI Summit" and now to "Data + AI Summit." The evolution of the event name tracks Databricks' own transition from the ...

Lakeflow Designer and Agent Bricks technology unveilings for building data pipeline workflows and AI agents, respectively, are on tap at Wednesday’s Databricks Data + Summit. With new technologies for ...

Today, at its annual Data + AI Summit, Databricks announced that it is open-sourcing its core declarative ETL framework as Apache Spark Declarative Pipelines, making it available to the entire Apache ...

Databricks will bring Genie, Lakeflow, Unity Gateway and Lakebase to SAP Databricks, as SAP adds a new way to reach data ...

A monthly overview of things you need to know as an architect or aspiring architect. Unlock the full InfoQ experience by logging in! Stay updated with your favorite authors and topics, engage with ...

Databricks has unveiled a new extract, transform, load (ETL) framework, dubbed Delta Live Tables, which is now generally available across the Microsoft Azure, AWS and Google Cloud platforms. According ...