# Databricks LTAP: Bringing PostgreSQL OLTP and Lakehouse Analytics Together


## Introduction

Modern applications continuously generate transactional data. PostgreSQL is commonly used as the operational database because applications need fast and reliable INSERT, UPDATE, DELETE, and SELECT operations.

At the same time, organizations want to use that same data for:

- Data engineering
- Historical analysis
- BI dashboards
- AI/ML
- Business reporting

The traditional approach is to copy the PostgreSQL data into a separate analytical environment using CDC and replication pipelines.

Databricks LTAP — Lake Transactional/Analytical Processing — provides a different architecture by bringing transactional and analytical workloads closer together through Lakebase and the lakehouse.

## The Problem: PostgreSQL Data Is Needed in Two Worlds

Consider an airport application. The backend continuously writes flight information into PostgreSQL — flight, airline, origin, destination, status, and delay.

The application needs PostgreSQL for transactional operations. The data engineering team needs the same data for analytics.

Traditionally, the flow looks like this:

**Application → PostgreSQL → CDC / Replication → Cloud / Data Lake → Transformations → Dashboard**

This introduces another data movement and synchronization layer. As data volume increases, the cost and operational complexity also increase.

## The Two Burning Costs

The main problem is not only storage. There are two major cost areas to think about.

**1. CDC and replication infrastructure**

A traditional architecture may require additional components for change capture, replication, streaming, pipeline compute, monitoring, and failure recovery. These components continuously consume resources.

**2. Moving large volumes of data**

If a large PostgreSQL database needs to be replicated into another analytical environment, organizations may repeatedly move and maintain large amounts of data:

**PostgreSQL → Large data movement → Cloud / Lakehouse copy**

If another system later needs the data back, another movement path can be introduced. The larger the dataset and the more frequently it changes, the more important this becomes.

## LTAP: A Different Approach

LTAP changes the architecture from "copy the operational data somewhere else and continuously synchronize it" toward "use a unified storage foundation for transactional and analytical workloads."

With Databricks Lakebase, PostgreSQL-compatible transactional workloads can operate alongside lakehouse analytics. Unity Catalog provides the governance and analytical access layer.

## How the LTAP Flow Works

In my flight-data POC, the backend continuously creates and modifies flight records:

- INSERT → New flight
- UPDATE → Flight status / delay changed
- DELETE → Flight removed

The backend connects directly to the Lakebase PostgreSQL database using standard PostgreSQL connectivity. A simplified Python connection looks like this:

```python
import psycopg

conn = psycopg.connect(
    host=PG_HOST,
    dbname=PG_DATABASE,
    user=PG_USER,
    password=PG_PASSWORD,
    sslmode="require"
)

cursor = conn.cursor()

cursor.execute("""
    INSERT INTO flights
    (flight_id, airline, origin, destination, status)
    VALUES (%s, %s, %s, %s, %s)
""", (101, "IndiGo", "MAA", "DEL", "Scheduled"))

conn.commit()
```

Lakebase supports standard PostgreSQL clients and drivers.

## Connecting Lakebase with Unity Catalog

After creating the Lakebase project and PostgreSQL database, the database can be registered with Unity Catalog:

**Lakebase Project → PostgreSQL Database → Unity Catalog Registration → Databricks Analytics**

The registered Lakebase database can then be accessed through Unity Catalog for analytical workloads. Databricks documents this registration as a read-only Unity Catalog representation of the Lakebase database.

This gives the data engineering team a governed way to work with the data without changing how the application performs its normal PostgreSQL transactions.

## Bronze, Silver, and Gold

Once the flight data is available to the analytical environment, normal data engineering practices can be applied.

**Bronze — store the raw flight data**

```sql
CREATE TABLE bronze_flights AS
SELECT *
FROM lakebase_catalog.public.flights;
```

**Silver — clean and standardize the data**

```sql
CREATE TABLE silver_flights AS
SELECT
    flight_id,
    UPPER(airline) AS airline,
    UPPER(origin) AS origin,
    UPPER(destination) AS destination,
    status,
    COALESCE(delay_minutes, 0) AS delay_minutes
FROM bronze_flights;
```

**Gold — create business-ready metrics**

```sql
CREATE TABLE gold_airline_metrics AS
SELECT
    airline,
    COUNT(*) AS total_flights,
    ROUND(AVG(delay_minutes), 2) AS avg_delay
FROM silver_flights
GROUP BY airline;
```

The Gold layer is then suitable for reporting and dashboard analysis.

## Flight Analytics Dashboard

The Gold data can be consumed by a Databricks SQL dashboard. For example:

**Airport Flight Analytics**

| Metric | Value |
|---|---|
| Total Flights | 125,430 |
| Average Delay | 14.8 min |
| Delayed Flights | 21,540 |

| Airline | Flights | Avg Delay |
|---|---|---|
| IndiGo | 52,430 | 12.4 |
| Air India | 38,210 | 16.8 |
| Vistara | 22,140 | 13.2 |

This creates a complete flow from application transaction → analytical transformation → business insight.

## Where Does CDC Fit?

This is the most important distinction.

LTAP is designed to reduce the need for a separate external CDC/replication architecture whose purpose is to maintain a second analytical copy of transactional data.

**Traditional approach:** PostgreSQL → External CDC → Replication → Data Lake

**LTAP approach:** Lakebase PostgreSQL → Unified storage architecture → Lakehouse Analytics

Databricks specifically describes LTAP as removing traditional synchronization infrastructure such as CDC pipelines and replication used between separate transactional and analytical systems.

However, this should not be interpreted as "CDC does not exist anywhere in Lakebase." Lakebase also provides a native Change Data Feed capability for use cases that specifically require row-level change information.

The key difference: LTAP reduces the need for an external CDC stack simply to keep separate OLTP and OLAP copies synchronized.

## Why This Can Reduce Cost

The cost benefit comes primarily from reducing unnecessary infrastructure and duplicated data movement.

**Traditional architecture:** PostgreSQL → CDC infrastructure → Replication → Cloud storage → Transformation

Every additional layer can introduce compute, storage, network transfer, monitoring, and maintenance costs.

With LTAP, the architecture is designed to bring these workloads closer together. So instead of paying for a continuously maintained synchronization path just to create another analytical copy, the organization can use the unified LTAP architecture and apply analytical processing where appropriate.

This can reduce:

- CDC infrastructure cost
- Replication cost
- Duplicate storage
- Pipeline maintenance
- Operational complexity
- Unnecessary data movement

The exact savings will depend on workload size, frequency of changes, retention, compute requirements, and the specific Databricks architecture.

## One Important Security and Connection Detail

For production applications, authentication should be designed carefully.

Lakebase supports PostgreSQL authentication mechanisms including OAuth and native PostgreSQL password authentication. OAuth credentials are temporary and need refreshing, while PostgreSQL passwords are suitable for clients that require longer-lived database credentials.

The connection architecture is essentially:

**Backend → PostgreSQL credentials → Lakebase PostgreSQL → Unity Catalog → Lakehouse Analytics**

The authentication used by the application to connect to PostgreSQL should not be confused with credentials used for managing Databricks/Lakebase APIs.

## Benefits of LTAP

LTAP is particularly interesting when an application has both transactional and analytical requirements. The main benefits are:

- Less dependency on external CDC/replication infrastructure
- Reduced duplicated data movement
- Reduced operational complexity
- PostgreSQL-compatible application development
- Lakehouse analytics using Databricks
- Unity Catalog governance
- Bronze/Silver/Gold data engineering
- Direct path to SQL dashboards and analytics

## Things to Consider

LTAP does not mean that every organization should immediately remove every existing ETL or CDC pipeline. The right architecture depends on:

- Existing databases
- Data volume
- Latency requirements
- Analytical workload
- Application requirements
- Integration with external systems
- Cloud architecture
- Cost model

Also, some Lakebase capabilities have specific availability and status depending on the Databricks cloud and release.

## Conclusion

Traditional data architectures often create a separation between OLTP and OLAP: **Application → PostgreSQL → CDC / ETL → Lakehouse**.

This separation can introduce additional infrastructure, duplicated data, and repeated movement of large datasets.

LTAP provides a modern alternative by bringing transactional and analytical workloads together around a unified storage architecture. My airport flight-data POC demonstrated this flow:

**Backend → Lakebase PostgreSQL → Unity Catalog → Bronze → Silver → Gold → Databricks Dashboard**

The biggest value proposition is therefore not simply "no CDC." It is: reduce the need for a separate CDC/replication stack and duplicated data movement between transactional databases and analytical platforms, while allowing application and data engineering workloads to work within the same Databricks data architecture.

That is what makes LTAP + Lakebase an interesting pattern for modern data engineering.


**Thank you for spending your valuable time reading my tech blog!** I hope this POC and my learnings around LTAP and Lakebase were useful.

I’m still learning and exploring this space, so if you find any mistakes, missing points, or areas that could be improved, please feel free to share your thoughts or ping me. I’d be happy to learn from the community and improve my understanding.

**Keep learning, keep building, and keep sharing! 🚀**  
**Happy Learning! 😊**