> For the complete documentation index, see [llms.txt](https://alex-25.gitbook.io/welcome/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://alex-25.gitbook.io/welcome/interview-prep/apache-hudi.md).

# Apache Hudi

## 🧩 Apache Hudi — Interview Questions & Answers

***

### 🔹 1️⃣ What is Apache Hudi and why is it used?

**Answer:** Apache Hudi (Hadoop Upserts Deletes and Incrementals) is a **data lake framework** that brings **ACID transactions**, **record-level upserts/deletes**, and **incremental ingestion** to data lakes on object stores like S3, GCS, or HDFS.

It bridges the gap between:

* **Data warehouses (ACID, updates, fast queries)**
* **Data lakes (scalable, cheap storage, flexible schema)**

**Use cases:**

* Change Data Capture (CDC) ingestion
* Incremental ETL pipelines
* GDPR deletions (right-to-be-forgotten)
* Near real-time data serving

✅ Hudi makes your **data lake behave like a database**.

***

### 🔹 2️⃣ What are Hudi’s main components?

| Component              | Role                                                                       |
| ---------------------- | -------------------------------------------------------------------------- |
| **Write Client**       | Handles insert, upsert, delete, and compaction operations.                 |
| **Storage Layer**      | Organizes files and metadata on disk (S3/HDFS).                            |
| **Timeline Service**   | Tracks all commits and actions on the dataset (provides ACID).             |
| **Query Engine Layer** | Allows reads via Spark, Presto, Hive, Trino, etc.                          |
| **Metadata Table**     | Stores file listings and stats to speed up queries and listing operations. |

***

### 🔹 3️⃣ What problems does Hudi solve?

| Problem                                            | Hudi Solution                                 |
| -------------------------------------------------- | --------------------------------------------- |
| Immutable data lakes → can’t update/delete records | Provides **record-level upserts and deletes** |
| Expensive full reloads                             | Supports **incremental queries**              |
| Schema changes                                     | Built-in **schema evolution**                 |
| Eventual consistency on S3                         | ACID transactions via **commit timeline**     |
| Long ingestion delays                              | **Streaming ingestion** with Spark/Flink      |
| Slow S3 list operations                            | **Metadata Table** optimization               |

***

### 🔹 4️⃣ What are Hudi’s table types?

| Table Type              | Description                                                         | Use Case                         |
| ----------------------- | ------------------------------------------------------------------- | -------------------------------- |
| **Copy-On-Write (COW)** | Data is rewritten to new Parquet files on update.                   | Read-heavy batch workloads.      |
| **Merge-On-Read (MOR)** | Writes new changes to Avro log files, later compacted into Parquet. | Write-heavy streaming workloads. |

**Example:**

* COW: nightly ETL batches → dashboards.
* MOR: CDC stream → near-real-time analytics.

***

### 🔹 5️⃣ What are the query types in Hudi?

| Query Type               | Description                                    | Use Case                   |
| ------------------------ | ---------------------------------------------- | -------------------------- |
| **Snapshot Query**       | Reads the latest committed view (base + logs). | Real-time analytics.       |
| **Read Optimized Query** | Reads only base Parquet files (no merge).      | Fast batch queries.        |
| **Incremental Query**    | Reads only new data since a specific commit.   | Incremental ETL pipelines. |

***

### 🔹 6️⃣ How does Hudi ensure ACID transactions on S3?

Hudi maintains a **timeline** (in `.hoodie/`) where every operation (commit, compaction, rollback) is an **instant** with a timestamp.

* Writes are first done in **staging**.
* On successful commit → the timeline is **atomically updated**.
* Readers only see **completed instants** → ensures snapshot isolation.
* If failure occurs → the instant is rolled back.

✅ This gives **atomic, consistent, isolated, durable (ACID)** semantics even on eventually consistent stores like S3.

***

### 🔹 7️⃣ What are “instants” in Hudi?

Instants represent all actions in the dataset’s **timeline**:

| Type            | Example                           | Purpose               |
| --------------- | --------------------------------- | --------------------- |
| **commit**      | `20250101123000.commit`           | Insert/upsert done.   |
| **deltacommit** | `20250101124000.deltacommit`      | MOR delta log write.  |
| **compaction**  | `20250101130000.compaction`       | Merge logs → Parquet. |
| **clean**       | Delete obsolete files.            |                       |
| **rollback**    | Undo failed commit.               |                       |
| **savepoint**   | Mark dataset for rollback safety. |                       |

Each instant moves from `requested` → `inflight` → `completed`.

***

### 🔹 8️⃣ What write operations does Hudi support?

| Operation             | Description                                    |
| --------------------- | ---------------------------------------------- |
| **insert**            | Add new records.                               |
| **upsert**            | Insert or update existing records (via index). |
| **bulk\_insert**      | High-speed initial load (no index lookup).     |
| **insert\_overwrite** | Replace an entire dataset or partition.        |
| **delete**            | Remove records by key.                         |

***

### 🔹 9️⃣ What is the difference between COW and MOR?

| Aspect            | COW                           | MOR                            |
| ----------------- | ----------------------------- | ------------------------------ |
| Storage           | Only Parquet                  | Parquet + Avro logs            |
| Write performance | Slower (rewrites entire file) | Faster (append logs)           |
| Read performance  | Faster (no merge)             | Slightly slower (merge needed) |
| Use case          | Batch, read-heavy             | Streaming, write-heavy         |

***

### 🔹 🔟 How does Hudi handle schema evolution?

* Uses **Avro schema** stored in the `.hoodie` metadata.
* New columns → allowed (added as nullable).
* Dropping columns → not recommended (need rewrite).
* Backward and forward compatibility handled automatically.

✅ Safe schema evolution = crucial for evolving pipelines.

***

### 🔹 11️⃣ What indexing mechanisms does Hudi use?

| Index Type                | Description                                  |
| ------------------------- | -------------------------------------------- |
| **Bloom Index (default)** | Stores record key hashes in Parquet footers. |
| **Global Bloom Index**    | Ensures uniqueness across all partitions.    |
| **Simple Index**          | Uses metadata table instead of file scans.   |
| **HBase Index**           | External index for ultra-large scale.        |
| **Metadata Table Index**  | Modern, fast index layer for file lookups.   |

***

### 🔹 12️⃣ What is the Metadata Table and why is it important?

The **Metadata Table** (stored under `.hoodie/metadata/`) caches:

* File listings
* Bloom filters
* Column stats

**Benefits:**

* Avoids expensive S3 `LIST` operations (reduces latency by 10–100×).
* Speeds up query planning and compaction.

***

### 🔹 13️⃣ What is compaction in Hudi?

* Applies only to **MOR tables**.
* Merges small delta log files into larger Parquet base files.
* Reduces read latency and small file count.
* Can be **inline** (during write) or **asynchronous** (background job).

***

### 🔹 14️⃣ What are cleaning and archival in Hudi?

| Process      | Purpose                                                                                       |
| ------------ | --------------------------------------------------------------------------------------------- |
| **Cleaning** | Deletes obsolete or unreferenced file versions.                                               |
| **Archival** | Moves older commits from `.hoodie/timeline` to `.hoodie/archive` to reduce metadata overhead. |

***

### 🔹 15️⃣ How does Hudi support incremental data ingestion?

Hudi tracks commits via the timeline.\
You can read **only the new data** since a given instant:

```python
spark.read.format("hudi") \\
  .option("hoodie.datasource.query.type", "incremental") \\
  .option("hoodie.datasource.read.begin.instanttime", "20250101000000") \\
  .load("s3://hudi/orders")
```

✅ This is the foundation of **incremental ETL pipelines**.

***

### 🔹 16️⃣ How does Hudi integrate with Spark and Flink?

| Engine             | Integration                       | Description                                      |
| ------------------ | --------------------------------- | ------------------------------------------------ |
| **Spark**          | DataSource API (`format("hudi")`) | Supports batch and streaming writes/reads.       |
| **Flink**          | Native connector                  | Real-time ingestion with exactly-once semantics. |
| **Presto / Trino** | Read connector                    | Fast analytical queries.                         |
| **Hive**           | Sync via Hive Sync Tool           | Enables Hive table queries on Hudi data.         |

***

### 🔹 17️⃣ What is clustering in Hudi?

Clustering re-organizes existing data to optimize layout and file sizes for query performance.

***

### 🔹 18️⃣ How do you tune Hudi performance?

| Tuning Area        | Parameter / Strategy                                                     |
| ------------------ | ------------------------------------------------------------------------ |
| **Parallelism**    | `hoodie.upsert.shuffle.parallelism`, `hoodie.insert.shuffle.parallelism` |
| **Compaction**     | Schedule asynchronously; tune parallelism                                |
| **File sizing**    | `hoodie.parquet.max.file.size`, `hoodie.small.file.limit`                |
| **Metadata Table** | Enable for large datasets                                                |
| **Bloom filter**   | Disable for purely append-only use cases                                 |
| **Bulk insert**    | Use for initial data load (no index lookup)                              |

***

### 🔹 19️⃣ How is Hudi different from Delta Lake and Iceberg?

| Feature                 | Hudi                        | Delta Lake                 | Iceberg                         |
| ----------------------- | --------------------------- | -------------------------- | ------------------------------- |
| **Design focus**        | Streaming + incremental ETL | DWH-like batch reliability | Large-scale metadata management |
| **Upsert/Delete**       | ✅ Native                    | ✅ Native                   | ✅ Native                        |
| **Table Types**         | COW, MOR                    | Single table type          | Single table type               |
| **Incremental Reads**   | ✅ Yes                       | ✅ Limited                  | ✅ Yes                           |
| **Metadata Management** | Metadata Table              | Delta Log                  | Manifest List                   |
| **Integrations**        | Spark, Flink, Hive          | Spark                      | Spark, Flink, Trino             |
| **Best For**            | Streaming CDC pipelines     | Batch ETL, BI              | Federated query lakes           |

***

### 🔹 20️⃣ Real-world Use Case Example

> You’re processing CDC data from Kafka to S3.\
> Each event is either an insert, update, or delete.

✅ **Solution:**

* Ingest CDC stream via **Spark Structured Streaming → Hudi upsert**.
* Maintain latest state table in **MOR**.
* Query real-time dashboard via **Snapshot reads**.
* Downstream batch ETL reads incrementally since last checkpoint.

***

## ✅ Summary

| Concept               | Description                           |
| --------------------- | ------------------------------------- |
| **Timeline**          | ACID metadata for commits & actions   |
| **Instants**          | Individual commit events              |
| **Table Types**       | COW & MOR                             |
| **Query Types**       | Snapshot, Read Optimized, Incremental |
| **Compaction**        | Merge delta logs (MOR only)           |
| **Metadata Table**    | Index + file listing optimization     |
| **Incremental Query** | Reads new data since last commit      |
| **Indexing**          | Bloom / Metadata / Global             |
| **Clustering**        | Layout optimization for performance   |

***

✅ **In summary:**

> Apache Hudi is a streaming data lake framework providing **ACID, incremental processing, and upserts/deletes** over object stores, designed for **real-time data engineering pipelines**.\
> It integrates deeply with **Spark** and **Flink**, making it ideal for building **incremental, low-latency, and fault-tolerant data lakes**.