Proposed execution policy

I designed a memory policy for an ELT stack using Dagster, Trino, DataFusion and PostgreSQL. The assumed server has AMD EPYC 9175F 16c/32t, 768GB DDR5-6400, RTX 6000 Pro Max-Q and NVMe 3.84TB. The plan reserves 512GB for ELT and reuses it step by step.

The wider data-platform idea supports local, offline-first flows such as Airbyte -> Iceberg / Trino -> visualization, with possible 48h Prototype and data-platform delivery formats. This is the proposed execution policy for that design.

Separating memory peaks

Even with 768GB, concurrent Trino queries, DataFusion transforms, PostgreSQL loads and Dagster workers can overlap in memory. The policy must account for OOM, swap and restarting partially failed work.

The proposed rule is:

Reuse a dedicated 512GB pool step by step, keep light paths in In-Memory, and fall back to Arrow IPC or Parquet when the workload gets large.

It defines both memory use and how failed work can resume.

Base Resource Assumptions

These are the hardware assumptions:

  • Server: AMD EPYC 9175F 16c/32t (5GHz) + 768GB DDR5-6400 + RTX 6000 Pro Max-Q + NVMe 3.84TB
  • ELT pool: a dedicated 512GB reserved for workflow execution
  • Bandwidth: memory bandwidth around ~600GB/s, NVMe I/O in multiple GB/s, network at 10GbE

The 512GB pool is assigned to the active ELT step. Workload budgets remain explicit as ingestion, monitoring, cataloging and visualization services are added.

Memory Operation Policy

Principle

The memory policy is simple on purpose:

  • Reuse the 512GB pool step by step
  • Prefer direct In-Memory handoff when everything stays in one process
  • If the work is split across processes or containers, keep the combined usage within the same 512GB ceiling

Heavy analytics steps run one at a time, release memory when finished and let the next step reuse the pool.

Switching Thresholds

I also wrote down explicit thresholds for intermediate data size:

Intermediate sizeRecommended modeNotes
<= 40-50% (200-250GB)In-Memory (in-process + InMemoryIOManager)Fastest path because it directly uses ~600GB/s memory bandwidth
50-80%Arrow IPC streamGood compromise with lower overhead and NVMe-backed safety
>80% or concurrent heavy opsParquet temporary outputBetter restart behavior and stronger safety margin

These thresholds determine whether intermediate data stays in memory or is written to Parquet.

Optimization by Component

DataFusion (Rust)

On the DataFusion side, the biggest rule is to avoid accidentally materializing huge intermediates too early.

  • Set MemoryPool limits per op using S/M/L profiles
  • Point DiskManager at NVMe under /opt/tmp so forced spill is available
  • Prefer the streaming API, especially execute_stream()
  • Align partition count with the physical core count (16c)

Streaming avoids collecting intermediates that do not need to be materialized.

Trino

Trino needs both internal limits and external scheduling discipline.

  JVM: -Xmx192g
max-spill-per-node=512GB
query.max-memory-per-node=128GB
query.max-total-memory-per-node=160GB

Trino uses a 256GB container with NVMe spill. Dagster allows only one heavy query at a time.

PostgreSQL

PostgreSQL handles persistence and bulk loading.

  shared_buffers=8-16GB
work_mem=128MB
maintenance_work_mem=2-4GB
wal_compression=lz4
checkpoint_timeout=20-30min
max_wal_size=12-16GB

The preferred load patterns are:

  • UNLOGGED TABLE -> COPY -> build indexes -> LOGGED
  • CTAS (CREATE TABLE AS SELECT)

synchronous_commit=off remains an option only during loading.

Dagster

Dagster selects the executor by workload size.

  • Light jobs: in_process_executor + InMemoryIOManager
  • Heavy jobs: multiprocess_executor + per-op Podman containers (--memory/--cpuset-cpus)

Then I tag workloads to control concurrency:

  • heavy operations (Trino, large DataFusion ops): max_concurrent=1
  • light operations such as metadata handling, dbt run, or logging: parallel execution is allowed

The same limits apply across jobs.

Process / Container Design

In-Memory (fastest path)

The memory path uses in-process executor and shallow-copy handoffs of pyarrow.Table or RecordBatch. Its proposed limit is about <=200GB, or roughly 50% of the ELT pool.

Beyond that size, downstream op startup must be included in the memory peak.

Container isolation per op

Each op can run with podman run --memory=... --cpuset-cpus=.... Process exit releases its memory for the next step in the 512GB pool.

This follows the wider plan to reproduce local workloads through Podman.

Release memory before child startup

If Python launches Rust child processes while the parent is still holding large buffers, combined peak memory becomes the real problem. A few specific safeguards:

  • release parent buffers with del
  • let scopes end before child startup
  • call gc.collect()
  • prefer Python spawn over fork

Losing fork Copy-on-Write can increase physical memory use.

I/O Optimization

Intermediate formats and storage locations are assigned as follows.

  • Arrow IPC: a fast spill path, close to zero-copy
  • Parquet: the safer fallback with zstd(1-3) and row groups at 128-256MB
  • NVMe: centralize spill directories under /opt/tmp
  • /dev/shm: increase to --shm-size 2-4GB for Arrow Plasma / IPC
  • cleanup: remove temporary files at job end, including failure paths

Arrow IPC prioritizes transfer speed; Parquet prioritizes saved state and retries.

Operating Rules

The execution rules are:

  1. Run heavy queries one at a time.
  2. End each op, free its memory, and reuse the same 512GB pool.
  3. Switch between In-Memory, Arrow IPC, and Parquet based on intermediate size.
  4. Use Parquet when restartability matters most.
  5. Use In-Memory shallow-copy or pointer-style transfer when raw speed matters most.

Different ELT jobs use the same criteria.

Expected benefits

The intended benefits are:

  • Speed: use the ~600GB/s memory bandwidth through In-Memory handoffs.
  • Stability: limit peaks with cgroups, spill and step-wise reuse.
  • Flexibility: choose Arrow IPC or Parquet by intermediate size.
  • Execution control: change modes with the Dagster executor.

These are design goals. Concurrency limits keep memory peaks within the single-server budget.

Summary

The plan reuses 512GB per step, keeps smaller workloads in In-Memory, and saves larger intermediates through Arrow or Parquet. Container exit releases op memory.

Configuration changes planned

The next planned step is to define podman-compose.yml CPU and memory profiles and Dagster executor settings. The surrounding Airbyte OSS, Great Expectations, OpenMetadata and MLflow services would follow the same policy.