Memory Estimation Heuristics

Module

pclean.utils.memory_estimate — heuristic RAM estimator for parallel CASA imaging workers.

API

Function

Purpose

estimate_worker_memory_gib()

Peak RAM for a single worker given imsize, nchan_per_task, gridder, deconvolver, nterms, nfields

estimate_peak_ram_gib()

Total system RAM for N concurrent workers (adds 0.5 GiB scheduler overhead)

recommend_nworkers()

Max workers that fit in available RAM (auto-detected or supplied), with a configurable safety factor

Core Model

Calibrated from IRC+10216 ALMA Band 6 cube-imaging logs (8000 × 8000, 40 antennas, 449 280 rows, gridder='standard', deconvolver='hogbom', 1 channel per task):

$$\text{mem_per_worker} = 0.7;\text{GiB} + \frac{n_x \cdot n_y \cdot n_\text{chan} \cdot 76;\text{B/pix} \cdot f_\text{gridder} \cdot n_\text{terms}^2}{2^{30}}$$

where:

  • 0.7 GiB — Python + Dask worker process baseline overhead (constant per worker).

  • 76 B/pixel/channel — empirical constant for the standard gridder, accounting for ~8–10 float/complex buffers CASA keeps simultaneously:

Buffer

Dtype

Bytes/pixel

Complex visibility grid

complex64

8

Weight grid

complex64

8

FFT workspace (in + out)

complex64

16

Residual image

float32

4

Model image

float32

4

PSF image

float32

4

Weight image (sumwt)

float32

4

Primary beam (PB)

float32

4

Mask

float32

4

Temporary / bookkeeping

mixed

~20

Scaling Factors

  • Mosaic gridder (f_gridder = 2.0): each pointing requires a convolution function (CF) table; memory also grows sub-linearly with the number of fields as 1 + 0.1 × (√nfields − 1).

  • MTMFS deconvolver: internal Hessian products scale as nterms². With nterms=1, memory matches hogbom.

  • Multi-channel sub-cubes: linear in nchan_per_task.

  • MS row count: negligible — visibilities are processed in row chunks occupying a few MB, dwarfed by multi-GiB image grids.

Gridder multipliers

Gridder

Factor

standard

1.0

wproject

1.2

widefield

1.3

mosaic

2.0

awproject

2.5

Calibration Point

From the IRC+10216 reference run:

4.9 GiB / (8000 × 8000 × 1 chan) ≈ 76 B/pix/chan

Each worker consumed ~4.9 GiB of C++ (unmanaged) memory + ~0.7 GiB Python/Dask overhead = ~5.6 GiB total.

Example Usage

from pclean.utils.memory_estimate import (
    estimate_worker_memory_gib,
    estimate_peak_ram_gib,
    recommend_nworkers,
)

# Single worker: 8000×8000, standard gridder, 1 chan
mem = estimate_worker_memory_gib(imsize=8000, nchan_per_task=1)
# ≈ 5.2 GiB

# 12 workers total system RAM
total = estimate_peak_ram_gib(nworkers=12, imsize=8000, nchan_per_task=1)
# ≈ 63 GiB

# How many workers fit in 64 GiB?
n = recommend_nworkers(available_ram_gib=64.0, imsize=8000)
# → 10

Tests

19 unit tests in tests/test_memory_estimate.py covering:

  • IRC+10216 calibration point validation (4.5–6.5 GiB range)

  • Scalar vs rectangular imsize handling

  • Linear nchan_per_task scaling

  • Mosaic > standard memory, multi-field sub-linear growth

  • MTMFS nterms² scaling, nterms=1 ≡ hogbom

  • Unknown gridder defaults to standard factor

  • Small images dominated by base overhead

  • recommend_nworkers with auto-detected RAM, safety factor control