DGX Spark's Unified Memory: Capacity vs. Speed

A single NVIDIA DGX Spark solves the capacity half of this problem and not the speed half: its 128GB of coherent LPDDR5x memory is shared between the GB10 Grace CPU and Blackwell GPU over NVLink-C2C, so model weights, a growing KV cache, and multi-agent state all live in one address space instead of fighting for a 24–32GB VRAM budget. What it does not do is make a large model generate tokens quickly once you actually use that headroom — decode throughput is capped by 273 GB/s of memory bandwidth, and that number doesn’t change no matter how much context you’re holding.

Skip to the picks

What the unified pool actually changes

On a discrete GPU, the model has to fit in VRAM alongside the KV cache, and when it doesn’t, you either quantize harder, truncate the context, or spill to system RAM over PCIe — the exact “performance collapse” a long agent session runs into. DGX Spark removes that specific failure mode by giving the CPU and GPU one coherent 128GB pool instead of a small dedicated one: there’s no PCIe copy between “system memory” and “GPU memory” because there’s only one memory. ServeTheHome reports the Grace-to-Blackwell NVLink-C2C die-to-die link on the GB10 superchip running at roughly 600 GB/s aggregate — well above the 273 GB/s the LPDDR5x itself delivers to either processor, which is the detail that matters here: the interconnect isn’t the bottleneck, the memory technology behind it is.

That’s why NVIDIA’s own product page lists local support for models “up to 200 billion parameters” and fine-tuning up to 70B on a single unit — the ceiling is what fits in 128GB, not what the interconnect can move.

The number that matters more than capacity: 273 GB/s

LMSYS’s benchmarks of a single DGX Spark show exactly where that bandwidth ceiling shows up. Llama 3.1 8B (FP8) under SGLang ran 7,991 tokens/sec prefill with 20.5 tokens/sec decode at batch size 1, scaling to 368 tokens/sec decode at batch 32. Llama 3.1 70B (FP8) — a model that actually needs the 128GB pool to fit — dropped to 803 tokens/sec prefill and 2.7 tokens/sec decode. GPT-OSS 120B in MXFP4 loaded and ran in Ollama, because it fits; it wasn’t fast for the same reason 70B wasn’t.

The pattern holds across every source that’s benchmarked it: DGX Spark is compute-bound and competitive on prefill, and memory-bandwidth-bound on decode the moment a model gets big enough to need the 128GB it’s built for. Speculative decoding narrows that gap — LMSYS measured up to a 2x end-to-end speedup from EAGLE3 on supported models — but it doesn’t erase it.

Budgeting the KV cache so a long session doesn’t collapse

For “massive context” specifically, the practical constraint isn’t the model — it’s how much room is left over for the KV cache once the model is loaded. A d-central.tech configuration guide for running 70B-class models on a single 128GB unit gives a concrete rule of thumb: a Llama 3.3 70B model at Q4 quantization takes roughly 42GB, a 32K-token context adds another 10–14GB of KV cache at 16-bit precision, and every additional 32K of context costs roughly another 8–14GB depending on the model’s layer count and GQA configuration. That leaves close to 70GB of headroom at a 32K context on a 70B model — plenty of room to go further, but the guide’s own advice is to move past 128K context with llama.cpp’s explicit context offload rather than stretching Ollama’s default handling, and to avoid loading two large models with big contexts simultaneously (OLLAMA_MAX_LOADED_MODELS=1) rather than letting the system silently thrash.

vLLM’s own deployment notes for DGX Spark make the same point from the serving side. Because the OS, container runtime, model weights, and KV cache all draw from the same 128GB pool, the recommended --gpu-memory-utilization is a conservative 0.85, leaving explicit room for everything else touching that memory. Running Nemotron-3-Super-120B-A12B-NVFP4 at --max-model-len 131072 (a 128K-token context), vLLM held decode throughput in a steady 22.7–23.7 tokens/sec band across test scenarios, with prefill scaling from 140 to nearly 1,900 tokens/sec depending on prompt length and time-to-first-token ranging from 0.42 to 3.85 seconds. That’s the direct answer to “does this collapse under massive context”: a 120B-class model at a 128K context stayed in a narrow, predictable throughput band rather than falling off a cliff — provided you keep --max-num-seqs low (4, in that configuration), since pushing more concurrent decode streams lets the per-token bandwidth tax outweigh whatever continuous-batching would otherwise gain.

The same source’s monitoring showed KV-cache utilization staying under 30% even under small-batch demo traffic, and its concrete advice for agent loops that keep growing is to compact the conversation before KV-cache usage nears the configured limit, rather than letting the session run until it truncates on its own.

When one Spark isn’t enough

Two units solve a different problem than context length — they solve model size. A single 200G QSFP56 direct-attach cable between the two ConnectX-7 ports (no switch required) pools the memory into 256GB, which multiple independent setup guides put at roughly 405B-parameter capacity depending on quantization; NVIDIA’s own multi-node clustering documentation extends that to “up to 700 billion parameters” across four linked units. One two-unit cluster benchmark, running Qwen3.5-122B-A10B at Q4_K_M, measured about 70 tokens/sec prefill and 20 tokens/sec generation with a 65,536-token context, with the model weights split proportionally across both nodes. A separate configuration running DeepSeek V4 Flash on the same two-unit link reported holding up to a 1M-token context at roughly 40–45 tokens/sec in a single session — the clearest existing demonstration that the “massive context” half of this question scales past a single box, provided the model and quantization are chosen to fit the pooled 256GB.

Getting there requires llama.cpp built with distributed RPC support (-DGGML_RPC=ON), not Ollama — Ollama has no distributed-RPC path across nodes, so a cluster’s master node runs llama-server and offloads to a worker node’s rpc-server over the 200Gb/s link instead. A second unit is worth adding when you’re routinely hitting the ceiling of a single 128GB pool — a dense model past roughly 70B with meaningful context, or a large MoE model — not as a default upgrade for a single agent running an 8–30B model, where the extra node and RPC setup add operational complexity for no throughput gain.

Scaling further than two nodes changes the caching strategy too: a goodput-optimization discussion on NVIDIA’s developer forums, tuning agent-coding workloads across two Sparks with LMCache, found that different model sizes needed different cache tiers — smaller models did well with peer-to-peer KV cache transfer over the fabric, while larger ones needed disk-backed cache as a second tier rather than relying on RAM pooling alone. That’s a genuine caveat: pooling memory across nodes buys capacity, but sustaining throughput as sessions grow into the hundreds of thousands of tokens still takes cache-tier tuning, not just more RAM.

Where a single high-bandwidth GPU is the better tool instead

None of this makes a fast discrete GPU the wrong choice in general — it makes it the wrong choice for this specific problem. A 32GB card like the RTX 5090 delivers 1,792 GB/s of GDDR7 bandwidth, roughly 6.5x DGX Spark’s 273 GB/s, and it will out-run a Spark on any model that comfortably fits in 32GB. The failure mode shows up exactly where this query’s “massive context” requirement lives: a 32GB VRAM ceiling has to hold both the model weights and the KV cache at once, and a large model with a genuinely long agent-loop context — tens to hundreds of thousands of tokens — simply doesn’t fit at any reasonable quantization. That forces the truncation or aggressive KV-cache eviction that unified memory exists to avoid. If you’re deciding between capacity and raw speed for models in the 8–20GB range, 18GB or 32GB for Local LLMs? Where 18GB Runs Out covers where that smaller VRAM budget actually runs out; for a 70B+ model with a long-running agentic session, that ceiling arrives much sooner than the model’s parameter count would suggest.

Software stack: what ships and what you still tune

DGX Spark ships running NVIDIA’s DGX OS with the AI software stack — including NIM microservices and CUDA — preinstalled, and vLLM, llama.cpp, SGLang, and Ollama all run on it without modification for single-node use. The choice between them matters for exactly the workload this question describes: Ollama is the simplest path for a single large model with a bounded context (set num_ctx deliberately, budget roughly 8–14GB of KV cache per additional 32K tokens, and cap OLLAMA_MAX_LOADED_MODELS at 1 when running anything near the 70B range), while vLLM’s paged KV cache and explicit --gpu-memory-utilization and --max-num-seqs controls give more predictable behavior once context regularly pushes past 128K tokens or multiple agent sessions run concurrently. Multi-node clustering requires stepping outside Ollama entirely to llama.cpp’s RPC backend, which is worth knowing before committing to a two-unit purchase on the assumption that the simpler single-node tooling will carry over unchanged.