My AMD inference lab
I build the machines, run the workloads, and measure where the time goes.
From hardware enthusiasm to serving experiments
My personal lab includes AMD GPU systems for local inference. I have worked on ROCm kernel tuning as well as runtime configuration. This case study focuses on the serving experiments for which I have preserved benchmark results and a Python harness.
What I tested
In August 2026, I benchmarked Qwen3.8-27B on one and two Radeon AI PRO R9700 32 GB cards. I compared quantization, tensor and layer splitting, speculative decoding depth, occupied context, and concurrent requests. The harness records actual prompt lengths, generation counts, timing, and repeated runs in JSON.
The controlled ROCm 7.14 retest used llama.cpp source commit daef7b6, Q4_K_M weights, Q8 K/V cache, MTP depth 3, and binaries compiled with RCCL enabled and disabled. These are personal-lab measurements on Radeon hardware, not Instinct benchmarks or customer production results.
A faster collective was not a faster interactive server
The non-RCCL tensor fallback produced a median 55.22 generated tokens/second across three short runs. The RCCL plus forced fine-grain PCIe configuration produced 6.86–6.88 tokens/second in two recorded short runs. Both generated 256 tokens; actual prompt lengths differed slightly, at 69 and 64 tokens respectively. This is a useful diagnostic comparison, not a universal speedup claim.
The lab notes recorded a host-bridge path and shared-memory transport rather than direct GPU-to-GPU access. I retained the fallback for interactive serving on this topology rather than assuming that enabling RCCL would improve every workload.
Prefill, decode, and total latency need separate measurements
At roughly 32K occupied context, the tensor fallback recorded 48.00 generated tokens/second and 45.52 seconds total wall time. The layer-split result in the experiment notes had slower decode but lower end-to-end latency because prompt ingestion was faster. Choosing a serving configuration requires measuring the whole request, not just its most flattering rate.
Context capacity is not the same as responsive context
I also tested long-context ingestion, concurrent KV sequences, and out-of-memory limits. The lab notes document cold-prefill costs and host-memory cache-restore delays. Those observations inform how I think about agent context: a context window that allocates successfully is not necessarily one that supports an interactive workload.
What this work demonstrates
Hands-on AMD serving, Python benchmark tooling, controlled build comparisons, and decisions based on measured latency and topology. My next documentation step is to recover the kernel-tuning patches and their before/after measurements so that work can be examined with the same specificity.
Agent systems and operational knowledge · The build in my yearly story