@@ -0,0 +1,96 @@ |
| 1 | +# AMD ROCm support |
| 2 | + |
| 3 | +DLM supports AMD GPUs via ROCm as a **Tier 2** backend: LoRA training |
| 4 | +and inference work, but QLoRA is refused and the CI coverage is |
| 5 | +weaker than the CUDA path. |
| 6 | + |
| 7 | +## What works |
| 8 | + |
| 9 | +- **Training**: LoRA on bf16-capable AMD GPUs. |
| 10 | +- **Inference**: `dlm prompt` uses the standard PyTorch path; MLX and |
| 11 | + bitsandbytes are not involved. |
| 12 | +- **Export**: llama.cpp produces GGUF quantized weights. For |
| 13 | + ROCm-accelerated quantization, rebuild llama.cpp with HIP (see |
| 14 | + below); otherwise the default CPU build is used. |
| 15 | + |
| 16 | +## Supported GPUs |
| 17 | + |
| 18 | +| Generation | Arch codes | Example SKUs | bf16 | FA2 (via `flash_attn`) | |
| 19 | +|------------|------------------|-----------------------------|------|------------------------| |
| 20 | +| CDNA2 | `gfx90a` | Instinct MI200/MI210/MI250 | yes | yes | |
| 21 | +| CDNA3 | `gfx942` | Instinct MI300 | yes | yes | |
| 22 | +| RDNA3 | `gfx1100`/`1101`/`1102` | RX 7900 XTX/XT/7800/7700 | yes | yes (experimental) | |
| 23 | +| RDNA4 | `gfx1200`/`1201` | RX 9000-series | yes | varies | |
| 24 | +| RDNA2 | `gfx1030`/`1031` | RX 6000-series | no | no | |
| 25 | +| CDNA1 | `gfx908` | MI100 | no | no | |
| 26 | +| Vega20 | `gfx906` | Radeon VII / MI50 | no | no | |
| 27 | + |
| 28 | +The bf16-capable allowlist is enforced in `dlm.hardware.capabilities` |
| 29 | +based on `torch.cuda.get_device_properties(0).gcnArchName`. |
| 30 | +Unsupported arches fall back to fp16 (still functional, just slower |
| 31 | +per token on weight-heavy layers). |
| 32 | + |
| 33 | +## What doesn't work |
| 34 | + |
| 35 | +**QLoRA is refused on ROCm.** `bitsandbytes` ROCm builds are |
| 36 | +upstream-unstable — 4-bit quantized matmuls silently return wrong |
| 37 | +values on several arch/driver combinations. We refuse the |
| 38 | +combination rather than risk corrupt gradients. Use `adapter: lora` |
| 39 | +in your `.dlm` frontmatter. |
| 40 | + |
| 41 | +**Multi-GPU ROCm** is out of scope for this sprint. Sprint 23's |
| 42 | +multi-GPU work targets CUDA first; ROCm multi-GPU is a follow-on. |
| 43 | + |
| 44 | +## Software prerequisites |
| 45 | + |
| 46 | +- **ROCm** ≥ 5.7; 6.0+ preferred. We test against 6.0 and 6.2. |
| 47 | +- **PyTorch** with HIP build — install via the ROCm wheels from |
| 48 | + pytorch.org. The `torch.version.hip` attribute must be non-None. |
| 49 | +- **FlashAttention 2 (optional)**: AMD's `flash_attn` fork is the |
| 50 | + package name on ROCm. Install for CDNA (MI200/MI300); RDNA3 |
| 51 | + support is experimental. If `flash_attn` is not importable or the |
| 52 | + arch is not on the allowlist, SDPA is used instead. |
| 53 | + |
| 54 | +## Determinism posture |
| 55 | + |
| 56 | +The doctor reports `determinism_class: best-effort` on ROCm. ROCm's |
| 57 | +deterministic kernels exist but are not as thorough as CUDA's; fp |
| 58 | +match may drift across PyTorch/ROCm upgrades even with a pinned |
| 59 | +seed. |
| 60 | + |
| 61 | +## Rebuilding llama.cpp with ROCm |
| 62 | + |
| 63 | +The default vendored llama.cpp binary is CPU-only. Build a ROCm |
| 64 | +version once for faster quantization: |
| 65 | + |
| 66 | +```bash |
| 67 | +# Set your GPU arch |
| 68 | +export AMDGPU_TARGETS="gfx1100" # RDNA3 |
| 69 | +# export AMDGPU_TARGETS="gfx90a" # MI200 |
| 70 | +# export AMDGPU_TARGETS="gfx942" # MI300 |
| 71 | + |
| 72 | +scripts/build-llama-cpp-rocm.sh |
| 73 | +``` |
| 74 | + |
| 75 | +The script writes to `vendor/llama.cpp/build-rocm/`. To make |
| 76 | +`dlm export` prefer this build, point the runner at it: |
| 77 | + |
| 78 | +```bash |
| 79 | +export DLM_LLAMA_CPP_BUILD=vendor/llama.cpp/build-rocm |
| 80 | +``` |
| 81 | + |
| 82 | +(Environment-variable plumbing in `dlm.export.vendoring` lands as |
| 83 | +part of the next ROCm polish pass — for now, manually invoke the |
| 84 | +ROCm binaries if you need them.) |
| 85 | + |
| 86 | +## CI / testing |
| 87 | + |
| 88 | +No default ROCm CI runner exists. Contributors with ROCm hardware can |
| 89 | +run the gated smoke: |
| 90 | + |
| 91 | +```bash |
| 92 | +DLM_ENABLE_ROCM_SMOKE=1 uv run pytest tests/integration/hardware/test_rocm_train_smoke.py -v |
| 93 | +``` |
| 94 | + |
| 95 | +A scheduled self-hosted runner is the intended deployment; contact |
| 96 | +the maintainers if you'd like to host one. |
