Multi-GPU (AMD) Performance Regression in Ollama with ROCm 6.3.1

### What is the issue? *Description* For the past two weeks, my second and third GPUs have been freezing during inference when running Ollama. After the crash, the VRAM remains full, but the GPUs stop processing. However, the system itself remains stable, and a soft reboot is no longer possible—only a hard reboot restores functionality. Additionally, GPU performance has drastically decreased. Previously, Qwen2.5-Coder 32B ran at ~17 tokens/sec, but now it barely reaches 4 tokens/sec. Even models that fully fit into VRAM are underperforming. System Information CPU: AMD Ryzen Threadripper 3960X RAM: 256 GB DDR4-3200 GPUs: 3 x AMD Radeon RX 7900 XTX OS: Ubuntu 24.04 LTS Server Ollama Version: (latest, post-update) ROCm Version: 6.3.1 (with ROCm-SMI 5.7.0) *Key Findings & Symptoms* **GPU Workload Distribution is Broken** GPUs are recognized but remain idle during inference. Power consumption stays under 100W per GPU, which is far too low. GPU clock speeds (sclk) remain at 0 MHz, meaning no actual computation occurs. CPU utilization is very high (~80-95%), even when models should be running entirely in VRAM. **Specific Models Trigger Freezes** Mistral-Large frequently crashes the GPUs, requiring a hard reboot. Llama3.3-70B is extremely slow but at least remains stable. DeepSeek R1 32B only uses ~50% of VRAM, yet the GPUs remain idle. **Multi-GPU Scaling is Failing** Performance does not improve with multiple GPUs. Even when a model fully fits into VRAM, Ollama does not utilize GPU compute units. Disabling multiple GPUs (CUDA_VISIBLE_DEVICES=0) sometimes improves stability. **Possible ROCm Regressio**n Previously (ROCm 5.7.1 or earlier), everything worked fine. After Ollama updated, ROCm was also upgraded to 6.3.1 automatically. It is unclear whether this issue is caused by Ollama’s inference engine or a ROCm 6.3.1 regression. Downgrading ROCm is not trivial, as Ollama depends on its installed version. **Troubleshooting Attempts** Setting performance mode to compute (rocm…