Qwen Image LoRA Training on Apple Silicon: It Works (and Here's How)


Phew, this was a runaround. Ok, what we have and what we know:

Running Qwen Image on Apple Silicon is solved. Ivan Fioravanti's qwen-image-mps has had inference working since August 2025. You can generate images on your Mac right now. That's not what this post is about.

Training it locally — actually fine-tuning LoRA adapters through a 20 billion parameter diffusion transformer on MPS — was not. Every training tool says you need NVIDIA. And getting to the point where I could confirm otherwise was a far messier journey than the clean result suggests.


The Mess

Since OpenAI confirmed DALL-E 2 and 3 API deprecation for May 12, 2026, I'd been researching with Claude to figure out the best path for training a style LoRA — 150+ blog posts need visual consistency going forward, and I'm not handing that to someone else's API.

Before any of this Mac work, I'd spent a full weekend fighting my RTX 5080. I cannot explain how frustrating that was. Claude Code cheerfully told me "just install PyTorch 2.9+ with CUDA 12.8." Sounded clean. It wasn't. Digging into the PyTorch forums and GitHub issues revealed that stable PyTorch builds still don't support sm_120 (the 5080's architecture). You need to build PyTorch from source, or use unstable nightly builds. And the really fun discovery — someone reverse-engineered NVIDIA's drivers with Ghidra and found the driver actively rejects native sm_120 kernel execution, silently falling back to Ada Lovelace code with a 30-40% performance hit. A weekend of kernel fights and yak-shaving before you even start training.

That's why I bought the Mac Studio. 256GB unified memory, no driver politics, no building toolchains from source. Just run things.

So — can I train on Mac? First we looked at DiffSynth-Studio — the reference implementation from Qwen's own team. CUDA only. Someone had already asked for MPS support and got explicitly rejected. Kohya — CUDA only. SimpleTuner — CUDA only. Every tutorial I found — including Furkan Gözükara's comprehensive guide — assumes Windows with an NVIDIA GPU.

Meanwhile, we'd been planning around Qwen Image 2.0 — smaller model (7B vs 20B), better quality, uses a Qwen3-VL encoder. Except the weights aren't public. It launched in February as API-only on Alibaba Cloud's BaiLian platform. There's literally a prediction market on Manifold about when the open weights will drop.

So: the reference tool rejected Mac support. The Windows GPU path was a CUDA toolchain nightmare. The model I actually wanted wasn't available for local training. Every direction I turned, something was blocked.


The Hunch

Somewhere in the middle of all this, Claude did what I should have done from the start — read the source code instead of the README.

The HuggingFace Diffusers training script for Qwen Image (train_dreambooth_lora_qwen_image.py) had explicit torch.backends.mps.is_available() checks buried in it. A bf16 blocking path for MPS. AMP disable flags for Metal. Someone on the Diffusers team had clearly thought about Apple Silicon. But I couldn't find any public confirmation that Qwen Image LoRA training actually works on MPS.

So I said... well... shall we just try? Claude Code and I designed a quick experiment: try it, document where it breaks, share the results either way.


Attempt 1: The Wrong Precision

First run, we used --mixed_precision="fp16" because that's what the README examples suggest. The model loaded. The VAE loaded. The latent caching worked. Training started.

Every step: loss=nan.

But here's the thing — the speed was shocking. 4 seconds per step. I'd expected 5–30 minutes based on earlier community reports of SDXL LoRA training on M3 Max taking 10 minutes per step. Qwen Image is roughly 6x larger than SDXL. We were braced for pain. We got 4 seconds of NaN.

The hardware was fine. The precision was the problem.

Why fp16 fails: the script disables AMP gradient scaling on MPS. Without a gradient scaler, fp16 gradients from a 20B model are tiny enough to underflow to zero. On CUDA, AMP handles this transparently. On MPS, it's a dead end.


Attempt 2: One Flag

We changed one thing:

--mixed_precision="no"

That's it. fp32 instead of fp16. Everything else identical.

Step 1:  loss: 0.150  (8.71s — warmup)
Step 2:  loss: 0.183  (4.62s)
Step 3:  loss: 0.122  (4.27s)
Step 4:  loss: 0.814  (4.88s)
Step 5:  loss: 0.098  (3.47s)

Real loss. Decreasing trend. Valid LoRA weights saved to disk.

No MPS fallback warnings appeared in our runs, and the full training loop completed successfully on Metal — which is the part I couldn't find anyone publicly confirming before this.

After all that research into CUDA toolchains, driver shenanigans, and API-only models — the answer was on the Mac the whole time. One flag.


What This Actually Means

If you have an Apple Silicon Mac with enough memory, you can train Qwen Image LoRAs right now. No cloud GPU. No NVIDIA card. No rental fees. No building PyTorch from source. The complete loop — curate data, train adapter, generate images — runs locally on a single machine.

Steps Time (3 images) Time (237 images) What That Gets You
100 ~8 min ~18 min Quick concept test
500 ~37 min ~92 min Standard LoRA
1000 ~75 min ~3 hrs Extended training

Speed scales with dataset size — our initial smoke test with 3 images ran at ~4.5s/step, but a real 237-image dataset runs at ~11s/step. Still firmly in "start it, grab coffee" territory.

On this setup, the fp32 penalty is surprisingly small. With unified memory and no PCIe hop between CPU and GPU memory, the slowdown was far less dramatic than I expected. The compute difference between fp16 and fp32 multiply-accumulate is real but small relative to the memory access patterns in a transformer.


How to Reproduce This

You'll need

  • Apple Silicon Mac (I used M3 Ultra, 256GB — lower memory configs untested)
  • Python 3.10+
  • PyTorch 2.10+ with MPS support (earlier versions like 2.6.0 crash with LLVM errors — MPS kernel coverage improves significantly with each release)
  • Diffusers installed from git (the training script isn't in the PyPI release)

Setup

# Check MPS is available
python3 -c "import torch; print(f'MPS: {torch.backends.mps.is_available()}')"

# Install diffusers from source
git clone https://github.com/huggingface/diffusers.git
cd diffusers && pip install -e .

# Install dependencies
pip install accelerate transformers peft
cd examples/dreambooth && pip install -r requirements_qwen_image.txt

Do not install bitsandbytes — it's CUDA-only and will cause import errors.

Pin transformers to 4.57.1 — older versions lack Qwen2_5_VLForConditionalGeneration, newer ones have a to_dict bug that breaks the pipeline. This will likely resolve as both libraries update, but right now: pip install transformers==4.57.1.

The command that works

export MODEL_NAME="Qwen/Qwen-Image"
export INSTANCE_DIR="./training_images"
export OUTPUT_DIR="./output"
export PYTORCH_ENABLE_MPS_FALLBACK=1

accelerate launch train_dreambooth_lora_qwen_image.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --mixed_precision="no" \
  --instance_prompt="a photo of sks person" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --learning_rate=2e-4 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --offload \
  --cache_latents \
  --seed="0"

The flags that matter

  • --mixed_precision="no" — Required. fp16 gives NaN loss. bf16 is blocked on MPS. fp32 is the only path.
  • --offload — Moves the text encoder and VAE to CPU when not in use. Reduces memory pressure.
  • --cache_latents — Pre-computes VAE encodings for your training images. Removes the VAE from the training loop.
  • PYTORCH_ENABLE_MPS_FALLBACK=1 — Safety net for any operations without Metal kernels. Never triggered in our tests, but set it anyway.

First run downloads the full model (~30 minutes). After that it loads from cache.


What I Haven't Done Yet

I'm being honest about where I'm at. We've confirmed training runs with both a small smoke test (3 images) and a full 237-image dataset — loss decreases, weights save, no MPS fallbacks. What I haven't yet done:

  • Evaluated the quality of generated images from the trained LoRA
  • Tested at 1024px resolution
  • Tested on machines with less than 256GB memory

Those are next. But the foundational question — can you train Qwen Image LoRAs on Apple Silicon at all — is answered. Yes. And it's fast.


Known Limitations

  • fp32 only. fp16 produces NaN loss (AMP gradient scaling is disabled on MPS). bf16 is blocked at the hardware level. You must train in fp32.
  • PyTorch 2.10+ required. Earlier versions (tested 2.6.0) crash with LLVM ERROR: Failed to infer result type(s) — the MPS matmul kernels simply aren't there yet. Use the latest stable or nightly.
  • Pin transformers==4.57.1. Older versions miss the Qwen2.5-VL class, newer ones have a to_dict serialisation bug. This window will shift as both libraries update.
  • No 8-bit Adam. bitsandbytes is CUDA-only. Standard AdamW only — fine with 256GB, may matter on smaller machines.
  • Untested below 256GB. The 20B model in fp32 is large. Whether 64GB or 96GB machines can handle it with offloading is unknown.

If You Hit loss=nan

You're using --mixed_precision="fp16". Change it to --mixed_precision="no". That's it. The script disables AMP gradient scaling on MPS, so fp16 gradients underflow to zero and everything collapses. fp32 fixes it completely.


The Landscape Right Now

For anyone searching for this (because I certainly was):

Inference (image generation) on Apple Silicon:

Tool Status
qwen-image-mps Works — mature, actively maintained
qwen-image-macos Works — Lightning LoRA acceleration
HuggingFace Diffusers Works

Training (LoRA fine-tuning) on Apple Silicon:

Tool Status
DiffSynth-Studio No — CUDA only, MPS support rejected
Kohya / sd-scripts No — CUDA only
SimpleTuner No — CUDA only
fal.ai Cloud API — works but not local
HuggingFace Diffusers Yes — fp32, with offload and cache_latents

If you try this on your own hardware, I'd love to hear about it — especially on different memory configurations. The MPS backend is maturing faster than the community seems to realise.


Experiment run: 2026-02-21 (smoke test), 2026-02-22 (full 237-image dataset). Mac Studio M3 Ultra, 256GB unified memory, PyTorch 2.10.0, Diffusers 0.37.0.dev0, transformers 4.57.1. Research and pipeline development done in collaboration with Claude (claude.ai and Claude Code).