Guide: Running Kohya_ss GUI on Windows 11 with Nvidia RTX 50-Series

[TensorKaze]

Guide: Setting Up and Running Kohya_ss GUI (bmaltais fork) on Windows 11 with Nvidia RTX 50-Series (like RTX 5090)

Yo, this guide’s for getting Kohya_ss GUI running on Nvidia RTX 50-series GPUs. These can throw issues like the “CUDA no kernel image is available…” error with standard PyTorch versions. The fix? A manual setup with specific versions, based on community tips and some trial-and-error.

A. Stuff You Gotta Install First

Make sure you’ve got ALL this installed:

-Python: Version 3.10.11, no exceptions. Grab it from python.org if you don’t have it. When installing, check the box for “Add Python 3.10 to PATH.”

-CUDA Toolkit: Go for version 12.8. Kohya might mention 12.4, but for PyTorch Nightly and newer GPUs, 12.8 seems to work better. Download it from NVIDIA’s archive and install.

-cuDNN: Get the latest version that plays nice with CUDA 12.8 from NVIDIA’s site (you’ll need a free developer account). Follow NVIDIA’s guide to install it, usually just copying some .dll, .h, and .lib files to the CUDA Toolkit folder (like C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.8).

-Git: Download and install from git-scm.com. Stick with the default settings.

-Microsoft Visual C++ Redistributables: Get the x64 versions for 2015-2022 from Microsoft’s site.

-Nvidia Drivers: Install the latest Studio Driver (preferred) or Game Ready Driver for your RTX 5090 from NVIDIA’s site. Restart your PC after.

B. Manual Setup Steps

-Make a Folder for the Install:
Open a terminal (CMD or PowerShell, admin mode’s best).

mkdir C:\Kohya_ss
cd C:\Kohya_ss

-Clone the Kohya_ss Repo (bmaltais fork):

git clone https://github.com/bmaltais/kohya_ss.git .

-Find Your Python 3.10.11 Path:
You’ll need this for the next step. In a new terminal, try:

py -3.10 -c "import sys; print(sys.executable)"
Or search “Python 3.10” in Windows, right-click -> Open file location (maybe twice), and copy the path to python.exe. Example: C:\Users\YourUser\AppData\Local\Programs\Python\Python310\python.exe.

-Set Up a Virtual Environment (.venv):
Using the Python path you found:

"C:\Path\To\Your\Python3.10\python.exe" -m venv .venv

-Activate the Virtual Environment:
In the main terminal where you’re working:

..venv\Scripts\activate
Check that your prompt now starts with (.venv).

ATTENTION: GitHub’s editor shows ..venv instead of the correct path (it removes the backslash before .venv). Type it as dot-backslash-dot-venv to get it right. Big thanks to @jmgmcm for catching this.

-Update Pip:

python -m pip install --upgrade pip

-Install PyTorch Nightly (Super Important!):
This grabs the preview version that works with RTX 50-series and CUDA 12.8.

pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128

-Tweak requirements.txt (Don’t Skip This!):

Open C:\Kohya_ss\requirements.txt in Notepad.
Find the last line (-e ./sd-scripts).
Add a # at the start: #-e ./sd-scripts.
Save it.

-Install Kohya’s Dependencies for Windows:

pip install -r requirements_windows.txt

-Install a Specific Version of Accelerate:

pip install -U accelerate==0.30.0

-Install bitsandbytes:

pip install -U bitsandbytes

--(Optional) Check Versions:
Run:

pip list | findstr "torch accelerate bitsandbytes"
You should see torch as 2.8.0.dev... (or some nightly version), accelerate as 0.30.0, and bitsandbytes.

C. Post-Install Tweaks

-Set Up Accelerate:
Make sure the venv is still active (shows (.venv)). Run:

accelerate config
Answer: This machine (0), No distributed training (0), NO, NO, NO, all (or 0), bf16.

-Edit gui.bat:

Open C:\Kohya_ss\gui.bat in Notepad.
Add --noverify (space, two dashes, noverify) to the end of the two lines that run python.exe kohya_gui.py %*.
Save it.

D. How to Launch Kohya_ss GUI the Right Way

Don’t use gui-uv.bat.
Don’t use .bat files that call gui-uv.bat or mess up the venv.

-Here’s what works:
Open a terminal in C:\Kohya_ss.

-Activate the venv:

..venv\Scripts\activate

ATTENTION: GitHub’s editor shows ..venv instead of the correct path (it removes the backslash before .venv). Type it as dot-backslash-dot-venv to get it right.

-Run the edited script:

gui.bat

-Open http://127.0.0.1:7860 in your browser.

E. GUI Setup Tip

When you’re in the Kohya interface setting up LoRA training:

-Go to the Training parameters tab -> Advanced section.
Find the Memory attention or Cross Attention optimization option.
Pick sdpa instead of xformers. We didn’t install xformers, and sdpa is PyTorch’s built-in option, seems more stable with nightly versions and new GPUs, according to the community.

F. You’re Good to Go!

If you followed these steps, your Kohya_ss setup should be ready to roll on your RTX 50-Series.

[PonyCrotTitLover]

this doesn't install sd scripts?

[TensorKaze]

The guide has a step to comment out (#) the line -e ./sd-scripts in the main requirements.txt. This is because that specific pip instruction for an "editable install" can fail if the sd-scripts folder isn't set up as a formal Python package (e.g., missing a setup.py at its root), which was the case in the specific setup for RTX 50-series needing PyTorch Nightly.

Commenting it out prevents that pip error and isn't strictly necessary for Kohya_ss to use the scripts, as they are called directly by path.

However, you could certainly try leaving that line uncommented. If your pip install -r requirements.txt runs through without error with that line active, then it's fine! If it errors out saying sd-scripts isn't a Python project, then you'll know you need to comment it out for your setup. Either way, the scripts are present from the clone.

[PonyCrotTitLover]

cloning does not seem to clone sd scripts for me so i just copied it over from my old install also ty out of the three others i tried this guide actually worked for my 5090

[ClaudeDidIt]

This is the first workflow that worked for me on my 5060 Ti with kohya. I needed to download the sd scripts and put them into the folder. I somehow had another venv inside my repo that I had to delete (maybe I already started a false batch script). I do not understand why I always have to run .venv\Scripts\activate before I run the gui.bat (because the command is already inside) but it works now fine! Thank you for sharing! :)

[TensorKaze]

Glad the guide worked for your 5060 Ti!
Regarding activate before gui.bat, you're right, gui.bat also calls activate. However, for some users (including during intense troubleshooting for this guide on a 5090), explicitly activating the venv in the terminal first seemed to provide a more stable environment, avoiding some elusive ModuleNotFound errors that only appeared with certain launch methods. Consider it a "just in case" step for maximum reliability based on that experience, but if running gui.bat directly from a fresh terminal works consistently for you, that's perfectly fine too!

[jmgmcm]

I have followed the steps, and it starts perfectly. However, I still have not tested training.

Just a small comment:
The step to activate the venv:

..venv\Scripts\activate

should be
..\.venv\Scripts\activate ( it seems the Git text editor removes the \ when followed by . )

[jmgmcm]

There are also mismatches in the gui.bat in venv path due to the same reason in the creation:
-Set Up a Virtual Environment (.venv):
Using the Python path you found:
"C:\Path\To\Your\Python3.10\python.exe" -m venv .venv

[TensorKaze]

@jmgmcm Thanks for the comment! You’re right, GitHub removes the \ when followed by dot, and I hadn’t noticed that. Glad it starts perfectly, and thanks for spotting the gui.bat mismatch too. Really appreciate the help! Let me know how training goes. Cheers!