This is a how-to article on Stable Diffusion.
In this article, we will introduce how to learn LoRA with FLUX using the Kohya LoRA GUI.
■ I sell and generate AI content in my profile
circle “AI Love create”.
We have received many requests for generation from individuals and other circles through crowdsourcing.
We send out useful information from the images we have actually generated and the work we have experienced.
If you subscribe to a membership (500 yen per month), you can read member-only articles and paid articles under 300 yen as much as you want.
Specs required for Flux LoRA training
It seems that Gravo needs to be able to support BF16 at VRAM10GB or more.
The specs seem to be fine if you clear the above, but please check the requirements on github.
https://github.com/RedRayz/Kohya_lora_param_gui/releases/tag/v1.13.2.2
Installing Kohya LoRA GUI
Before you can learn LoRA with Flux, you need to install the Kohya LoRA GUI.
The installation method is explained in a separate article, so please refer to this article.
Preparation before learning LoRA with FLUX
Preparation of models, clips, and VAEs
If you have already installed Flux, please skip it.
When learning LoRA, it is necessary to set the model, clip, and VAE respectively.
If you haven’t downloaded it, please download it from the respective link.
If you can specify the file, you can put it anywhere, but for the time being, I also introduce the path to put it by UI.
・Flux model
dev:https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/flux1-dev.safetensors
Schnell: https://huggingface.co/black-forest-labs/FLUX.1-schnell/blob/main/flux1-schnell.safetensors
The path of the model depends on the UI.
forge: C:\stable-diffusion-webui-forge\models\Stable-diffusion
ComfyUI: C:\ComfyUI_windows_portable\ComfyUI\models\unet
copy
・ CLIP
clip_l is required, and either t5xxl is required.
comfyanonymous/flux_text_encoders at mainWe’re on a journey to advance and democratize artificial intehuggingface.co
・VAE
ae.safetensors · black-forest-labs/FLUX.1-schnell at mainWe’re on a journey to advance and democratize artificial intehuggingface.co
Please place forge in the following path for both CLIP and VAE.
C:\stable-diffusion-webui-forge\models\VAE
copy
ComfyUI is as follows:
clip: C:\ComfyUI_windows_portable\ComfyUI\models\clip
VAE: C:\ComfyUI_windows_portable\ComfyUI\models\vae
copy
Regenerating a venv
Regenerate the venv with the tools in the Kohya LoRA GUI.
You need to install Torch 2.4, so delete the venv folder in the sd-scripts folder and then start the Kohya LoRA GUI.
Open Tools → Utilities from the menu bar.
Check Install Torch2.4 and click on venv regeneration.
Once the installation is complete, the regeneration of the venv is complete.
Switching Branches
Switch the branch on the sd-scripts side to sd3.
Branches allow you to keep track of versions and work on different topics.
In the main branch, there is no training file “flux_train_network.py” for flux, so flux cannot be learned.
Currently, the files for flux and sd3 are developed separately, so if you switch to that branch, you will be able to learn.
First, open a command prompt and navigate to the sd-scripts path.
cd C:\sd-scripts
copy
All that’s left to do is run the command to switch branches.
git checkout sd3
copy
This will allow the files in the sd3 branch to be used locally.
If you want to change it back to main, do the following:
git checkout main
copy
Installing dependencies
It seems that the requirements.txt has been updated, so I will install it as well.
Open a command prompt and navigate to the sd-scripts path to activate the virtual environment.
cd C:\sd-scripts
./venv/scripts/activate
copy
Install the dependencies with the following command:
pip install -r requirements.txt
copy
This will install the packages necessary for learning flux.
How to learn LoRA in flux using Kohya LoRA GUI
Prepare images and captions
・About images
I’m not sure what the cause is, but I couldn’t learn if the image size was 512×512.
Therefore, it is necessary to prepare 512×512 or more images.
It seems that the number of sheets is about 10 ~ 20.
・Caption data
How to create caption data is explained in a separate article, so please refer to here.
Configuring the Kohya LoRA GUI
I don’t understand the settings at all, and I’m referring to the settings of a person named muooon.
Presets are also distributed, so if you want to start learning quickly, please download the presets from this link and import them.
Flux LoRA Guide 01 | LowVRAM | RedRayz / Kohya_lora_param_gui | CivitaiFlux LoRA Guide 01 RedRayz / Kohya_lora_param_gui @ RTX-3060civitai.com
The following is a summary of the parts that I set up.
・Main UI
- Learning source model: FLUX model Teacher
- Image folder: Path above the image
- Resolution: 512
- Module type: LoRA
- Model type: FLUX.1
- Optimizer: Prodigy
・Detailed settings・Page 3
- Loss function type: L2 (default)
– Detailed settings and performance
- Gradient checkpointing: Check
- Mixed accuracy: Load bf16
- model in fp8: Check
- Caching Text Encoder output: Check
- Cache Text Encoder output to disk: Check
・ Detailed settings / Path
- VAE：ae.safetensors
- CLIP L：clip_l.safetensors
- T5XXL：t5xxl_fp8_e4m3fn.safetensors
・Detailed settings, etc.
- Accuracy at store: FP16
・Detailed settings・DiT
Split mode: block to check
and learn: single only
The number of learning depends on the person, but it seems that it can be reproduced to some extent even with around 1000 steps.
Results of LoRA training
It took about 1.4 hours with 1200 steps with the same settings as muooon.
Also, in forge, it was not applied well even if LoRA was used, so it was generated by ComfyUI.
This time, we studied 10 images of Boa Hancock reproduced in Animagine XL.
・Images generated by Flux using LoRA
It’s a little blurry and the texture isn’t good, but I feel like it’s a pretty good feeling in terms of reproducibility.
However, when using LoRA, I couldn’t generate characters, which is a feature of flux, so I may be doing something strange.
I think there is room for various adjustments, such as parameters and weight at the time of learning.
For the time being, I think that LoRA learning itself can be done with this, so if you are interested, please give it a try.
This is how to learn flux LoRA with the Kohya LoRA GUI.
In this circle, we disseminate various information about AI.
If you subscribe to a membership, you can read some paid articles as much as you want.
If you are interested in improving AI technology, monetizing methods, etc., please consider it.
