Paper | Project Page | Demo
PairCustomization.mp4
Added Flux compatibility!!! Check out the Flux README for more info :)
Environment Setup
-
First clone our environment and cd into the corresponding folder (make sure you use python >= 3.10):
git clone https://github.com/PairCustomization/PairCustomization.git cd PairCustomization -
We provide a txt file that contains all the required dependencies.
pip install -r requirements.txt -
This can be used for a virtual environment:
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt -
Alternatively, you can create a conda environment in the following way:
conda create --name PairCustomization conda activate PairCustomization pip install -r requirements.txt
Make sure to confirm that torch.cuda.is_available() is true
Evaluating a model
We provide example trained model weights at this link. They should go in a folder named example_loras inside of the PairCustomization folder
mkdir example_loras
cd example_loras
wget https://www.cs.cmu.edu/~model-weight-storage/lora-dog-digital-art-style.safetensors
wget https://www.cs.cmu.edu/~model-weight-storage/lora-man-painting-style.safetensors
To evaluate a model, run
python evaluation_scripts/evaluate.py
Generating stylized images may take ~1.5x as long due to the updated inference path used for generation, which is detailed in section 3.3 of the paper. If you still experience overfitting to the training image, try lowering the "lora_guidance_scale" term to 3.0
We also allow for multiple LoRAs to be combined via equation (11) in section 3.3. For this funcitonality, run:
python evaluation_scripts/evaluate_multiple_adapters.py
We also allow for controlnet conditioning. For this, run:
python evaluation_scripts/evaluate_with_controlnet.py
Finally, we allow for real image editing via:
python evalution_scripts/evaluate_real_image.py
Training a model
To train a model, first initialize an 🤗Accelerate environment via:
accelerate config
For a default Accelerate environment, you can run:
accelerate config default
The training code is currently set up to train using the following image pair:
curtesy of Jack Parkhouse
To run the training script, you can simply run:
sh training_scripts/train_pair_customization_lora_sdxl.sh
See the sh file itself for more details. Currently training is on the dog_digital_art_style images in the data folder. After training, the model should get stored in the example_loras folder. Training/generating validation images should take ~16 minutes on an NVIDIA A5000 24GiB GPU.