File size: 8,794 Bytes
79ccf3a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
---
license: cc-by-nc-4.0
---
# ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text
(March. 2025)
Fundemental issue for this repository: [ColorizeDiffusion (e-print)](https://arxiv.org/abs/2401.01456).
Version 1 - trained with 512px (WACV 2025): [ColorizeDiffusion](https://openaccess.thecvf.com/content/WACV2025/html/Yan_ColorizeDiffusion_Improving_Reference-Based_Sketch_Colorization_with_Latent_Diffusion_Model_WACV_2025_paper.html) Basic reference-based training. Released.
Version 1.5 - trained with 512px (CVPR 2025): [ColorizeDiffusion 1.5 (e-preprint)](https://arxiv.org/html/2502.19937v1) Solving spatial entangelment. Released.
Version 2 - trained with 768px, paper and code: Enhancing background and style transfer. Available soon.
Version XL - trained with 1024px : Enhancing embedding guidance for character colorization, geometry disentanglement. Ongoing.
Model weights are available: https://huggingface.co/tellurion/colorizer.
## Implementation Details
The repository offers the implementation of ColorizeDiffusion.
Now, only the noisy model introduced in the paper, which utilizes the local tokens.
## Getting Start
To utilize the code in this repository, ensure that you have installed the required dependencies as specified in the requirements.
### To install and run:
```shell
conda env create -f environment.yaml
conda activate hf
```
## User Interface:
We also provided a Web UI based on Gradio UI. To run it, just:
```shell
python -u app.py
```
Then you can browse the UI in http://localhost:7860/.
### Inference:
-------------------------------------------------------------------------------------------
#### Important inference options:
| Options | Description |
|:--------------------------|:----------------------------------------------------------------------------------|
| Mask guide mode | Activate mask guided attention and corresponding lora weights for colorization. |
| Crossattn scale | Used to diminish all kinds of artifacts caused by the distribution problem. |
| Pad reference with margin | Used to diminish spatial entanglement, pad reference to T times of current width. |
| Reference guidance scale | Classifier-free guidance scale of the reference image, suggested 5. |
| Sketch guidance scale | Classifier-free guidance scale of the sketch image, suggested 1. |
| Attention injection | Strengthen similarity with reference. |
| Visualize | Used for local manipulation. Visualize the regions selected by each threshold. |
For artifacts like spatial entanglement (the distribution problem discussed in the paper) like this
Please activate background enhance (optionally with foreground enhance).
### Manipulation:
The colorization results can be manipulated using text prompts.
For local manipulations, a visualization is provided to show the correlation between each prompt and tokens in the reference image.
The manipulation result and correlation visualization of the settings:
Target prompt: the girl's blonde hair
Anchor prompt the girl's brown hair
Control prompt the girl's brown hair,
Target scale: 8
Enhanced: false
Thresholds: 0.5、0.55、0.65、0.95
As you can see, the manipluation unavoidably changed some unrelated regions as it is taken on the reference embeddings.
#### Manipulation options:
| Options | Description |
| :----- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Group index | The index of selected manipulation sequences's parameter group. |
| Target prompt | The prompt used to specify the desired visual attribute for the image after manipulation. |
| Anchor prompt | The prompt to specify the anchored visaul attribute for the image before manipulation. |
| Control prompt | Used for local manipulation (crossattn-based models). The prompt to specify the target regions. |
| Enhance | Specify whether this manipulation should be enhanced or not. (More likely to influence unrelated attribute). |
| Target scale | The scale used to progressively control the manipulation. |
| Thresholds | Used for local manipulation (crossattn-based models). Four hyperparameters used to reduce the influnece on irrelevant visual attributes, where 0.0 < threshold 0 < threshold 1 < threshold 2 < threshold 3 < 1.0. |
| \<Threshold0 | Select regions most related to control prompt. Indicated by deep blue. |
| Threshold0-Threshold1 | Select regions related to control prompt. Indicated by blue. |
| Threshold1-Threshold2 | Select neighbouring but unrelated regions. Indicated by green. |
| Threshold2-Threshold3 | Select unrelated regions. Indicated by orange. |
| \>Threshold3 | Select most unrelated regions. Indicated by brown. |
|Add| Click add to save current manipulation in the sequence. |
## Code reference
1. [Stable Diffusion v2](https://github.com/Stability-AI/stablediffusion)
2. [Stable Diffusion XL](https://github.com/Stability-AI/generative-models)
3. [SD-webui-ControlNet](https://github.com/Mikubill/sd-webui-controlnet)
4. [Stable-Diffusion-webui](https://github.com/AUTOMATIC1111/stable-diffusion-webui)
5. [K-diffusion](https://github.com/crowsonkb/k-diffusion)
6. [Deepspeed](https://github.com/microsoft/DeepSpeed)
7. [sketchKeras-PyTorch](https://github.com/higumax/sketchKeras-pytorch)
## Citation
```
@article{2024arXiv240101456Y,
author = {{Yan}, Dingkun and {Yuan}, Liang and {Wu}, Erwin and {Nishioka}, Yuma and {Fujishiro}, Issei and {Saito}, Suguru},
title = "{ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text}",
journal = {arXiv e-prints},
year = {2024},
doi = {10.48550/arXiv.2401.01456},
}
@InProceedings{Yan_2025_WACV,
author = {Yan, Dingkun and Yuan, Liang and Wu, Erwin and Nishioka, Yuma and Fujishiro, Issei and Saito, Suguru},
title = {ColorizeDiffusion: Improving Reference-Based Sketch Colorization with Latent Diffusion Model},
booktitle = {Proceedings of the Winter Conference on Applications of Computer Vision (WACV)},
year = {2025},
pages = {5092-5102}
}
@article{2025arXiv250219937Y,
author = {{Yan}, Dingkun and {Wang}, Xinrui and {Li}, Zhuoru and {Saito}, Suguru and {Iwasawa}, Yusuke and {Matsuo}, Yutaka and {Guo}, Jiaxian},
title = "{Image Referenced Sketch Colorization Based on Animation Creation Workflow}",
journal = {arXiv e-prints},
year = {2025},
doi = {10.48550/arXiv.2502.19937},
}
``` |