Ximing Xing1

Chuang Wang1

Haitao Zhou1

1Beihang University   2The University of Hong Kong

  *Corresponding author

Paper status - Accepted by NIPS 2023 🎉🎉🎉

Visualizations of the vector sketches generated by our proposed method, DiffSketcher. For each example, we show two sketches with a different number of strokes.


Even trained mainly on images, we discover that pretrained diffusion models show impressive power in guiding sketch synthesis. In this paper, we present DiffSketcher, an innovative algorithm that creates vectorized free-hand sketches using natural language input. DiffSketcher is developed based on a pre-trained text-to-image diffusion model. It performs the task by directly optimizing a set of Bézier curves with an extended version of the score distillation sampling (SDS) loss, which allows us to use a raster-level diffusion model as a prior for optimizing a parametric vectorized sketch generator. Furthermore, we explore attention maps embedded in the diffusion model for effective stroke initialization to speed up the generation process. The generated sketches demonstrate multiple levels of abstraction while maintaining recognizability, underlying structure, and essential visual details of the subject drawn. Our experiments show that DiffSketcher achieves greater quality than prior work.



DiffSketcher generates free-hand sketches from a text prompt such as "A photo of Eiffel Tower". To synthesize a sketch that matches the given text prompt, we optimize the parameters of the differentiable rasterizer R that produces the raster sketch S, such that the resulting sketch is close to a sample from the frozen latent diffusion model (the blue part of the picture). Since the diffusion model directly predicts the update direction, we do not need to backpropagate through the diffusion model; the model simply acts like an efficient, frozen critic that predicts image-space edits.



Macaw full color, ultra detailed, realistic,
insanely beautiful

Very detailed masterpiece painting of baby
yoda hoding a lightsaber

Sailboat sailing in the sea on a clear day

More Results





Painterly Rendering

Colorful Results


oil painting



  title={DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models},
  author={Xing, Ximing and Wang, Chuang and Zhou, Haitao and Zhang, Jing and Yu, Qian and Xu, Dong},
journal={arXiv preprint arXiv:2306.14685},