DiffuSculpt

LogoSmall

This project focuses on the development of virtual terrains and explores the potential for automating their creation process while preserving the user's degree of freedom in design, enabling the realization of their desired ideas. In the 21st century, virtual objects, characters, scenes, and entire worlds have become ubiquitous. Digital landscapes permeate various aspects of our lives, ranging from product advertisements that transport viewers to fantastical realms, to films featuring mythical creatures and knights, and science fiction video games that enable free exploration of distant planets. However, the creation process for these virtual worlds is often labor-intensive and requires specialized knowledge.

This projects investigates the prospective advancement of automation and accessibility in this domain, by creating a website that would allow users to quickly generate various terrains using only prompts and additional sliders and buttons.

Project was done using a multitude of technologies and is made up of several parts:

  • Hyper-tuned Stable diffusion version 1.5 to generate images using PyTorch
  • X4 Stable diffusion upscaler to create more detailes images using PyTorch
  • uniquely trained img2img GAN model to generate heightmaps using tensorflow 2
  • an another unique GAN model to create generate using tensorflow 2
  • a novel python script that combines all the data to create a GLTF model representation
  • Website coded using Next.js stack
  • Visualization of the model entirely developed using Three-Fiber libraries

Video showcasing the capabilities of this project: