Lvmin Zhang is a key figure in AI-driven computational art, known for his ground-breaking tools like ControlNet, Fooocus, and Style2Paint, as well as his influential research and community engagement. His ability to blend academic rigour with practical applications has earned him global recognition, while his hobby projects like YGOPro2 highlight his creative versatility.
As of April 2025, he is a student in Computer Science at Stanford University, where he has been studying since 2022 under the supervision of Professor Maneesh Agrawala. Before this, he served as a Research Assistant at the Chinese University of Hong Kong from 2021 to 2022, working with Professor Tien-Tsin Wong, and completed his Bachelor of Engineering in Computer Science at Soochow University in 2021, supervised by Professors Yi Ji and Chunping Liu.
Education and Research Positions
Zhang’s academic journey began around 2017 when he likely enrolled in a Bachelor of Engineering program at Soochow University, Suzhou, China, graduating in 2021. During his undergraduate years, he collaborated with Professors Yi Ji and Chunping Liu, laying the foundation for his research in computational art. From 2021 to 2022, he worked as a Research Assistant at the Chinese University of Hong Kong, furthering his expertise in computer graphics and image processing under Professor Tien-Tsin Wong. Since 2022, Zhang has been pursuing a Ph.D. at Stanford University, advised by Professor Maneesh Agrawala, focusing on advanced AI applications in creative domains.
Research Interests and Contributions
Zhang’s research spans computational art and design, interactive content creation, computer graphics, image and video processing, and anime-style artwork. He is particularly noted for developing AI tools that democratise artistic creation, making them accessible to a broad audience. His most significant contribution is ControlNet, a neural network framework for controlling diffusion models, published in 2023 and recognised with the best paper award at the International Conference on Computer Vision (ICCV2023). This framework has been widely adopted for its ability to enhance text-to-image diffusion models.
Other notable projects include Fooocus, a user-friendly web interface based on Stable Diffusion for generating high-quality images from text prompts, released in September 2023. Style2Paint, a framework for colourising sketches with appropriate colour, texture, and gradient, has outperformed state-of-the-art techniques and is one of Zhang’s most popular projects. Additional tools include IC-Light, which generates AI-driven lighting effects; Omost, focused on advanced image manipulation; FramePack, for image-to-video processing; and Paints-UNDO, a tool for reversing and refining painting processes, launched in July 2024.
Notable Projects and Their Impact
Zhang’s projects have gained significant traction, as evidenced by the popularity of his GitHub repositories under the username lllyasviel. The following table summarises his key projects and their impact as of April 2025:
| Project | Description | GitHub Stars |
|---|---|---|
| ControlNet | Neural network for controlling diffusion models, ICCV2023 best paper winner | 32,100 |
| Style2Paint | Framework for sketch colourisation with colour, texture, and gradient | 18,100 |
| FramePack | Tool for image-to-video and frame processing | 9,700 |
| IC-Light | AI-driven lighting effect generator for images | 7,900 |
| Omost | Project for advanced image manipulation | 7,600 |
| Paints-UNDO | Tool for reversing and refining painting effects | 3,900 |
These projects have not only advanced academic research but also found practical applications in creative industries, making Zhang a key figure in AI-driven art.
Publications and Awards
Zhang has authored 13 scientific papers, accumulating 4,996 citations on Google Scholar and 618 highly influential citations on Semantic Scholar. His work has been published in prestigious venues, including ACM Transactions on Graphics. Key publications include:
- Two-stage sketch colourisation (2018), introducing a semi-automatic framework for sketch colourisation.
- Generating digital painting lighting effects via RGB-space geometry (2020), advancing digital painting techniques.
- *Adding conditional
control to text-to-image diffusion models* (2023), detailing the ControlNet framework, which earned the ICCV2023 best paper award.
These publications highlight Zhang’s ability to bridge theoretical research with practical applications, contributing to his reputation as a leading researcher.
Online Presence and Community Engagement
Zhang maintains an active online presence, sharing his work and engaging with the tech community. On Hugging Face, he has over 7,700 followers and has published models such as FramePackI2V_HY and flux_redux_bfl, which are widely used in AI-driven image processing. His GitHub profile hosts 51 repositories, with ControlNet and Style2Paint being among the most starred. He also contributes to platforms like Model Database and OpenReview, where he is recognised for expertise in image processing and diffusion models.
Zhang leads the Style2Paints Research group, a special interest initiative dedicated to advancing AI-driven art and design, as detailed on his personal website. This group fosters collaboration and innovation in computational creativity.
Hobbies and Broader Impact
In his leisure time, Zhang enjoys game development, showcasing his versatility beyond academic research. He is the creator of YGOPro2, a Unity-based card game for the Yu-Gi-Oh! franchise, which gained popularity online around 2018. This hobby project reflects his creative talent and ability to apply technical skills in diverse domains. His interest in anime also informs his research, particularly in developing AI tools for anime-style artwork, further blurring the lines between art and technology.
Zhang’s contributions extend beyond academia, influencing creative industries and empowering users worldwide to engage with AI-driven art. His tools have made complex processes like image generation and sketch colourisation accessible, while his open-source ethos has fostered a collaborative tech community.
History Timeline
Zhang’s career trajectory is marked by significant milestones, as outlined below:
- 2017–2021: Undergraduate Studies at Soochow University Zhang began his Bachelor of Engineering in Computer Science around 2017, graduating in 2021. He published Two-stage sketch colourisation in 2018 and Generating digital painting lighting effects via RGB-space geometry in 2020, both in ACM Transactions on Graphics. During this period, he developed YGOPro2, a Unity card game that became popular by 2018.
- 2021–2022: Research Assistant at the Chinese University of Hong Kong Zhang worked under Professor Tien-Tsin Wong, focusing on computer graphics and image processing, bridging his undergraduate and doctoral studies.
- 2022–Present: Ph.D. Studies at Stanford University Since ascended to Ph.D. Student at Stanford University, advised by Professor Maneesh Agrawala. Key achievements include:
- 2023: Developed ControlNet, winning the best paper award at ICCV2023.
- 2023: Released Fooocus v2, a popular image generation tool, available by September.
- 2024: Launched Paints-UNDO, a tool for reversing drawing processes, shared on 10 July.
Zhang’s work continues to shape the fields of AI and computer graphics, with his tools and research driving innovation in creative technology.
