Reference-Based 3D-Aware Image Editing with Triplanes

Bahri Batuhan Bilecen1, Yigit Yalin1, Ning Yu2, Aysegul Dundar1
1Bilkent University, Computer Science Department
2Netflix Eyeline Studios
Paper Code arXiv

Our approach excels in reference-based edits, faithfully reproducing the copied reference parts with a single source and reference image. Leveraging 3D-aware triplanes, our edits are versatile and 3D consistent, allowing for rendering from various viewpoints. We show results for human and animal faces, 360-degree human heads, cross-domain edits with cartoon portraits, and virtual try-ons.

Full 360-degree head edits on PanoHead
Human face edits on EG3D
Try-on edits on AG3D

Abstract

Generative Adversarial Networks (GANs) have emerged as powerful tools for high-quality image generation and real image editing by manipulating their latent spaces. Recent advancements in GANs include 3D-aware models such as EG3D, which feature efficient triplane-based architectures capable of reconstructing 3D geometry from single images. However, limited attention has been given to providing an integrated framework for 3D-aware, high-quality, reference-based image editing. This study addresses this gap by exploring and demonstrating the effectiveness of the triplane space for advanced reference-based edits. Our novel approach integrates encoding, automatic localization, spatial disentanglement of triplane features, and fusion learning to achieve the desired edits. Additionally, our framework demonstrates versatility and robustness across various domains, extending its effectiveness to animal face edits, partially stylized edits like cartoon faces, full-body clothing edits, and 360-degree head edits. Our method shows state-of-the-art performance over relevant latent direction, text, and image-guided 2D and 3D-aware diffusion and GAN methods, both qualitatively and quantitatively.

Qualitative Results and Comparisons

Comparisons with the competing editing methods for glasses addition and hair edits. Ours, HisD, VecGAN++, Barbershop, StyleFusion, HairCLIPv2, and Paint by Ex. use reference images for editing. InterFaceGAN, StyleCLIP, SFE, and NoiseCLR use previously calculated latent directions. LEDITS++ and InfEdit use text prompts. N/A indicates the model is incapable of such edits.


Additional editing examples from the CelebA dataset showcasing our method's ability to seamlessly incorporate features such as lips, eyes, and nose from reference to source, despite pose differences and interference like eyeglasses.


BibTeX

@misc{bilecen2024referencebased,
      title={Reference-Based 3D-Aware Image Editing with Triplanes}, 
      author={Bahri Batuhan Bilecen and Yigit Yalin and Ning Yu and Aysegul Dundar},
      year={2024},
      eprint={2404.03632},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}