Line Drawings for Face Portraits from Photos using Global and Local Structure based GANs
Published in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
Abstract: Despite significant effort and notable success of neural style transfer, it remains challenging for highly abstract styles, in particular line drawings. In this paper, we propose APDrawingGAN++, a generative adversarial network (GAN) for transforming face photos to artistic portrait drawings (APDrawings), which addresses substantial challenges including highly abstract style, different drawing techniques for different facial features, and high perceptual sensitivity to artifacts. To address these, we propose a composite GAN architecture that consists of local networks (to learn effective representations for specific facial features) and a global network (to capture the overall content). We provide a theoretical explanation for the necessity of this composite GAN structure by proving that any GAN with a single generator cannot generate artistic styles like APDrawings. We further introduce a classification-and-synthesis approach for lips and hair where different drawing styles are used by artists, which applies suitable styles for a given input. To capture the highly abstract art form inherent in APDrawings, we address two challenging operations-(1) coping with lines with small misalignments while penalizing large discrepancy and (2) generating more continuous lines-by introducing two novel loss terms: one is a novel distance transform loss with nonlinear mapping and the other is a novel line continuity loss, both of which improve the line quality. We also develop dedicated data augmentation and pre-training to further improve results. Extensive experiments, including a user study, show that our method outperforms state-of-the-art methods, both qualitatively and quantitatively.
Recommended citation: Ran Yi, Mengfei Xia, Yong-Jin Liu*, Yu-Kun Lai, Paul L. Rosin. Line Drawings for Face Portraits from Photos using Global and Local Structure based GANs. IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI (identifier) 10.1109/TPAMI.2020.2987931, 2020.