This invention aims to tackle the multi-domain image to image translation task using novel deep learning-based adversarial networks. In particular, this invention can transfer facial attributes (e.g., hair color, gender, age) as well as morphing facial expressions and changing facial attributes arrangement and shapes. This technology provides sharp and realistic results with additional morphable features. 
Technology Overview
This invention uses novel deep learning-based adversarial networks called Segmentation Guided Generative Adversarial Networks, which fully leverage semantic segmentation information to guide the image translation process. This model detects faces from an input image and extracts corresponding semantic segmentations. Then the translation process implements the trained models by using novel deep learning network. 
The proposed SG GAN model consists of three networks i.e., generator, discriminator and segmentor: 
(1) Generator takes a given image, multiple attributes and target segmentation as input to generate a target image.
(2) Discriminator pushes the generated images towards target domain distribution and utilizes an auxiliary attribute classifier to enable SGGAN to generate images with multiple attributes.
(3) Segmentor imposes semantic information on the generation process. This framework is trained using a large data‑set of face images with attribute-level labels.
- It can generate more realistic results with better image quality (sharper and clearer details) after image translation.
- Additional morphing features (face attributes reallocation, changing face shape, making the person gradually smile) are provided in the translation process while no other existing method can achieve the same effects.
- It can generate facial semantic segmentations directly from given input face images that have traditionally been achieved by converting the results from pre-trained face landmark detector.
- Can be applied in many interesting face apps for entertainment that people can change the attributes of their faces such as changing hair color, gender or their age after they upload their photos. 
- Automatic criminal sketch and forensic tools; human tracking, missing children verification and recognition.
- License
- Partnering
- Research collaboration
Patent Information:
For Information, Contact:
Barbara Finer
Northeastern University
Yun Fu
Songyao Jiang