Abstract: There are provided systems and methods for facial landmark detection using a convolutional neural network (CNN). The CNN comprises a first stage and a second stage where the first stage produces initial heat maps for the landmarks and initial respective locations for the landmarks. The second stage processes the heat maps and performs Region of Interest-based pooling while preserving feature alignment to produce cropped features. Finally, the second stage predicts from the cropped features a respective refinement location offset to each respective initial location. Combining each respective initial location with its respective refinement location offset provides a respective final coordinate (x,y) for each respective landmark in the image. Two-stage localization design helps to achieve fine-level alignment while remaining computationally efficient.

Abstract: There are provided systems and methods for facial landmark detection using a convolutional neural network (CNN). The CNN comprises a first stage and a second stage where the first stage produces initial heat maps for the landmarks and initial respective locations for the landmarks. The second stage processes the heat maps and performs Region of Interest-based pooling while preserving feature alignment to produce cropped features. Finally, the second stage predicts from the cropped features a respective refinement location offset to each respective initial location. Combining each respective initial location with its respective refinement location offset provides a respective final coordinate (x,y) for each respective landmark in the image. Two-stage localization design helps to achieve fine-level alignment while remaining computationally efficient.

Abstract: GANs based generators are useful to perform image to image translations. GANs models have large storage sizes and resource use requirements such that they are too large to be deployed directly on mobile devices. Systems and methods define through conditioning a student GANs model having a student generator that is scaled downwardly from a teacher GANs model (and generator) using knowledge distillation. A semantic relation knowledge distillation loss is used to transfer semantic knowledge from an intermediate layer of the teacher to an intermediate layer of the student. Student generators thus defined are stored and executed by mobile devices such as smartphones and laptops to provide augmented reality experiences. Effects are simulated on images, including makeup, hair, nail and age simulation effects.

Abstract: A system and method implement deep learning on a mobile device to provide a convolutional neural network (CNN) for real time processing of video, for example, to color hair. Images are processed using the CNN to define a respective hair matte of hair pixels. The respective object mattes may be used to determine which pixels to adjust when adjusting pixel values such as to change color, lighting, texture, etc. The CNN may comprise a (pre-trained) network for image classification adapted to produce the segmentation mask. The CNN may be trained for image segmentation (e.g. using coarse segmentation data) to minimize a mask-image gradient consistency loss. The CNN may further use skip connections between corresponding layers of an encoder stage and a decoder stage where shallower layers in the encoder, which contain high-res but weak features are combined with low resolution but powerful features from deeper decoder layers.

Abstract: There is described a deep learning supervised regression based model including methods and systems for facial attribute prediction and use thereof. An example of use is an augmented and/or virtual reality interface to provide a modified image responsive to facial attribute predictions determined from the image.

Abstract: Disruption of computerized face detection includes receiving a source image that contains a representation of a face and computing a perturbation for the source image. The perturbation is specific to the source image and is configured for a target face detector. A perturbed image is then generated by adding the perturbation to the source image and then the perturbed image may be outputted instead of the source image.

Abstract: Techniques are provided for computing systems, methods and computer program products to produce efficient image-to-image translation by adapting unpaired datasets for supervised learning. A first model (a powerful model) may be defined and conditioned using unsupervised learning to produce a synthetic paired dataset from the unpaired dataset, translating images from a first domain to a second domain and images from the second domain to the first domain. The synthetic data generated is useful as ground truths in supervised learning. The first model may be conditioned to overfit the unpaired dataset to enhance the quality of the paired dataset (e.g. the synthetic data generated). A run-time model such as for a target device is trained using the synthetic paired dataset and supervised learning. The run-time model is small and fast to meet the processing resources of the target device (e.g. a personal user device such as a smart phone, tablet, etc.

Abstract: There are provided methods and computing devices using semi-supervised learning to perform end-to-end video object segmentation, tracking respective object(s) from a single-frame annotation of a reference frame through a video sequence of frames. A known deep learning model may be used to annotate the reference frame to provide ground truth locations and masks for each respective object. A current frame is processed to determine current frame object locations, defining object scoremaps as a normalized cross-correlation between encoded object features of the current frame and encoded object features of a previous frame. Scoremaps for each of more than one previous frame may be defined. An Intersection over Union (IoU) function, responsive to the scoremaps, ranks candidate object proposals defined from the reference frame annotation to associate the respective objects to respective locations in the current frame. Pixel-wise overlap may be removed using a merge function responsive to the scoremaps.

Abstract: Provided is a method and system of providing a cosmetics enhancement interface. The method comprises showing, at the display screen of a computing device having a memory and a processor: a digital photograph including facial features; an interactive dialog portion reflecting a conversational input received and a subsequent response provided thereto from the computing device; and a product display portion; receiving an inquiry, as reflected in the interactive dialog portion, related to a cosmetic product for application onto a selected facial feature; receiving a selection of the cosmetic product based on a matching to the at least one facial feature according to a predefined rule; displaying, at the product display portion, a product representation associated with the selected cosmetic product; receiving an update request; and updating the digital photograph showing a modification to the facial feature on the display screen by simulating application of the selected cosmetic product thereon.

Abstract: This document relates to hybrid eye center localization using machine learning, namely cascaded regression and hand-crafted model fitting to improve a computer. There are proposed systems and methods of eye center (iris) detection using a cascade regressor (cascade of regression forests) as well as systems and methods for training a cascaded regressor. For detection, the eyes are detected using a facial feature alignment method. The robustness of localization is improved by using both advanced features and powerful regression machinery. Localization is made more accurate by adding a robust circle fitting post-processing step. Finally, using a simple hand-crafted method for eye center localization, there is provided a method to train the cascaded regressor without the need for manually annotated training data. Evaluation of the approach shows that it achieves state-of-the-art performance.

Abstract: Systems and methods process images to determine a skin condition severity analysis and to visualize a skin analysis such as using a deep neural network (e.g. a convolutional neural network) where a problem was formulated as a regression task with integer-only labels. Auxiliary classification tasks (for example, comprising gender and ethnicity predictions) are introduced to improve performance. Scoring and other image processing techniques may be used (e.g. in assoc. with the model) to visualize results such as highlighting the analyzed image. It is demonstrated that the visualization of results, which highlight skin condition affected areas, can also provide perspicuous explanations for the model. A plurality (k) of data augmentations may be made to a source image to yield k augmented images for processing. Activation masks (e.g. heatmaps) produced from processing the k augmented images are used to define a final map to visualize the skin analysis.

Abstract: A computer-implemented method for correcting a makeup or skin effect to be rendered on a surface region of an image of a portion of a body of a person. The method and system correcting the makeup or skin effect by accounting for image-specific light field parameters, such as a light profile estimate and minimum light field estimation, and rendering the corrected the makeup or skin effect on the image to generate a corrected image.

Abstract: This document relates to hybrid eye center localization using machine learning, namely cascaded regression and hand-crafted model fitting to improve a computer. There are proposed systems and methods of eye center (iris) detection using a cascade regressor (cascade of regression forests) as well as systems and methods for training a cascaded regressor. For detection, the eyes are detected using a facial feature alignment method. The robustness of localization is improved by using both advanced features and powerful regression machinery. Localization is made more accurate by adding a robust circle fitting post-processing step. Finally, using a simple hand-crafted method for eye center localization, there is provided a method to train the cascaded regressor without the need for manually annotated training data. Evaluation of the approach shows that it achieves state-of-the-art performance.

Abstract: Presented is a convolutional neural network (CNN) model for fingernail tracking, and a method design for nail polish rendering. Using current software and hardware, the CNN model and method to render nail polish runs in real-time on both iOS and web platforms. A use of Loss Mean Pooling (LMP) coupled with a cascaded model architecture simultaneously enables pixel-accurate fingernail predictions at up to 640×480 resolution. The proposed post-processing and rendering method takes advantage of the model's multiple output predictions to render gradients on individual fingernails, and to hide the light-colored distal edge when rendering on top of natural fingernails by stretching the nail mask in the direction of the fingernail tip. Teachings herein may be applied to track objects other than fingernails and to apply appearance effects other than color.

Abstract: Provided are systems and methods to perform colour extraction from swatch images and to define new images using extracted colours. Source images may be classified using a deep learning net (e.g. a CNN) to indicate colour representation strength and drive colour extraction. A clustering classifier is trained to use feature vectors extracted by the net. Separately, pixel clustering is useful when extracting the colour. Cluster count can vary according to classification. In another manner, heuristics (with or without classification) are useful when extracting. Resultant clusters are evaluated against a set of (ordered) expected colours to determine a match. Instances of standardized swatch images may be defined from a template swatch image and respective extracted colours using image processing. The extracted colour may be presented in an augmented reality GUI such as a virtual try-on application and applied to a user image such as a selfie using image processing.

Abstract: There are provided systems and methods for facial landmark detection using a convolutional neural network (CNN). The CNN comprises a first stage and a second stage where the first stage produces initial heat maps for the landmarks and initial respective locations for the landmarks. The second stage processes the heat maps and performs Region of Interest-based pooling while preserving feature alignment to produce cropped features. Finally, the second stage predicts from the cropped features a respective refinement location offset to each respective initial location. Combining each respective initial location with its respective refinement location offset provides a respective final coordinate (x,y) for each respective landmark in the image. Two-stage localization design helps to achieve fine-level alignment while remaining computationally efficient.

Abstract: A system and method implement deep learning on a mobile device to provide a convolutional neural network (CNN) for real time processing of video, for example, to color hair. Images are processed using the CNN to define a respective hair matte of hair pixels. The respective object mattes may be used to determine which pixels to adjust when adjusting pixel values such as to change color, lighting, texture, etc. The CNN may comprise a (pre-trained) network for image classification adapted to produce the segmentation mask. The CNN may be trained for image segmentation (e.g. using coarse segmentation data) to minimize a mask-image gradient consistency loss. The CNN may further use skip connections between corresponding layers of an encoder stage and a decoder stage where shallower layers in the encoder, which contain high-res but weak features are combined with low resolution but powerful features from deeper decoder layers.

Abstract: There is provided a framework including systems and methods for analyzing skin parameters from images or videos showing skin. Using a series of Hierarchical Differential Image Filters (HDIF), it becomes possible to detect different skin features such as wrinkles, spots, and roughness. The hierarchical differential image filter computes two enhancements to an image showing skin at two different levels of enhancement, determines a differential image using the two enhancements and computes the skin analysis rating using the differential image. These skin ratings are comparably accurate to actual ratings by dermatologists.

Abstract: A system and method compute, store and use relativity measures between events in datasets where the measures are stored in a relational memory for querying. From user data and e-commerce shopping session data, relativity measures are computed for a plurality of subsets of data attributes of the user data and session data, each subset comprising two or more data attributes. The relativity measures individually or when combined represent conditional relativities between a set of events within the session data. Only the relativity measures are stored to the relational memory. The measures may be queried for results and applied to the e-commerce service (e.g. to determine which specific product data to present or an order of the presentation of the specific product data). The relativity measures may be computed only for pre-selected relations between particular data attributes which give desired trends and insights into user shopping using the e-commerce service.

Abstract: There is shown and described a deep learning based system and method for skin diagnostics as well as testing metrics that show that such a deep learning based system outperforms human experts on the task of apparent skin diagnostics. Also shown and described is a system and method of monitoring a skin treatment regime using a deep learning based system and method for skin diagnostics.