Abstract: Aspects of the disclosed technology include computer-implemented systems and methods for machine-learned multimodal models. A machine-learned multimodal model includes one or more embedding layers configured to generate one or more image tokens and one or more text tokens in response to the imagery and the text, a transformer encoder configured to receive the one or more image tokens and the one or more text tokens and generate one or more fused image tokens and one or more fused text tokens, a heatmap predictor configured to obtain the one or more fused image tokens and generate at least one image heatmap, and a sequence predictor configured to obtain the one or more fused image tokens and the one or more fused text tokens and generate a predicted sequence associated with the image.

Abstract: Aspects of the disclosed technology include computer-implemented systems and methods for machine-learned multimodal models for feedback predictions for synthetic content. A machine-learned multimodal model is configured to generate a feature map based at least in part on fusion of image information and text information from a synthetic image and a text prompt. The model is configured to generate a set of text tokens based at least in part on fusion of the image information and the text information. The model is configured to generate at least one misalignment or implausibility heatmap based at least in part on the at least one feature map. The model is configured to generate at least one predicted misalignment sequence based at least in part on the set of text tokens.

Abstract: Improved methods are provided for generating heatmaps or other summary map data from multiple users' data (e.g., probability distributions) in a manner that preserves the privacy of the users' data while also generating heatmaps that are visually similar to the ‘true’ heatmap. These methods include decomposing the average of the users' data (the ‘true’ heatmap) into multiple different spatial scales, injecting random noise into the data at the multiple different spatial scales, and then reconstructing the privacy-preserving heatmap based on the noisy multi-scale representations. The magnitude of the noise injected at each spatial scale is selected to ensure preservation of privacy while also resulting in heatmaps that are visually similar to the ‘true’ heatmap.

Abstract: Improved methods are provided for generating heatmaps or other summary map data from multiple users' data (e.g., probability distributions) in a manner that preserves the privacy of the users' data while also generating heatmaps that are visually similar to the ‘true’ heatmap. These methods include decomposing the average of the users' data (the ‘true’ heatmap) into multiple different spatial scales, injecting random noise into the data at the multiple different spatial scales, and then reconstructing the privacy-preserving heatmap based on the noisy multi-scale representations. The magnitude of the noise injected at each spatial scale is selected to ensure preservation of privacy while also resulting in heatmaps that are visually similar to the ‘true’ heatmap.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for gaze position prediction using neural networks. One of the systems includes a neural network comprising one or more neural network layers, wherein the neural network is configured to obtain a collection of input facial images of a user, wherein the collection of input facial images of the user comprises (i) a query image of the user, (ii) one or more calibration images of the user, and (iii) a respective calibration label that labels a known gaze position of the user for each of the one or more calibration images of the user; and process the received collection of input facial images of the user using the one or more neural network layers to generate a neural network output that characterizes a gaze position of the user in the query image.

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for gaze position prediction using neural networks. One of the systems includes a neural network comprising one or more neural network layers, wherein the neural network is configured to obtain a collection of input facial images of a user, wherein the collection of input facial images of the user comprises (i) a query image of the user, (ii) one or more calibration images of the user, and (iii) a respective calibration label that labels a known gaze position of the user for each of the one or more calibration images of the user; and process the received collection of input facial images of the user using the one or more neural network layers to generate a neural network output that characterizes a gaze position of the user in the query image.

Abstract: Multimedia features extracted from display advertisements may be integrated into a click prediction model for improving click prediction accuracy. Multimedia features may help capture the attractiveness of ads with similar contents or aesthetics. Numerous multimedia features (in addition to user, advertiser and publisher features) may be utilized for the purposes of improving click prediction in ads with limited or no history.

Abstract: Evaluating a web design includes: receiving input that includes page elements; deriving a plurality of key page elements from the input; running a saliency model on the input to derive hot spots representing those items that are most likely to initially grab (or obtain) a viewer's attention; comparing positions of the hot spots to placement of the plurality of the key page elements to determine effectiveness of the placement of the key page elements; and presenting a saliency map depicting the hot spots in the page elements.