Abstract: The present disclosure relates to analysis of perturbed subjects using semantic embeddings. One example embodiment includes a method. The method includes applying a respective perturbation to each of a plurality of subjects in a controlled environment. The method also includes producing a respective visual representation for each of the perturbed subjects using at least one imaging modality. Further, the method includes obtaining, by a computing device for each of the respective visual representations, a corresponding semantic embedding associated with the respective visual representation. The semantic embedding associated with the respective visual representation is generated using a machine-learned, deep metric network model. In addition, the method includes classifying, by the computing device based on the corresponding semantic embedding, each of the visual representations into one or more groups.