Abstract: Embodiments provide systems, methods, and computer storage media for generating a 3D model from a target 2D image or 3D point cloud (e.g., generated by a 3D scan). Given a particular target, a retrieval network retrieves or identifies a source model from a database, and a deformation network deforms the source model to fit the target. In some cases, joint learning is employed to enable the retrieval and deformation networks to jointly learn a deformation-aware retrieval embedding space and an individualized deformation space for each source model. In some cases, the retrieval network retrieves based on distance in the deformation-aware retrieval embedding space, enabling the retrieval module to retrieve a source model that best fits to the target after deformation. In some cases, a deformation is decomposed into a plurality of per-part deformations, and/or and the retrieval embedding space is used to select training data.