Abstract: A method and system to generate a probabilistic prediction of the presence/absence of cancer in longitudinal and current image datasets, and/or multimodal image datasets, and the location of the cancer, is described. The method and system uses an ensemble of deep learning models. The ensemble includes a global model in the form of a 3D convolutional neural network (CNN) extracting features in the datasets indicative of the presence of cancer on a global basis. The ensemble also includes a two-stage prediction model which includes a first stage or detection model which identifies cancer detection candidates (different cropped volumes of 3D data in the a dataset containing candidates which may be cancer) and a second stage or probability model which incorporates the longitudinal datasets (or multimodal images in a multimodal dataset) and the extracted features from the global model and assigns a cancer probability p to each of the cancer detection candidates.