Abstract: Systems and methods for incorporating machine learning to predict post operative right ventricular failure using echocardiograms are described. In an embodiment, the system obtains echocardiography video data describing a patient's heart, generates several dense trajectory descriptors based on the echocardiography video data, reduces the dense trajectory descriptors to a bag-of-words representation, generates a first prediction metric of RV failure based on the bag-of-words representation, generates a second prediction metric based on the echocardiography video data using a neural network, and generates an output prediction metric by applying a weighted classifier to the first prediction metric and the second prediction metric.