Abstract: Described herein are systems, methods, and instrumentalities associated with processing mammogram images using machine learning based techniques. An apparatus as described herein may obtain a mammographic image of a person, extract features from the mammographic image using a feature encoder, and predict a health condition of the person based on the extracted features. The feature encoder may be trained using a self-supervised technique and based on multi-view mammogram images that may belong to a same person or to different people.

Abstract: Described herein are systems, methods, and instrumentalities associated with automatic determination of hemodynamic characteristics. An apparatus as described may implement a first artificial neural network (ANN) and a second ANN. The first ANN may model a mapping from a set of 3D points associated with one or more blood vessels to a set of hemodynamic characteristics of the one or more blood vessels, while the second ANN may generate, based on a geometric relationship of the set of points in a 3D space, parameters for controlling the mapping. The apparatus may obtain a 3D anatomical model representing at least one blood vessel of a patient based on one or more medical images of the patient, and determine, based on the first ANN and the second ANN, a hemodynamic characteristic of the at least one blood vessel of the patient at a target location of the 3D anatomical model.

Abstract: Disclosed herein are deep-learning based systems, methods, and instrumentalities for medical decision-making. A system as described herein may implement an artificial neural network (ANN) that may include multiple encoder neural networks and a decoder neural network. The multiple encoder neural networks may be configured to receive multiple types of patient data (e.g., text and image based patient data) and generate respective encoded representations of the patient data. The decoder neural network (e.g., a transformer decoder) may be configured to receive the encoded representations and generate a medical decision, a medical summary, or a medical questionnaire based on the encoded representations. In examples, the decoder neural network may be configured to implement a large language model (LLM) that may be pre-trained for performing the aforementioned tasks.

Abstract: Described herein are systems, methods, and instrumentalities associated with image segmentation such as tubular structure segmentation. An artificial neural network is trained to segment tubular structures of interest in a medical scan image based on annotated images of a different type of tubular structures that may have a different contrast and/or appearance from the tubular structures of interest. The training may be conducted in multiple stages during which a segmentation model learned from the annotated images during a first stage may be modified to fit the tubular structures of interest in a second stage. In examples, the tubular structures of interest may include coronary arteries, catheters, guide wires, etc., and the annotated images used for training the artificial neural network may include blood vessels such as retina blood vessels.

Abstract: Described herein are systems, methods, and instrumentalities associated with medical image enhancement. The medical image may include an object of interest and the techniques disclosed herein may be used to identify the object and enhance a contrast between the object and its surrounding area by adjusting at least the pixels associated with the object. The object identification may be performed using an image filter, a segmentation mask, and/or a deep neural network trained to separate the medical image into multiple layers that respectively include the object of interest and the surrounding area. Once identified, the pixels of the object may be manipulated in various ways to increase the visibility of the object. These may include, for example, adding a constant value to the pixels of the object, applying a sharpening filter to those pixels, increasing the weight of those pixels, and/or smoothing the edge areas surrounding the object of interest.

Abstract: The present disclosure provides a system and method for fetus monitoring. The method may include obtaining ultrasound data relating to a fetus collected by an ultrasound imaging device; generating a 4D image of the fetus based on the ultrasound data; directing a display component of a virtual reality (VR) device to display the 4D image to an operator; detecting motion of the fetus based on the ultrasound data; and directing a haptic component of the VR device to provide haptic feedback with respect to the motion to the operator.

Abstract: An apparatus may obtain a sequence of medical images of a target structure and determine, using a first ANN, a first segmentation and a second segmentation of the target structure based on a first medical image and a second medical image, respectively. The first segmentation may indicate a first plurality of pixels that may belong to the target structure. The second segmentation may indicate a second plurality of pixels that may belong to the target structure. The apparatus may identify, using a second ANN, a first subset of true positive pixels among the first plurality of pixels that may belong to the target structure, and a second subset of true positive pixels among the second plurality of pixels that may belong to the target structure. The apparatus may determine a first refined segmentation and a second refined segmentation of the target structure based on the true positive pixels.

Abstract: Described herein are systems, methods, and instrumentalities associated with processing complex-valued MRI data using a machine learning (ML) model. The ML model may be learned based on synthetically generated MRI training data and by applying one or more meta-learning techniques. The MRI training data may be generated by adding phase information to real-valued MRI data and/or by converting single-coil MRI data into multi-coil MRI data based on coil sensitivity maps. The meta-learning process may include using portions of the training data to conduct a first round of learning to determine updated model parameters and using remaining portions of the training data to test the updated model parameters. Losses associated with the testing may then be determined and used to refine the model parameters. The ML model learned using these techniques may be adopted for a variety of tasks including, for example, MRI image reconstruction and/or de-noising.

Abstract: Disclosed herein are systems, methods, and instrumentalities associated with magnetic resonance (MR) image reconstruction. An under-sampled MR image may be reconstructed through an iterative process (e.g., over multiple iterations) based on a machine-learning (ML) model. The ML model may be obtained through a reinforcement learning process during which the ML model may be used to predict a correction to an input MR image of at least one of the multiple iterations, apply the correction to the input MR image to obtain a reconstructed MR image, determine a reward for the ML model based on the reconstructed MR image, and adjust the parameters of the ML model based on the reward. The reward may be determined using a pre-trained reward neural network and the ML model may also be pre-trained in a supervised manner before being refined through the reinforcement learning process.

Abstract: A magnification system for magnifying an image based on trained neural networks is disclosed. The magnification system receives a first user input associated with a selection of a region of interest (ROI) within an input image of a site and a second user input associated with a first magnification factor of the selected ROI. The first magnification factor is associated with a magnification of the ROI in the input image. The ROI is modified based on an application of a first neural network model on the ROI. The modification of the ROI corresponds to a magnified image that is predicted in accordance with the first magnification factor. A display device is controlled to display the modified ROI.

Abstract: The motion estimation of an anatomical structure may be performed using a machine-learned (ML) model trained based on medical training images of the anatomical structure and corresponding segmentation masks for the anatomical structure. During the training of the ML model, the model may be used to predict a motion field that may indicate a change between a first training image and a second training image, and to transform the first training image and a corresponding first segmentation mask based on the motion field. The parameters of the ML model may then be adjusted to maintain a correspondence between the transformed first training image and the second training image and between the transformed first segmentation mask or a second segmentation mask associated with the second training image. The correspondence may be assessed based on at least a boundary region shared by the anatomical structure and one or more other anatomical structures.

Abstract: Disclosed herein are systems, methods, and instrumentalities associated with cardiac motion tracking and/or analysis. In accordance with embodiments of the disclosure, the motion of a heart such as an anatomical component of the heart may be tracked through multiple medical images and a contour of the anatomical component may be outlined in the medical images and presented to a user. The user may adjust the contour in one or more of the medical images and the adjustment may trigger modifications of motion field(s) associated with the one or more medical images, re-tracking of the contour in the one or more medical images, and/or re-determination of a physiological characteristic (e.g., a myocardial strain) of the heart. The adjustment may be made selectively, for example, to a specific medical image or one or more additional medical images selected by the user, without triggering a modification of all of the medical images.

Abstract: Described herein are systems, methods, and instrumentalities associated with denoising medical images such as fluoroscopic images using deep learning techniques. A first artificial neural network (ANN) is trained to denoise an input medical image in accordance with a provided target noise level. The training of the first ANN is conducted by pairing a noisy input image with target denoised images that include different levels of noise. These target denoised images are generated using a second ANN as intermediate outputs of the second ANN during different training iterations. As such, the first ANN may learn to perform the denoising task in an unsupervised manner without requiring noise-free training images as the ground truth.

Abstract: An apparatus for stent visualization includes a hardware processor that is configured to input one or more stent images from a sequence of X-ray images and corresponding balloon marker location data to a cascaded spatial transform network. The background is separated from the one or more stent images using the cascaded spatial transform network and a transformed stent image with a clear background and a non-stent background image is generated. The stent layer and non-stent layer are generated using a neural network without online optimization. A mapping function f maps the inputs, the sequence images and marker coordinates, into the two single image outputs.

Abstract: Described herein are systems, methods, and instrumentalities associated with using a multi-layer perceptron (MLP) neural network to process medical images of an anatomical structure. The processing may include padding an input image in accordance with the training of the MLP neural network, splitting the input image (e.g., the padded input image) into patches of a same size, and processing the patches through the MLP neural network over one or more iterations. During an iteration of the processing, the patches may be processed separately and re-combined into an intermediate image before the intermediate image is shifted to concatenate portions of the image that are derived from different patches. This way, global features of the anatomical structure may be learned and used to improve the quality of the image generated by the MLP neural network, without incurring significant computation or memory costs.

Abstract: Described herein are systems, methods, and instrumentalities associated with reconstruction of multi-contrast magnetic resonance imaging (MRI) images. The reconstruction may be performed based on under-sampled MRI data collected for the multiple contrasts using corresponding sampling patterns. The sampling patterns and the reconstruction operations for the multiple contrasts may be jointly optimized using deep learning techniques implemented through one or more neural networks. An end-to-end reconstruction optimizing framework is provided with which information collected while processing one contrast may be stored and used for another contrast. A differentiable sampler is described for obtaining the under-sampled MRI data from a k-space and a novel holistic recurrent neural network is used to reconstruct MRI images based on the under-sampled MRI data.

Abstract: Mammography may provide multi-view information about the healthy state of a person's breasts, and described herein are deep learning based techniques for obtaining mammographic images associated with multiple views, extracting cross-view features from the images, and automatically determining the existence or non-existence of a medical abnormality based on the extracted features. The cross-view features may be determined using an encoding module of an artificial neural network (ANN) and the encoded features may be decoded using a decoding module of the ANN to generate a prediction (e.g., a classification label, a bounding shape, a segmentation mask, a probability map, etc.) about the medical abnormality.

Abstract: Deblurring and denoising a medical image such as X-ray fluoroscopy images may be challenging, and deep-learning based techniques may be employed to meet the challenge. An artificial neural network (ANN) may be trained using training images with synthetic noise and as well as training images with real noise. The parameters of the ANN may be adjusted during the training based on at least a first loss designed to maintain continuity between consecutive medical images generated by the ANN and a second loss designed to maintain similarity of patches inside a medical image generated by the ANN. The parameters of the ANN may be further adjusted based on a third loss that may be calculated from ground truth associated with the synthetic training images. Transfer learning between the synthetic images and the real images may be accomplished using these techniques.

Abstract: Described herein are systems, methods, and instrumentalities associated with automatically annotating a tubular structure (e.g., such as a blood vessel, a catheter, etc.) in medical images. The automatic annotation may be accomplished using a machine-learning image annotation model and based on a marking of the tubular structure created or confirmed by a user. A user interface may be provided for a user to create, modify, and/or confirm the marking, and the ML model may be trained using a training dataset that comprises marked images of the tubular structure paired with ground truth annotations of the tubular structure.

Abstract: A neural network system implements a model for generating an output image based on a received input image. The model is learned through a training process during which parameters associated with the model are adjusted so as to maximize a difference between a first image predicted using first parameter values of the model and a second image predicted using second parameter values of the model, and to minimize a difference between the second image and a ground truth image. During a first iteration of the training process the first image is predicted and during a second iteration the second image is predicted. The first parameter values are obtained during the first iteration by minimizing a difference between the first image and the ground truth image, and the second parameter values are obtained during the second iteration by maximizing the difference between the first image and the second image.