Abstract: A method of generating an augmented reality lens comprises: causing to display a list of lens categories on a display screen of a client device; receiving a user choice from the displayed list; causing to prepopulate a lens features display on the display device based on the user choice, wherein each lens feature comprises image transformation data configured to modify or overlay video or image data; receiving a user selection of a lens feature from the prepopulated lens display; receiving a trigger selection that activates the lens feature to complete the lens; and saving the completed lens to a memory of a computer device.

Abstract: Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, an image generated by a camera of a user; displaying an option to crop an object depicted in the image, determining whether the object is depicted in the image; selectively enabling selection of the option to crop the object based on determining whether the object is depicted in the image; receiving input that selects the option to crop an object depicted in the image when the option is enabled; in response to receiving the input, segmenting the image to separate the object from other portions depicted in the image; extracting the object from the image; and adding the extracted object to a new image as a virtual object.

Abstract: A surgical system for use in a surgical procedure is disclosed. The surgical system includes at least one imaging device and a control circuit configured to identify an anatomical organ targeted by the surgical procedure, generate a virtual three-dimensional (3D) construct of at least a portion of the anatomical organ based on visualization data from the at least one imaging device, identify anatomical structures relevant to the surgical procedure from the visualization data from the at least one imaging device, couple the anatomical structures to the virtual 3D construct, and overlay onto the virtual 3D construct a layout plan of the surgical procedure determined based on the anatomical structures.

Abstract: A surgical system for use in a surgical procedure is disclosed. The surgical system includes at least one imaging device and a control circuit configured to identify an anatomical organ targeted by the surgical procedure, generate a virtual three-dimensional (3D) construct of at least a portion of the anatomical organ based on visualization data from the at least one imaging device, identify anatomical structures relevant to the surgical procedure from the visualization data from the at least one imaging device, couple the anatomical structures to the virtual 3D construct, and overlay onto the virtual 3D construct a layout plan of the surgical procedure determined based on the anatomical structures.

Abstract: An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

Abstract: Aspects of the invention relate to systems, devices and methods for performing a surgical step or surgical procedure with visual guidance using an optical head mounted display.

Abstract: A display apparatus is provided. The display apparatus includes a flexible display configured to display an image, a driver configured to change a screen size of the flexible display, and a processor configured to control the driver to display an image corresponding to a content, at a screen size corresponding to a size of a content to be displayed in the flexible display.

Abstract: The invention relates to a viewing system for use in a surgical environment. Various real object detection devices detect locations of real objects in a real environment, such as a patient and body part of patient, medical staff, robots, a cutting tool on a robot, implant transferred by robot into body part, surgical tools, and disposable items. A map generator generates a map that forms a digital representation or a digital twin of the real environment. Various guiding modules including a room setup module, an anatomy registration module, a surgical planning module, and a surgical execution module make use of the digital representation to guide virtual or real objects based on the digital representation.

Abstract: The present invention relates to a computer-vision based autonomous steerable catheter device and methods of using the same for targeted delivery of a liquid microvolume into the interior of a lung.

Abstract: An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

Abstract: A method of designing an orthopedic implant comprising: (a) iteratively evaluating possible shapes of a dynamic orthopedic implant using actual anatomical shape considerations and kinematic shape considerations; and, (b) selecting a dynamic orthopedic implant shape from one of the possible shapes, where the dynamic orthopedic implant shape selected satisfies predetermined kinematic and anatomical constraints.

Abstract: Methods and devices are disclosed for intra-operative viewing of pre- and intra-operative 3D patient images.

Abstract: The present disclosure provides systems and methods for calibration-free instant motion tracking useful, for example, for rending virtual content in augmented reality settings. In particular, a computing system can iteratively augment image frames that depict a scene to insert virtual content at an anchor region within the scene, including situations in which the anchor region moves relative to the scene. To do so, the computing system can estimate, for each of a number of sequential image frames: a rotation of an image capture system that captures the image frames; and a translation of the anchor region relative to an image capture system, thereby providing sufficient information to determine where and at what orientation to render the virtual content within the image frame.

Abstract: Systems, apparatuses, and methods may provide for technology to process graphics data in a virtual gaming environment. The technology may identify, from graphics data in a graphics application, redundant graphics calculations relating to common frame characteristics of one or more graphical scenes to be shared between client game devices of a plurality of users and calculate, in response to the identified redundant graphics calculations, frame characteristics relating to the one or more graphical scenes. Additionally, the technology may send, over a computer network, the calculation of the frame characteristics to the client game devices.

Abstract: In accordance with various implementations, a method is performed at an electronic device with one or more processors, a non-transitory memory, and a display. The method includes determining an engagement score that characterizes a level of engagement between a user and a first object. The first object is located at a first location on the display. The method includes determining an interruption priority value that characterizes a relative importance of signaling a presence of a second object to the user. The second object is detectable by the electronic device. In some implementations, the method includes presenting the second object according to one or more output modalities of the electronic device. The method includes, in response to determining that the engagement score and the interruption priority value collectively together satisfy an interruption condition, presenting a notification.

Abstract: A System for image processing (IPS), in particular for lung imaging. The system (IPS) comprises an interface (IN) for receiving at least a part of a 3D image volume (VL) acquired by PAT an imaging apparatus (IA1) of a lung (LG) of a subject (PAT) by exposing the subject (PAT) to a first interrogating signal. A layer definer (LD) of the system (IPS) is configured to define, in the 3D image volume, a layer object (LO) that includes a representation of a surface (S) of the lung (LG). A renderer (REN) of the system (IPS) is configured to render at least a part of the layer object (LO) in 3D at a rendering view (Vp) for visualization on a display device (DD).

Abstract: Methods and apparatus for projecting augmented reality (AR) enhancements to real objects in response to user gestures detected in a real environment are disclosed. An example apparatus includes one or more processors to execute computer-readable instructions to identify a user gesture within a real environment based on data obtained from a motion sensor. The user gesture is associated with a target real object from among one or more real objects located within the real environment. The user gesture represents a desired shape of a desired virtual drawing to be projected to the target real object. The one or more processors are further to execute the instructions to determine an AR enhancement based on the user gesture and the target real object. The AR enhancement includes a virtual drawing having a shape corresponding to the desired shape of the desired virtual drawing.

Abstract: In one embodiment, a method includes using one or more cameras of a mobile computing device to capture one or more images of a first user wearing a VR display device in a real-world environment. The mobile computing device transmits a pose of the mobile computing device with respect to the VR display device to a VR system. The mobile computing device receives from the VR system a VR rendering of a VR environment. The VR rendering is from the perspective of the mobile computing device with respect to the VR display device. The method includes segmenting the first user from the one or more images and generating, in real-time responsive to capturing the one or more images, a MR rendering of the first user in the VR environment. The MR rendering of the first user is based on a compositing of the segmented one or more images of the first user and the VR rendering.

Abstract: Presented herein are systems, methods, and architectures related to augmented reality (AR) surgical visualization of one or more dual-modality probe species in tissue. As described herein, near infrared (NIR) images are detected and rendered in real time. The NIR images are registered and/or overlaid with one or more radiological images (e.g., which were obtained preoperatively/perioperatively) by a processor [e.g., that uses an artificial neural network (ANN) or convolutional neural network (CNN) reconstruction algorithm] to produce a real-time AR overlay (3D representation). The AR overlay is displayed to a surgeon in real time. Additionally, a dynamic motion tracker tracks the location of fiducial tracking sensors on/in/about the subject, and this information is also used by the processor in producing (e.g., positionally adjusting) the AR overlay.

Abstract: A waveguide apparatus includes a planar waveguide and at least one optical diffraction element (DOE) that provides a plurality of optical paths between an exterior and interior of the planar waveguide. A phase profile of the DOE may combine a linear diffraction grating with a circular lens, to shape a wave front and produce beams with desired focus. Waveguide apparati may be assembled to create multiple focal planes. The DOE may have a low diffraction efficiency, and planar waveguides may be transparent when viewed normally, allowing passage of light from an ambient environment (e.g., real world) useful in AR systems. Light may be returned for temporally sequentially passes through the planar waveguide. The DOE(s) may be fixed or may have dynamically adjustable characteristics. An optical coupler system may couple images to the waveguide apparatus from a projector, for instance a biaxially scanning cantilevered optical fiber tip.