Patents Assigned to Nevermind Capital LLC
  • Patent number: 12088932
    Abstract: Methods and apparatus for capturing, communicating and using image data to support virtual reality experiences are described. Images, e.g., frames, are captured at a high resolution but lower frame rate than is used for playback. Interpolation is applied to captured frames to generate interpolated frames. Captured frames, along with interpolated frame information, are communicated to the playback device. The combination of captured and interpolated frames correspond to a second frame playback rate which is higher than the image capture rate. Cameras operate at a high image resolution but slower frame rate than images could be captured with the same cameras at a lower resolution. Interpolation is performed prior to delivery to the user device with segments to be interpolated being selected based on motion and/or lens FOV information. A relatively small amount of interpolated frame data is communicated compared to captured frame data for efficient bandwidth use.
    Type: Grant
    Filed: February 13, 2023
    Date of Patent: September 10, 2024
    Assignee: Nevermind Capital LLC
    Inventors: Ramesh Panchagnula, David Cole, Alan Moss
  • Patent number: 12081723
    Abstract: Methods and apparatus for streaming or playing back stereoscopic content are described. Camera dependent correction information is communicated to a playback device and applied in the playback device to compensate for distortions introduced by the lenses of individual cameras. By performing lens dependent distortion compensation in the playback device edges which might be lost if correction were performed prior to encoding are preserved. Distortion correction information maybe in the form of UV map correction information. The correction information may indicate changes to be made to information in a UV map, e.g., at rendering time, to compensate for distortions specific to an individual camera. Different sets of correction information maybe communicated and used for different cameras of a stereoscopic pair which provide images that are rendered using the same UV map. The communicated correction information is sometimes called a correction mesh since it is used to correct mesh related information.
    Type: Grant
    Filed: September 13, 2021
    Date of Patent: September 3, 2024
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss, Hector M. Medina
  • Patent number: 11991418
    Abstract: Methods and apparatus for implementing a playback system capable of operating as a virtual or augmented reality device are described. In various embodiment one or more environmental triggers are detected and an action is taken based on the detected trigger, user input and/or other environmental conditions. The methods allow a user who is subject to an immersive experience to smoothly transition in/or out of the virtual environment and respond to environmental triggers which may require a user to take an action.
    Type: Grant
    Filed: July 12, 2019
    Date of Patent: May 21, 2024
    Assignee: Nevermind Capital LLC
    Inventors: Matt McCarty, Hector Medina, David Cole, Alan Moss
  • Patent number: 11924394
    Abstract: Methods and apparatus for using selective resolution reduction on images to be transmitted and/or used by a playback device are described. Prior to transmission one or more images of an environment are captured. Based on image content, motion detection and/or user input a resolution reduction operation is selected and performed. The reduced resolution image is communicated to a playback device along with information indicating a UV map corresponding to the selected resolution allocation that should be used by the playback device for rendering the communicated image. By changing the resolution allocation used and which UV map is used by the playback device different resolution allocations can be made with respect to different portions of the environment while allowing the number of pixels in transmitted images to remain constant. The playback device renders the individual images with the UV map corresponding to the resolution allocation used to generate the individual images.
    Type: Grant
    Filed: March 14, 2022
    Date of Patent: March 5, 2024
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss, Hector M. Medina
  • Patent number: 11902493
    Abstract: Methods and apparatus for collecting user feedback information from viewers of content are described. Feedback information is received from viewers of content. The feedback indicates, based on head tracking information in some embodiments, where users are looking in a simulated environment during different times of a content presentation, e.g., different frame times. The feedback information is used to prioritize different portions of an environment represented by the captured image content. Resolution allocation is performed based on the feedback information and the content re-encoded based on the resolution allocation. The resolution allocation may and normally does change as the priority of different portions of the environment change.
    Type: Grant
    Filed: April 15, 2022
    Date of Patent: February 13, 2024
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss, Hector M Medina
  • Patent number: 11870967
    Abstract: Methods and apparatus for supporting the capture of images of surfaces of an environment visible from a default viewing position and capturing images of surfaces not visible from the default viewing position, e.g., occluded surfaces, are described. Occluded and non-occluded image portions are packed into one or more frames and communicated to a playback device for use as textures which can be applied to a model of the environment where the images were captured. An environmental model includes a model of surfaces which are occluded from view from a default viewing position but which maybe viewed is the user shifts the user's viewing location. Occluded image content can be incorporated directly into a frame that also includes non-occluded image data or sent in frames of a separate, e.g., auxiliary content stream that is multiplexed with the main content stream which communicates image data corresponding to non-occluded environmental portions.
    Type: Grant
    Filed: March 7, 2022
    Date of Patent: January 9, 2024
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan Mckay Moss
  • Patent number: 11871085
    Abstract: Content delivery and playback methods and apparatus are described. The methods and apparatus are well suited for delivery and playback of content corresponding to a 360 degree environment and can be used to support streaming and/or real time delivery of 3D content corresponding to an event, e.g., while the event is ongoing or after the event is over. Portions of the environment are captured by cameras located at different positions. The content captured from different locations is encoded and made available for delivery. A playback device selects the content to be received based in a user's head position.
    Type: Grant
    Filed: April 6, 2022
    Date of Patent: January 9, 2024
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss
  • Patent number: 11770558
    Abstract: Methods and apparatus for stereoscopic image encoding and decoding are described. Left and right eye images are encoded following an entropy reduction operation being applied to one of the eye images when there is a difference between the left and right images of an image pair. Information about regions of negative parallax within the entropy reduced image of an image pair is encoded along with the images. Upon decoding a sharpening filter is applied to the image in an image pair which was subjected to the entropy reduction operation. In addition edge enhancement filtering is performed on the regions of the recovered entropy reduced image which are identified in the encoded image data as regions of negative parallax. Interleaving of left and right eye images at the input of the encoder combined with entropy reduction allows for efficient encoding, storage, and transmission of 3D images.
    Type: Grant
    Filed: June 11, 2020
    Date of Patent: September 26, 2023
    Assignee: Nevermind Capital LLC
    Inventors: David Michael Cole, Alan McKay Moss, David Roller, Jr.
  • Patent number: 11736673
    Abstract: An unobstructed image portion of a captured image from a first camera of a camera pair, e.g., a stereoscopic camera pair including fisheye lenses, is combined with a scaled extracted image portion generated from a captured image from a second camera in the camera pair. An unobstructed image portion of a captured image from the second camera of the camera pair is combined with a scaled extracted image portion generated from a captured image from the first camera in the camera pair. As part of the combining obstructed image portions which were obstructed by part of the adjacent camera are replaced in some embodiments. In some embodiments, the obstructions are due to adjacent fisheye lens. In various embodiments fish eye lenses which have been cut to be flat on one side are used for the left and right cameras with the spacing between the optical axis approximating the spacing between the optical axis of a human person's eyes.
    Type: Grant
    Filed: June 30, 2020
    Date of Patent: August 22, 2023
    Assignee: Nevermind Capital LLC
    Inventors: Alan McKay Moss, Hector M. Medina, Ryan Michael Sheridan, Matthew Yaeger, David Ibbitson, David Cole
  • Patent number: 11688124
    Abstract: A background image is also generated, e.g., by filling portions of a captured image where a foreground object was extracted and communicated to the playback device, Foreground objects are identified and point cloud representations of the foreground objects are generated and communicated to a playback device so that they can be used in generating images including the background which is communicated separately. In the case of a point cloud representation a number of points in an environment, e.g., 3D space, are communicated to the playback device along with color information. Thus in some embodiments a foreground object is represented as a set of points with corresponding color information on a per point basis. Foreground object information is communicated and processed in some embodiments at a different rate, e.g., faster rate, then the background textures. The playback device renders images which are sent to the display by first rendering a background layer using the communicated background information, e.g.
    Type: Grant
    Filed: February 14, 2022
    Date of Patent: June 27, 2023
    Assignee: Nevermind Capital LLC
    Inventors: Hector Medina, David Cole, Alan Moss
  • Patent number: 11582384
    Abstract: Methods and apparatus for capturing, communicating and using image data to support virtual reality experiences are described. Images, e.g., frames, are captured at a high resolution but lower frame rate than is used for playback. Interpolation is applied to captured frames to generate interpolated frames. Captured frames, along with interpolated frame information, are communicated to the playback device. The combination of captured and interpolated frames correspond to a second frame playback rate which is higher than the image capture rate. Cameras operate at a high image resolution but slower frame rate than images could be captured with the same cameras at a lower resolution. Interpolation is performed prior to delivery to the user device with segments to be interpolated being selected based on motion and/or lens FOV information. A relatively small amount of interpolated frame data is communicated compared to captured frame data for efficient bandwidth use.
    Type: Grant
    Filed: April 24, 2020
    Date of Patent: February 14, 2023
    Assignee: Nevermind Capital LLC
    Inventors: Ramesh Panchagnula, David Cole, Alan Moss
  • Patent number: 11575870
    Abstract: Methods and apparatus relating to encoding and decoding stereoscopic (3D) image data, e.g., left and right eye images, are described. Various pre-encoding and post-decoding operations are described in conjunction with difference based encoding and decoding techniques. In some embodiments left and right eye image data is subject to scaling, transform operation(s) and cropping prior to encoding. In addition, in some embodiments decoded left and right eye image data is subject to scaling, transform operations(s) and filling operations prior to being output to a display device. Transform information and/or scaling information may be included in a bitstream communicating encoded left and right eye images. The amount of scaling can be the same for an entire scene and/or program.
    Type: Grant
    Filed: June 8, 2020
    Date of Patent: February 7, 2023
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss
  • Patent number: 11526211
    Abstract: Methods and Apparatus for controlling, implementing and supporting trick Play in an augmented reality (AR) device are described. Detected changes in AR device orientation and/or AR device position are detected and used in controlling temporal playback operations.
    Type: Grant
    Filed: May 2, 2019
    Date of Patent: December 13, 2022
    Assignee: Nevermind Capital LLC
    Inventors: Hector Medina, Alan McKay Moss, David Cole
  • Patent number: 11470299
    Abstract: A method and system for encoding a stereoscopic image pair is disclosed. Groups of pixels are analyzed to determine the depth of each pixel group. The number of bits per pixel used to encode each pixel group is selected based on the depth of that pixel group. Therefore, images of objects closer to the camera pair, which appear closer to the viewer, are encoded with a larger number of bits per pixel than objects perceived to be farther from the viewer. The number of bits per pixel may also be increased based on a number of objects depicted or motion detected. The size of prediction blocks used to encode image portions may also be determined based on an angular distance of an image portion relative to the center of the frame. Therefore, smaller prediction blocks may be used to encode image portions closer to the center of the frame.
    Type: Grant
    Filed: September 25, 2020
    Date of Patent: October 11, 2022
    Assignee: Nevermind Capital LLC
    Inventors: Ramesh Panchagnula, Alan Moss
  • Patent number: 11388384
    Abstract: Methods and apparatus for using selective resolution reduction on images to be transmitted and/or used by a playback device are described. Prior to transmission one or more images of an environment are captured. Based on image content, motion detection and/or user input a resolution reduction operation is selected and performed. The reduced resolution image is communicated to a playback device along with information indicating a UV map corresponding to the selected resolution allocation that should be used by the playback device for rendering the communicated image. By changing the resolution allocation used and which UV map is used by the playback device different resolution allocations can be made with respect to different portions of the environment while allowing the number of pixels in transmitted images to remain constant. The playback device renders the individual images with the UV map corresponding to the resolution allocation used to generate the individual images.
    Type: Grant
    Filed: July 17, 2018
    Date of Patent: July 12, 2022
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan Moss, Hector Medina
  • Patent number: 11381801
    Abstract: Methods and apparatus for using selective resolution reduction on images to be transmitted and/or used by a playback device are described. Prior to transmission one or more images of an environment are captured. Based on image content, motion detection and/or user input a resolution reduction operation is selected and performed. The reduced resolution image is communicated to a playback device along with information indicating a UV map corresponding to the selected resolution allocation that should be used by the playback device for rendering the communicated image. By changing the resolution allocation used and which UV map is used by the playback device different resolution allocations can be made with respect to different portions of the environment while allowing the number of pixels in transmitted images to remain constant. The playback device renders the individual images with the UV map corresponding to the resolution allocation used to generate the individual images.
    Type: Grant
    Filed: September 21, 2020
    Date of Patent: July 5, 2022
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss, Hector M. Medina
  • Patent number: 11381802
    Abstract: Customer wide angle lenses and methods and apparatus for using such lenses in individual cameras as well as pairs of cameras intended for stereoscopic image capture are described. The lenses are used in combination with sensors to capture different portions of an environment at different resolutions. In some embodiments ground is capture at a lower resolution than sky which is captured at a lower resolution than a horizontal area of interest. Various asymmetries in lenses and/or lens and sensor placement are described which are particularly well suited for stereoscopic camera pairs where the proximity of one camera to the adjacent camera may interfere with the field of view of the cameras.
    Type: Grant
    Filed: June 30, 2020
    Date of Patent: July 5, 2022
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss, Michael P. Straub
  • Patent number: 11375090
    Abstract: A camera apparatus, e.g., a stereoscopic camera apparatus, includes a dual element mounting plate, a pair of individual lens mounts, and a pair of sensor holders, each sensor holder holding an image sensor. The lens mounts and sensor holders are secured to the dual element mounting plate. Multiple dual element mounting plates corresponding to camera pairs may be secured to a base plate. Various features facilitate achieving and maintaining alignment between a lens and a corresponding image sensor, as well as between camera pairs and between sets of camera pairs.
    Type: Grant
    Filed: March 15, 2021
    Date of Patent: June 28, 2022
    Assignee: Nevermind Capital LLC
    Inventor: Ryan Michael Sheridan
  • Patent number: 11348252
    Abstract: Methods for capturing and generating information about objects in a 3D environment that can be used to support augmented reality or virtual reality playback operations in a data efficient manner are described. In various embodiments one or more frames including foreground objects are generated and transmitted with corresponding information that can be used to determine the location where the foreground objects are to be positioned relative to a background for one or more frame times are described. Data efficiency is achieved by specifying different locations for a foreground object for different frame times avoiding in some embodiments the need to transmit an image and depth information defining the same of the foreground for each frame time. The frames can be encoded using a video encoder even though some of the information communicated are not pixel values but alpha blending values, object position information, mesh distortion information, etc.
    Type: Grant
    Filed: July 15, 2019
    Date of Patent: May 31, 2022
    Assignee: Nevermind Capital LLC
    Inventors: Alan Moss, Krishnaprasad Jagadash, Hector Medina, Fabricio Tofoli, Joel Phillips, Jun Liu, Ramesh Panchagnula, Vassilios Mavros
  • Patent number: 11350080
    Abstract: Methods and apparatus for using a display in a manner which results in a user perceiving a higher resolution than would be perceived if a user viewed the display from a head on position are described. In some embodiments one or more displays are mounted at an angle, e.g., sometimes in range a range from an angle above 0 degrees to 45 relative to a user's face and thus eyes. The user sees more pixels, e.g., dots corresponding to light emitting elements, per square inch of eye area than the user would see if the user were viewing the display head on due to the angle at which the display or displays are mounted. The methods and display mounting arrangement are well suited for use in head mounted displays, e.g., Virtual Reality (VR) displays for stereoscopic viewing (e.g., 3D) and/or non-stereoscopic viewing of displayed images.
    Type: Grant
    Filed: June 30, 2020
    Date of Patent: May 31, 2022
    Assignee: Nevermind Capital LLC
    Inventors: David Cole, Alan McKay Moss