Abstract: A system and a method are disclosed for displaying content collected in a pen-based computing system. Content collected in the pen-based computing system includes stroke data collected by a smart pen as well as other contextual data collected by the pen-based computing system. The collected stroke data are grouped into snippets and used to create paper strips. Paper strips are arranged based on metadata associated with the collected content.

Abstract: A system and a method are disclosed for organizing content collected in a pen-based computing system. Content collected in the pen-based computing system includes stroke data collected by a smart pen as well as other contextual data collected by the pen-based computing system. The collected stroke data includes timestamps and positional data. Based on correlations of spatial and temporal data, collected stroke data is grouped into clusters of stroke data. The clusters of stroke data, along with contextual data, are further grouped into snippets based on temporal correlations.

Abstract: A system and a method are disclosed for organizing content collected in a pen-based computing system. A plurality of groups of stroke data are obtained. Each group of stroke data is associated with a timestamp based on the timing of strokes made by the smart pen. The groups of stroke data are grouped into one or more snippets based on a temporal proximity of the timestamps and spatial proximity of the stroke data. A request to associate a snippet to a tag is received. An association between the snippet and the tag is stored responsive to the request.

Abstract: Imaging systems may include camera modules that include multiple image sensor pixel arrays. A transparent lens substrate may be formed over the image pixel arrays. Lenses may be formed in the lens substrate such that each lens transmits light to a corresponding image sensor pixel array. Total internal reflection mitigation structures such as groove structures may be formed in one or more surfaces of the lens substrate between each of the lenses. The groove structures may include concentric ring shaped grooves in a surface of the lens substrate so that each lens is surrounded by a respective group of concentric ring shaped grooves. The groove structures may have a depth, angle, shape, and spacing that prevents total internal reflection of image light between the lenses so that high incident angle image light incident on a given pixel array is not captured by an adjacent pixel array.

Abstract: An imaging system may include a camera module with an image sensor having an array of image sensor pixels and one or more lenses that focus light onto the array of image sensor pixels. The array of image sensor pixels may include a corresponding array of color filter elements. The system may include circuitry configured to detect and mitigate flare artifacts in image data captured using the image sensor. Detecting the flare artifacts may include detecting color mismatches in the captured image data and performing edge-detection operations to determine whether the color mismatches are in an edge region of the image data. If the color mismatches are detected in an edge region, no flare-mitigation operations may be performed. If the color mismatches are in a flat region of the captured image, flare-mitigation operations may be performed.

Abstract: Some embodiments provide a method for processing metadata associated with digital video in a multi-state video computer readable medium. The method specifies a set of rules for propagating the metadata between different states in the video computer readable medium. It then propagates the metadata between the states based on the specified set of rules.

Abstract: Some embodiments provide a method of capturing digital video. According to this method, the digital video is received at a first rate. The received digital video includes metadata. The method uses the metadata to store the digital video at a second rate that is different from the first rate. In some embodiments, the first rate is an encoded rate, while the second rate is the digital video's native shooting rate. The digital video's native shooting rate can be any arbitrary rate up to, but not including, the encoded rate. Some embodiments provide a method of outputting digital video from a computer. From a storage structure of the computer, this method retrieves a video clip. In this video clip, the method embeds a metadata structure and records a set of metadata in this structure. The method then outputs the video clip with the embedded metadata.

Abstract: Some embodiments of the invention provide a method for synchronizing an audio stream with a video stream. This method involves searching in the audio stream for audio data having values that match a distinct set of audio data values and synchronizing the audio stream with the video stream based on the search. In some embodiments, the distinct set of audio data values is defined by a predetermined distinct tone. In other embodiments, the distinct set of audio data values is defined by audio data contained in the video stream.

Abstract: Some embodiments provide a method for processing metadata associated with digital video in a multi-state video computer readable medium. The method specifies a set of rules for propagating the metadata between different states in the video computer readable medium. It then propagates the metadata between the states based on the specified set of rules.

Abstract: Electronic devices may include camera modules. A camera module may include an array camera having an array of lenses and a corresponding array of image sensors. The array of image sensors may include a monochromatic image sensor such as a green image sensor and a polychromatic image sensor such as a red and blue image sensor. The red and blue image sensor may include a color filter array of red and blue color filter elements formed over an array of image pixels. The red and blue color filter elements in the polychromatic image sensor may be arranged in a checkerboard pattern, a zigzag pattern that extends vertically from the top of the pixel array to the bottom of the pixel array, or a diagonal strip pattern. The electronic device may include processing circuitry for combining monochromatic images from the monochromatic image sensor with polychromatic images from the polychromatic image sensor.

Abstract: Electronic devices may include camera modules. A camera module may be formed from an array of lenses and corresponding image sensors. The array of image sensors may include three color image sensors for color imaging and a fourth image sensor positioned to improve image depth mapping. Providing a camera module with a fourth image sensor may increase the baseline distance between the two most distant image sensors, allowing parallax and depth information to be determined for objects a greater distance from the camera than in a conventional electronic device. The fourth image sensor may be a second green image sensor positioned at a maximal distance from the green color image sensor used for color imaging. The fourth image sensor may also be a clear image sensor, allowing capture of improved image depth information and enhanced image resolution and low-light performance.

Abstract: A system and a method are disclosed for calibrating writing on a writing surface to a digital document. One or more calibration parameters associated with a writing surface and a digital document of a display device are determined. The calibration parameters indicate a spatial offset between a reference position on the writing surface and a reference position in the digital document. A gesture captured by a smart pen is received. The gesture includes a sequence of spatial positions representing movement of the smart pen with respect to the writing surface. The sequence of spatial positions is mapped to a sequence of spatial positions in the digital document based on the calibration parameters.

Abstract: A system and method receives a plurality of data streams from a smart pen device and a computing device, and indexes the data streams to a synchronized time index. A processor receives a first data stream representing gesture data captured by a smart pen device, and a second data stream representing a sequence of states associated with applications executing on a computing device, such that each state identifies content displayed by a computing device while the gesture data is captured. For example, a state could be a particular page of a digital document displayed by the computing device. After receiving the first and second data streams, the processor indexes the data streams to a synchronized time index, and stores the indexed data streams in a memory.

Abstract: A system and method are disclosed for interacting with digital workbooks. An identifier, identifying a physical workbook, is received. The physical workbook is associated with a digital book and can be displayed on a display screen of a computing system. Captured interactions between the smart pen and the writing surface of the workbook are received. One or more completed areas of the workbook are identified based on the one or more captured interactions. Based on the one or more completed areas of the workbook, a portion of the digital book is selected and displayed.

Abstract: A central device concurrently receives handwriting gestures from a plurality of smart pen devices. Each set of handwriting gestures includes a sequence of spatial positions of the corresponding smart pen device with respect to a writing surface. Representations of the handwriting gestures are displayed on a display screen, and the representations show relative timing between the different sets of handwriting gestures. In one embodiment, a portion of the received handwriting gestures is outputted for display.

Abstract: A system and method establishes a connection between a smart pen and a computing device, and establishes a privilege level that regulates data requests for specific data from the smart pen. The smart pen determines whether a connection should be established between the smart pen and a computing device, based on device information received from the computing device. If a connection is established, a privilege level is established for an application executing on the computing device based on the device information, which determines whether a request from the application for specific data from the smart pen is allowed or denied.

Abstract: Some embodiments provide a method of processing digital video in a computer. The method receives a first video clip. It stores the first video clip at a first rate in a storage of the computer. From the first video clip, the method generates a second video clip that is at a second rate different than the first rate. The method then outputs the second video clip from the computer. In some embodiments, the first rate is the native rate that a camera originally shot at least a portion of the first video clip, and the second rate is an encoded rate. In some embodiments, the method generates the second video clip from the first video by using the 2:3:3:2 encoding format. Also, in some embodiments, the method outputs the second video clip as the method generates the second video clip.

Abstract: Method for transforming an input data structure to an output data structure according to requirements specified in a transformation template. The method includes translating an input data structure in an input container format to an output data structure in a different output container format and converting an input data structure to an output data structure where the structures have the same container format. During transformation, modifications to the structure and/or content of the input data structure may be made. The resulting output data structure is dynamically determined based on the structure and content of the input data structure. The input data structure is represented by a data structure tree, each node of the tree having an associated scope dictionary containing zero or more key-value pairs. The tree is traversed downwards as scope dictionaries are created and/or modified and traversed upwards as data of the output data structure tree is created.

Abstract: The invention includes methods and apparatus for correcting shading non-uniformity in camera systems. A method includes capturing at least two sets of flatfield images from at least two sets of camera modules under first and second illuminant, respectively. Pixels for each image in the sets of flatfield images are then averaged to form first and second averaged flatfield images, respectively. The first averaged flatfield image is transformed using the second averaged flatfield image to create a transform image. The transform image is then saved in memory for calibrating the shading non-uniformity of the camera module.

Abstract: Some embodiments of the invention provide a method for synchronizing an audio stream with a video stream. This method involves searching in the audio stream for audio data having values that match a distinct set of audio data values and synchronizing the audio stream with the video stream based on the search. In some embodiments, the distinct set of audio data values is defined by a predetermined distinct tone. In other embodiments, the distinct set of audio data values is defined by audio data contained in the video stream.