Abstract: In a pen-based computing system, multi-modal data is transferred between a paper domain and a digital domain. Data initially generated in the paper domain is captured by a smart pen and a digital file including the captured data is generated. For example, a computing system coupled to the smart pen generates a digital file including the captured data. A paper representation of the digital file is subsequently generated. The digital file can subsequently be modified by editing the paper representation of the digital file using the smart pen. Edits to the paper representation of the digital file are captured by the smart pen and converted to the digital domain where they are used to edit the content of the digital file.

Abstract: In a pen-based computing system, use of paradigms similar to those used with physical paper to organize user generated content captured by a smart pen is disclosed. Data, such as handwriting gestures, is captured by the smart pen and transferred to a digital domain, such as by being transferred to a computing system. Once in the digital domain, the captured content is organized as virtual pages or virtual notebooks. Hence, content captured from various sources, such as different notebooks or different physical pieces of paper, is assembled into a virtual page or virtual notebook. User input or automatic application of rules can be used to assemble the captured content into a virtual page or virtual notebook.

Abstract: In a pen-based computing system, a microphone on the smart pen device records audio to produce audio data and a gesture capture system on the smart pen device records writing gestures to produce writing gesture data. Both the audio data and the writing gesture data include a time component. The audio data and writing gesture data are combined or synchronized according to their time components to create audio ink data. The audio ink data can be uploaded to a computer system attached to the smart pen device and displayed to a user through a user interface. The user makes a selection in the user interface to play the audio ink data, and the audio ink data is played back by animated the captured writing gestures and playing the recorded audio in synchronization.

Abstract: A system and method for identifying captured written data associated with a writing surface using a pen-based computing system are described. In one embodiment, a series of writing surfaces, such as notebooks, are produced, each including a plurality of pages that have an encoded pattern. In an embodiment, each writing surface is also associated with an identifier recognizable by a smart pen or other data capture device. When the smart pen captures written data from a writing surface, the smart pen determines whether the pen-based computing system includes stored written data previously captured from a writing surface having the same encoded pattern as the writing surface from which written data is currently being captured.

Abstract: In a pen-based computing system, a pair of microphones captures audio concurrently with a smart pen device capturing handwriting gestures. The audio captured by the pair of microphones is processed to produce an enhanced audio recording. Noise cancellation techniques may be applied to reduce noise generated by the smart pen device interacting with the writing surface to produce a higher quality audio recording. In addition, beam steering techniques may be applied to adjust the gain of portions of the audio signal from different directions and allow the user to focus between multiple audio sources.

Abstract: In a pen-based computing system, a printed version of a document having preexisting content is annotated using a smart pen. The smart pen captures handwriting gestures to obtain an electronic representation of the annotations. The smart pen computing system identifies a digital version of the document having the preexisting content and stores the electronic representation of the annotations in association with the digital document. The smart pen computing system may overlay the electronic representation of the annotations with the preexisting content to provide a digital representation of the annotated document.

Abstract: A pen based computing system concurrently captures handwriting gestures and records audio using binaural recording. A binaural headset communicatively coupled to the smart pen device uses at least two microphones. A left microphone is placed in or near the left ear and the right microphone is placed in or near the right ear, each facing outward. Speakers are integrated into a shared housing with the microphones facing inward towards the ear canal to play back the audio recordings. By recording audio with microphones placed close to the ears, the system provides realistic sounding playback and allows users to more easily differentiate between multiple sources of audio.

Abstract: In a pen-based computing system, a microphone on the smart pen device records audio to produce audio data and a gesture capture system on the smart pen device records writing gestures to produce writing gesture data. Both the audio data and the writing gesture data include a time component. The audio data and writing gesture data are combined or synchronized according to their time components to create audio ink data. The audio ink data can be uploaded to a computer system attached to the smart pen device and displayed to a user through a user interface. The user makes a selection in the user interface to play the audio ink data, and the audio ink data is played back by animated the captured writing gestures and playing the recorded audio in synchronization.

Abstract: In a pen-based computing system, a microphone on the smart pen device records audio to produce audio data and a gesture capture system on the smart pen device records writing gestures to produce writing gesture data. Both the audio data and the writing gesture data include a time component. The audio data and writing gesture data are combined or synchronized according to their time components to create audio ink data. The audio ink data can be uploaded to a computer system attached to the smart pen device and displayed to a user through a user interface. The user makes a selection in the user interface to play the audio ink data, and the audio ink data is played back by animated the captured writing gestures and playing the recorded audio in synchronization.

Abstract: Systems for removing or attenuating friction and/or click noise generated during writing with a smart pen are disclosed. In a smart pen, the writing cartridge is biased towards a first side of the smart pen internal cavity by a low-force biasing system without affecting writing cartridge movement. For example, the writing cartridge is comprised of a magnetic material and a magnet is embedded into the first side of the smart pen internal cavity to bias the writing cartridge towards the first side. Alternatively, the material comprising the writing cartridge or smart pen internal cavity is modified, reducing friction between the components. As another alternative, a DSP method identifies analyzes one or more parameters of received audio data to differentiate click noise from audio data, and may receive data from a movement sensor indicating when the writing cartridge has moved to reduce power consumption for identifying click noise.

Abstract: Interactive digital content for printed materials is provided using applications executing in a smart pen computing system. The applications and the printed materials may be developed by separate entities but appear well integrated from the perspective of a user of the smart pen computing system. Printed content produced by a first entity is obtained. The printed content is associated with a dot pattern. An application associated with the printed content and linked to the dot pattern is produced by a separate second entity. The application includes computer program instructions for, responsive to a selection within the dot pattern, invoking a function related to the printed content. The selection within the dot pattern may be made using a smart pen device, and the function may include presenting output to a user through an output device (e.g., a display or speaker) of the smart pen device.

Abstract: Interactive digital content for printed materials is provided using applications executing in a smart pen computing system. The applications and the printed materials may be developed by separate entities but appear well integrated from the perspective of a user of the smart pen computing system. Printed content produced by a first entity is obtained. The printed content is associated with a dot pattern. An application associated with the printed content and linked to the dot pattern is produced by a separate second entity. The application includes computer program instructions for, responsive to a selection within the dot pattern, invoking a function related to the printed content. The selection within the dot pattern may be made using a smart pen device, and the function may include presenting output to a user through an output device (e.g., a display or speaker) of the smart pen device.

Abstract: Interactive digital content for printed materials is provided using applications executing in a smart pen computing system. The applications and the printed materials may be developed by separate entities but appear well integrated from the perspective of a user of the smart pen computing system. Printed content produced by a first entity is obtained. The printed content is associated with a dot pattern. An application associated with the printed content and linked to the dot pattern is produced by a separate second entity. The application includes computer program instructions for, responsive to a selection within the dot pattern, invoking a function related to the printed content. The selection within the dot pattern may be made using a smart pen device, and the function may include presenting output to a user through an output device (e.g., a display or speaker) of the smart pen device.

Abstract: Systems for removing or attenuating friction and/or click noise generated during writing with a smart pen are disclosed. In a smart pen, the writing cartridge is biased towards a first side of the smart pen internal cavity by a low-force biasing system without affecting writing cartridge movement. For example, the writing cartridge is comprised of a magnetic material and a magnet is embedded into the first side of the smart pen internal cavity to bias the writing cartridge towards the first side. Alternatively, the material comprising the writing cartridge or smart pen internal cavity is modified, reducing friction between the components. As another alternative, a DSP method identifies analyzes one or more parameters of received audio data to differentiate click noise from audio data, and may receive data from a movement sensor indicating when the writing cartridge has moved to reduce power consumption for identifying click noise.

Abstract: A system and method for identifying captured written data associated with a writing surface using a pen-based computing system are described. In one embodiment, a series of writing surfaces, such as notebooks, are produced, each including a plurality of pages that have an encoded pattern. In an embodiment, each writing surface is also associated with an identifier recognizable by a smart pen or other data capture device. When the smart pen captures written data from a writing surface, the smart pen determines whether the pen-based computing system includes stored written data previously captured from a writing surface having the same encoded pattern as the writing surface from which written data is currently being captured.

Abstract: The described embodiments present a system and method for obtaining a portion of a digital document from a printed version of the document. Locations where a smart pen device interacts with a digital bookclip aligned relative the printed document are identified via coordinate sets. A digital document associated with the printed document and the coordinate sets from the smart pen device and digital bookclip are used to identify locations in the digital documents corresponding to locations in the printed document. Data from the identified locations in the digital documents is retrieved and presented to a user and can then be copied, saved, mailed electronically, and/or printed to a new paper document.

Abstract: Interactive digital content for printed materials is provided using applications executing in a smart pen computing system. The applications and the printed materials may be developed by separate entities but appear well integrated from the perspective of a user of the smart pen computing system. Printed content produced by a first entity is obtained. The printed content is associated with a dot pattern. An application associated with the printed content and linked to the dot pattern is produced by a separate second entity. The application includes computer program instructions for, responsive to a selection within the dot pattern, invoking a function related to the printed content. The selection within the dot pattern may be made using a smart pen device, and the function may include presenting output to a user through an output device (e.g., a display or speaker) of the smart pen device.

Abstract: An instruction is presented to a user for making a target gesture. The target gesture may be a portion of an exemplary or model symbol. The instruction may be presented in various ways, such as being printed on a writing surface or being played in audio format through a speaker of a smart pen device. A writing gesture made on a writing surface by the user is digitally captured using a smart pen device. The captured writing gesture is compared to the target gesture and feedback is determined based on this comparison. This feedback may indicate the correctness of the user's writing gesture. The feedback may be presented to the user through various means, including through the smart pen device. The comparison may also be used to determine a next instruction for the user to follow.

Abstract: Embodiments of the invention present a system and method for controlling audio capture by a smart pen based computing system. An audio capture mechanism that is independent from a gesture capture system is included on the smart pen to control audio capture by one or more microphones included on the smart pen. In one embodiment, the audio capture mechanism comprises a shared function button, such as a power button. For example, a user interaction with the shared function button initiates audio capture by the one or more microphones on the smart pen and a second user interaction with the shared audio function stops audio capture. Alternatively, audio capture is stopped after completion of a predefined time interval after the user interaction with the shared function button.

Abstract: Embodiments of the invention present a system and method for generating digital links, or “ink tags,” from handwritten notes on a writing surface. Handwritten data, including one or more ink tags identifying digital content, is captured by a smart pen device and used to retrieve a digital document associated with the handwritten data. The handwritten data is analyzed to identify the ink tags. In one embodiment, a computing system performs optical character recognition on the handwriting data to identify a plurality of characters included in the handwriting data and compares one or more characters to formats associated with digital content. Responsive to identifying an ink tag, a digital link between the ink tag and a source for the digital content identified by the ink tag is generated, allowing the digital content to be retrieved by an interaction with the ink tag.