Abstract: In a pen-based computing system, a smart pen allows user interaction with the pen-based computing system using multiple modalities. Generally, the modalities are categorized as input (or command) modalities and output (or feedback) modalities. Examples of input modalities for the smart pen include writing with the smart pen to provide written input and/or speaking or otherwise providing sound to give audio input to the smart pen.