System and method for interaction of graphical objects on a computer controlled system
A user interface method and system for performing sensemaking tasks using visual information objects. The present invention provides techniques for the system to perform operations for creating, rearranging visual information objects, reallocating display space and scaling user input automatically so that users need not interrupt their creative activities. Such operations are intuitive and non-intrusive. Certain of the behavior results of the operations are specified in contracts associated with each visual information object.
Latest Xerox Corporation Patents:
- Transfer of print queues via machine readable codes
- Color-changing particulate compositions for additive manufacturing and methods associated therewith
- Method for generating document with specialty imaging and computing device associated therewith
- System and method to separate gases using high viscosity liquid sorbents in a spray contactor
- Semiconductor array imager for printing systems
This is a Division of application Ser. No. 10/371,017 filed Feb. 21, 2003, which claims the benefit of U.S. Provisional Application No. 60/357,850 filed Feb. 21, 2002. The disclosure of the prior application is hereby incorporated by reference herein in its entirety.CROSS REFERENCE TO RELATED APPLICATIONS
Reference is made to the following commonly-assigned copending U.S. patent applications: (1) Ser. No. 10/371,263, entitled “System and Method for Moving Graphical Objects on a Computer Controlled System,” by Lance E. Good et al., now U.S. Pat. No. 7,068,288, (2) Ser. No. 10/369,613, entitled “Method and System for Incrementally Changing Text Representation,” by Benjamin B. Bederson et al., (3) Ser. No. 10/369,614, entitled “Method and System for Incrementally Changing Text Representation,” by Benjamin B. Bederson et al., now U.S. Pat. No. 7,549,114, (4) Ser. No. 10/369,612, entitled “Methods and Systems for Navigating a Workspace,” by Lance E. Good et al., now U.S. Pat. No. 7,228,507, (5) Ser. No. 10/369,624, entitled “Methods and Systems for Interactive Classification of Objects,” by Lance E. Good et al., and (6) Ser. No. 10/396,617, entitled “Methods and Systems for Indicating Invisible Contents of Workspace,” by Lance E. Good et al., now U.S. Pat. No. 7,487,462 all filed herewith and the disclosures of which are incorporated herein.FIELD OF THE INVENTION
The present invention is related to the field of computer controlled display systems, in particular techniques which optimize use of available display space and minimize user disruption when performing tasks causing the interaction of graphical objects on the computer controlled display system.BACKGROUND OF THE INVENTION
Blackboards, electronic whiteboards and other two-dimensional free form workspaces have long been used for supporting brainstorming, sensemaking, and other generative tasks where information is created and organized incrementally. Sensemaking is a term used to describe activities that interweave retrieving, creating and using information. As the amount of information placed in these workspaces grows, the competition for display space increases. If users need to interrupt their ideation to rearrange the information in the workspace, make space for new information, or perform other “housekeeping” functions to enable use of the workspace, they risk disruption of their train of thought or “losing the flow” resulting in the possible loss of ideas. Often, the information contained in the workspace is represented as graphical objects. Thus, there is a general need in systems providing a two-dimensional (2D) workspace for techniques that provide intuitive and timely manipulation and interaction of graphical objects.
The use of computer controlled two-dimensional workspaces for organizing information is well known in the art. Some of these systems include the Alternate Reality Kit (SMITH, R. Experiences with the Alternate Reality Kit: an example of the tension between literalism and magic. Proceedings of CHI/GI '87, April 1987, pp. 61-67), Aquanet (M
In order to minimize “losing the flow” when using such systems, the manner in which a user interacts and the system responds is critical. A number of techniques have been disclosed in the art which purportedly facilitate the use of such systems. Some techniques provide for certain operations to be automatically performed. One such operation is the handling of related information in some coordinated fashion, e.g. as a list or group. For example, PowerPoint, Tivoli, and VIKI all have notions of a list structure to automate the alignment and ordering of list items. Although PowerPoint and Tivoli also use this list structure to assist in inserting items into the list, neither of these systems provide an interactive preview of the resulting insertion. Such a preview is useful in understanding the effect of updating a list structure. Storyspace and VIKI also have the notion of a subspace (i.e. a separate and distinct interaction space within another space) to automate the maintenance of a spatial, hierarchical collection of objects.
Several systems have also employed features for automating object interactions. The Flatland system uses a bumping concept to avoid the occlusion of objects on a whiteboard surface. Moving or growing an object could cause it to transitively bump other objects and if an object hits the edge of the whiteboard (i.e. the visible display area) it would automatically shrink. Data Mountain (R
Another technique known in the art is to provide specialized tools to enable movement or the organization of objects. The alignment stick (R
A computer controlled system embodying a method for graphical object interaction is disclosed. The system provides intuitive and non-intrusive techniques to perform operations for rearranging graphical objects, reallocating display space and scaling user input automatically so that users need not interrupt their creative activities. The user interface techniques may be applied to any application which includes interaction between graphical visual information objects. An important principle is that graphical objects should not unnecessarily occlude or overlap other graphical objects when they are created or manipulated so that they interact with other graphical objects.
The system of the present invention is comprised of: one or more input elements for enabling creation and interaction of said visual information objects; a display for showing a viewed space that is contained within a populated space, each of said visual information objects positioned within said populated space; a processing element for generating display information for said display so that visual information objects within said viewed space maintain user perceptible information throughout the creation and interaction of visual information objects.
A system and method for graphical object interaction that enables a user to effectively and efficiently organize and process large amounts of data is described herein. When utilizing the system a user is able to reduce disruptions to their activity caused by the limited display space constraints of a computer controlled display system. So for example, when a user is entering or organizing information on a workspace area of the system, they need not be distracted by the need to rearrange items in order to enter new information. This allows the user to stay “in the flow” of providing such new information.
The system and method of the currently preferred embodiment are implemented in a sensemaking system as described below. In such a sensemaking environment, instances of information content are referred to herein as “visual information objects”. Visual information objects are in turn handled by the system as graphical objects. It should also be noted and emphasized that certain of the techniques described herein may be applied generally to the field of user interfaces for computer controlled display systems and may be utilized for other types of applications that utilize graphical objects other than the sensemaking activities described herein.Description Of A Sensemaking Activity
It has been determined that users transition around such a graph and between the two workspaces in the course of developing their ideas. When the transition steps are too cumbersome, or when resource conflicts in external cognition interfere with progress, the steps take longer or must be repeated. This may cause the user to “lose the flow” or become disrupted from their sensemaking activity.
It has been determined that user interface techniques for working with such workspaces can help a user stay in the flow. For example, when generating content, a key requirement for staying in the flow is to maintain a visible region of work—that is, to keep the item(s) that the user is working on, together with as much context as possible, visible to the user. The system may adjust the view or move objects to accomplish this. A policy favoring limited occlusion or non-occlusion helps prevent items from getting lost during system manipulations.
The following are assumptions about key cognitive resources for a sensemaking activity on a workspace. A cognitive resource is one that aids in the performance of some mental task.
Display space. The system's display space is a primary resource for external cognition. Information requires display space and display space is limited. In general, a display space is too small to show all of the information at once. Many of the design elements in the currently preferred embodiment involving zooming, panning, and peripheral awareness indicators are intended to make trade-offs in the allocation of display space. In making these trade-offs the user and system determine what information elements to display and how much display space to allocate to each element in order to support the active sensemaking task. The management of a display space influences the cost structure of all three activity steps in the writing workspace: generating, arranging, and navigating.
Persistence of spatial arrangement. A user uses a system's interactive blackboard-like writing workspace to create and arrange items. The meaning of the information is held both in the content (that is, words or graphical objects such as diagrams, graphs, etc.) as well as in the spatial arrangement. A user tends to locate closely related items near each other so that they can be conveniently used together. Parallel arrangements of items suggest parallel meanings. Persistence of arrangement speeds access in that the user roughly remembers the location of items and does not need to search for them. It also speeds understanding when a user uses parallel visual structures to imply parallel meanings. The user invests time to arrange items, and presumably benefits both from convenient use and speedier access because he remembers where things are. If the system automatically changes the arrangement radically or in some non-intuitive or illogical fashion, the user will invest more time and energy to learn the new arrangement and possibly to rearrange it.
Limited focus. A user cannot think about everything at once. The system provides several kinds of “focus+context” mechanisms to support a user in focusing on what is important. These mechanisms include grouping operators for objects, peripheral awareness indicators on spaces, and automatic reduction of the scale of the contents of hierarchical spaces so that users can focus on the “big picture” or overall categories while maintaining limited awareness about internal details. The management of this resource especially influences the navigation step in the workspace.
Further illustrated is a menu bar 210 which provides a plurality of function menus that may be selected in a point and click or pull down fashion, as well as a toolbar 211 for directly invoking commonly used functions. The manner of user operation of the menu bar 210 and corresponding function menus and toolbar 211 is according to well known user interface techniques, thus further description of their operation is not deemed necessary. The various functions provided in menu bar 210 include those for creating or manipulating visual information objects, viewing or modifying contracts and entering a repair cycle. Toolbar 211 includes functions for entering text, creating space objects and moving objects.System Operation Overview
During the course of a sensemaking activity, the user will alternate between steps of generating content, arranging the visual information objects, and navigating between different parts of the workspace. The system manages key resources for external cognition with contracts on its visual information objects to minimize “housekeeping functions” and to help keep a user in the flow.
The first step is the generation of content. In the system, for example, content corresponds to the words contained in text boxes or various organizational structures. These could also be graphical information objects, such as images, charts, etc. The contracts in the system's visual information objects are intended to support a user in staying in the flow while generating content. The system does not require that a user “make space” before entering information or otherwise generating content. A user can start at any point in the workspace and start typing or otherwise entering objects. As a text box or a list grows, the system bumps other objects or pans the containing space as needed so that the user can maintain focus and stay in the flow of generating content. Similarly, as a user is adding elements to a list, the contracts make space for the new items in the list and enforce vertical alignment. The user does not need to stop entering to arrange the items.
The system also simplifies the controls for arranging objects. Contracts are active during content generation and automatically arrange objects without additional user action using bumping, transitive bumping, and panning. The system also simplifies the “move” operation by providing visual feedback of the process. The simple pick-up-and-move operation has visual feedback with drop shadows that provide cues showing which objects are being moved. As the user moves an object or set of objects over a space, the moving objects are scaled while in motion to the appropriate scale for the space so that the user can see how objects will fit in. When a user moves objects to put them into a list, the list opens up and makes room for them. In one smooth action, the user can move an object and decide just where to drop it in the list. The list dynamically moves its elements around to prepare room for it. When the user releases the mouse button, all of the objects are dropped into the selection point in the list and scaled to have the appropriate font size for the list. If a user drops an object in a space in a way that would occlude some other objects, an action suggestive of stepping on a bar of soap moves objects out of the way. In addition, if a user wants to create some space in the middle of arranging objects, the user can dynamically turn the movement of an object into a “snowplow” mode. The drop shadow indicating selection disappears and the user can push other objects out of the way. This action does not require the user to fetch a tool or to drop the object first and then select the objects to be moved. The user uses the object to push other objects to make room. These automatic actions keep the user in the flow during arrangement, without requiring the distraction of selecting, moving, or resizing other objects.
The system also provides simplified navigation. Navigation or panning, is the work by a user to get an appropriate view of some part of the workspace. One way that the system supports navigation is by the automatic reduced scaling of the contents of nested workspaces. This gives the usual benefits of a hierarchical view where it is easier to see either the higher level structure or to drill down to see details. The system provides a simple one-click interface to dive down to a subspace so that it grows to fill the display space or to return back to the containing space. The system also provides “City Lights” peripheral awareness indicators to help keep a user oriented to the presence of objects that are clipped and not currently in view.
The system has been implemented for operation on computer based systems having a graphical user interface such as those utilizing the Windows® Operating System from Microsoft, Inc. or the Mac OS operating System from Apple Computer, Inc. Such graphical user interfaces have the characteristic that a user may interact with the computer system using a cursor control device and/or via a touch-screen display, rather than solely via keyboard input device. Such systems also have the characteristic that they may have multiple “windows” wherein discrete operating functions or applications may occur. Further, such computer based system will be preferably of a “free form” type wherein a user may create and manipulate objects independent of positional constraints. Examples of such free form systems include electronic whiteboards or paint and/or draw application programs. A more complete description of such a computer-based system upon which the present invention may be implemented is described below with reference to
To support sensemaking activities, three basic types of visual information objects are commonly used—text objects, lists, and spaces. A fourth type, graphical objects such as images, diagrams, drawings and the like, may also be used. However, the currently preferred embodiment will be described with respect to the first three types, but it would be apparent to one of skill in the art to use all or any subset of the four types. The behavior of these objects under different kinds of interactions such as creation, movement or growing, is governed by policies, represented as contracts. Each type of visual information object has its own kind of contract. Further, each instance of a visual information object may have its own instance of a contract that is different from contracts of other visual information objects of the same type.
Text objects are visual information objects that contain textual information. Instantiations of text objects are referred to as a “text box(es)”. List objects are visual information objects that arrange and present items in some ordered sequence, e.g. in a vertical tabular format. Instantiations of list objects are referred to as a “list(s)”. A space object is a visual information object that is used to arrange other visual information objects. Instantiations of space objects are referred to as a “space(s)”.
Contracts in the currently preferred embodiment of the present invention govern both objects themselves and the “children” or elements of composite objects. For example, contracts for a list govern both the behavior of the visual list object and also the behavior of objects that are members of the list. Note that if an element is “removed” as an element of a composite object it is no longer governed by the parent object. Such relationships can be established explicitly (e.g. in a list), or implicitly (e.g. subspace within a space).
Listed below are examples of properties of visual information objects governed by contracts.
Alignment. It is desirable to have constraints that govern the vertical or horizontal alignment of objects with other objects. For example, the left boundaries of the elements of a list should be vertically aligned.
Occlusion. It has been determined that for sensemaking activities, contracts often preclude occlusion. This prevents temporary “loss” of information when one objects obscures another. However, it is recognized that there may be instances where some occlusion may be desirable, e.g. “piles” or “stacks” of related visual information objects.
Clipping. Because display space is limited, clipping is inevitable. Clipping refers to objects that cannot be completely displayed, typically as a result of performing some operation. Contracts about clipping determine what information is portrayed about clipped objects.
Growth. The size of an object can change if the amount of information that it holds changes or if the user stretches the object in the display space. Objects differ in their growth contracts.
Scale. Objects can be shown at different scales. There is a default scale for objects. The scale of an object can be changed by explicit user actions. Scale contracts can also be transformed when an object is moved into a region where additional scaling contracts apply. Scale contracts can also apply differently to different parts of an object.
Aspect Ratio. The ratio of height to width of an object is preserved by most operations.
Type. Most operations preserve the type of an object.
Representation of Detail. Contracts governing level of detail can control the use of overviews and summaries.
The visual information objects of the present invention are implemented using well known software development techniques and data structures and are described in more detail below.The No-Occlusion Principle
When the populated space contains more information than can be displayed in the view space using a reasonable scale, something has to give. Other systems have addressed this by using 2.5D or 3D techniques such as having windows that partially overlap or cover up other windows. Using such systems, users can cause occlusion unintentionally, such as when a new object is created on top of an existing one. In such cases, the user is focused on manipulating some objects and may not notice that another object has been occluded. As a result, objects that the user needs for the next steps of sensemaking may become difficult to access or may have disappeared for no obvious reason. Detecting occlusions requires the user's attention and resolving them may require breaking their flow of concentration.
Many of the contracts described below enforce a no-occlusion principle. In practice, this often means that instead of objects covering up other objects, they “bump” them aside. Bumping behavior arises when objects are moved, when objects grow to contain more information, and when they are re-sized by the user.Text Box Objects
Text box objects are visual information objects for containing text. An easy way to start developing and capturing ideas is to create text boxes anywhere in an open space for the ideas that come to mind, as shown in
What happens to a text box when a user types depends on which contract is in force. In a usual case for the currently preferred embodiment, the user just begins typing. In this case, the vertical line disappears and a cursor appears at the point where text is entered as the user enters text. As suggested by the text box 402, a user can type and the text box will grow to the right as the user adds characters, and shrink to the left if the user backspaces. In the currently preferred embodiment, the box can grow to the right until reaching a maximum number of default characters, e.g. 33. If the user continues typing, line wrap will occur and a second line of text will appear left aligned and below the first line. Referring briefly to
In another stretching variant interaction shown in
None of the operations relating to the “length” of the text box affect the visual size (roughly “font size”) of the text itself. In the currently preferred embodiment a text box scaling operation that uses predetermined keys on the keyboard, e.g. the Page Up and Page Down keys, can be used to grow or shrink the scale of the text box and the text within it. The scaling contract preserves the aspect ratio and line wrapping within a text box.Lists
A list is a visual information object in the system that arranges items in an ordered sequence. In the currently preferred embodiment, this arrangement is vertically oriented. A list has a title (written in a title bar) and zero (0) or more members.
Lists can be created by selecting a set of items, e.g. text boxes, and invoking an operation to cause them to be treated as a list. In the currently preferred embodiment of the present invention, this is accomplished by using a “listify” menu operation on a set of selected items.
In the currently preferred embodiment, an alternative way to make a list is to start typing an item in a text box, and then ending with a carriage return (typically the enter key of a keyboard coupled to a computer controlled display system).
The title “List” is the default for a list structure that is provided upon initial creation of the list structure. The title of the list can then be set by a separate operation—either dragging one of the list elements into the title position to make it the title or positioning the cursor and entering text directly in the title position.
The basic contract for lists causes all members to be left aligned. Items in a list have a default size (font size) and new items that are added to the list are given the same size. As new items are added to a list, the list grows downwards.
Lists are recursive in that they can have other lists as members. This is illustrated in
Like text boxes, lists have contracts that govern scaling. In the currently preferred embodiment, lists can be scaled by first selecting them and then using the Page Up and Page Down keys. When the scale of a list is reduced, the system preferentially reduces the size of the list members more than the size of the title, as shown in
The top-level writing workspace of the system of the currently preferred embodiment is itself an instance of a space. A space is conceptually similar to a blackboard for creating and arranging visual information objects, such as text boxes, lists, and other spaces.
One feature of spaces in the system is that nested spaces reduce the scale of their members. This is illustrated in
It should be noted that spaces and lists are both composite objects. Their behaviors are different in several important ways and it is instructive to compare some of the differences in their contracts. Like lists, spaces have members. Unlike lists, spaces do not enforce a vertical alignment property for arranging the members. Further, spaces do not grow in size when more members (i.e. information) are added to them.
One way to understand a space is to think of it as a finite window over an infinite canvas. Because the window has a finite size, any visual information objects on the canvas that are not directly under the window are clipped. That part of the canvas on which there are visual information objects is referred to as “populated space” and that part which can be seen at any given time is referred to as “viewed space.” Many of the contract elements for spaces are intended to govern trade-offs in managing what information about populated space appears in the viewed space.
To provide peripheral awareness of clipped objects in the populated space, a peripheral awareness indicator that is referred to as “City Lights” is provided. City Lights are described more generally in co-pending application entitled “Methods and Systems for Indicating Invisible Contents of Workspace”, Ser. No. XX/XXX,XXX. As the name suggests, a peripheral awareness indicator operates such that a person can see the “glow” of the lights of a distant city in the night sky before they can see the city itself. A peripheral awareness indicator provides information about clipped objects. Such information about a clipped object could include its direction from the boundary of the viewed space, its size, its distance, its type, its recency or time since last edit, or a summary of its information content.
The peripheral awareness indicator of the currently preferred embodiment provides limited information about the size, direction, and distance of clipped objects as illustrated by
Preferably, the computation of the peripheral awareness indicators in the currently preferred embodiment is fast enough to allow them to be computed and re-displayed in real time when a space is being panned.
As a user works in the writing space, it is sometimes important to be able to zoom down to work on the details of a contained subspace. A zooming operation on a space will cause it to expand to fill the display area of the entire writing workspace.
Operations Governing Interaction Behaviors
The contracts described above focus mainly on the allocation of static display space. In the following, what is described are operations that govern the moving, growing, and bumping of information display objects.
Since the spatial organization of objects is used to convey meaning, the system provides operations for moving objects. The user operations for moving an object follows a sequence typical for those used in graphical user interfaces. The user first moves the cursor over the target object and clicks on it to select it. Still pressing the mouse button, the user then moves the object so that the cursor is over the place where the object is to be placed. Releasing the mouse button drops the object. It should be noted that in the currently preferred embodiment, that prior to releasing the mouse button and dropping the object, the described actions will occur. This could enable a user to decide if this is the desired course of action to be taken. If the user does not drop the object, the system, and the visually displayed objects, will revert back to their original state. This is illustrated in
Automatic Scaling: Moving Objects Into Subspaces
In this example, the user proceeds to move the text box 1701 into the “Places to Eat” space 1702. In the system of the currently preferred embodiment, spaces reduce the scale for objects contained within them, so the floating image of the text box 1701 immediately reduces in size as shown in
The user next moves the “Plaza Gardens” text box 1701 over the list “Main Street Restaurants” 1704. The list 1704 fluidly grows to make space for the new item.
These examples show how the contracts for display of an object interact in a fluid way when objects are moved in the system—reducing the scale of objects as they move over spaces or making room for an object in a list. Collectively, they support the user in being able to gracefully move an object using a single, simple gesture, without needing to attend to details of scaling, making room, or re-arranging other objects.
Stepping on the Soap: Placing an Object Between Two Objects
Interactions can also occur over white space. When an object is dropped between two objects and there is not enough room to display the object in that location, the objects move out of the way to make room for it. This action is referred to as “stepping on the soap”, to suggest the way a slippery bar of soap can shoot out from under your foot if you step on it in the shower. The system selects an object to move and a direction to move it that minimizes the overall perturbation as illustrated in
“Snowplow” is an alternate contract for moving an object “on” the surface, and may be instigated by the user selecting an object and initiating an action indicating that the snowplow mode be invoked. Invoking the snowplow mode causes the drop shadow to disappear and the user can use the moving object to push other objects around. When any edge of the moving object touches another object, the other object is pushed in whatever direction the user pushes. The user can push a number of objects into any configuration. Terminating the snowplow mode, e.g. by lifting the shift key, ends snowplow mode and causes the moving object to again float above the surface and display a drop shadow. In either mode, the moving object is released when the cursor control button is released indicating the object is no longer selected.
The “snowplow” operation is illustrated with reference to
Many of the actions of movement—making space in a list, stepping on the soap, or snowplow mode—are all effects of the “no occlusion” rule in contracts for visual information objects. Another consequence of this rule is that whenever the movement of one object causes it to bump into another, the second object is also moved. Bumping is transitive: all of the objects that bump cause the other objects that they bump to move.
The no occlusion rule in contracts can have effects for any operation that changes the area that a visual information object might occupy. The previous section illustrated actions triggered when a sensemaker moves an object, including scaling and bumping. Described below are further effects of the no occlusion rule.Automatic Bumping: Growing an Object Into Another
Bumping can also be triggered when an object grows. A text box can grow when a user adds more text to it and a list grows when a user adds more items to it. Any visual information object can grow if the user explicitly changes its scale. An object can also occupy more space if a user selects it and then reallocates the space assigned to it. Grabbing and stretching the corners of a text box can cause it to change its aspect ratio. Grabbing and stretching the corners of a space changes the area available for the “viewed space.” When any of these operations would cause an object to occlude another object, the system bumps the second object so that no occlusion occurs.Automatic Panning: Keeping Text Entry in View
Growth of a visual information object can also cause panning of the viewed space.
The same effect occurs if a user adds information to a list that is near the edge of a containing space. The space pans automatically while the user is working on the list and restores the viewed space to the original pan position when the user selects another object.Zoom-Anti-Zoom: An Alternative to Bumping and Panning for Adding Material
An alternative technique for governing the growth of text boxes is one that is termed “Zoom-Anti-Zoom” or “ZAZ.” The intended effect of ZAZ is to enable a user to stay focused on entering new text without causing the text box to bump into other objects. Instead, the ZAZ contract automatically changes the scale (roughly the font size) and camera position for rendering of the text boxes. The manner of representation and rendering of visual information objects is described in greater detail below. In effect, the text gets smaller as the box gets more full. Since this approach would eventually have the effect of making the text too small to see, the system also “moves the user closer” to the text so that its apparent size stays about the same. When the user selects another object, the containing space is restored to its original scale and the text appears in its greatly reduced size. The user can then move it or scale it as desired.
This combination of scaling the text and the background simultaneously is illustrated by the sequence in
In the currently preferred embodiment, contracts are made explicit to represent interaction and display policies as parameterized contracts. The term contracts and policies in this context is meant to indicate how the visual information object will interact with other objects and be displayed when it is itself being interacted with directly or indirectly by the user. The parameters within the contracts may apply to all the visual information objects. Alternatively, a contract may be created to control behavior of an individual visual information object. A description of exemplary user interfaces to these contracts is presented below. It would have been apparent to one of ordinary skill in the art to utilize other well known user interface techniques to provide parameters for such contracts,Contracts for Spaces
Contracts for Lists
Lists and spaces in the system of the currently preferred embodiment are similar in that both kinds of objects are composite—holding other visual information objects in their interiors. Lists differ from spaces in that they impose an alignment on their interior objects. The contract for growth for lists and for spaces is similar.
The user interface to the interior of a list offers controls for formatting,. such as whether the elements of a list have a leading “bullet” 4001 and whether items are indented 4002. It also offers controls governing the magnification of items moved into a list 4003. Behavior controls 4004 are offered that can govern scrolling and panning behavior when viewing the contents of a large list based on known techniques such as a fish-eye, focus-context approach. Finally, the radio buttons 4005 and 4006 govern whether the list attempts to display all objects in the area provided for the list, or whether it employs clipping. The resulting behaviors are similar as those for the Space object.Contracts for Text Boxes
Selectable Contracts and the Repair Cycle
A useful feature of the system is a “repair cycle” for easily managing the system's automated behaviors. The user has the option of entering the repair cycle whenever the system makes automated contract adjustments in response to a user action. Once in the repair cycle, the user can undo this previous transaction under the current contract and replay the action under other contracts to find the most satisfying result. A repair cycle is illustrated in
In the example sequence shown in
The repair cycle is invoked using conventional well known means such as a function option via the menu structure of the sensemaking application and/or a predetermined key combination or sequence on the keyboard. When invoked the user will be presented with an interface to undo the last transaction performed. Once the transaction is “undone”, the contract user interface (as described with respect to
The currently preferred embodiment is implemented on a computer based system having a “windows” supported Graphical User Interface (GUI), such as Windows from Microsoft Corporation of Redmond Washington, or MAC OS from Apple Computer of Cupertino, California. Such a GUI will typically support operations such as “tap” for selecting/deselecting a graphic object or “double tap” for invoking an operation that may be associated with the graphic object being tapped on. Once an object is selected it may be dragged and dropped to a desired location in the workspace. Mechanisms for providing such user interface operations are well known in the art. Thus no further description for creating such UI operations is deemed necessary.
The currently preferred embodiment has been implemented using object oriented programming techniques of the Java programming environment in combination with a 2D graphics toolkit available from the University of Maryland known as JAZZ. The JAZZ toolkit can be used to create graphics based applications. JAZZ utilizes a “scene graph” model that contains the graphic object information used to render a scene. In this case, the scene being rendered is the sensemaking workspace. A scene graph is comprised of a tree structure that contains information about the objects to be rendered for display as well as “cameras” for providing different views. In the currently preferred embodiment, each Visual Information object and camera is represented by a node on the tree structure. As objects are moved, their position within the scene graph will correspondingly change. JAZZ provides a set of tools that may be used to build applications with a zoomable interface. This feature is exploited when implementing the various panning and zooming operations. Further information regarding JAZZ can be found on the World Wide Web at the website for the University of Maryland.
Each Visual Information Object type is defined by a class and each Visual Information Object is an instance of a class. So in the currently preferred embodiment, there is an object class defined for each of the space, list and text objects. As described above, each Visual Information Object has a distinct user interface. In the system of the currently preferred embodiment, such user interface is implemented using tools provided in the Graphics User Interface (GUI) Library known as SWING (which is available on the Java 2 platform from Sun Microsystems). Such user interface tools are part of the class definition for each Visual Information Object type.
Input from a user is responded to by an Event Handler that is also provided by JAZZ. The input could correspond to movement of the cursor control device, selection, deletion, movement or creation of a Visual Information Object.
Movement of Visual Information Objects
The system then determines where the visual information object is positioned on the workspace and respond accordingly. In one instance it looks to see if it is positioned over a List structure, step 4409. If it is positioned over a list structure, then a preview of the insertion is presented, step 4410. The system then detects whether it is “dropped” into the list, step 4411. If it is dropped into the List structure, then it is “incorporated” into the list structure, step 4412 and any transitive bumping of other visual information objects as a result of the growth of the list occurs, step 4413. Note that incorporation means that the visual information object dropped takes on various characteristics or attributes of the list into which it is inserted. For example, if a text box is inserted into a text list, then the scale and font of the inserted text box takes on the scale and font of the list.
As movement continues, it is detected if the visual information object is dropped into a space visual information object, step 4414. If it is, then it is incorporated and displayed into the space visual information object, step 4415. Note that in this instance incorporation not only includes incorporating attributes of the space, e.g. scaling, but it will also determine whether or not there is room on the area within the space visual information object and bumping of other visual information objects as described above may occur.
It is determined that the visual information object is to be dropped on an open area on the workspace, step 4416. It is first checked to see if there is enough room in the open area to accommodate the visual information object, step 4417. If yes, the visual information object is dropped and displayed in normal unselected mode, step 4419. If there is not enough room, visual information objects are bumped to make space, step 4418. The visual information object is then displayed in original mode per step 4419. If the visual information object is not dropped, then the system continues detecting for user action per the steps starting at step 4403.Computer Controlled Display System
The computer controlled display system as may be utilized in the present invention is embodied in the computer based system illustrated with reference to
Also coupled to the bus 4501 is a display 4504 and a pointing device 4505. The pointing device 4505 may be a stylus, mouse, track-ball or other cursor control device. The network connection 4508 provides means for attaching to a network, e.g. a Local Area Network (LAN) card or modem card with appropriate software. Such access may be to the Internet or to local Intranets or both. Besides communication access, the network connection 4508 may be used to access various resources (i.e. servers) such as shared computing, storage or printing resources.
Moreover, the computer controlled display upon which the computer controlled system may be implemented includes the various forms of interactive systems such as electronic whiteboards, laptop or workstation computers, Personal Digital Assistants, Notebook computers, mainframe systems, client-server systems and the like. Further, the system may be “dedicated” to the sensemaking activity or could be part of a general purpose computer based system.
Finally, it should be noted that the invention has been described in relation to software implementations, but the invention may be implemented with specialized hardware built to operate in a systems architecture as described above.
Although the invention has been described in relation to various implementations, together with modifications, variations, and extensions thereof, other implementations, modifications, variations, and extensions are within the scope of the invention. The invention is therefore not limited by the description contained herein or by the drawings, but only by the claims.
1. A computer controlled free form sensemaking system for user creation and user control of visual information objects, the visual information objects being automatically responsive to user interactions involving objects or the information in them, the system behavior for automatic responses being governed by parameterized contracts, and the system comprising a repair element for enabling a finished interaction to be replayed differently resulting in a different automatic system behavior according to selection of a different value in a parameter in a contract wherein the repair element further comprises:
- a) a recording element for recording interactions amongst said visual information objects;
- b) an undo element for undoing an automatic interaction behavior;
- c) a contract parameter selection element to enable selection of different values for one or more contract parameters, and
- d) a replay element that carries out the user interactions under the different selected contract parameters resulting in a different automatic system behavior without requiring the interactions to be recorded again.
2. The computer controlled free form sensemaking system of claim 1, wherein the parameter selected is for an object with which the user has directly interacted.
3. The computer controlled free form sensemaking system of claim 1, wherein the parameter selected is for an object other than an object with which the user has directly interacted.
4. The computer controlled free form sensemaking system of claim 1, wherein the selectable parameters govern behavior involving one or more of automatic alignment, occlusion, clipping, growth, scaling, shading, object arrangement, aspect ratio, panning, peripheral awareness indicators, or representation of detail.
5. The computer controlled free form sensemaking system of claim 4, wherein the scaling parameter controls objects that are moved into a nested space of objects.
6. The computer controlled free form sensemaking system of claim 4, wherein the shading parameter controls shading of objects that are visually nested inside other objects.
7. The computer controlled free form sensemaking system of claim 1, wherein when user interaction with an object increases the amount of information in the object then the parameter selection governs the automatic behavior of the object for growing the object in its containing object by at least one of altering the scale used for information inside the object, clipping information inside the object, moving information in the populated space to a space outside of the viewed space, or automatic panning to keep the interacting objects in view.
8. The computer controlled free form sensemaking system of claim 1, wherein the parameters that govern the behavior of text objects select between line wrapping or horizontal growth.
9. The computer controlled free form sensemaking system of claim 1, wherein the parameters that govern the behavior of text object select between bumping versus zoom-anti-zoom.
10. The computer controlled free form sensemaking system of claim 1, wherein the parameters that govern the behavior of text objects select between preservation of aspect ratio or font scaling.
11. The computer controlled free form sensemaking system of claim 1, wherein the parameters that govern scaling allow selection of scaling parameters for each type of information object contained within an object.
12. The computer controlled free form, sensemaking system of claim 1, wherein each of said visual information objects is an instance of one of a plurality of visual information object types, said plurality of information object types including one or more of:
- a space object type for indicating a content based relationship amongst visual information objects;
- a text object type for representing textual information having textual behaviors; or
- a structure object type for representing sets of visual information objects having a predetermined structure.
13. The computer controlled free form, sensemaking system as recited in claim 1, wherein said predetermined structure of said structure object type is a list structure.
14. The computer controlled free form, sensemaking system as recited in claim 1, wherein said space object type is displayed as a view area of a larger populated area wherein information objects are positioned.
15. A method of operating a computer controlled free form sensemaking system comprising the steps of:
- a) generating and displaying user perceptible representations of a plurality of visual information objects in a viewed space that is contained within a populated space during user free form sensemaking wherein each user perceptible representation of the plurality of visual information objects has one or more associated contracts for defining automatic behavior responsive to an interaction with other visual information objects;
- b) recording interactions amongst said visual information objects;
- c) undoing an automatic interaction behavior;
- d) detecting a user input indicating a selection of different values for one or more contract parameters; and
- e) replaying the interactions under the different selected contract parameters resulting in a different automatic system behavior without requiring the interactions to be recorded again.
Filed: Jul 13, 2009
Publication Date: Dec 3, 2009
Applicant: Xerox Corporation (Norwalk, CT)
Inventors: Lance E. Good (Cupertino, CA), Mark J. Stefik (Portola Valley, CA), Patrick Baudisch (Seattle, WA), Jock D. Mackinlay (Palo Alto, CA), Polle T. Zellweger (Palo Alto, CA)
Application Number: 12/458,460
International Classification: G09G 5/00 (20060101); G06F 3/048 (20060101);