Application sharing
Ones of the windows associated with a software process are identified in a screen layout on a local display of a sharer network node. On the sharer network node, an image of the identified windows as they are arranged in the screen layout and free of obscuration by any other windows in the screen layout is generated. The image is transmitted from the sharer network node to a viewer network node. Also, commands derived from input device events on sharer and viewer network nodes are received. The received commands are processed into a command sequence, which is passed to a shared process executing on the sharer network node. Windows associated with the shared process are presented on the sharer network node in accordance with the received command sequence. An image of the presented windows is generated and transmitted from the sharer network node to the viewer network node.
Latest Sococo, Inc. Patents:
This application is an application for reissue of U.S. Pat. No. 8,407,605 and relates to the following co-pending patent applications, the entirety of each of which is incorporated herein by reference:
U.S. patent application Ser. No. 12/354,709, filed Jan. 15, 2009;
U.S. patent application Ser. No. 61/120,372, filed Dec. 5, 2008;
U.S. patent application Ser. No. 61/042,714, filed Apr. 5, 2008;
U.S. patent application Ser. No. 11/923,629, filed Oct. 24, 2007; and
U.S. patent application Ser. No. 11/923,634, filed Oct. 24, 2007.
BACKGROUND OF THE INVENTIONWhen face-to-face communications are not practical, people often rely on one or more technological solutions to meet their communications needs. These solutions typically are designed to simulate one or more aspects of face-to-face communications. Traditional telephony systems enable voice communications between callers. Instant messaging (also referred to as “chat”) communications systems enable users to communicate text messages in real time through instant message computer clients that are interconnected by an instant message server. Some instant messaging systems additionally allow users to be represented in a virtual environment by user-controllable graphic objects (referred to as “avatars”). Interactive virtual reality communication systems enable users in remote locations to communicate over multiple real-time channels and to interact with each other by manipulating their respective avatars in three-dimensional virtual spaces. Each of these modes of communications typically can handle some form of data sharing between the communicants.
A common form of data sharing is application sharing, which involves transmitting application data from one node (referred to as the “sharer node”) to one or more other nodes (referred to as “viewer nodes”). Application sharing has a variety of useful applications, including providing remote technical support, remote collaboration, and remote presentation of demonstrations, documents, and images. In some proposed systems, an application sharing program on the sharer node periodically collects drawing commands (e.g., GDI calls for drawing lines and curves, rendering fonts and handling palettes) from a chained display driver process on the sharer node, packages the drawing commands into an order packet, and sends the order packet to a respective counterpart application sharing program on each of the viewer nodes that accurately constructs the shared view of the sharer's display. Such an application sharing approach, however, requires each viewer node to render its own version of the shared application by passing the drawing commands in the order packet to a display process (e.g., the GDI interface provided by a Microsoft® Windows® operating system).
What are needed are improved application sharing apparatus and methods.
BRIEF SUMMARY OF THE INVENTIONIn one aspect, the invention features a method in accordance with which ones of the windows that are associated with a software process are identified in a screen layout on a local display of a sharer network node. On the sharer network node, a composite image of the identified windows as they are arranged in the screen layout and free of obscuration by any other windows in the screen layout is generated. The composite image is transmitted from the sharer network node to a viewer network node.
In one aspect, the invention features a method in accordance with which locally-generated commands that are derived from local input device events on a sharer network node are received. Remotely-generated commands that are derived from remote input device events on a remote viewer network node also are received. The received commands are processed into a command sequence. The command sequence is passed to a shared process executing on the sharer network node. In a screen layout on a local display of the sharer network node, one or more windows that are associated with the shared process are presented in accordance with the received command sequence. An image of the one or more windows as they are presented in the screen layout is generated. The image is transmitted from the sharer network node to the viewer network node.
The invention also features apparatus operable to implement the inventive methods described above and computer-readable media storing computer-readable instructions causing a computer to implement the inventive methods described above.
Other features and advantages of the invention will become apparent from the following description, including the drawings and the claims.
In the following description, like reference numbers are used to identify like elements. Furthermore, the drawings are intended to illustrate major features of exemplary embodiments in a diagrammatic manner. The drawings are not intended to depict every feature of actual embodiments nor relative dimensions of the depicted elements, and are not drawn to scale.
I. DEFINITION OF TERMSA “window” is a visual area of a display that typically includes a user interface. A window typically displays the output of a software process and typically enables a user to input commands or data for the software process. A window that has a parent is called a “child window.” A window that has no parent, or whose parent is the desktop window, is called a “top-level window.” A “desktop” is a system-defined window that paints the background of a graphical user interface (GUI) and serves as the base for all windows displayed by all software processes.
The term “window scraping” refers to a process of extracting data from the display output of another software process. The extraction process is performed by a “scraper” software process.
A “layering order” (also referred to as the “z-order”) is an ordering of overlapping two-dimensional objects, such as windows in a graphical user interface (GUI), along an axis (typically referred to as the “z-axis”) that is perpendicular to a display on which the GUI is presented.
“Compositing” is the combining of visual elements from separate sources into a single composite image (also referred to herein as a “frame”).
The term “obscuration” means the act or process of concealing or hiding by or as if by covering.
A “communicant” is a person who communicates or otherwise interacts with other persons over one or more network connections, where the communication or interaction may or may not occur in the context of a virtual area. A “user” is a communicant who is operating a particular network node that defines a particular perspective for descriptive purposes. A “sharer” is a communicant who is operating a sharer network node. A “viewer” is a communicant who is operating a viewer network node.
A “realtime contact” of a user is a communicant or other person who has communicated with the user via a realtime communications platform.
A “computer” is any machine, device, or apparatus that processes data according to computer-readable instructions that are stored on a computer-readable medium either temporarily or permanently. An “operating system” is a software component of a computer system that manages and coordinates the performance of tasks and the sharing of computing and hardware resources. A “software process” (also referred to as software, an application, computer software, a computer application, a program, and a computer program) is a set of instructions that a computer can interpret and execute to perform one or more specific tasks. A software process may have one or more “threads” of execution. A “shared software process” is a software process whose output is shared with a viewer network node. A “computer data file” is a block of information that durably stores data for use by a software application.
A “database” is an organized collection of records that are presented in a standardized format that can be searched by computers. A database may be stored on a single computer-readable data storage medium on a single computer or it may be distributed across multiple computer-readable data storage media on one or more computers.
A “data sink” (referred to herein simply as a “sink”) is any of a device (e.g., a computer), part of a device, or software that receives data.
A “data source” (referred to herein simply as a “source”) is any of a device (e.g., a computer), part of a device, or software that originates data.
A “network node” (also referred to simply as a “node”) is a junction or connection point in a communications network. Exemplary network nodes include, but are not limited to, a terminal, a computer, and a network switch. A “server” network node is a host computer on a network that responds to requests for information or service. A “client” network node is a computer on a network that requests information or service from a server. A “network connection” is a link between two communicating network nodes. The term “local network node” refers to a network node that currently is the primary subject of discussion. The term “remote network node” refers to a network node that is connected to a local network node by a network communications link. A “connection handle” is a pointer or identifier (e.g., a uniform resource identifier (URI)) that can be used to establish a network connection with a communicant, resource, or service on a network node. A “sharer network node” is a network node that is sharing content with another network node, which is referred to as a “viewer network node. A “network communication” can include any type of information (e.g., text, voice, audio, video, electronic mail message, data file, motion data stream, and data packet) that is transmitted or otherwise conveyed from one network node to another network node over a network connection.
A “communicant interaction” is any type of direct or indirect action or influence between a communicant and another network entity, which may include for example another communicant, a virtual area, or a network service. Exemplary types of communicant interactions include communicants communicating with each other in realtime, a communicant entering a virtual area, and a communicant requesting access to a resource from a network service.
“Presence” refers to the ability and willingness of a networked entity (e.g., a communicant, service, or device) to communicate, where such willingness affects the ability to detect and obtain information about the state of the entity on a network and the ability to connect to the entity:
A “realtime data stream” is data that is structured and processed in a continuous flow and is designed to be received with no delay or only imperceptible delay. Realtime data streams include digital representations of voice, video, user movements, facial expressions and other physical phenomena, as well as data within the computing environment that may benefit from rapid transmission, rapid execution, or both rapid transmission and rapid execution, including for example, avatar movement instructions, text chat, realtime data feeds (e.g., sensor data, machine control instructions, transaction streams and stock quote information feeds), and file transfers.
A “virtual area” (also referred to as an “area” or a “place”) is a representation of a computer-managed space or scene. Virtual areas typically are one-dimensional, two-dimensional, or three-dimensional representations; although in some embodiments a virtual area may correspond to a single point. Oftentimes, a virtual area is designed to simulate a physical, real-world space. For example, using a traditional computer monitor, a virtual area may be visualized as a two-dimensional graphic of a three-dimensional computer-generated space. However, virtual areas do not require an associated visualization to implement switching rules. A virtual area typically refers to an instance of a virtual area schema, where the schema defines the structure and contents of a virtual area in terms of variables and the instance defines the structure and contents of a virtual area in terms of values that have been resolved from a particular context.
A “virtual area application” (also referred to as a “virtual area specification”) is a description of a virtual area that is used in creating a virtual environment. The virtual area application typically includes definitions of geometry, physics, and realtime switching rules that are associated with one or more zones of the virtual area.
A “virtual environment” is a representation of a computer-managed space that includes at least one virtual area and supports realtime communications between communicants.
A “zone” is a region of a virtual area that is associated with at least one switching rule or governance rule. A “switching rule” is an instruction that specifies a connection or disconnection of one or more realtime data sources and one or more realtime data sinks subject to one or more conditions precedent A switching rule controls switching (e.g., routing, connecting, and disconnecting) of realtime data streams between network nodes communicating in the context of a virtual area. A governance rule controls a communicants access to a resource (e.g., an area, a region of an area, or the contents of that area or region), the scope of that access, and follow-on consequences of that access (e.g., a requirement that audit records relating to that access must be recorded). A “renderable zone” is a zone that is associated with a respective visualization.
A “position” in a virtual area refers to a location of a point or an area or a volume in the virtual area. A point typically is represented by a single set of one-dimensional, two-dimensional, or three-dimensional coordinates (e.g., Cartesian coordinates, polar coordinates, or spherical coordinates) that define a spot in the virtual area. A coordinate can be defined as any single or plurality of numbers that establish location. An area typically is represented by the three-dimensional coordinates of three or more coplanar vertices that define a boundary of a closed two-dimensional shape in the virtual area. A volume typically is represented by the three-dimensional coordinates of four or more non-coplanar vertices that define a closed boundary of a three-dimensional shape in the virtual area.
A “spatial state” is an attribute that describes where a user has presence in a virtual area. The spatial state attribute typically has a respective value (e.g., a zone_ID value) for each of the zones in which the user has presence.
A “placemark” is a stored reference (e.g., a hyperlink) to a location in a virtual area. A placemark typically can be selected to present a view of the associated location in the virtual area to a user. The verb “placemark” means the act or operation of creating a placemark.
In the context of a virtual area, an “object” is any type of discrete element in a virtual area that may be usefully treated separately from the geometry of the virtual area. Exemplary objects include doors, portals, windows, view screens, and speakerphone. An object typically has attributes or properties that are separate and distinct from the attributes and properties of the virtual area. An “avatar” is an object that represents a communicant in a virtual area.
The term “double-click” refers to the act or operation of entering or inputting an execution command (e.g., double-clicking the left computer mouse button or by single-clicking a user interface button associated with an execute command, e.g., enter zone or view object). The term “shift-click” refers to the act or operation of entering or inputting a selection command (e.g., clicking the left computer mouse button) while the Shift key of an alphanumeric input device is activated. The term “shift-double-click” refers to the act or operation of entering or inputting an execution command while the Shift key of an alphanumeric input device is activated.
As used herein, the term “includes” means includes but not limited to, the term “including” means including but not limited to. The term “based on” means based at least in part on.
II. INTRODUCTIONThe embodiments that are described herein enable application sharing with high fidelity, realtime performance, viewer immersion, and privacy protection. In some embodiments, the screen content that is associated with each thread of a software process can be determined and composited into a respective composite image (or frame) that is free of other window content. The contents of the windows that are associated with one or more software application threads on a network node can be broadcasted to other network nodes without risk of obscuration by the screen content (e.g., windows containing application content, messages, or dialog boxes) that might be generated by other software processes, thereby preventing corruption of shared window content by overlapping screen content that sometimes is generated by processes that are outside of the user's immediate control. This feature avoids the need for sharers to interrupt a presentation in order to remove obscuring screen content, thereby creating a more immersive collaboration experience for viewers of the shared window content. In addition, sharers don't have to worry that private information inadvertently will be shared along with the intended screen content, thereby maintaining the sharer's privacy.
Some embodiments also enable multichannel application sharing in which two or more communicants share applications and screen content with each other at the same time. These embodiments typically include an interface that allows each receiver to distinguish one shared window from another.
III. APPLICATION SHARINGA. Introduction
Embodiments of the communications application may implement one or more of the following application sharing modes:
-
- Share all windows created by a given process;
- Share only one window and do not share any other windows;
- Share a given window and all child windows, where the child windows may belong to the same process that created the given window or they may belong to a different process.
- Share multiple applications. For instance, composing windows created by different processes into a single frame instead of sharing each of the applications independently.
In accordance with the method of
The communications application 26 generates a composite image of the identified windows as they are arranged in the screen layout and free of obscuration by any other windows in the screen layout (
In addition to preventing obscuration by other windows, some embodiments also prevent obscuration of the selected windows as a result of the windows being partially or completely off-screen (e.g., outside the visible desktop window that contains the screen layout). For example, a respective image of each of the identified windows is stored in a respective memory buffer, and the process of generating the composite image involves retrieving each of the images from the respective memory buffers and compositing the retrieved images into the composite image. For example, in some exemplary embodiments, each of the windows is a layered window whose screen data is stored in a respective memory buffer through a programmatic call to the Microsoft® Win32 application programming interface (API), which is available in Microsoft® Windows® operating systems versions 2000 and later. These operating systems provide an extended window style that is invoked by setting the WS_EX_LAYERED window style bit. The WS_EX_LAYERED style bit that is associated with a particular window may be set by the shared software process at window creation time (e.g., via the CreateWindowEx API call) or it may be set by the communications application 26 after creation time (e.g., via SetWindowLong API call with GWL_EXSTYLE). With the WS_EX_LAYERED window style bit set for a window, the operating system redirects the drawing of the window into an off-screen bitmap and buffer, which can then be accessed by the communications application 26 for generating the composite image. Similar layered windows functionality is available from other operating systems (e.g., X-Windows on a UNIX based operating system).
After the composite image has been generated, the communications application 26 transmits the composite image to the viewer one of the first and second network nodes (i.e., the one of the first and second network nodes that receives the composite image) 12, 14 (
B. Application Sharing Embodiments
1. Introduction
In some embodiments, application sharing is initiated after a sharer network node has published one or more applications or documents that are available to share and at least one viewer has subscribed to at least one of the published applications or documents. In some embodiments, the sharer can publish a shared application or document to a viewscreen object that is associated with a virtual area and the viewer can subscribe to the shared content by activating the viewscreen object in the virtual area (e.g., by double-clicking on the viewscreen object with a user input device).
The viewer typically is granted one of two types of access to the shared content: view access, which allows the viewer to only passively view the shared content; and control access, which allows the viewer to view, control, edit, and manipulate the shared content. The type of access that is granted to the viewer can be set by the sharer or by one or more governance rules that are associated with the context in which the sharing takes place (e.g., a governance rule that is associated with a zone of a virtual area, as described below in section IV).
The shared content typically is streamed from the sharer network node to the viewer network node in the form of streaming bitmaps of the windows on the sharers display that are associated with the shared application or document The bitmap of each window can be streamed separately or already composited. The bitmaps typically are compressed before being streamed. If the viewer has only view access, then the viewer can only passively view images of the shared windows on the sharers display. If the viewer has control access, then the viewer network node can transmit remote control commands that are generated by a user input device (e.g., a keyboard, a computer mouse, a touchpad, and a touch screen) to the sharer network node for controlling, editing, and manipulating the shared content on the sharers network node.
2. Application Sharing Service
In some embodiments, the application sharing functionality of the communications application 26 is provided by a Scraper module, which is a plug-in for an Application Sharing Service that implements a platform specific part of the application sharing. This embodiment implements the application sharing mode in which all windows created by a shared application automatically are shared with subscribing network nodes. This section describes an exemplary embodiment of the Scraper module and the Application Sharing Service that are implemented in a Microsoft® Windows® application environment that provides layered windows functionality.
a. Classes
b. Methods
(i) The Methods Start and Stop
Before Application Sharing Service calls any method on the Scraper module it calls the start method and, during shutdown, it calls the stop method. In the illustrated embodiment no calls can be made before the start method was called and after the stop method was called.
In the start method, the Scraper module starts the thread that listens for WinEvents. The Scraper module listens for WinEvents to get notifications when windows and menus are created/destroyed and shown/hidden.
In the stop method, the Scraper module stops all the application monitors and then shuts down the thread that is listening for WinEvents.
When a WinEvent notification is received, the Scraper module gets the thread identifier (ID) and the process ID for the window. The Scraper module then looks up a monitor by the process ID and notifies the application monitor about the event. In response to a notification that a window was destroyed, the Scraper module notifies all of the application monitors of the event since the process and thread IDs are not available.
(ii) The Method Get_Sharable_Application
As shown in
(iii) The Method Start_Share_Application
Referring to
(iv) The Method Stop_Share_Application
Referring to
(v) The Method Get_Shared_Applications
This method returns a list of applications being shared.
(vi) The Method Subscribe
Referring to
(vii) The Method Unsubscribe
Referring to
c. Window Scraping
Each application monitor has a thread that wakes up on a periodic basis and performs window scraping.
In accordance with the method of
The scraper module determines a bounding rectangle that encompasses all the windows that are associated with the shared software process (
For each of the windows that are associated with the shared software process, the Scraper module calls a scrape function (
allows individual window compressed bitmaps to be sent, instead of just the compressed composited bitmap.
After completing the scraping process, the Scraper module creates a device independent bitmap of the composite image (
The Scraper module starts with an initial list of all top-level windows that are associated with the shared software process (
The Scraper module determines a z-order of all the windows that currently are associated with the shared software process (
The Scraper module sorts the initial list of top-level windows that are associated with the software process according to the determined z-order (
The Scraper module appends to the sorted list any of the windows in the initial list of top-level windows that are not included in the sorted list (
The Scraper module replaces the initial list with the sorted list of top-level windows (
d. Remote Access
If the viewers have only view access, then the Application Sharing Service on the sharer network node only transmits the composite images of the shared window content (in the form of samples) to the subscribing ones of the viewer network nodes. The viewers on the viewer network nodes can only passively view the composite images of the shared windows on the sharers display.
If the viewers have control access, on the other hand, then the Application Sharing Service on the sharer network node transmits the composite images of the shared window content (in the form of samples) to the subscribing ones of the viewer network nodes. In addition, the Scraper module combines commands that are received from the viewer network nodes with the commands that are generated by the sharer on the sharer network node, and passes the combined set of commands to the shared application. This allows the viewers to control, edit, and manipulate the shared application on the sharer network node. The commands typically are derived from events that are generated by one or more user input devices (e.g., a keyboard, a computer mouse, a touchpad, and a touch screen) on the viewer and sharer network nodes.
The display processes 620, 628 provide the display facilities of the sharer network node 610 and the viewer network node 612, respectively. The display facilities control the writing of visual contents on the sharer and viewer displays 618, 626. In some embodiments, the display facilities include a graphic device interface (e.g., the GDI available in Microsoft® Windows® application environments) that provides functions that can be called by software processes in order to present visual content on the displays 618, 626.
The network layers 624, 632 provide the networking facilities of the sharer network node 610 and the viewer network node 612, respectively. The network facilities include, for example, networking communications protocol stacks and networking hardware that perform processes associated with sending and receiving information over the network 18.
The communications applications 622, 630 respectively provide various communications facilities (including application sharing facilities) to the sharer network node 610 and the viewer network node 612. In the illustrated embodiment, the communications application 622 on the sharer network node 610 generates a composite image 634 of the shared window content 614 on the sharer's display, transmits the composite image 634 over the network 18 to the viewer network node 612 for presentation on the viewer's display 626, and grants the viewer remote control access to the shared window content 614. The communications application 630 on the viewer network node 612 controls the presentation of the composite image 634 on the display 626, transforms user input into commands, and transmits the commands to the sharer network node 610.
In accordance with the method of
The communications application 622 receives commands that are derived from local input device events generated on the sharer network node 610 (
The operating systems on the sharer and viewer network nodes 610, 612 typically convert the pointer input device events into user commands, where the location parameter values are defined with respect to the coordinate system of the main window of the graphical user interface (e.g., the desktop window in a Microsoft® Windows® application environment). The sharer's input commands that are received by the communications application 622 (
The communications application 622 processes the received commands into a command sequence (
The communications application 622 passes the command sequence to the shared process 616 (
The shared process 616 calls one or more graphical device interface functions that are provided by the display process 620 to present the windows that are associated with the shared software process 616 on the sharer display 618 in accordance with the received command sequence (
The process (
A. System Architecture
1. Introduction
In some embodiments, the network infrastructure service environment 30 manages sessions of the first and second client nodes 12, 14 in a virtual area 32 in accordance with a virtual area application 34. The virtual area application 34 is hosted by the virtual area 32 and includes a description of the virtual area 32. The communications applications 26 operating on the first and second client network nodes 12, 14 present respective views of the virtual area 32 in accordance with data received from the network infrastructure service environment 30 and provide respective interfaces for receiving commands from the communicants. The communicants typically are represented in the virtual area 32 by respective avatars, which move about the virtual area 32 in response to commands that are input by the communicants at their respective network nodes. Each communicants view of the virtual area 32 typically is presented from the perspective of the communicants avatar, which increases the level of immersion experienced by the communicant. Each communicant typically is able to view any part of the virtual area 32 around his or her avatar. In some embodiments, the communications applications 26 establish realtime data stream connections between the first and second client network nodes 12, 14 and other network nodes sharing the virtual area 32 based on the positions of the communicants' avatars in the virtual area 32.
The network infrastructure service environment 30 also maintains a relationship database 36 that contains records 38 of interactions between communicants. Each interaction record 38 describes the context of an interaction between a pair of communicants.
2. Network Environment
The network 18 may include any of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN) (e.g., the internet). The network 18 typically includes a number of different computing platforms and transport facilities that support the transmission of a wide variety of different media types (e.g., text, voice, audio, and video) between network nodes.
The communications application 26 (see
3. Network Infrastructure Services
The network infrastructure service environment 30 typically includes one or more network infrastructure services that cooperate with the communications applications 26 in the process of establishing and administering network connections between the client nodes 12, 14 and other network nodes (see
Account Service
The account service manages communicant accounts for the virtual environment. The account service also manages the creation and issuance of authentication tokens that can be used by client network nodes to authenticate themselves to any of the network infrastructure services.
Security Service
The security service controls communicants' access to the assets and other resources of the virtual environment. The access control method implemented by the security service typically is based on one or more of capabilities (where access is granted to entities having proper capabilities or permissions) and an access control list (where access is granted to entities having identities that are on the list). After a particular communicant has been granted access to a resource, that communicant typically uses the functionality provided by the other network infrastructure services to interact in the network communications environment 10.
Area Service
The area service administers virtual areas. In some embodiments, the area service remotely configures the communications applications 26 operating on the first and second client network nodes 12, 14 in accordance with the virtual area application 34 subject to a set of constraints 47 (see
The area service also manages network connections that are associated with the virtual area subject to the capabilities of the requesting entities, maintains global state information for the virtual area, and serves as a data server for the client network nodes participating in a shared communication session in a context defined by the virtual area 32. The global state information includes a list of all the objects that are in the virtual area and their respective locations in the virtual area. The area service sends instructions that configure the client network nodes. The area service also registers and transmits initialization information to other client network nodes that request to join the communication session. In this process, the area service may transmit to each joining client network node a list of components (e.g., plugins) that are needed to render the virtual area 32 on the client network node in accordance with the virtual area application 34. The area service also ensures that the client network nodes can synchronize to a global state if a communications fault occurs. The area service typically manages communicant interactions with virtual areas via governance rules that are associated with the virtual areas.
Rendezvous Service
The rendezvous service manages the collection, storage, and distribution of presence information and provides mechanisms for network nodes to communicate with one another (e.g., by managing the distribution of connection handles) subject to the capabilities of the requesting entities. The rendezvous service typically stores the presence information in a presence database. The rendezvous service typically manages communicant interactions with each other via communicant privacy preferences.
Interaction Service
The interaction service maintains the relationship database 36 that contains the records 38 of interactions between communicants. For every interaction between communicants, one or more services of the network infrastructure service environment 30 (e.g., the area service) transmit interaction data to the interaction service. In response, the interaction service generates one or more respective interaction records and stores them in the relationship database. Each interaction record describes the context of an interaction between a pair of communicants. For example, in some embodiments, an interaction record contains an identifier for each of the communicants, an identifier for the place of interaction (e.g., a virtual area instance), a description of the hierarchy of the interaction place (e.g., a description of how the interaction room relates to a larger area), start and end times of the interaction, and a list of all files and other data streams that are shared or recorded during the interaction. Thus, for each realtime interaction, the interaction service tracks when it occurred, where it occurred, and what happened during the interaction in terms of communicants involved (e.g., entering and exiting), objects that are activated/deactivated, and the files that were shared.
The interaction service also supports queries on the relationship database 36 subject to the capabilities of the requesting entities. The interaction service presents the results of queries on the interaction database records in a sorted order (e.g., most frequent or most recent) based on virtual area. The query results can be used to drive a frequency sort of contacts whom a communicant has met in which virtual areas, as well as sorts of who the communicant has met with regardless of virtual area and sorts of the virtual areas the communicant frequents most often. The query results also may be used by application developers as part of a heuristic system that automates certain tasks based on relationships. An example of a heuristic of this type is a heuristic that permits communicants who have visited a particular virtual area more than five times to enter without knocking by default or a heuristic that allows communicants who were present in an area at a particular time to modify and delete files created by another communicant who was present in the same area at the same time. Queries on the relationship database 36 can be combined with other searches. For example, queries on the relationship database may be combined with queries on contact history data generated for interactions with contacts using a communication system (e.g., Skype, Facebook, and Flickr) that is outside the domain of the network infrastructure service environment 30.
4. Virtual Areas
The communications application 26 and the network infrastructure service environment 30 typically administer the realtime connections with network nodes in a communication context that is defined by an instance of a virtual area. The virtual area instance may correspond to an abstract (non-geometric) virtual space that is defined with respect to abstract coordinates. Alternatively, the virtual area instance may correspond to a visual virtual space that is defined with respect to one-, two- or three-dimensional geometric coordinates that are associated with a particular visualization. Abstract virtual areas may or may not be associated with respective visualizations, whereas visual virtual areas are associated with respective visualizations.
As explained above, communicants typically are represented by respective avatars in a virtual area that has an associated visualization. The avatars move about the virtual area in response to commands that are input by the communicants at their respective network nodes. In some embodiments, the communicants view of a virtual area instance typically is presented from the perspective of the communicants avatar, and each communicant typically is able to view any part of the visual virtual area around his or her avatar, increasing the level of immersion that is experienced by the communicant.
As explained in detail below, the virtual area 66 includes zones 74, 76, 78, 80, 82 that are associated with respective rules that govern the switching of realtime data streams between the network nodes that are represented by the avatars 68-72 in the virtual area 66. (During a typical communication session, the dashed lines demarcating the zones 74-82 in
A virtual area is defined by a specification that includes a description of geometric elements of the virtual area and one or more rules, including switching rules and governance rules. The switching rules govern realtime stream connections between the network nodes. The governance rules control a communicants access to resources, such as the virtual area itself, regions with the virtual area, and objects within the virtual area. In some embodiments, the geometric elements of the virtual area are described in accordance with the COLLADA-Digital Asset Schema Release 1.4.1 April 2006 specification (available from http://www.khronos.org/collada/), and the switching rules are described using an extensible markup language (XML) text format (referred to herein as a virtual space description format (VSDL)) in accordance with the COLLADA Streams Reference specification described in U.S. application Ser. Nos. 11/923,629 and 11/923,634.
The geometric elements of the virtual area typically include physical geometry and collision geometry of the virtual area. The physical geometry describes the shape of the virtual area. The physical geometry typically is formed from surfaces of triangles, quadrilaterals, or polygons. Colors and textures are mapped onto the physical geometry to create a more realistic appearance for the virtual area. Lighting effects may be provided, for example, by painting lights onto the visual geometry and modifying the texture, color, or intensity near the lights. The collision geometry describes invisible surfaces that determine the ways in which objects can move in the virtual area. The collision geometry may coincide with the visual geometry, correspond to a simpler approximation of the visual geometry, or relate to application-specific requirements of a virtual area designer.
The switching rules typically include a description of conditions for connecting sources and sinks of realtime data streams in terms of positions in the virtual area. Each rule typically includes attributes that define the realtime data stream type to which the rule applies and the location or locations in the virtual area where the rule applies. In some embodiments, each of the rules optionally may include one or more attributes that specify a required role of the source, a required role of the sink, a priority level of the stream, and a requested stream handling topology. In some embodiments, if there are no explicit switching rules defined for a particular part of the virtual area, one or more implicit or default switching rules may apply to that part of the virtual area. One exemplary default switching rule is a rule that connects every source to every compatible sink within an area, subject to policy rules. Policy rules may apply globally to all connections between the client nodes or only to respective connections with individual client nodes. An example of a policy rule is a proximity policy rule that only allows connections of sources with compatible sinks that are associated with respective objects that are within a prescribed distance (or radius) of each other in the virtual area.
In some embodiments, governance rules are associated with a virtual area to control who has access to the virtual area, who has access to its contents, what is the scope of that access to the contents of the virtual area (e.g., what can a user do with the contents), and what are the follow-on consequences of accessing those contents (e.g., record keeping, such as audit logs, and payment requirements). In some embodiments, an entire virtual area or a zone of the virtual area is associated with a “governance mesh.” In some embodiments, a governance mesh is implemented in a way that is analogous to the implementation of the zone mesh described in U.S. application Ser. Nos. 11/923,629 and 11/923,634. A governance mesh enables a software application developer to associate governance rules with a virtual area or a zone of a virtual area. This avoids the need for the creation of individual permissions for every file in a virtual area and avoids the need to deal with the complexity that potentially could arise when there is a need to treat the same document differently depending on the context.
In some embodiments, a virtual area is associated with a governance mesh that associates one or more zones of the virtual area with a digital rights management (DRM) function. The DRM function controls access to one or more of the virtual area or one or more zones within the virtual area or objects within the virtual area. The DRM function is triggered every time a communicant crosses a governance mesh boundary within the virtual area. The DRM function determines whether the triggering action is permitted and, if so, what is the scope of the permitted action, whether payment is needed, and whether audit records need to be generated. In an exemplary implementation of a virtual area, the associated governance mesh is configured such that if a communicant is able to enter the virtual area he or she is able to perform actions on all the documents that are associated with the virtual area, including manipulating the documents, viewing the documents, downloading the documents, deleting the documents, modifying the documents and re-uploading the documents. In this way, the virtual area can become a repository for information that was shared and discussed in the context defined by the virtual area.
Additional details regarding the specification of a virtual area are described in U.S. application Ser. No. 61/042,714 (which was filed on Apr. 4, 2008), Ser. No. 11/923,629 (which was filed on Oct. 24, 2007), and Ser. No. 11/923,634 (which was filed on Oct. 24, 2007).
5. Communications Application
In some embodiments, the communications application 26 includes:
a. local Human Interface Devices (HIDs) and audio playback devices;
b. a So3D graphical display, avatar, and physics engine;
c. a system database and storage facility.
a. Local Human Interface Devices (HIDS) and Audio Playback Devices
The local HIDs enable a communicant to input commands and other signals into the client network node while participating in a virtual area communications session. Exemplary HIDs include a computer keyboard, a computer mouse, a touch screen display, and a microphone.
The audio playback devices enable a communicant to playback audio signals that are received during a virtual area communications session. Exemplary audio playback devices include audio processing hardware (e.g., a sound card) for manipulating (e.g., mixing and applying special effects) audio signals, and speakers for outputting sounds.
b. So3D Graphical Display, Avatar, and Physics Engine
The So3D engine is a three-dimensional visualization engine that controls the presentation of a respective view of a virtual area and objects in the virtual area on a display monitor. The So3D engine typically interfaces with a graphical user interface driver and the HID devices to present the views of the virtual area and to allow the communicant to control the operation of the communications application 26.
In some embodiments, the So3D engine receives graphics rendering instructions from the area service. The So3D engine also may read a local communicant avatar database that contains images needed for rendering the communicants avatar in the virtual area. Based on this information, the So3D engine generates a visual representation (i.e., an image) of the virtual area and the objects in the virtual area from the point of view (position and orientation) of the communicants avatar in the virtual area. The visual representation typically is passed to the graphics rendering components of the operating system, which drive the graphics rendering hardware to render the visual representation of the virtual area on the client network node.
The communicant can control the presented view of the virtual area by inputting view control commands via a HID device (e.g., a computer mouse). The So3D engine updates the view of the virtual area in accordance with the view control commands. The So3D engine also updates the graphic representation of the virtual area on the display monitor in accordance with updated object position information received from the area service 26.
c. System Database and Storage Facility
The system database and storage facility stores various kinds of information that is used by the platform. Exemplary information that typically is stored by the storage facility includes the presence database, the relationship database, an avatar database, a real user id (RUID) database, an art cache database, and an area application database. This information may be stored on a single network node or it may be distributed across multiple network nodes.
6. Client Node Architecture
A communicant typically connects to the network 18 from a client network node. The client network node typically is implemented by a general-purpose computer system or a dedicated communications computer system (or “console”, such as a network-enabled video game console). The client network node executes communications processes that establish realtime data stream connections with other network nodes and typically executes visualization rendering processes that present a view of each virtual area entered by the communicant.
A communicant may interact (e.g., input commands or data) with the computer system 120 using one or more input devices 130 (e.g. one or more keyboards, computer mice, microphones, cameras, joysticks, physical motion sensors such Wii input devices, and touch pads). Information may be presented through a graphical user interface (GUI) that is presented to the communicant on a display monitor 132, which is controlled by a display controller 134. The computer system 120 also may include other input/output hardware (e.g., peripheral output devices, such as speakers and a printer). The computer system 120 connects to other network nodes through a network adapter 136 (also referred to as a “network interface card” or NIC).
A number of program modules may be stored in the system memory 124, including application programming interfaces 138 (APIs), an operating system (OS) 140 (e.g., the Windows XP® operating system available from Microsoft Corporation of Redmond, Wash. U.S.A.), the communications application 26, drivers 142 (e.g., a GUI driver), network transport protocols 144, and data 146 (e.g., input data, output data, program data, a registry, and configuration settings).
7. Server Node Architecture
In some embodiments, the one or more server network nodes of the virtual environment creator 16 are implemented by respective general-purpose computer systems of the same type as the client network node 120, except that each server network node typically includes one or more server software applications.
In other embodiments, the one or more server network nodes of the virtual environment creator 16 are implemented by respective network devices that perform edge services (e.g., routing and switching).
B. Exemplary Communication Session
Referring back to
In some embodiments, the area service maintains global state information that includes a current specification of the virtual area, a current register of the objects that are in the virtual area, and a list of any stream mixes that currently are being generated by the network node hosting the area service. The objects register typically includes for each object in the virtual area a respective object identifier (e.g., a label that uniquely identifies the object), a connection handle (e.g., a URI, such as an IP address) that enables a network connection to be established with a network node that is associated with the object, and interface data that identifies the realtime data sources and sinks that are associated with the object (e.g., the sources and sinks of the network node that is associated with the object). The objects register also typically includes one or more optional role identifiers for each object; the role identifiers may be assigned explicitly to the objects by either the communicants or the area service, or may be inferred from other attributes of the objects or the user. In some embodiments, the objects register also includes the current position of each of the objects in the virtual area as determined by the area service from an analysis of the realtime motion data streams received from the network nodes associated with objects in the virtual area. In this regard, the area service receives realtime motion data streams from the network nodes associated with objects in the virtual area, tracks the communicants' avatars and other objects that enter, leave, and move around in the virtual area based on the motion data. The area service updates the objects register in accordance with the current locations of the tracked objects.
In the process of administering realtime data stream connections with other network nodes, the area service maintains for each of the client network nodes a set of configuration data, including interface data, a zone list and the positions of the objects that currently are in the virtual area. The interface data includes for each object associated with each of the client network nodes a respective list of all the sources and sinks of realtime data stream types that are associated with the object. The zone list is a register of all the zones in the virtual area that currently are occupied by the avatar associated with the corresponding client network node. When a communicant first enters a virtual area, the area service typically initializes the current object positions database with position initialization information. Thereafter, the area service updates the current object positions database with the current positions of the objects in the virtual area as determined from an analysis of the realtime motion data streams received from the other client network nodes sharing the virtual area.
C. Interfacing with a Spatial Virtual Communication Environment
In addition to the local Human Interface Device (HID) and audio playback devices, the So3D graphical display, avatar, and physics engine, and the system database and storage facility, the communications application 26 also includes a graphical navigation and interaction interface (referred to herein as a “seeker interface”) that interfaces the user with the spatial virtual communication environment. The seeker interface includes navigation controls that enable the user to navigate the virtual environment and interaction controls that enable the user to control his or her interactions with other communicants in the virtual communication environment. The navigation and interaction controls typically are responsive to user selections that are made using any type of input device, including a computer mouse, a touch pad, a touch screen display, a keyboard, and a video game controller. The seeker interface is an application that operates on each client network node. The seeker interface is a small, lightweight interface that a user can keep up and running all the time on his or her desktop. The seeker interface allows the user to launch virtual area applications and provides the user with immediate access to realtime contacts and realtime collaborative places (or areas). The seeker interface is integrated with realtime communications applications and/or realtime communications components of the underlying operating system such that the seeker interface can initiate and receive realtime communications with other network nodes. A virtual area is integrated with the users desktop through the seeker interface such that the user can upload files into the virtual environment created by the virtual environment creator 16, use files stored in association with the virtual area using the native client software applications independently of the virtual environment while still present in a virtual area, and more generally treat presence and position within a virtual area as an aspect of their operating environment analogous to other operating system functions rather than just one of several applications.
The spatial virtual communication environment typically can be modeled as a spatial hierarchy of places (also referred to herein as “locations”) and objects. The spatial hierarchy includes an ordered sequence of levels ranging from a top level to a bottom level. Each of the places in a successive one of the levels of the spatial hierarchy is contained in a respective one of the places in a preceding one of the levels. Each of the objects in the spatial hierarchy is contained in a respective one of the places. The levels of the spatial hierarchy typically are associated with respective visualizations that are consistent with a geographical, architectural, or urban metaphor, and are labeled accordingly. The zones of each virtual area are defined by respective meshes, some of which define elements of a physical environment (e.g., spaces, such as rooms and courtyards, that are associated with a building) that may contain objects (e.g., avatars and props, such as view screen objects and conferencing objects).
The navigational controls of the seeker interface allow the user to traverse a path through the virtual environment in accordance with a navigational model that is tied to the underlying spatial hierarchy of places and objects. The network infrastructure service environment 30 records the path traversed by the user. In some embodiments, the network infrastructure service environment 30 records a history that includes a temporally ordered list of views of the virtual area that are presented to the user as the user navigates through the virtual area. Each view typically corresponds to a view of a respective renderable zone of the virtual area. In these embodiments, the navigation controls enable the user to move to selected ones of the zones in the history. The navigation controls also include a graphical representation of a depth path that shows the location in the spatial hierarchy that corresponds to the users current view of the virtual area. In some embodiments, the graphical representation of the depth path includes a respective user-selectable link to a respective view of each of the preceding levels in the spatial hierarchical model of the virtual area above the current view.
The interaction controls of the seeker interface allow the user to manage interactions with other communicants. The interaction options that available to the user typically depend on the zones in which the user has a presence. In some embodiments, the interaction options that are available to communicants who have presence in a particular zone are different from the options that are available to other communicants who do not have presence in that zone. The level of detail and interactivity of the user typically depend on whether or not the user has a presence the particular zone. In one exemplary embodiment, if the user is outside the virtual area, the user is provided with a minimal level of detail of the interactions occurring within the virtual area (e.g., the user can see an outline of the floorplan, background textures, and plants of the area, but the user cannot see where other communicants are present in the area); if the user is within the virtual area but outside a particular zone of the area, the user is provided with a medium level of detail of the interactions occurring within the particular zone (e.g., the user can see where other communicants are present in the area, see a visualization of their current states—talking, typing a chat message, whether or not their headphones and microphones are turned-on—and see whether any of the view screens are active); if the user is within the particular zone of the area, the user is provided with full level of detail of the interactions occurring with the particular zone (e.g., the user can see a thumbnail of the file being shared on a view screen, hear and speak with other communicants in the area, and see elements of a log of chat messages that were generated by communicants in the zone). In some embodiments, the switching and governance rules that are associated with the zones of the virtual area control how the network infrastructure services distinguish between those who have presence in the particular zone from those who do not.
Each of the tabs 164 typically is associated with a respective view of the virtual environment. In the illustrated embodiment the view presented in the table 164 (labeled “My Areas”) is associated with a respective set of virtual areas, which may be a default set of virtual areas in the virtual environment or it may be a set of virtual areas that is identified by a respective filter on the interaction database. In particular, the tab 164 is associated with a set of three virtual areas (i.e., Acme, Sococo Help Area, and Personal Space), which may be a default set of areas that are associated with the user or may be identified by a filter that identifies all of the areas that are associated with the user (e.g., all of the areas in which the user has interacted). Additional tabs may be created by selecting the “+” button 170.
The browsing area 166 of each tab shows graphic representations of the elements of the virtual environment that are associated with the tab. For example, in the illustrated embodiment, the browsing area 166 shows top-level views 172, 174, 176 of the virtual areas that are associated with the tab 164. The user may navigate to the next lower level in the spatial hierarchical model of any of the virtual areas by selecting the corresponding graphic representation of the virtual area.
The toolbar 168 includes an adaptive set of navigational and interaction tools that automatically are selected by the seeker interface based on the current location of the user in the virtual environment In the illustrated embodiment, the toolbar 168 includes a back button 178, a forward button 180, a placemarks button 182, and a home button 184. The back button 178 corresponds to a backward control that enables the user to incrementally move backward to preceding ones of the zones in the history of the zones that were traversed by the user. The forward button 180 corresponds to a forward control that enables the user to incrementally move forward to successive ones of the zones in the history of the zones that were traversed by the user. The placemarks button 182 provides a placemarking control for storing links to zones and a placemark navigation control for viewing a list of links to previously placemarked zones. In response to a user selection of the placemarking control, a placemark is created by storing an image of the location shown in the current view in association with a hyperlink to the corresponding location in the virtual area. In response to a user selection of the placemark navigation control, a placemarks window is presented to the user. The placemarks window includes live visualizations of all locations that have been placemarked by the user. Each of the images in the placemarks window is associated with a respective user-selectable hyperlink. In response to user selection of one of the hyperlinks in the placemarks window, a view of the virtual area corresponding to the location associated with the selected hyperlink is automatically displayed in the browsing area 166 of the seeker interface window 162. The home button 184 corresponds to a control that returns the user to a view of a designated “home” location in the virtual environment (e.g., the view shown in
Referring to
The seeker interface shows a top or floorplan view of the Acme virtual area in the browsing area 166 of the tab 164 and provides the user with a default set of interaction options. In the illustrated embodiment, a presence automatically is established in a courtyard zone 190 of the virtual area, and the user's microphone and default speakers (e.g., headphones) are turned-on. In the floorplan view shown in
In addition to the backward button 178, the forward button 180, the placemarks button 182, and the home button 184, the toolbar 168 also includes a series of one or more breadcrumb buttons 207 that originate from and include the home button 184. The breadcrumb buttons 207 correspond to a hierarchical sequence of successive, user-selectable links. Each of the successive links corresponds to a view of a respective level in the hierarchical model of the virtual area in which each successive level is contained by preceding ones of the levels. In the illustrated embodiment, the breadcrumb buttons 207 include the home button 184 and an Acme button 208 that corresponds to the current view of the Acme virtual area shown in
When an area is selected or in focus, the button 210 appears as an iconographic representation of two people and is labeled “members,” and allows members and moderators to see the list of people associated with an area. When an audio or chat zone is in focus, the button 210 has a different image (e.g., an image of an arrow pointing downward onto a plane to represent the operation of getting) and is labeled “get”. In response to a user selection of the button 210, a list of all the members of the Acme virtual area 166 is displayed in a user interface. The user may select any of the communicants in the list and click a get button that is presented in the user interface; in response, the platform transmits an invitation to the selected communicant to join the user in a designated one of the zones.
The settings button 212 provides the user with access to a set of controls for specifying default settings that are associated with the current area.
The user may navigate from the view of the Acme virtual area shown in
The user may navigate to any of the zones of the Acme virtual area. In some embodiments, in order to move to a zone, the user transmits a command to execute one of the zones displayed on the monitor, (e.g., by selecting the zone and then clicking the Enter button (in the toolbar), or, as a shortcut, double-clicking the zone) and, in response, the platform depicts the user's avatar in the zone corresponding to the zone object. In response to the zone execution command, the seeker interface outlines the zone (indicating to the user that it is selected) and updates the breadcrumb buttons 207 to show the selected zone location in the hierarchy. Toolbar buttons specific to the selection will also appear to the right of the breadcrumb buttons 207.
The user also may interact with any objects (e.g., a screen, table, or file) that are present in a zone. In some embodiments, in order to interact with an object, the user transmits a command to execute one of the objects displayed on the monitor, (e.g., by selecting the object and then clicking the View button (in the toolbar), or, as a shortcut, double-clicking the object) and, in response, the platform performs an operation with respect to the object (e.g., present a zoomed-in view of the object, open an interaction interface window, etc.). In response to the object execution command, the seeker interface outlines or otherwise highlights the prop (indicating to the user that it is selected) and updates the breadcrumb buttons 207 to show the selected object location in the hierarchy. Toolbar buttons specific to the selection will also appear to the right of the breadcrumb buttons 207.
Referring to
In response to the user's command to execute the wall object 290, the seeker interface presents in the browsing area 166 of the tab 164 a 2.5-dimensional view of the contents of the wall object 290 and areas of the Main space 213 surrounding the wall object 290. In the embodiment shown in
The interface also shows in the minimap 256 a view of the Main space 213 and areas of the Acme space surrounding the Main space 213. The minimap 256 also shows a highlighted view 292 of the selected North Wall object 290 in the Main space 213.
The breadcrumb buttons 207 shown in the toolbar 168 of the tab 164 include a North Wall button 294 that corresponds to the current level in the hierarchical spatial model of the virtual area. The toolbar 168 includes a rotate left button 296 and a rotate right button 298 that allow the user to rotate the current view left and right by ninety degrees (90°) so that the user can view the contents of different walls of the Main space in the central viewing area of the 2.5-dimensional view of the Main space 213. The user also can double-click a different one of the walls that are shown in the minimap 256 in order to change the contents that are presented in the central viewing area of the 2.5-dimensional view of the Main space 213.
Referring to
Referring to
One or more viewers can subscribe to the windows of the shared application showing the contents of the selected data file by clicking (or double-clicking) on the thumbnail image shown on the viewscreen object 291. Each viewer can view, control, edit, and manipulate the shared window content presented on the viewscreen object 291 subject to any governance rules that are associated with the selected data file or the zone containing the viewscreen object 291. If a viewer has control access to the shared window content, the viewer can input commands to the shared process executing on the sharer network node by using one or more input devices on the viewer's network node as described above in section IV. Assuming that realtime performance can be achieved over the respective network connections between the sharer network node and the viewer network nodes, the edits and other manipulations of the shared data file typically will appear to each of the collaborators as if they were made on the same network node.
V. CONCLUSIONThe embodiments that are described herein enable application sharing with high fidelity, realtime performance, viewer immersion, and privacy protection. Some embodiments also enable multichannel application sharing in which two or more communicants share applications and screen content with each other at the same time. These embodiments typically include an interface that allows each receiver to distinguish one shared window from another.
Other embodiments are within the scope of the claims.
Claims
1. A method, comprising, by a sharer network node:
- on a local display of the sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, a representation of a second viewscreen object in the virtual area, a representation of a user who is present in the virtual area and operating the sharer network node, and a respective representation of each of one or more other communicants who are present in the virtual area and respectively operating one or more other network nodes;
- in a screen layout of windows on the local display, selecting multiple windows that are associated with a first software process based on user input in connection with the first viewscreen object, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- generating a first sequence of composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;
- transmitting the first sequence of composite images from the sharer network node to a particular one of the one or more other network nodes in connection with the first viewscreen object; and
- contemporaneously with the transmitting, communicating a second sequence of composite images of windows associated with a second software process between the sharer network node and the particular one of the one or more other network nodes in connection with the second viewscreen object.
2. The method of claim 1, wherein the selecting comprises selecting all of the windows in the screen layout that are associated with the first software process.
3. The method of claim 1, wherein the selecting comprises selecting ones of the windows in the screen layout that match a handle that is assigned to the first software process.
4. The method of claim 3, wherein the selecting comprises selecting a parent window and at least one associated child window that are created by the first software process.
5. The method of claim 1, wherein the generating comprises for each of the composite images in the first sequence:
- determining a layering order of the selected windows in relation to each other that corresponds to the relative layering order of the selected windows in the screen layout;
- for each of the selected windows, retrieving a respective image of the window; and
- compositing the retrieved images into the composite image in accordance with the determined layering order.
6. The method of claim 5, wherein the determining comprises:
- generating a z-order list that associates a respective z-order value with a respective window handle for each of the windows in the screen layout; and
- deriving the layering order of the selected windows from the z-order list.
7. The method of claim 6, wherein the deriving comprises:
- for each of the z-order values in the z-order list, matching the associated window handle with the window handle of a respective one of the selected windows; and
- ordering the selected windows in the layering order in accordance with the respective z-order values in the z-order list associated with the ones of the z-order values determined to match the window handles of the selected windows.
8. The method of claim 5, wherein the generating further comprises determining two-dimensional locations of the selected windows in the screen layout for each of the composite images in the first sequence.
9. The method of claim 1, wherein:
- the selecting comprises selecting ones of the windows that are associated with a designated group of software processes; and
- the generating comprises, for each of the composite images in the first sequence: determining a layering order of the selected windows in relation to each other that corresponds to the relative layering order of the selected windows in the screen layout; for each of the selected windows, retrieving a respective image of the window; and compositing the retrieved images into the composite image in accordance with the determined layering order.
10. The method of claim 1, wherein a respective image of each of the selected windows is stored in a respective memory buffer, and the generating comprises retrieving each of the images from the respective memory buffers and compositing the retrieved images into the composite image for each of the composite images in the first sequence.
11. The method of claim 1, further comprising by the sharer network node setting the windows that are associated with the first software process to a layered windows style such that a respective image of each of the selected windows is stored in a respective memory buffer.
12. The method of claim 1, wherein the transmitting is performed based on a request from the viewer network node to view screen data associated with the first viewscreen object.
13. The method of claim 12, further comprising transmitting the first sequence of composite images from the sharer network node to multiple of the other network nodes based on a respective request from each of the multiple other network nodes to view screen data associated with the first viewscreen object, wherein the first sequence of composite images is transmitted to the multiple other network nodes contemporaneously.
14. The method of claim 1, wherein the virtual area comprises a plurality of zones, each zone of the plurality of zones supporting establishment of communicant presence in the zone; and further comprising:
- on the local display, displaying a visualization of the virtual area, wherein the visualization comprises a spatial layout of the zones;
- on the local display, presenting navigation controls and interaction controls, wherein the navigation controls enable the user to specify where to establish a presence in the virtual area and the interaction controls enable the user to manage interactions with the one or more other communicants in the virtual area;
- establishing a respective presence of the user in one or more of the zones of the virtual area in response to input received via the navigation controls; and
- on the local display, depicting the representations of the user and the one or more other communicants in the zones in which the user and the one or more other communicants are present.
15. The method of claim 1, further comprising presenting the first sequence of composite images on the local display in connection with the first viewscreen object.
16. The method of claim 1, further comprising by the sharer network node:
- receiving locally-generated commands that are derived from local input device events on the sharer network node;
- receiving remotely-generated commands that are derived from remote input device events on a remote viewer network node;
- processing the received locally-generated commands and the received remotely-generated commands into a command sequence;
- passing the command sequence to the first software process;
- presenting the windows in the screen layout on the local display in accordance with the received command sequence.
17. The method of claim 16, wherein the processing comprises remapping screen location parameter values in the remotely-generated commands from a coordinate system of a shared application image received from the sharer network node to a coordinate system of a main window on the local display.
18. The method of claim 17, wherein the processing comprises arranging into the command sequence (i) the remotely-generated commands with the remapped screen location parameter values and (ii) the locally-generated commands.
19. The method of claim 18, wherein the arranging comprises ordering (i) the remotely-generated commands with the remapped screen location parameter values and (ii) the locally-generated commands based on respective times that the locally-generated commands are derived from the local input device events and respective times that the remotely-generated commands are received.
20. The method of claim 17, further comprising remapping screen location parameter values in the commands derived from the remote input device events from a coordinate system of a main window on a remote display of the viewer network node to a coordinate system of the shared application image.
21. The method of claim 20, wherein the remapping is performed on the viewer network node, and further comprising transmitting the remotely-generated commands with the remapped screen location parameter values to the sharer network node.
22. The method of claim 16, wherein the command sequence controls operation of the shared process.
23. The method of claim 1, wherein:
- in the screen layout, at least one of the selected windows is obscured by a particular unselected one of the windows in the screen layout; and
- the generating comprises retrieving a respective image of each of the selected windows from a respective memory buffer for each of respective ones of the composite images in the first sequence, and, from the retrieved images, generating the composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by the particular unselected window in the screen layout.
24. The method of claim 1, wherein the transmitting is based on a subscription of the particular one of the other network nodes to content associated with the first viewscreen object.
25. The method of claim 24, wherein
- multiple of the communicant operating respective ones of the other network nodes are contemporaneously subscribed to content associated with the first viewscreen object; and
- the transmitting comprises contemporaneously transmitting the sequence of composite images to the network nodes being operated by the communicants who are subscribed to the content associated with the first viewscreen object.
26. The method of claim 24, wherein the virtual area is associated with a governance rule that controls access to the sequence of composite images.
27. The method of claim 26, wherein the governance rule defines one or more conditions on abilities of communicants in the virtual area to subscribe to the transmitted sequence of composite images.
28. The method of claim 26, wherein the governance rule defines one or more conditions on one or more types of access to the transmitted sequence of composite images.
29. The method of claim 28, wherein the governance rule defines a condition on abilities of communicants in the virtual area to view the transmitted sequence of composite images.
30. The method of claim 28, wherein the governance rule defines a condition on abilities of communicants in the virtual area to control the transmitted sequence of composite images.
31. The method of claim 1, further comprising selecting one or more windows that are associated with a second software process, and generating the first sequence of composite image from the selected windows of the first and second software processes as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any unselected ones of the windows in the screen layout.
32. The method of claim 1, wherein the communicating comprises by the sharer network node receiving the second sequence of composite images from the particular one of the one or more other network nodes.
33. The method of claim 1, wherein the communicating comprises by the sharer network node transmitting the second sequence of composite images to the particular one of the one or more other network nodes.
34. A method, comprising:
- in a screen layout of windows on a local display of a sharer network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- on the sharer network node, generating a composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout, wherein the generating comprises generating the composite image of the selected windows that includes any contents of the selected windows that are located outside a visible desktop window that contains the screen layout; and
- transmitting the composite image from the sharer network node to a viewer network node.
35. A method, comprising:
- on a local display of a sharer network node, displaying a visualization of a virtual area comprising a plurality of zones, wherein the visualization comprises a spatial layout of the plurality of zones, each zone of the plurality of zones supports establishment of communication sessions between network nodes associated with the zone, a sharer at the sharer network node and a viewer at the viewer network node both have presence in a particular one of the zones of the plurality that comprises viewscreen objects, and the displaying comprises displaying respective representations of the viewer, the sharer, and the viewscreen objects in the virtual area;
- on the local display, presenting navigation controls and interaction controls, wherein the navigation controls enable the sharer to specify where to establish a presence in the virtual area and the interaction controls enable the sharer to manage interactions with the viewer in the virtual area;
- in response to input received via the navigation controls, establishing a presence of the sharer in the particular zone of the virtual area;
- in a screen layout of windows on the local display, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- on the sharer network node, generating a composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;
- transmitting the composite image from the sharer network node to a viewer network node;
- performing the selecting, the generating, and the transmitting with respect to one or more other software processes executing on one or more other network nodes; and
- on the local display, presenting the respective composite images transmitted from the one or more other network nodes in connection with different respective ones of the viewscreen objects in the particular zone of the virtual area at the same time.
36. The method of claim 35, further comprising for each of the viewscreen objects in connection with which a respective one of the composite images is presented, on the local display showing the graphical depiction of the respective sharer who is operating the network node from which the respective composite image is transmitted.
37. A method, comprising:
- in a screen layout of windows on a local display of a sharer network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- on the sharer network node, generating a composite image of the selected windows as they are arranged in the screen layout but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;
- transmitting the composite image from the sharer network node to a viewer network node;
- sending a request to share screen data associated with the a software process executing on the viewer network node; and
- at the sharer network node, receiving a composite image of shared screen data from the viewer network node, wherein the receiving and the transmitting are performed contemporaneously.
38. At least one non-transitory, tangible computer-readable medium having computer-readable program code embodied therein, the computer-readable program code adapted to be executed by a computer to perform operations comprising:
- on a local display of the sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, a representation of a second viewscreen object in the virtual area, a representation of a user who is present in the virtual area and operating the sharer network node, and a respective representation of each of one or more other communicants who are present in the virtual area and respectively operating one or more other network nodes;
- in a screen layout of windows on the local display, selecting multiple windows that are associated with a first software process based on user input in connection with the first viewscreen object, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- generating a first sequence of composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout;
- transmitting the first sequence of composite images from the sharer network node to a particular one of the one or more other network nodes in connection with the first viewscreen object; and
- contemporaneously with the transmitting, communicating a second sequence of composite images of windows associated with a second software process between the sharer network node and the particular one of the one or more other network nodes in connection with the second viewscreen object.
39. An apparatus, comprising:
- a memory storing computer-readable instructions; and
- a data processing unit coupled to the memory, operable to execute the instructions, and based at least in part on the execution of the instructions operable to perform operations comprising on a local display of the sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, a representation of a second viewscreen object in the virtual area, a representation of a user who is present in the virtual area and operating the sharer network node, and a respective representation of each of one or more other communicants who are present in the virtual area and respectively operating one or more other network nodes; in a screen layout of windows on the local display selecting multiple windows that are associated with a first software process based on user input in connection with the first viewscreen object, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout; generating a first sequence of composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the unselected ones of the windows in the screen layout; transmitting the first sequence of composite images from the sharer network node to a particular one of the one or more other network nodes in connection with the first viewscreen object; and contemporaneously with the transmitting, communicating a second sequence of composite images of windows associated with a second software process between the sharer network node and the particular one of the one or more other network nodes in connection with the second viewscreen object.
40. A method, comprising:
- on a local display of a sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, and a representation of a second viewscreen object in the virtual area;
- in a screen layout of windows on the local display of the sharer network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- based on user input, associating the software process with the first viewscreen object;
- on the sharer network node, generating a first sequence of successive composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration by any other windows in the screen layout;
- transmitting the first sequence of composite images from the sharer network node to a first viewer network node based on a subscription of the first viewer network node to content associated with the first viewscreen object;
- selecting a second set of one or more of the windows in the screen layout;
- generating a second sequence of successive composite images of the selected windows in the second set as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the windows in the screen layout outside the second set; and
- based on a subscription of a second viewer network node to content associated with the second viewscreen object, transmitting the second sequence of composite images from the sharer network node to the second viewer network node, wherein the first sequence of composite images and the second sequence of composite images are respectively transmitted to the first and second viewer network nodes contemporaneously.
41. A method, comprising:
- on a local display of a sharer network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, and a representation of a second viewscreen object in the virtual area;
- in a screen layout of windows on the local display of the sharer network node, selecting ones of the windows that are associated with a first software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- based on user input, associating the first software process with the first viewscreen object;
- on the sharer network node, generating a first sequence of successive composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration by any other windows in the screen layout;
- transmitting the first sequence of composite images from the sharer network node to a first viewer network node based on a subscription of the first viewer network node to content associated with the first viewscreen object;
- based on user input, associating a second software process with the second viewscreen object;
- selecting a second set of one or more of the windows that associated with the second software process;
- generating a second sequence of successive composite images of the selected windows in the second set as they are arranged in the screen layout at different respective times but free of obscuration in any area obscured by any of the windows in the screen layout outside the second set; and
- based on a subscription of a second viewer network node to content associated with the second viewscreen object, transmitting the second sequence of composite images from the sharer network node to the second viewer network node, wherein the first sequence of composite images and the second sequence of composite images are respectively transmitted to the first and second viewer network nodes contemporaneously.
42. A method, comprising:
- on a local display of a first network node, displaying a representation of a virtual area, a representation of a first viewscreen object in the virtual area, and a representation of a second viewscreen object in the virtual area, wherein a first communicant is operating the first network node and a second communicant is operating a second network node, the first and second communicants are present in a particular zone of the virtual area comprising the first viewscreen object and the second viewscreen object, and the displaying comprises displaying in the virtual area a representation of the first communicant and a representation of the second communicant;
- in a screen layout of windows on the local display of the first network node, selecting ones of the windows that are associated with a software process, wherein at least one of the selected windows is at least partially obscured by one or more unselected ones of the windows in the screen layout;
- based on input from the first communicant, the first network node associating the software process with the first viewscreen object;
- the first network node generating a first sequence of successive composite images of the selected windows as they are arranged in the screen layout at different respective times but free of obscuration by any other windows in the screen layout;
- transmitting the first sequence of composite images from the first network node to the second network node based on a subscription of the second network node to content associated with the first viewscreen object;
- based on input from the first communicant, the first network node subscribing to content associated with the second viewscreen object;
- the first network node receiving from the second network node a second sequence of composite images of one or more windows associated with a software process executing on the second network node, wherein the transmitting of the first sequence of composite images from the first network node to the second network node is contemporaneous with the receiving of the second sequence of composite images by the first network node from the second network node; and
- presenting the received second sequence of composite images on the local display in association with the second viewscreen object.
| 5471318 | November 28, 1995 | Ahuja |
| 5491743 | February 13, 1996 | Shiio et al. |
| 5627978 | May 6, 1997 | Altom |
| 5727155 | March 10, 1998 | Dawson |
| 5737533 | April 7, 1998 | Hond |
| 5758110 | May 26, 1998 | Boss |
| 5764916 | June 9, 1998 | Busey |
| 5793365 | August 11, 1998 | Tang et al. |
| 5938724 | August 17, 1999 | Pommier |
| 5949414 | September 7, 1999 | Namikata et al. |
| 5956038 | September 21, 1999 | Rekimoto |
| 5982372 | November 9, 1999 | Brush, II et al. |
| 5995096 | November 30, 1999 | Kitahara et al. |
| 5999206 | December 7, 1999 | Inoue et al. |
| 5999208 | December 7, 1999 | McNerney et al. |
| 6005578 | December 21, 1999 | Cole |
| 6047314 | April 4, 2000 | Pommier et al. |
| 6057856 | May 2, 2000 | Miyashita et al. |
| 6119147 | September 12, 2000 | Toomey et al. |
| 6215498 | April 10, 2001 | Filo |
| 6219045 | April 17, 2001 | Leahy et al. |
| 6226655 | May 1, 2001 | Borman et al. |
| 6304283 | October 16, 2001 | Kitagawa |
| 6308199 | October 23, 2001 | Katsurabayashi |
| 6335927 | January 1, 2002 | Elliott et al. |
| 6342906 | January 29, 2002 | Kumar et al. |
| 6380952 | April 30, 2002 | Mass et al. |
| 6392760 | May 21, 2002 | Ahuja |
| 6396509 | May 28, 2002 | Cheng |
| 6426778 | July 30, 2002 | Valdez |
| 6473096 | October 29, 2002 | Kobayashi et al. |
| 6480191 | November 12, 2002 | Balabanovic |
| 6567980 | May 20, 2003 | Jain et al. |
| 6570587 | May 27, 2003 | Efrat et al. |
| 6572476 | June 3, 2003 | Shoji et al. |
| 6580441 | June 17, 2003 | Schileru-Key |
| 6704784 | March 9, 2004 | Matsuda et al. |
| 6708172 | March 16, 2004 | Wong et al. |
| 6714222 | March 30, 2004 | Chincholle et al. |
| 6721741 | April 13, 2004 | Eyal et al. |
| 6731314 | May 4, 2004 | Cheng et al. |
| 6735708 | May 11, 2004 | Busey |
| 6772195 | August 3, 2004 | Hatlelid et al. |
| 6784901 | August 31, 2004 | Harvey et al. |
| 6833863 | December 21, 2004 | Clemens |
| 6862625 | March 1, 2005 | Busey |
| 6909443 | June 21, 2005 | Robertson et al. |
| 7007235 | February 28, 2006 | Hussein |
| 7036082 | April 25, 2006 | Dalrymple |
| 7058896 | June 6, 2006 | Hughes |
| 7080096 | July 18, 2006 | Imamura |
| 7086005 | August 1, 2006 | Matsuda |
| 7145898 | December 5, 2006 | Elliott |
| 7165213 | January 16, 2007 | Busey |
| 7168048 | January 23, 2007 | Goossen et al. |
| 7181690 | February 20, 2007 | Leahy et al. |
| 7184037 | February 27, 2007 | Gallery et al. |
| 7188317 | March 6, 2007 | Hazel |
| 7240093 | July 3, 2007 | Danieli et al. |
| 7260784 | August 21, 2007 | Crichton |
| 7263526 | August 28, 2007 | Busey |
| 7293243 | November 6, 2007 | Ben-Shachar et al. |
| 7305438 | December 4, 2007 | Christensen et al. |
| 7336779 | February 26, 2008 | Boyer |
| 7356563 | April 8, 2008 | Leichtling |
| 7415502 | August 19, 2008 | Vishik et al. |
| 7418664 | August 26, 2008 | Ben-Shachar |
| 7451181 | November 11, 2008 | Sasaki |
| 7467356 | December 16, 2008 | Gettman et al. |
| 7474741 | January 6, 2009 | Brunson |
| 7503006 | March 10, 2009 | Danieli |
| 7516411 | April 7, 2009 | Grossner |
| 7533346 | May 12, 2009 | McGrath et al. |
| 7616624 | November 10, 2009 | John |
| 7640300 | December 29, 2009 | Wohlgemuth et al. |
| 7680098 | March 16, 2010 | John |
| 7680480 | March 16, 2010 | John |
| 7680885 | March 16, 2010 | Schauser |
| 7734692 | June 8, 2010 | Kaplan et al. |
| 7765259 | July 27, 2010 | MacVarish |
| 7769806 | August 3, 2010 | Wie et al. |
| 7813488 | October 12, 2010 | Kozdon |
| 7840668 | November 23, 2010 | Sylvain |
| 7844724 | November 30, 2010 | Wie et al. |
| 7908663 | March 15, 2011 | Horvitz et al. |
| 7958453 | June 7, 2011 | Taing |
| 7979574 | July 12, 2011 | Gillo et al. |
| 8191001 | May 29, 2012 | Wie et al. |
| 8578044 | November 5, 2013 | Wie et al. |
| 8621079 | December 31, 2013 | Wie et al. |
| 8930472 | January 6, 2015 | Leacock et al. |
| 20010023450 | September 20, 2001 | Chu |
| 20010034740 | October 25, 2001 | Kerne |
| 20020019833 | February 14, 2002 | Hanamoto |
| 20020026388 | February 28, 2002 | Roebuck |
| 20020033844 | March 21, 2002 | Levy et al. |
| 20020033848 | March 21, 2002 | Sciammarella et al. |
| 20020049814 | April 25, 2002 | Yoo |
| 20020052918 | May 2, 2002 | Rekimoto et al. |
| 20020080195 | June 27, 2002 | Carlson et al. |
| 20020097267 | July 25, 2002 | Dinan et al. |
| 20020112028 | August 15, 2002 | Colwill |
| 20020116458 | August 22, 2002 | Bricklin et al. |
| 20020165922 | November 7, 2002 | Wei |
| 20020169829 | November 14, 2002 | Shuster |
| 20020178072 | November 28, 2002 | Gusler et al. |
| 20020188959 | December 12, 2002 | Piotrowski |
| 20030009469 | January 9, 2003 | Platt et al. |
| 20030037110 | February 20, 2003 | Yamamoto |
| 20030043200 | March 6, 2003 | Faieta et al. |
| 20030065558 | April 3, 2003 | Shaw et al. |
| 20030077561 | April 24, 2003 | Alsop et al. |
| 20030149731 | August 7, 2003 | Ohwa |
| 20030158953 | August 21, 2003 | Lal |
| 20030184579 | October 2, 2003 | Zhang et al. |
| 20030189599 | October 9, 2003 | Ben-Shachar et al. |
| 20030192049 | October 9, 2003 | Schneider et al. |
| 20030195928 | October 16, 2003 | Kamijo et al. |
| 20030197739 | October 23, 2003 | Bauer |
| 20030215779 | November 20, 2003 | Dupont et al. |
| 20040078444 | April 22, 2004 | Malik |
| 20040128350 | July 1, 2004 | Topfl et al. |
| 20040179038 | September 16, 2004 | Blattner et al. |
| 20040183827 | September 23, 2004 | Putterman et al. |
| 20040210847 | October 21, 2004 | Berson et al. |
| 20040268451 | December 30, 2004 | Robbin et al. |
| 20050021624 | January 27, 2005 | Herf et al. |
| 20050058094 | March 17, 2005 | Lazaridis et al. |
| 20050075885 | April 7, 2005 | Danieli |
| 20050080866 | April 14, 2005 | Kent |
| 20050086612 | April 21, 2005 | Gettman et al. |
| 20050088452 | April 28, 2005 | Hanggie et al. |
| 20050132299 | June 16, 2005 | Jones et al. |
| 20050138570 | June 23, 2005 | Good et al. |
| 20050144247 | June 30, 2005 | Christensen et al. |
| 20050154574 | July 14, 2005 | Takemura et al. |
| 20050215252 | September 29, 2005 | Jung et al. |
| 20050232168 | October 20, 2005 | Schauser |
| 20050261980 | November 24, 2005 | Hadi |
| 20060005187 | January 5, 2006 | Neil |
| 20060031779 | February 9, 2006 | Theurer |
| 20060041684 | February 23, 2006 | Daniell et al. |
| 20060117264 | June 1, 2006 | Beaton et al. |
| 20060136837 | June 22, 2006 | Ben-Shachar |
| 20060161624 | July 20, 2006 | Montgomery et al. |
| 20060167996 | July 27, 2006 | Orsolini et al. |
| 20060184886 | August 17, 2006 | Chung et al. |
| 20060230156 | October 12, 2006 | Shappir et al. |
| 20060248159 | November 2, 2006 | Polan |
| 20060271877 | November 30, 2006 | Theurer |
| 20060293103 | December 28, 2006 | Mendelsohn |
| 20070011232 | January 11, 2007 | Manion et al. |
| 20070047700 | March 1, 2007 | Mohler |
| 20070061399 | March 15, 2007 | Schmieder et al. |
| 20070070066 | March 29, 2007 | Bakhash |
| 20070101282 | May 3, 2007 | Goossen |
| 20070136686 | June 14, 2007 | Price |
| 20070156908 | July 5, 2007 | Szomolanyi et al. |
| 20070160129 | July 12, 2007 | Fujisawa et al. |
| 20070162432 | July 12, 2007 | Armstrong |
| 20070184855 | August 9, 2007 | Klassen et al. |
| 20070192427 | August 16, 2007 | Berstis et al. |
| 20070198726 | August 23, 2007 | Marco et al. |
| 20070204047 | August 30, 2007 | Parker et al. |
| 20070220568 | September 20, 2007 | Fukasawa |
| 20070226357 | September 27, 2007 | McMurry et al. |
| 20070233785 | October 4, 2007 | Abraham |
| 20070234212 | October 4, 2007 | de Souza |
| 20070279484 | December 6, 2007 | Derocher et al. |
| 20070291034 | December 20, 2007 | Dones |
| 20070291706 | December 20, 2007 | Miller et al. |
| 20070299778 | December 27, 2007 | Haveson et al. |
| 20080019285 | January 24, 2008 | John |
| 20080021949 | January 24, 2008 | John |
| 20080033941 | February 7, 2008 | Parrish |
| 20080052373 | February 28, 2008 | Pousti |
| 20080086696 | April 10, 2008 | Prakash et al. |
| 20080091692 | April 17, 2008 | Keith et al. |
| 20080098121 | April 24, 2008 | Wu |
| 20080101561 | May 1, 2008 | Choi et al. |
| 20080133580 | June 5, 2008 | Wanless et al. |
| 20080163090 | July 3, 2008 | Cortright |
| 20080168154 | July 10, 2008 | Skyrm et al. |
| 20080209075 | August 28, 2008 | Shamma |
| 20080221998 | September 11, 2008 | Mendelsohn et al. |
| 20080244458 | October 2, 2008 | Brugiolo |
| 20080250115 | October 9, 2008 | Iyer |
| 20080252637 | October 16, 2008 | Berndt et al. |
| 20080262910 | October 23, 2008 | Altberg et al. |
| 20080262911 | October 23, 2008 | Altberg et al. |
| 20080263460 | October 23, 2008 | Altberg et al. |
| 20080301557 | December 4, 2008 | Kotlyar |
| 20090018912 | January 15, 2009 | Altberg et al. |
| 20090079816 | March 26, 2009 | Qvarfordt et al. |
| 20090089364 | April 2, 2009 | Hamilton, II et al. |
| 20090089685 | April 2, 2009 | Mordecai et al. |
| 20090096810 | April 16, 2009 | Green |
| 20090112997 | April 30, 2009 | Parker, II et al. |
| 20090113066 | April 30, 2009 | Wie et al. |
| 20090113314 | April 30, 2009 | Dawson et al. |
| 20090128567 | May 21, 2009 | Shuster et al. |
| 20090132943 | May 21, 2009 | Minsky et al. |
| 20090193077 | July 30, 2009 | Horii et al. |
| 20090199095 | August 6, 2009 | Nicol, II |
| 20090222742 | September 3, 2009 | Pelton et al. |
| 20090235180 | September 17, 2009 | Liu et al. |
| 20090241037 | September 24, 2009 | Hyndman |
| 20090247196 | October 1, 2009 | Kim et al. |
| 20090251457 | October 8, 2009 | Walker et al. |
| 20090254840 | October 8, 2009 | Churchill et al. |
| 20090254843 | October 8, 2009 | Wie et al. |
| 20090259948 | October 15, 2009 | Hamilton, II et al. |
| 20090286605 | November 19, 2009 | Hamilton, II et al. |
| 20090288007 | November 19, 2009 | Leacock et al. |
| 20090300521 | December 3, 2009 | Jerrard-Dunne et al. |
| 20090307620 | December 10, 2009 | Hamilton, II et al. |
| 20100058202 | March 4, 2010 | Rostom |
| 20100058229 | March 4, 2010 | Mercer |
| 20100077034 | March 25, 2010 | Alkov et al. |
| 20100131868 | May 27, 2010 | Chawla |
| 20100146085 | June 10, 2010 | Wie et al. |
| 20100146118 | June 10, 2010 | Wie |
| 20100162121 | June 24, 2010 | Yoakum |
| 20100164956 | July 1, 2010 | Hyndman |
| 20100169796 | July 1, 2010 | Lynk |
| 20100169799 | July 1, 2010 | Hyndman |
| 20100169801 | July 1, 2010 | Blattner et al. |
| 20100169837 | July 1, 2010 | Hyndman |
| 20100211880 | August 19, 2010 | Haggar et al. |
| 20100221693 | September 2, 2010 | Gupta |
| 20100228547 | September 9, 2010 | Scott et al. |
| 20100228560 | September 9, 2010 | Balasaygun |
| 20100235501 | September 16, 2010 | Klemm |
| 20100241432 | September 23, 2010 | Michaelis |
| 20100246570 | September 30, 2010 | Chavez |
| 20100246571 | September 30, 2010 | Geppert |
| 20100246800 | September 30, 2010 | Geppert |
| 20100251119 | September 30, 2010 | Geppert |
| 20100251124 | September 30, 2010 | Geppert |
| 20100251127 | September 30, 2010 | Geppert |
| 20100251142 | September 30, 2010 | Geppert |
| 20100251158 | September 30, 2010 | Geppert |
| 20100251177 | September 30, 2010 | Geppert |
| 20100257450 | October 7, 2010 | Go et al. |
| 20100262550 | October 14, 2010 | Burritt |
| 20100274848 | October 28, 2010 | Altmaier et al. |
| 20100287274 | November 11, 2010 | Martinez et al. |
| 20100322395 | December 23, 2010 | Michaelis |
| 20110029898 | February 3, 2011 | Malik |
| 20110029907 | February 3, 2011 | Bakhash |
| 20110106662 | May 5, 2011 | Stinchcomb |
| 20110169927 | July 14, 2011 | Mages et al. |
| 20110196930 | August 11, 2011 | Chawla et al. |
| 20110231781 | September 22, 2011 | Betzler et al. |
| 20110274104 | November 10, 2011 | Cozzi et al. |
| 20110302509 | December 8, 2011 | Leacock et al. |
| 20120066306 | March 15, 2012 | Leacock et al. |
| 20120124486 | May 17, 2012 | Robinson |
| 20120179672 | July 12, 2012 | Wie et al. |
| 20120216131 | August 23, 2012 | Moyers et al. |
| 20120246582 | September 27, 2012 | Leacock et al. |
| 20120254858 | October 4, 2012 | Moyers et al. |
| 20130073978 | March 21, 2013 | Butler et al. |
| 20130109418 | May 2, 2013 | Lazaridis et al. |
| 20140129644 | May 8, 2014 | Mohan |
| 20140213309 | July 31, 2014 | Lazaridis et al. |
| 1678994 | October 2005 | CN |
| 1701568 | November 2005 | CN |
| 101499080 | August 2009 | CN |
| 1964597 | September 2008 | EP |
| 1964597 | September 2008 | EP |
| 2237537 | October 2010 | EP |
| 2239930 | October 2010 | EP |
| 2002123479 | April 2002 | JP |
| 2002149580 | May 2002 | JP |
| 2002224447 | August 2002 | JP |
| 2003067317 | March 2003 | JP |
| 2004272579 | September 2004 | JP |
| 2005182331 | July 2005 | JP |
| 2005286749 | October 2005 | JP |
| 2007506309 | March 2007 | JP |
| 2007184871 | July 2007 | JP |
| 2007251380 | September 2007 | JP |
| 2008182670 | August 2008 | JP |
| 2010535363 | November 2010 | JP |
| 1019990078775 | November 1999 | KR |
| 1020000030491 | June 2000 | KR |
| 1020010100589 | November 2001 | KR |
| 1020030054874 | July 2003 | KR |
| 1020040011825 | February 2004 | KR |
| 1020060060788 | June 2006 | KR |
| 1020070105088 | October 2007 | KR |
| 1020090016692 | February 2009 | KR |
| 2006127429 | November 2006 | WO |
| 2008008806 | January 2008 | WO |
| 2008106196 | September 2008 | WO |
| 2009000028 | December 2008 | WO |
- Search report and written opinion issued on Aug. 13, 2010, in counterpart PCT Application No. PCT/US2010/020596.
- International Search Report and Written Opinion issued in counterpart International Application No. PCT/US2012/030766 (mailed Oct. 19, 2012).
- 01-Allowance Notice dated Oct. 9, 2012 in related U.S. Appl. No. 12/354,709, filed Jan. 15, 2009.
- 02-IDS dated Sep. 20, 2012 in related U.S. Appl. No. 12/354,709, filed Jan. 15, 2009.
- 03-RCE dated Sep. 20, 2012 in related U.S. Appl. No. 12/354,709, filed Jan. 15, 2009.
- 04-Allowance Notice dated Aug. 28, 2012 in related U.S. Appl. No. 12/354,709, filed Jan. 15, 2009.
- 05-Amendment dated Mar. 18, 2012 in related U.S. Appl. No. 12/354,709, filed Jan. 15, 2009.
- 06-Non-final Office Action dated Sep. 7, 2011 in related U.S. Appl. No. 12/354,709, filed Jan. 15, 2009.
- 07-Non-final Office action dated Dec. 27, 2011 in related U.S. Appl. No. 12/354,709, filed Jan. 15, 2009.
- 11-Final Office Action dated Sep. 10, 2012 in related U.S. Appl. No. 12/509,658, filed Jul. 27, 2009.
- 12-Amendment dated May 30, 2012 in related U.S. Appl. No. 12/509,658, filed Jul. 27, 2009.
- 13-Non-Final Office Action dated Mar. 1, 2012 in related U.S. Appl. No. 12/509,658, filed Jul. 27, 2009.
- 14-Pre-Appeal Brief Request for Review dated Nov. 10, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 15-Advisory Action dated Oct. 16, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 16-Response After Final dated Sep. 14, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 17-Final Rejection dated Sep. 10, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 18-Amendment dated Jul. 20, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 19-Non-Final Office Action dated Apr. 22, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 20-Notice of Allowance dated Jun. 20, 2016 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 21-Appeal Brief dated Dec. 8, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- 22-Pre-Brief Appeal Conference Decision dated Nov. 27, 2015 in related U.S. Appl. No. 13/666,717, filed Nov. 1, 2012.
- “Microsoft Computer Dictionary”, Third Edition, 1997. Definitions for“audit,” “auditing,” and “audit trail” (p. 36), “log” (p. 288, “security log” (p. 426), “user account,” and “user profile” (p. 488).
- International search report and written opinion received in counterpart International application No. PCTIUS2010/028088, mailed Oct. 27, 2010.
Type: Grant
Filed: Mar 23, 2015
Date of Patent: Feb 14, 2017
Assignee: Sococo, Inc. (Eugene, OR)
Inventors: Alexander Say Go (Sammamish, WA), Vladimir Petter (Bellevue, WA)
Primary Examiner: Dennis Bonshock
Application Number: 14/666,298
International Classification: G06F 3/00 (20060101); G06F 3/048 (20130101); G06F 15/16 (20060101); G06F 3/0481 (20130101); G06F 3/14 (20060101); G06F 9/44 (20060101); H04L 29/06 (20060101); H04L 29/08 (20060101); H04L 12/18 (20060101); G09G 5/14 (20060101);
