RAPID PROTOTYPING AND MACHINE VISION FOR RECONFIGURABLE INTERFACES

Abstract

A system and method including software to aid in generation of panels and control instruments rapidly generates a station that can support a variety of control interfaces. Rapid-Prototyped Panels, or RP-Panels, replicate existing systems (for simulation, training, gaming, etc.) or from new designs (for human factors testing, as functional product, etc.). The controls have tactile and visual characteristics similar or identical to their functional component counterparts such as buttons, knobs, switches, pedals, joysticks, steering wheels, and touch panels but are modular and use alternative data transfer modes (potentiometers, fiber optics, RFID, machine vision, etc.) to track and analyze the response of the controls. The response is then transmitted to the host programs. With this method a user can design and fabricate a reconfigurable interface to interact with virtual environments for various applications such as simulation, training, virtual instrumentation, gaming, human factors testing, etc.

Description

REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No. 12/351,332, filed Jan. 9, 2009, which claims priority from U.S. Provisional Patent Application Ser. No. 61/020,013, filed Jan. 9, 2008, the entire content of which is incorporated herein by reference.

GOVERNMENT SUPPORT

This invention was made with Government support under Contract No. N61339-07-C-0085 awarded by the United States Navy. The Government has certain rights in the invention.

FIELD OF THE INVENTION

This invention relates generally to human-machine interfaces and, in particular, to interfaces that use rapid prototyping technology to fabricate interfaces with desired functional and haptic responses.

BACKGROUND OF THE INVENTION

In many situations, a complex simulation interface is useful for training. For example, pilots would like to train in a flight simulator using mission specific data. These simulators often take up a large amount of space, particularly if one simulator exists for each type of interface (e.g. each aircraft cockpit).

What is needed is a reconfigurable simulator that can faithfully replicate the look and feel (sense of touch) of a number of different interfaces. Simulating the sense of touch, or haptic, aspect of a range of interfaces is a particularly difficult task due to the sensitivity of the human tactile system. Current haptic interface technologies are unable to faithfully recreate the large range of tactile stimuli encountered when interacting with interface components such as buttons, knobs, etc. common to interfaces.

SUMMARY OF THE INVENTION

This invention relates generally to human-machine interfaces and, in particular, to interfaces that use rapid prototyping technology to fabricate interfaces with desired functional and haptic responses. The embodiments include software to aid in generation of panels and control instruments rapidly construct panels that can support a variety of control interfaces. These panels can replicate existing systems (for simulation, training, gaming, etc.) or create new designs (for human factors testing, as functional product, etc.).

The controls may have tactile and/or visual characteristics similar or identical to their functional component counterparts such as buttons, knobs, switches, pedals, joysticks, steering wheels, and touch panels but are modular and use alternative data transfer modes (potentiometers, fiber optics, RFID, machine vision, etc.) to track and analyze the response of the controls. The response is then transmitted to the host programs. By virtue of the invention, a user can design and fabricate a reconfigurable interface to interact with virtual environments for various applications such as simulation, training, virtual instrumentation, gaming, human factors testing, etc.

A method of forming a human-machine interface according to the invention includes the step of providing a scaffold enabling different interface components to be positioned at different locations on the scaffold. One or more of the interface components are fabricated with a rapid prototyping technology. Such components feature a physical structure that provides a specified functional response. The components are mounted on the scaffold in accordance with a given application.

The functional response may include a haptic response. For example, the interface component may be a pushbutton, in which case the haptic response may include the depth of travel or the stiffness of the pushbutton. In a specific embodiment, the interface component may be a pushbutton with a frame, a button portion, and compliant tabs coupling the button portion to the frame, and the haptic response may include the depth of travel or the stiffness of the pushbutton determined by the size, shape or composition of the compliant tabs.

Alternatively, the interface component may include a location to be touched by a user, in which case the specified functional response is determined through machine vision extraction of the touch location. A human-machine interface constructed in accordance with the invention broadly includes a plurality of user input devices, at least certain of which are fabricated with a rapid prototyping technology, wherein at least some of the user input devices are fabricated with a rapid prototyping technology include physical features that are sized, shaped or composed to yield a desired functional response.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows scaffolding for a multifunction display;

FIG. 1B shows a rapidly prototyped assembled button mechanism, where compliant tabs provide a specified haptic response (depth of travel, stiffness) as well as an optical change enabling machine vision extraction;

FIG. 1C shows the button mechanism of FIG. 1B in an exploded form;

FIG. 1D shows a rapidly prototyped touch panel; and

FIG. 2 depicts a machine vision interface state extraction embodiment.

DETAILED DESCRIPTION OF THE INVENTION

This invention takes advantage of progress made in rapid prototyping technology. Rapid prototyping machines are increasing in prevalence due to their increase in resolution, speed, and value. These machines are essentially 3D printers. Given a software description of a solid component, a rapid prototyping machine can build up a physical solid to those specifications in a manner of minutes to hours. Applicable technologies include Selective Laser Sintering (SLS); Fused Deposition Modeling (FDM); Stereolithography (SLA); Photopolymer Laminated Object Manufacturing (LOM); Electron Beam Melting (EBM); and “3D Printing” (3DP). This invention is not limited in this regard however, as it may take advantage of these as well as any yet-to-be developed alternatives.

The inventive approach to reconfigurable displays take advantage of these rapid prototyping machines, where new interface structures and components can be printed then easily configured into a working simulation. Because haptic fidelity is a priority for those components that need to be accessed “by touch,” this invention uses a combination of technologies enabling an actual button or knob component to give perfect haptic fidelity in conjunction with rapidly prototyped interface components. This allows for tradeoffs between haptic fidelity and storage space—using actual components for those high priority buttons and knobs that need to be found “by touch,” and using rapidly prototyped components for lower priority buttons that do not need to be physically stored until time of use.

According to the invention, certain human interface components are fabricated with rapid prototyping technology. Figure IA shows scaffolding for a multifunction display, where actual components can be positioned precisely. The scaffolding of FIG. 1A includes a frame 100 with various holes 102 and an aperture 104 enabling user controls and a central display to be ‘snapped’ into position. These controls may either be stored physical components or rapidly prototyped elements, depending upon operational criticality or other factors.

FIG. 1B shows a rapidly prototyped button mechanism in an assembled form, where compliant tabs 110, 112 provide a specified haptic response (depth of travel, stiffness) as well as an optical change enabling machine vision extraction. FIG. 1C shows the button mechanism of FIG. 1B in an exploded form. The tabs 110, 112 interact with a frame 120. By providing inputs associated with the desired haptic response, the components may be ‘written’ with rapid prototyping technology to provide the dimensions, thicknesses and other physical parameters to achieve a target ‘look and feel’ and functional response.

Another advantage of rapidly prototyping interface components is that monolithic, functional interfaces can be printed by rapid prototyping machines in a single pass, requiring no assembly before use. FIG. 1D shows a rapidly prototyped touch panel, where an interference grating is used to transform a small applied force into a large optical change, again enabling machine vision extraction of touch location. The structure of FIG. 1D uses only rapid prototyped components. This design demonstrates how rapid prototyping can combine structural with functional interface components.

A machine vision interface state extraction embodiment is shown in FIG. 2. Cameras are used to extract position of actual buttons and switches, maintaining haptic feel, without having to reconnect every button and switch upon reconfiguration. When used in conjunction with rapidly prototyped interface components, these components can be optimized to undergo a large optical change during a state change (e.g. button press).

This invention also incorporates machine vision technologies that enable this optimally “mixed fidelity” approach to haptically simulating cockpit interfaces. The main difficulty with using either real components or rapidly-prototyped components is that electrical wiring each interface component would be too time consuming to be practical. Instead, we have developed machine vision technologies that can extract button positions (or knob orientations, etc.) from a camera view of the interface. This way, no wiring is needed, and different cockpit interface panels can be interchanged quickly with no loss in functionality.

A primary advantage of using physical structures for interface components is that the haptic realism is near perfect, and cannot be matched by today's haptic simulation technology. These components can also be reused and quickly reconfigured simply by placing rapidly prototyped panels into specified locations.

Claims

1. A reconfigurable human-machine interface, comprising:

a plurality of user-input devices fabricated with a rapid prototyping technology;

a scaffold having a plurality of locations configured to receive the user-input devices;

at least one video camera having a field of view including the scaffold and the user-input devices;

wherein the camera optically determines the interaction between a user and the user-input devices, thereby controlling a machine without the need for wiring between the user-input devices and the machine.

2. The reconfigurable human-machine interface of claim 1, wherein:

the scaffold and user-input devices form a touch panel; and

one or more cameras are used to determine where a user touches the panel.

3. The reconfigurable human-machine interface of claim 1, wherein at least some of the user input devices include physical features that are sized, shaped or composed to simulate the controls of a particular machine.

4. The reconfigurable human-machine interface of claim 3, wherein:

the particular machine is an aircraft; and

the controls simulate a cockpit.

5. The reconfigurable human-machine interface of claim 1, wherein the user-input devices snap into the scaffold at different locations.

6. The reconfigurable human-machine interface of claim 1, wherein the user-input devices are buttons, knobs, switches, pedals, joysticks, or steering wheels.

7. The reconfigurable human-machine interface of claim 1, wherein at least some of the user input devices are fabricated to provide a desired haptic response.

8. The reconfigurable human-machine interface of claim 1, wherein:

the user-input device is a pushbutton; and

the desired haptic response includes the depth of travel or the stiffness of the pushbutton.

9. The reconfigurable human-machine interface of claim 1, wherein the rapid prototyping technology includes Stereolithography (SLA); Photopolymer Laminated Object Manufacturing (LOM), or 3D Printing (3DP).