US 7363398 B2
A system for interfacing a user with an electronic device. An accessor device provides a user interface matched to the needs, abilities and intentions of the user and translates the user input commands and data into commands for the electronic device. An intelligent access port translates commands from the accessor into an input format required by the electronic device. A communication link connects the accessor device with the intelligent access port to send the translated user input commands and data from the accessor to the intelligent access port. The accessor device contains an intelligent access module and an interactive display. The intelligent access module contains a driver device for routing message packets, an interaction processor for processing incoming message packets and generating a message packet for the electronic device, and a communications channel. Communications between system components take place over a universal serial bus or over a wireless fidelity or ZigBee device. The interactive display can include a touch screen, browser, wireless transceiver and flash storage.
1. A system for interfacing a user with an electronic device, comprising:
a plurality of accessor devices with each accessor device classified by a level at which the accessor device is connected, wherein each accessor device (i) comprises an input sensor, including at least one of a keyboard, a keypad, a mouse, an electronic tablet, a handheld device, a pointing device, a head tracker, an eye gaze tracker, a microphone, a speech recognizer, a sonar device, a camera, a biosensor, and a electromagnetic sensor, for providing a user interface matched to an access need and an ability of the user, (ii) determines an intent from a user's input actions, (iii) translates the user input actions and data into commands for the electronic device, and (iv) transfers the commands to an intelligent access port that translates the commands received from the accessor devices into an input format required by the electronic device; and
a communication link that connects each accessor device with the intelligent access port to send the translated user input actions and data from each accessor device to the intelligent access port.
2. The system for interfacing a user with an electronic device of
3. The system for interfacing a user with an electronic device of
4. The system for interfacing a user with an electronic device of
5. The system for interfacing a user with an electronic device of
6. The system for interfacing a user with an electronic device of
7. The system for interfacing a user with an electronic device of
8. The system for interfacing a user with an electronic device of
9. The system for interfacing a user with an electronic device of
10. The system for interfacing a user with an electronic device of
11. The system for interfacing a user with an electronic device of
12. The system for interfacing a user with an electronic device of
13. The system for interfacing a user with an electronic device of
14. The system for interfacing a user with an electronic device of
15. The system for interfacing a user with an electronic device of
16. The system for interfacing a user with an electronic device of
17. The system for interfacing a user with an electronic device of
The present patent application is a formalization of a previously filed, co-pending provisional application entitled “Intelligent Total Access System,” filed Aug. 16, 2002 as U.S. Patent Application Ser. No. 60/404,034, by the inventor named in this patent application. This patent application claims the benefit of the filing date of the cited provisional patent application according to the statute and rules governing provisional patent application, 35 USC § 119(e)(1) and 37 CFR §§ 1.789(a)(4) and (a)(5). The specification and drawings of the provisional patent application are specifically incorporated herein by reference.
This invention was made with Government support under contract 0071126 awarded by the National Science Foundation. The U. S. Government has certain rights in the invention.
The invention relates generally to a system and methods for interfacing human users with electronic devices. More specifically, it relates to a system and techniques for providing users with access to electronic devices using a personal interface matched to the needs, abilities, preferences and limitations of the user.
The computer industry is based on a premise that all users can be served by a single basic user interface paradigm. While each manufacturer has a different interpretation of what that paradigm should be, there is some consistency in the current approach based on a Graphical User Interface (GUI) that is accessed with a keyboard and mouse. While this works for many people, there are those for whom it presents major accessibility problems. For example, many physically disabled people are unable to use the keyboard or mouse, blind people cannot see the objects displayed on a screen, deaf people cannot hear alarms or spoken messages, and learning disabled or cognitively impaired people may be confused by the visual metaphors that are used.
Traditional methods for making information technology (IT) equipment accessible to individuals with disabilities required hardware or software modifications to specific IT devices to make them accessible to specific users. While these user-specific solutions worked well in many environments, they were usually costly to implement and required building the IT equipment from the ground up for each user and each work situation. Version obsolescence is an ongoing problem because new or updated versions of existing software invariably break existing access solutions. Furthermore, different solutions are required for each computer platform and each version of its operating system. In short, these solutions enabled a person with special needs to work with a constrained range of equipment and software but did not give them the freedom to move about.
The Intelligent Total Access System (ITAS) was developed to provide individuals with access to any type of computer-based Information Technology (IT) using a personal interface matched to his or her particular needs, abilities, preferences and culture. The concept of intent driven user interfaces dramatically simplifies user interfaces by eliminating the need for users to learn specific operating scripts.
Motivation for developing the ITAS grew out of the need for people with disabilities to have the same freedom of access to IT equipment as people who have no disabilities. To achieve this, access tools must work properly with any type of IT and software and they must be portable to enable the user move about freely. The ITAS achieves this by breaking the access solution into two parts: (i) a small device called the Intelligent Total Access Port (ITAP) attaches to the IT device and emulates the standard user interfaces provided by the manufacturer, and (ii) a personal “accessor” that provides each user with his or her preferred interaction strategies and translates information and commands into formats that are recognizable by the specific user and any ITAP.
The specific objectives of the ITAS are: (i) to provide the mechanisms that enable individuals who have disabilities with the freedom to use interfaces of their own choice with any IT device that is equipped with an ITAP; (ii) to develop intelligent user interfaces that eliminate the need for users to learn a different interaction strategy for each IT device; (iii) to incorporate intent-driven technologies that enable interfaces to respond to natural human text and gestures; and (iv) to implement these intelligent, intent-driven interfaces with small processors that can be easily embedded into low-cost appliances and tools.
These and other advantages and aspects of the present invention will become apparent and more readily appreciated from the following detailed description of the presently preferred exemplary embodiments of the invention taken in conjunction with the accompanying drawings, of which:
The following description of the invention is provided as an enabling teaching of the invention and its best, currently known embodiment. Those skilled in the art will recognize that many changes can be made to the embodiments described while still obtaining the beneficial results of the present invention. It will also be apparent that some of the desired benefits of the present invention can be obtained by selecting some of the features of the present invention without utilizing other features. Accordingly, those who work in the art will recognize that many modifications and adaptations of the invention are possible and may even be desirable in certain circumstances and are part of the present invention. Thus, the following description is provided as illustrative of the principles of the invention and not in limitation thereof since the scope of the present invention is defined by the claims.
Various figures show different aspects of the system, and, where appropriate, reference numerals illustrating like components in different figures are labeled similarly. It is understood that various combinations of components other than those specifically shown are contemplated. Further, separate components are at times described with reference to a particular system embodiment, and while such description is accurate, it is understood that these components, with the variants described, are independently significant and have patentable features that are described separate and apart from the system in which they are described.
U.S. Pat. No. 6,083,270 having the same inventor and the same assignee describes devices and methods for interfacing human users with electronic devices. In particular, it describes a communication system that includes a speech synthesis and recognition system which is connected to an accessor total access port, a device connected to a target total access port, and a total access link between the accessor and target total access ports. Universal data packets are transmitted between total access ports over the access link. This patent is incorporated by reference in its entirety herein.
Overview of the ITAS
The ITAS is a real-time, multimodal, distributed, plug-and-play command and control system. Design choices were influenced by the need for including the following capabilities: ease of use, multimodal inputs, low latency, true plug-and-play, natural interaction processing, low cost, immunity against obsolescence, distributed intelligence, ease of design, and isolation between the ITAS functions and the interactive display.
The ITAS is intended as a universal interface strategy that requires minimal learning by the user. The ITAS must automatically handle the addition and removal of accessors and ITAPs without requiring intervention from the user.
The ITAS generates output commands by combining different user inputs thereby placing stringent requirements on recognizing the relative timing of when inputs are received.
Latency is the time delay between a user expressing an intention and the target device receiving a command. Commands must be recognized and passed to the output in the minimum possible time.
True plug-and-play capability means being able to add or remove an accessor or an ITAP without disturbing the normal operation of the ITAS. Accessors and ITAPs must not be dependent upon having prior knowledge of other components in the ITAS.
To handle natural interaction processing, there must be flexibility in the way the ITAS does word filtering and combining of the different input channels.
An affordable ITAS provides a practical framework for implementing ubiquitous computing.
To provide immunity against obsolescence, it must be possible to update or replace individual accessors and ITAPs without disturbing normal operation.
Manufacturers of intelligent accessors and ITAPs must be able to develop and test their products independently of other ITAS components (distributed intelligence). New ITAPs, for example, contain knowledge of their intended target device or appliance.
Developers must be able to create new accessors and ITAPs without requiring a detailed understanding of the complete ITAS system.
An important aspect is isolation between the ITAS functions and the physical interactive display. Users can choose screen size and interaction modality without disturbing any ITAS functions.
As depicted in
The operation of the ITAS can be considered from two different perspectives. From a functional perspective, the ITAS provides a reliable connection between a person 30 and a target IT device 40. From a user interaction perspective, the ITAS interprets human intentions and translates them into commands for a specific target IT device 40. The basic operational steps employed in the ITAS are depicted in
Operations at the user interaction level are totally automatic and are independent of the computer platform, operating system, and applications used in the target device.
An accessor translates user activities into commands for an ITAP. Individual accessors may be designed to handle a single input modality or may combine several complementary modalities such as speech recognition and head tracking.
As depicted in
Humans are capable of using multiple modalities for normal communication. An accessor provides a convenient way for incorporating different input-sensing technologies to detect human actions such as typing, speaking or gesturing. Typical input sensors include: keyboard, keypad, switch, mouse, head tracker, eye gaze tracker, microphones, sonar, still camera, movie camera, bio sensors, and electromagnetic field sensors. Accessors may be designed to incorporate any number of input sensors.
While a single input sensor is often sufficient for determining user intent, there are situations in which it is either necessary or more convenient to combine the data received from several input sensors. Satisfying the access needs of individual users while at the same time meeting the product quantity requirements of manufacturers leads to conflicting design requirements. On the one hand, each user wants an accessor containing a combination of access strategies that precisely satisfy his or her needs. On the other hand, manufacturers must achieve a viable scale of production for each product by minimizing the number of design variants they offer. One of the design objectives for accessors, therefore, is to make them sufficiently flexible that they meet the broad range of individual access needs while, at the same time, constraining the range of different accessors that are necessary so that manufacturers can achieve viable production quantities. The ITAS strategy for resolving these conflicting design requirements is described below.
The input processor 14 amplifies and processes input data received from sensors 12 to transform electrical signals into human language descriptions of the current actions of the user. Typing and speech recognition, for example, automatically generate words. Other input modalities, however, may need to be processed to generate word-based descriptions, such as a pointing device naming the object that is pointed to instead of an x,y,z direction vector.
The operation of an IMNIP (Integration Manager and Natural Interaction Processor) is described in more detail below. Its most significant purpose, however, is to enable people to control IT devices using their own words and gestures to express their intentions. The IMNIP 16 performs three essential functions: (i) it merges multiple streams of incoming text; (ii) it filters out all of the incoming words that are meaningless to the ITAPs with which it is currently able to communicate; and (iii) it extracts user intent from the stream of filtered text. The IMNIP 16 can decode commands consisting of single words, phrases with words in any order, and phrases in which words must be received in a defined order. Outputs from the IMNIP 16 can be either words or phrases that are meaningful to an attached ITAP, or command codes for controlling settings and operation of the accessor. Outputs from the IMNIP 16 can specify either single commands or groups of commands to be executed sequentially (macros).
The accessor wireless transceiver 18 provides bi-directional streaming data communications between an accessor 10 and one or more ITAPs 20. Operation of the wireless system is fully automatic and is invisible to the user. Natural language messages are used between accessors 10 and ITAPs 20. In the simplest configuration, the wireless transceiver 18 connects a single accessor 10 to a single ITAP 20. A more typical configuration, however, will have single person multiple accessors or multiple people, each using one or more accessors and working together collaboratively such as students working in groups in a classroom, and groups of students working collaboratively with other groups. For example, a speech recognition accessor can be used to enter text and spoken commands, and a head tracking accessor can be used to move the mouse cursor.
An ITAP 20 translates between the ITAS protocol and the proprietary protocol required by a target system. Wherever possible, the ITAPs emulate standard input devices such as a keyboard, keypad, mouse or joystick. ITAPs currently exist for interfacing to the PS2 (mouse) and USB ports on a PC, to infrared inputs on audiovisual equipment and the X10 modules that are used to control appliances in the home. In situations where there is no clearly defined standard, an ITAP 20 can be configured to emulate the actual buttons, switches, and pointing controls included in an actual device.
As depicted in
The ITAP wireless transceiver 22 provides bi-directional streaming data communications between an ITAP 20 and one or more accessors 10. Operation of the wireless system is fully automatic and is invisible to the user. Natural language messages are used between accessors 10 and ITAPs 20. In the simplest configuration, the ITAP wireless transceiver 22 will connect a single accessor 10 to a single ITAP 20. It will be more typical, however, for multiple accessors and ITAPs to be linked into a single ITAS network. In a more typical configuration, an ITAP will be linked to multiple accessors to form a network in which a user may use several accessors in a collaborative manner. For example, a speech recognition accessor can be used to enter text and spoken commands, and a head tracking accessor can be used to move the mouse cursor.
The ITAP wireless transceiver 22 is also able to communicate with other ITAPs to establish addresses and priorities whenever an accessor 10 or ITAP 20 joins or leaves the particular ITAS network. Whenever more than one ITAP is present in a network, the ITAP with the lowest physical address is configured as the master that coordinates communications between accessors and ITAPs and maintains a timestamp clock that is used as the timing reference by all of the ITAPs and accessors. If present, a shared resource ITAP will have the lowest address and therefore will be the master or Node Coordinator of the ZigBee wireless network. If not present, the ITAP with the next highest address will be the master.
The operation of an IMNIP is described in more detail below. When used in an ITAP, the IMNIP 24 performs three essential functions: (i) it merges incoming text messages from multiple accessors; (ii) it filters out incoming words that are meaningless to the IT device to which it is connected; and (iii) it translates natural language commands into the specific control codes required by the IT device. The IMNIP 24 can decode text commands including single words, phrases with words in any order, and phrases in which words must be received in a defined order. Outputs from the ITAP IMNIP 24 are either single commands, or macros consisting of a sequence of commands.
IT device manufacturers usually adhere to industry standards such as Universal Serial Bus (USB) for user interface ports. It is not uncommon, however, to find products that use either modified versions of a standard protocol, or proprietary protocols that are not based on any external standard. The device control interface 26 translates between the ITAS protocol used for communication between accessors 10 and ITAPs 24, and the protocol that the IT device 40 uses. The ITAP 20 presents a standard interface to the accessors while emulating the real interface devices used by the particular IT equipment.
There is a basic conflict between the needs of individual users and manufacturers. The user needs are matched to his or her input or output needs whereas manufacturers want each accessor produced to satisfy the needs of a large number of users. ITAS provides a unique resolution of these conflicting requirements.
Where it makes practical and economic sense to do so, popular input strategies such as speech recognition and head tracking may be combined in a single accessor 10. Specialized input strategies, such as eye gaze trackers or biosensors, may be implemented in a single-function accessor. All accessors have one or more cascading inputs for receiving the output of another accessor, as shown in
The exemplary accessor configuration shown in
The function performed by a particular accessor is classified by the level at which the accessor is connected.
For consistency in the addressing scheme used by the ITAS, ITAPs 200 are designated as level 5. It is not necessary, however, for accessors to be present at all levels in a particular installation. For example, a system that is only used by one person to access one target device 40 could consist of a single “individual” accessor 10 and a single ITAP 20 as depicted in
There are some situations in which accessors 10 connect to the target device 40 without an ITAP 20. This forgoes command translation and macro functions normally provided by an ITAP, but still provides basic keyboard and mouse functions. For example, a speech accessor can be connected directly to the USB port of a PC for entry of text and spoken commands.
When a new accessor is introduced to an existing ITAS, it can only bond to accessors that are at a higher level. The accessor can only bond to one other higher-level accessor at a time. The user must make a selection when there are multiple accessors at the next higher level. It is possible to leave out one or more levels if there is no reason to include them. The example shown in
After it has been physically bonded to an accessor at a higher level, it is still possible for an accessor to bypass one or more levels by setting bypass flags in the header of any data packet. An augmenting accessor 100, for instance, can send data packets directly to an ITAP 200 by setting the bypass flags for levels 2, 3 and 4. There are two reasons for bypassing higher-level accessors. The first reason is that the packet contains information that is not to be interpreted by the IMNIPs in the bypassed levels. This enables an individual accessor, for instance, to alternate between sending control-packets to a group accessor 104 or to the target device 40. The second reason is that the packet contains streaming data that is already in the correct format for the next level that is not bypassed. An augmenting accessor 100 that produces standard mouse codes, for example, could control the cursor on an individual accessor 102, a group accessor 104 or the target device 40 simply by setting the appropriate bypass flags.
The primary function of an augmenting accessor 100 is to add special or customized interaction modalities to individual accessors 102. This strategy makes it viable for larger manufacturers to focus on high volume production of popular accessors that meet a broad range of user needs while also allowing smaller manufacturers to focus on low volume or custom production of specialized accessors that satisfy a narrow or range of needs. It also provides a convenient way for researchers to evaluate new input or output strategies before investing in the design of a full accessor. An augmenting accessor 100 may or may not include an IMNIP.
A group accessor 104 is used to combine the outputs of multiple individual accessors in situations where there is either one person using several individual accessors collaboratively to control one or more ITAPs or, if there are several people, each using an individual accessor to control one or more ITAPs. Multiple accessors can be arranged as a single group. In conference room, meetings, or classroom situations, a group accessor 104 gives participants access to shared tools such as remote keyboards and pointing devices as well as integrating any individual accessors 102 they may have.
In situations where a large number of people may be using accessors, such as in a large lecture hall, at a conference, or in a museum, there may be many groups of accessors 104 that are formed into one or more clusters. A cluster accessor 106 has cascading inputs for merging groups of accessors 104 but it does not necessarily have its own user inputs.
Four different ITAP configurations are described in the following paragraphs. These are wireless ITAP, networked ITAPs, multiple target ITAPs, and a shared resource ITAP.
There are situations in which it is more cost effective to use a single wireless communications system with several networked ITAPs, as depicted in
The multiple target ITAP depicted in
A shared resource ITAP depicted in
Physical Components of the ITAS
An ITAS installation includes a collection of accessors and ITAPs arranged as a hierarchical tree. Between two and six levels may be cascaded to provide the flexibility to accommodate many users and many target devices. More levels may be used in special circumstances, such as when there are many users with very different access requirements, accessing many different types of target devices.
Integration Manager and Natural Interaction Processor (IMNIP)
This section describes a technique for implementing an Integration Manager and Natural Interaction Processor. An IMNIP can be implemented in conventional desktop and notebook computers or embedded within small microprocessor-based control devices. While specifically designed for the Intelligent Total Access System (ITAS), the IMNIP can be used in any situation requiring a person to command and control a computer-based device. The IMNIP operates independently of the input modality and is able to integrate inputs from multiple sources such as keyboards, speech recognizers, touch panels, Morse code and gestures. The strategies developed for the IMNIP provide significant advantages over earlier agent based NIP designs, as follows:
The IMNIP performs all of the message merging and decoding functions for the ITAS. One embodiment of the ITAS system 50 is shown in
The ITAS 50 uses two slightly different implementations of the IMNIP 16. An accessor IMNIP 24 that combines and extracts intent from multiple streams of input from a user, and an ITAP IMNIP that merges natural text messages received from multiple accessors and generates the device commands required to perform the intended operation.
An accessor IMNIP 16, as depicted in
An ITAP IMNIP 24, as depicted in
It is important to distinguish between a multiword command and a macro. The words in a multiword command can be received in any order. The words in a macro must be received in a defined order with no other words attached before or after the macro. Macros are used for performing critical functions on the target system.
As depicted in
The IMNIP overcomes these problems by allowing a user to describe an intended outcome in his or her own words as if asking another person to perform the action. Whenever an IMNIP recognizes an intended action, it instructs the system to perform all of the necessary steps without requiring the user to participate in their execution or to even know what the steps are.
While a single IMNIP can perform all of the necessary merging and decoding operations for a particular combination of user inputs and output devices this results in very specialized and inflexible system. It is much more versatile when the IMNIP functions are distributed between the accessor which handles the human-centered functions and the ITAP which handles the machine-centered functions. A serial, or cascaded configuration of the IMNIPs enables systems to handle ad hoc combinations of users and target devices with minimum overhead. The first IMNIP handles user dependent details independently of the device requirements and the second IMNIP handles device specific details independently of the human factors. Parallel operation of IMNIPs, as depicted by the four accessors or the four ITAPs in
As further depicted in
The accessor IMNIP 16, as depicted in
The following table shows the relationships between inputs and outputs that can be generated by an accessor IMNIP.
The ITAP IMNIP 24, as depicted in
Accessors and ITAPs can be matched to the level of complexity that is to be handled. Any level of accessor can operate with any level of ITAP. Not all capabilities will be available in all cases. Table 2 provides characteristics of different levels of accessors and ITAPs.
As depicted in
The complexity of the ITAS is kept low by partitioning functionality and distributing intelligence across a selection of relatively simple modules. The overall ITAS can be considered as an asynchronous system in which each module (i.e., accessor or ITAP) is responsible for processing all input data in the shortest possible time and passing the results on to the next module in the chain. The internal operation of each module is optimized for a particular function and is independent of all other modules. In most cases, natural human language is used for communication between modules. The exception to this is when an accessor is emulating a time-dependent peripheral, like a mouse, and sends its output encoded as standard mouse packets. These packets bypass the internal processing of any intermediate modules and are conveyed to the target device with minimum possible delay.
This modular organization eliminates the need for a centralized operating system and dependence on third party software developers. Each module is designed to perform its designated function with minimum dependency on other modules, and to present its results in a format that can be used by any other module in the system. Detailed design concepts for the various modules are described below.
Timing considerations have dominated the design of the ITAS since its overriding purpose is to enable a person to interact with and control one or more target IT devices or appliances in real time. Communications between ITAS components, and between ITAS and the devices that are being controlled, can be wired, wireless, or a combination of both. Wired systems are simpler to design and implement but wireless systems are more convenient to install and use. Convenience is driving the transition to a wireless infrastructure and several emerging standards, such as BlueTooth, IEEE 802.11b (WiFi), and IEEE 802.15 (ZigBee) are competing for market share. There are significant economic advantages in embracing one or more of these general-purpose RF standards for the ITAS. However, they do introduce a number of problems due to complexity, power consumption and high latency under some operating conditions.
The real-time command and control requirements make low system latency essential from the time the user expresses an intention until a target device receives a command. Some accessors emulate mouse-like devices that produce streaming data, which must be passed directly to the target device without interpretation by an IMNIP and with minimum delays. Commands that are built from multimodal inputs depend on the relative timing of the various inputs. Consistent timing is required for the processing performed within individual accessors to enable targeted ITAPs to resolve competing or collaborating commands received from multiple accessors. Some target devices require the ITAP to deliver sequential commands within tight timing constraints.
Latency is important in the ITAS because people are attuned to receiving an instant response when they perform simple actions such as typing a character, clicking a mouse button or turning on a light. For this type of activity, latency of up to about one hundred milliseconds goes unnoticed but above this, the operation becomes unnatural and disconcerting. A latency of 100 to 200 milliseconds can normally be tolerated for computer inputs. Video gaming systems, however, require the latency to be less than 20 milliseconds.
There are three main factors that determine latency in the ITAS. The first factor is the time required for the wireless system to sniff out available target modules and to bond one module to another. Bonding typically takes several seconds but should only be necessary at the beginning of a session. Some power-saving strategies, however, terminate and restart sessions frequently. While making sessions persistent minimizes the time required for bonding, it increases the power consumption and limits the operational time available in battery-powered devices. The second factor is the time required for processing information within each module before it is ready to be transferred to the next module in the chain. The ITAS reduces processing delays by distributing intelligence across all of the modules. This allows decisions about user intent to made as close as possible to the source of the information upon which the decision is based. For instance, each of the accessors shown in
Obsolescence is one of the main economic driving forces of the computer industry. Potential users of the ITAS such as disabled and aging people do not have the resources to continually purchase new equipment. ITAS has been designed from the beginning with the goal of making each functional module immune to obsolescence. There are several strategies for achieving this goal. One strategy is using natural human language to pass commands from one accessor to another. The word “on,” for example, has the same meaning regardless of the type of accessor that generates or receives it. Word meaning is also independent of the order in which accessors are cascaded. Another strategy is encapsulating specialized functions within individual accessors. A speech accessor, for instance, uses a microphone to detect speech and outputs textual commands and content. The internal operation of the speech accessor is immaterial to the ITAS as long as all command words are consistent with the ITAS corpus. The IMNIP in a speech accessor translates spoken user commands into ITAS words. Spoken words that do not match the ITAS corpus may be optionally discarded or passed on for interpretation by a higher-level accessor or ITAP.
The ITAS communications must support two distinct requirements that cannot be satisfied completely with any one of the currently available wireless protocols. These requirements can be summarized as follows:
The wireless communications industry is still in a state of confusion with several emerging standards vying for domination in each of the identifiable market spaces. This has made it very difficult to choose an optimum communications strategy for the ITAS. It is already clear, however, that no single protocol will satisfy all of the ITAS requirements. While there is a strong temptation to create a new ITAS-specific communications solution, this would limit options for incorporating low-cost, off-the-shelf components and products as they become available. There is also the unresolved question of whether optical techniques (infrared or lasers) or radio frequency techniques might be better for particular applications within the ITAS. There is also the question of whether Ultra Wide Band (UWB) technologies will provide the best overall solution for ITAS in the longer term.
The current version of the ITAS protocol defines the necessary communication functions within the framework of three standards, one wired and two wireless:
The performance advantages of using three standardized communication protocols within the ITAS far outweigh the additional cost and complexity. For example, low-speed user input does not interfere with the screen updates and large web page or data files can be transferred to and from accessors without impacting critically-timed control messages to target devices.
The system is also much more stable since each module performs all processing functions independently and asynchronously. This allows each module to gather available inputs and process the data as quickly as possible without any dependencies on what is happening within other modules. As soon as they are generated, results are placed in an output buffer and transferred to the next module on the very next wireless cycle.
Each of the regions outlined with dashed lines between plural accessor modules in
The ITAS driver performs all of the functions required for selecting the communications mode and managing data transfers.
The same results can be achieved by connecting two accessors 110, 112 through a USB cable as shown in
All ITAS communication functions are inhibited when a USB cable connects an accessor 110 to a non-ITAS computer or IT appliance 120, as depicted in
The process of moving the Node Coordinator to the root accessor in the hierarchically organized ITAS network applies regardless of whether or not all of the intermediate levels are present.
In a fully configured ITAS, the Node Coordinator will normally reside in the shared resource ITAP (see level 5 in
An advantage of this configuration is that, within each ZigBee network, only two hops are required to link any accessors and ITAPs—one from the first node to the Node Coordinator, and one from the Node Coordinator to the second node. While this doubles the time required to move between adjacent levels, it has the advantage of eliminating the need for messages to pass through accessors that do not perform any IMNIP functions. This reduces the latency on messages that must be passed quickly from a source accessor to a target ITAP. Streaming mouse signals, for instance, will reach their destination in just two hops rather than being passed through every accessor in the chain. Latency is less of an issue for messages that are processed by an IMNIP because these are generally limited by the relatively slow rate at which a human can enter words into an accessor or enter coordinated inputs on several accessors.
As depicted in
With reference to
The ITAS/USB Manager 130 receives, routes, and transmits ITAS and HID packets through either USB 252 or Zigbee 250 communication channels. The core of the ITAS/USB Manager 130 is the ITAS driver 152. The ITAS driver 152 is responsible for selecting and maintaining the correct communications channels for accessor-to-accessor and accessor-to-ITAP interactions. A standard USB hub 170 allows multiple lower-level accessors to be connected to the accessor. The ITAS driver 152 handles three types of packets: (i) ITAS packets, (ii) HID packets, and (iii) internal representations of packets.
The IMNIP 160 integrates multiple data streams received from the ITAS/USB Manager 130 and performs the NIP processes specified by the ITAS header. Four types of output are provided by the IMNIP: (i) ITAS packets destined for lower-level or higher-level accessors or ITAPs; (ii) ITAS packets destined for the attached interactive display, (iii) keyboard or mouse commands for controlling the attached interactive display; and (iv) keyboard or mouse commands for controlling the next higher-level accessor.
The Zigbee wireless system 250 connects to lower-level and higher-level accessors and to ITAPs. Within each accessor, the Zigbee wireless system can function either as a Node Coordinator, when it is in the highest-level device in the network or as a node when there is already a higher-level device in the network. A different protocol stack is used for each operating node.
The ITAS driver 152 performs the following functions:
The input section of the ITAS driver 152 manages the transfer of information between an accessor and all lower-level accessors by performing the following functions: (i) selecting input source; (ii) discovering potential communication partners; (iii) bonding to lower-level accessors; (iv) transferring data; (v) bypassing the IMNIP; and (vi) managing errors.
An ITAS driver 152 can simultaneously manage wired USB and ZigBee wireless communication channels. At any time, however, there can be only one active ITAS communication channel between any pair of accessors. Whenever an accessor is connected to another accessor by a USB cable 252, wireless communications between these accessors are turned off and all ITAS functions are performed through the USB channel. The ITAS driver is responsible for recognizing when another ITAS driver 152 can be reached over a USB channel.
Any accessor can communicate with one or more lower-level accessors. Discovery begins when an ITAS driver 152 receives a response from an ITAS driver that has a lower level number and is not bonded to another accessor. Both accessors indicate that a potential communication partner exists and wait for either an automatic or user-initiated response.
Discovered accessors must be bonded to each other before they can begin exchanging commands and data. Bonding can be initiated automatically if the two accessors recognize each other from ID numbers stored in their “Accessor Table.” The ID in the table may be for a specific accessor, or for any member of a specified class of accessors. User intervention is required if matching IDs are not found in the accessor tables. A visual or textual description of available accessors is displayed on each accessor. The user may make a selection on either accessor to initiate bonding. To allow automatic bonding in the future, the user is given an option to load the accessor identities into the Accessor Table upon completion of the bonding process.
Each higher-level accessor periodically scans through all bonded lower-level accessors to exchange the contents of the output buffer in the lower-level accessor with the contents of an input buffer in the higher-level accessor. The higher-level accessor has a separate input buffer for each connected lower-level accessor.
The IMNIP 160 in an accessor does not process any packets that are received with their Bypass flag set. The ITAS driver 152 tests the Bypass flag as each data packet is received. If the flag is set, the contents of the input buffer are moved directly to the output buffer as soon as space is available. The ITAS driver 152 checks for logical errors in all incoming data packets. A detected error causes the complete buffer to be resent.
The output section of the ITAS driver 152 manages the transfer of information between: (i) an accessor and one higher-level accessor, (ii) an accessor and an ITAP, or (iii) an accessor and a target IT device that has a USB port 174.
An ITAS driver 152 manages the connection to a single higher-level device. The connection may be over a USB cable 252 or a ZigBee wireless channel 250. If a USB cable connection 252 is detected, the ITAS driver 152 will test to see if it is attached to a higher-level accessor or ITAP with an active ITAS driver. If it is, the USB cable 252 becomes the default connection and the ZigBee wireless channel 250 between the two devices is turned off. If no ITAS driver is detected, the accessor emulates standard USB keyboard and mouse devices if instructed to do so by the user. An accessor cannot send USB keyboard or mouse signals to a lower-level device.
Discovered accessors must be bonded to each other before they can begin exchanging commands and data. Bonding can be initiated automatically if the ID of each accessor is contained in an “Accessor Table” in the other accessor. The table ID may be for a specific accessor, or for any member of a specified class of accessors. User intervention is required if matching IDs are not found in the accessor tables. A visual or textual description of available accessors is displayed on each accessor. When the user selects a desired connection, the ITAS drivers will complete the bonding process. The user is given the option of loading the accessor identities into the Accessor Tables to allow automatic bonding in future.
Each higher-level accessor periodically scans through all bonded lower-level accessors to exchange the contents of the output buffer in the lower-level accessor with the contents of an input buffer in the higher-level accessor. The higher-level accessor has a separate input buffer for each connected lower-level accessor. Organization of the input 162 and output buffers 164 is illustrated in
The IMNIP 160 in an accessor does not process any packets that are received with their Bypass flag set. The ITAS driver 152 tests the Bypass flag as each data packet is received. If the flag is set, the contents of the input buffer 162 are moved directly to the output buffer 164 as soon as space is available. The ITAS driver 152 checks for logical errors in all incoming data packets. A detected error causes the complete buffer to be resent.
The functions performed by the interactive display 15 are shown in
The capabilities of the interactive display 15 are deliberately limited to the following essential functions: USB support 182 for HID keyboard, HID mouse, and ITAS packets; a touch screen 180; a browser 188 with flash viewer; WiFi communications 184 for delivery of web pages and files; and flash memory 186 for file storage. This limited and stable range of required functions allows for a relatively simple operating system 190 to be used. This operating system will support instant-on capabilities.
The performance requirements for an accessor browser 188 are much simpler than those of the browser in normal computers or IT appliances. Flash storage 186 is used in an interactive display 15 to minimize cost and to support instant on capabilities. USB drivers 182 are required for a standard HID keyboard, a standard HID mouse, and for custom ITAS data transfer. ITAS commands and data flow back and forth between the ITAS module 50 and the interactive display 15 over the USB connection 252. WiFi communications 184 enable the interactive display 15 to retrieve and display web pages and flash movies from the shared resource module or the Internet.
The touch screen 180 provides a basic interaction path for the user. Active areas on the touch screen 180 can be selected by touching them directly or by moving a curser over them using external mouse or keyboard commands received from the ITAS module 50 via the USB/ITAS connection 252. In some cases, active areas on the touch screen 180 generate commands for controlling operation of the interactive display 15 or interacting with web pages. In other cases, active areas of the touch screen 180 generate commands or parameters that are transferred to the ITAS module 50 via the USB/ITAS connection 182. The ITAS module 50 can take over selected areas of the screen 180 in the interactive display 15 as interactive control panels that enable the user to set up and control the operation of the ITAS module 50.
The ZigBee stack shown in
Information is transferred between accessors by exchanging an output buffer 164 in the lower-level accessor with an input buffer 162 in the higher-level accessor. The two buffers contain the same number of bytes. As depicted in
As shown in
ZigBee—Communications from ITAPs to Target Systems
ITAPs 200 translate ITAS commands into control codes that are specific for each target device 40. The ITAP uses an IMNIP to derive the control codes from messages provided by one or more accessors. In most situations, the ZigBee wireless protocol is used to transfer control codes from the ITAPs 200 to the target devices 40.
From a closer look at the proposed ZigBee standard, it appears that it will be a better choice than wireless USB for connecting accessors to support human interface devices. While it has lower bandwidth, it has much a much faster turn-on and much lower latency. Most significantly, it has very low power requirements. Widespread industry adoption is anticipated for user interface devices and device control.
When used in conjunction with WiFi for screen delivery, browsing, and file transfer, the resulting ITAS system handles input from the user and outputs to the user in the most efficient manner. A brief summary of the specification for ZigBee is provided in Table 3.
The ITAS system has conflicting requirements for communicating inputs from users and outputs to users. Text and commands are best handled by the ZigBee standard protocol and display requirements are best handled by one of the 802.11x protocols. This division of functions leads to an efficient design in which one channel provides low latency and low power consumption with a large number of available channels, and the other channel provides a high-speed network that will transfer video, still images, graphics, audio, and large data files and support applications such as Web browsing and email. As depicted in
Example of Mixed ZigBee and WiFi Operations.
Consider the following scenario: Mary is a school student with a physical disability that prevents her from using a conventional keyboard and mouse. She participates in normal school activities by using a personal accessor that enables her to access all of the same information resources as her fellow students. Her personal accessor is based on a small tablet computer that would normally be controlled by a touch screen. Mary's classroom environment is depicted in
When Mary's teacher asks her to show her home page to the group she works with, she speaks a command “talk to the group” that instructs her personal accessor to rout the outputs of her speech recognizer 302 and head tracker 304 through to the group accessor 306 as standard USB keyboard and mouse packets (arrows 1, 2 and 3). Mary now has full control of the group accessor 306 and is able to open its browser by pointing to its icon with the head tracker and saying “open this.” She can dictate the URL of her home page, or she may invoke a macro such as “show my home page.” The group accessor 306 will access the Internet via the WiFi connection (arrow 4) and allow Mary to move on from there.
If Mary's teacher then tells her to show the whole class, Mary would say, “talk to the class” and the personal accessor would pass the ITAS packets from the speech recognition accessor 302 and head tracking accessor 304 through to the group accessor 306 as depicted in
As indicated in
Many diagrams, similar to
The ITAS 50 uses two basic types of packet: (i) ITAS packets to communicate between the ITAS drivers 152 in each ITAS module 50, and (ii) USB HID packets to control the standard keyboard and mouse functions in any attached interactive displays 15 or USB equipped target devices such as a personal computer. Table 4 identifies how the different types of packets are used in the ITAS and how they may be may be routed to the various devices in the system. The addressing method and flags that specify the source and destination for individual packets are discussed below.
Sessions provide a persistent framework for interaction between accessors at different levels. In
The addressing scheme adopted for the ITAS is based on the concept of levels and on the constraints imposed by USB and ZigBee addressing schemes as illustrated in
A bypass_flag byte is included in the header of each packet to indicate the effective source of the packet and the intended target level. The least significant bit in the bypass_flag signifies whether the targeted IMNIP is to be dedicated to processing this packet or can be simultaneously processing packets from multiple accessors. If there is a sharing conflict with the targeted IMNIP the incoming packets are processed in the order they are received.
The following examples show how the bypass flag enables the effective source accessor to specify the target level and to indicate whether the packet is to be passed on to a higher level if the targeted level is not populated.
Conceptually, ITAPs 200 are connected to the Master ITAP. To provide a more convenient strategy for connecting to, and controlling any of, the equipment that makes up or supports an ITAS installation, ITAPs may also be connected to the ITAS Module in any accessor.
The accessor to which an ITAP is connected is addressed with the normal ITAS address but with all lower-level address bytes set to zero as shown in
The ITAS supports message passing between any accessor or ITAP in the system. A unique capability of the ITAS is that messages may be passed as literal messages or as intent. Message packets labeled as intent are passed through a separate message IMNIP in the target ITAS module, which expands them into natural text messages in whatever language and format is required or preferred by the user. Spoken output can be provided for blind and visually impaired users, for example, and animated sign language movies can be provided for deaf users. Sending feedback information and error messages to an IMNIP as intent has three major advantages: (i) intent messages may be very short and cryptic, (ii) messages from several sources may be combined by the IMNIP, and (iii) the user messages produced by the IMNIP is interesting, natural language that can be varied each time to maintain user attention and interest. The message passing system uses the same addressing scheme as the ITAP addressing system described herein.
Those skilled in the art will appreciate that many modifications to the exemplary embodiment of the present invention are possible without departing from the spirit and scope of the present invention. In addition, it is possible to use some of the features of the present invention without the corresponding use of the other features. Accordingly, the foregoing description of the exemplary embodiment is provided for the purpose of illustrating the principles of the present invention and not in limitation thereof since the scope of the present invention is defined solely by the appended claims.