US 6845338 B1
A telemetric contextually based spatial audio system is integrated into a mobile terminal including a headset and antenna. The headset includes an electronic compass which determines head orientation data for the user. The terminal's geographical position can either be determined locally (e.g. onboard GPS) or by network triangulation. The orientation and position data are sent to a server which overlays the position and head orientation data onto a locally stored “floor plan”, describing an object(s) of interest. The floor plan relates the user's viewpoint to a database of contextual audio clips. A regionally significant portion of the database and respective triggers are sent to the terminal. The terminal using the database information recalculates which audio clip(s) to play. The terminal then convolves the respective audio clip(s) via an impulse response function so as to be perceived by the user as emanating from the exact spatial position of the object specified by the system.
1. A telemetric contextually based audio system integrated into a mobile terminal in a wireless system, comprising:
a) A user transported mobile terminal coupled to a server via an air link;
b) a headset coupled to the terminal and including directional means indicating the user's head orientation relative to an object viewed by the user;
c) transmitting means transmitting signals to the server indicative of the user's geographical position and head orientation;
d) mapping means mapping the user's head orientation and geographical position to a physical layout of the object viewed;
e) selecting and transmitting means selecting and transmitting to the terminal clips of audio sound descriptive of the object viewed based upon the user's mapped position and head orientation; and
f) convolving means convolving the audio sound to give the user a perception that the audio sound is actually emanating from a point in the object viewed by the user.
2. The system of
g) wherein the convolving means continuously convolves the audio sound as the user's geographical position and head orientation move relative to the object viewed.
3. The system of
h) wherein the terminal provides the audio sound to the head set as 3D audio.
4. The system of
5. The system of
i) data means relating user's head and position information to an impulse response function for the object viewed by the user.
6. The system of
j) database means coupled to the server and storing the physical layout of the object viewed by the user.
7. The system of
8. The system of
9. The system of
10. The system of
11. The system of
12. The system of
13. The system of
14. The system of
15. The system of
16. The system of
17. A telemetric method to provide a user individualized audio information of an object viewed based upon the user's head orientation and geographical position in a mobile terminal wireless system, comprising:
a) coupling a user transported mobile terminal to a server via an air link;
b) coupling a headset to the terminal, the headset including directional means indicating the user's head orientation relative to the object viewed by the user;
c) transmitting signals to the server indicative of the user's geographical position and head orientation;
d) mapping the user's head orientation and geographical position to a physical layout of the object viewed;
e) selecting and transmitting to the terminal clips of audio sound descriptive of the object viewed based upon the user's mapped position and head orientation; and
f) convolving the audio sound to give the user a perception that the audio sound is actually emanating from a point in the object viewed by the user.
18. The method of
g) continuously convolving the audio sound as the user's geographical position and head orientation move relative to the object viewed.
19. The method of
h) providing the audio sound to the head set as 3D audio.
20. The method of
21. The method of
i) relating user's head and position information for the object viewed by the user to an impulse response function.
22. The method of
j) coupling database means to the server and storing the physical layout of the object viewed by the user.
23. The method of
k) storing clips of audio sounds in the database.
24. The method of
l) using an electronic compass as a directional means.
25. The method of
m) determining the user's geographical position from radio triangulation.
26. The method of
n) convolving an audio stream to support a dynamic positional and head orientation relationship between the user and an audio emanation point.
27. The method of
o) obtaining the geographical position from the Global Positioning System.
28. The method of
p) playing a clip of audio sound which describes what the user is viewing or playing spatial audio redirecting the user's attention to an item adjacent to the object viewed.
29. The method of
q) accessing a database to obtain clips of audio sound regionally related in the physical layout.
30. The method of
r) providing the user binaural stereo audio sound.
31. The method of
s) altering the perceived relative distance between the object viewed and the user by controlling the sound level and reverberations.
32. A telemetric method to provide a user individualized audio information of an object viewed based upon the user's head orientation and geographical position in a mobile terminal wireless system, comprising:
a) generating and storing impulse response function in the terminal for a user;
b) activating a server and storing in a database a physical layout of an object viewed by the user;
c) activating and storing in the terminal head orientation and geographical position data of an object viewed by the user;
d) transmitting the stored head orientation and geographical position data to the server via an air link;
f) mapping the transmitted head orientation and geographical data to a physical layout of the object view from a database;
g) transmitting to the terminal and caching locally a database subset of clips of audio sound based upon the mapped head orientation and geographical data;
h) accessing a table of impulse response functions in the terminal based upon the mapped head orientation and geographical data; and
i) convolving the audio sound to give the user a perception that the audio sound is actually emanating from a point in the object viewed by the user.
33. The method of
(j) returning to step (h) and repeating the process when the user moves outside the perimeter of the locally cached database subset.
34. A telemetric contextually based audio system, a terminal comprising:
a) a transceiver including a compression/decompression amplifier for receiving audio signals;
b) a processor coupled to the transceiver and to an antenna via an I/O circuit;
c) a non-volatile memory coupled to the processor and storing program instructions for operating the terminal and processing generic audio clips into spatial representations;
d) a volatile memory coupled to the processor and storing data subsets descriptive of objects viewable by a user; user head orientation obtained from a directional device and geographical positional of the user obtained from a geographical position determining source; and
e) convolving means convolving the audio signals to give the user a perception that an audio sound is actually emanating from a point in the object viewed by the user.
35. The terminal of
36. The terminal of
37. The terminal of
38. The terminal of 35 wherein HTRFs are based on a user training sequence.
39. In a telemetric contextually based audio system, a server comprising:
a) a processor coupled to a signal source for receiving positional data of a terminal;
b) a storage device for storing a floor plan of objects viewable by a user;
c) audio related to viewable objects by a user stored in the floor plan;
e) receiving terminal transmissions providing users' head orientation of the user relative to a viewable object;
mapping the head orientation and geographical position to the floor plan; and
g) selecting means selecting a regionally significant part of the database content related to audio for transmission to the terminal.
Ser. No. 09/961,373 entitled “Three Dimensional (3D) Object Locator For Items or Sites Using An Intuitive Sound Beacon: System and Method of Operation”, filed Sep. 25, 2001, assigned to the same assignee as that of the present invention, and fully incorporated herein by reference.
1. Field of the Invention
This invention relates to mobile information systems and methods of operation. More particularly, the invention relates to a telemetric contextually based spatial audio system integrated into a mobile terminal wireless system.
2. Description of Prior Art
Self-guided tours have long been used to enhance an individual's experience at art exhibits, industry trade shows, museums, cruise lines, historic sites and the like. In the past, self guided tours required the user to go from location to location and typically read written tags on each item or from a brochure. In many instances, this activity places too much of an imposition on the user and is slowly being replaced by localized audio broadcasts. Under this model, audio clips are played based on the proximity of the user to the specific item. Though easier for the individual, this model still lacks the animated and interactive experience desired in viewing items, exhibits and the like. Further, self-guided tours require the user to get within the proximity of the exhibit in order to find out what the exhibit is about, once again an imposition on the individual. In addition, once the user is at the location, the user must listen to the generalized audio even though they may only be interested, or may be particularly interested, in specific aspects of the location. Furthermore, the audio play-out has no method of directing the user's attention to particular features at the location other than to obtrusively give the user directions on where to look, which is particularly difficult since the system does not know the exact location, or the direction in which the user is facing. Finally, there is no way to personalize the audio of a particular exhibit to the individual user since the system has no ephemeral data on the user (e.g. knowing that a user has been present at a specified location, looking in a particular direction for a prolonged period of time can be construed as an area of interest) or a priori knowledge of the individual.
What is needed in the art is a system and method to animate the experience of a user relative to an item, exhibit; enable the user to hear about an item or exhibit based on their location and field of view; personalize the audio presentation based on specific interests of the individual; provide intuitive directional cues to bring attention to items of interest, and to guide the user to a desired location.
Prior art related to mobile communication systems, includes:
U.S. Pat. No. 3,934,202 issued Jan. 20, 1976 discloses an electronic tour guide system including means for suppressing objectionable crosstalk and noise. Crosstalk suppression is achieved by means of a compressor in the transmitter and an expander in the receiver. Objectionable noise between loops is eliminated by means of a squelch circuit, which reduces power drain whenever the received signal is below a certain threshold. The transmitter and receiver are designed to handle a plurality of information signals simultaneously. For example, a listener can select one of several different languages broadcast within an antenna loop. The transmitting loop antenna is matched in impedance to the output of the channel means in the transmitter so that maximum power transfer is obtained. Each receiver is further equipped with a gravity operated mercury switch which turns off power to the receiver when the receiver chassis is inverted. It is anticipated that this system will find application in museums, art galleries, archeological sites and the like where it is desirable to guide people from location to location and to address them in their respective languages.
U.S. Pat. No. 5,717,392 issued Feb. 10, 1998 discloses a position-responsive information presentation system and control. The system and control program automatically provides multimedia output that is responsive to a user's location, speed, acceleration, and directional orientation. Also disclosed is a system that receives position information from various systems, including satellite global position systems. As disclosed, the system and control program may provide video, audio and tactile outputs to users.
U.S. Pat. No. 6,459,388 issued Oct. 1, 2002 discloses a location database system provides information about nearby sites to a user. The system includes a processor configured to identify the present location of the system using the Global Positioning System. The processor is also associated with a clock providing real-time information. The clock may be the internal clock of a microprocessor serving as the processor of the system. The processor may also communicate with a remote database of sites using a wireless connection. The database includes relevant information about the sites, which may be of interest to a user of the system, such as locational information, imaging information and images, and/or tour-guide type information. A display provides a visual display of the relevant information to the user, and the system may provide setting information for the camera and also may provide travel directions to a particular site, based on a selected view as seen on a display.
U.S. Pat. No. 6,181,371 issued Jan. 30, 2001 discloses a head guide with a display is attitudinally controlled for guiding the head of a passive viewer wherein the display is for viewing images that are emulative of images viewed by a cameraman with head mounted cameras whose head attitude is monitored for controlling the head guide in synchronism with the images gathered by the cameras. Additionally, the viewer's eyes may be induced to follow a sequence of visual fixations at the same time as the passive viewer's head is induced to execute attitudinal movements consistent therewith.
U.S. Pat. No. 6,199,045 issued Mar. 6, 2001 discloses a system for providing position-related information to a mobile user includes a mobile unit and a central site server. The mobile unit includes circuitry for determining present position information from position signals, which may emanate from, for example, GPS satellites. The mobile unit further includes circuitry for establishing a wireless bi-directional communications link with the central site server via a terrestrial network, which may be accessed via a cellular telephone network. The central site server includes circuitry for receiving the present position information from the mobile unit. A table stored at the central site server includes different response information in correspondence with possible positions of the mobile unit. The response information may further be in correspondence with user preferences. The central site server uses the received present position information to retrieve corresponding response information from the table, and sends the retrieved response information to the mobile unit via the bi-directional communications link. The mobile unit further includes circuitry, such as a loudspeaker, for supplying the response information to the mobile user. In another aspect of the invention, communications between the mobile unit and the central site server are encrypted. Furthermore, the mobile unit may include components for preventing position information from being supplied to the loudspeaker, thereby eliminating distracting noise from being presented to the user.
USP Application 0020091793 published Jul. 11, 2002 discloses a method and system for tourist guiding, including both navigation and narration, utilizing mobile computing devices such as electronic books, palm computers, smart cellular phones, and future wearable computers. Loaded with, or Internet-connected to a Digital Guidebook, the system navigates the tourist to and within tour sites using an invented Visual Navigation method, and provides detailed multimedia narrations on objects along the tour. The system also navigates the tourist to other tourist destinations such as hotels and restaurants. An additional preferred embodiment combines GPS and CPS navigation with Visual Navigation. The system also creates a Digital Tour Album that captures the time and the places where the tourist has visited. The system also allows Armchair Touring and provides Tour Reviews on DVD. The Electronic Tourist Guide is based on latest mobile and wireless communication technologies and allows Internet channeling of live, timely information such as special events, latest prices and schedules. Digital Guidebooks can be downloaded to the tourist's device memory, or streamed to the tourist's device in real-time during the tour. The present invention entails two technologies: studio production and distribution of multimedia Digital Guidebooks, and use of mobile digital tourist guidebooks. The latter field entails mobile computing devices, wireless broadband Internet, and Geographic Information Systems.
None of the prior art discloses a system and method to animate the experience of a user relative to an item, exhibit, thereby enabling the user to hear about an item or exhibit based on their field of view, and personalizing the audio presentation based on specific interests. None of the prior art discloses intuitive methods of using spatial directional audio to bring the user's attention to a specific aspect of the location.
Moreover, none of the prior art discloses a mobile terminal capable of dynamically creating spatial audio sounds by using a set of locally stored impulse-response functions emulating angular positions of a sound relative to the user's head. And to this end, convolving such sounds in real-time according to the individual's head position relative to the item (e.g. to give the user a perception that the audio sound is actually emanating from a point in the object viewed by the individual).
For clarity, we refer to the spatial location at which the audio appears to be emanating from, as the “audio emanation point” (AEP). The AEP may map to a physical object, can be used as a stimuli to draw or direct the user's attention to a specified direction, or to animate the users experience using other spatial audio effects.
This system enables a user: (a) to obtain individualized audio information based on their position and head orientation (e.g. descriptions of merchandize items for sale, trade show displays, etc.), b) to have an enhanced audio-visual experience by having sounds perceived as being emitted from stationery objects, moving objects, or any point in 3D space (e.g. in a museum exhibit, a squawking sound appears to be emanating from a bird on the user's left side while a growl emanates from a bear as perceived on the user's right), c) to have the user's attention drawn to other items using spatially directed audio and based on their current and past actions (e.g. a user looking at a set of dishes is drawn to another location using spatially directed sound to describe a matching set of cups), and d) to provide a virtual tour-guide through spatial audio (e.g. a user is directed to a conference room by following the direction of a synthesized voice which is spatially located and continually updated so as to guide the user in the proper direction).
The user is equipped with a mobile terminal including a binaural stereo audio headset and an antenna for wireless communication with a server via an access point. The headset includes an electronic compass or device providing equivalent angular positioning, which determines the user's head orientation (e.g. azimuthal direction the user is looking). Head orientation data is sent to a server via a wireless access point. The terminal's position can either be determined locally (e.g. onboard GPS) or by using the network infrastructure to triangulate the terminal's transmission. In either scenario, all terminal position information is conveyed to the server.
The server overlays the user's geographical position and head orientation information onto a locally stored “floor plan.” The floor plan relates the user's viewpoint to a database of contextual audio clips based on position, orientation, and other event triggers (e.g. time duration at one position). The database also includes, or the server determines dynamically, the relative angular position and distance that the spatially directed audio should emanate from (AEP) based on the given user's position. A regionally significant portion of the database and respective triggers are sent to the terminal. The terminal updates its ephemeral data (current location and head orientation), and using the database information recalculates which audio clip(s) to play. Before playing the clip(s), the terminal updates its relative distance and angular positioning relative to the object from which the sound is to emanate. The terminal then convolves (see below) the respective audio clip(s) so as to be perceived by the user as emanating from the exact spatial position of the object specified by the system (AEP).
Based on the geographical position of the user and users head orientation, the terminal either plays an audio clip describing what the user is directly viewing (broadside) and/or plays a spatial audio redirecting the users attention to an item at a known off-center location from the user's view. The audio is perceived by the user as emanating from the item or direction of interest. To create this sound perception, the terminal accesses a table of impulse response functions. Each impulse response function is a generalized transfer function representing filter characteristics sufficient to transform a sound so as to be perceived by the user as arriving at their head from a specific angle (generally measured as 0 degrees directly in front of the person) and distance. Given an audio clip to play and knowing the relative angle and distance between the user and the object (AEP), the terminal convolves the audio clip with the respective (angle and distance) impulse response function and plays the resultant audio to the user. This gives the user the perception that the sound is actually emanating from that point in space (AEP). As the user changes their head orientation and/or geographical position, the audio is continuously re-convolved with the respective impulse response function, making the sound perceived by the user as continuously being emitted from the AEP. Constant monitoring of the telemetry data enables the spatially mapped sounds to maintain a constant relative position to the user. Note that the system can also create moving sounds relative to the user. In addition, by controlling volume levels and reverberation, the perceived relative distance between the object and the user can be altered.
An aspect of the invention is a mobile terminal storing a series of impulse response functions used to create 3 dimensional (3D) sounds that are perceived by the user as being spatially located at a defined relative angular and distance position (as if emanating from the AEP).
Another aspect is a terminal capable of convolving an audio stream(s) to support a dynamic positional relationship between the user and the AEP (i.e. a user's head is turning relative to a fixed AEP or the AEP is moving with respect to a static head orientation).
Another aspect is the ability to locate a mobile device using either a wireless network infrastructure (through triangulation) or incorporating locating capabilities within the mobile device (e.g. GPS). Location information is sent to a server, which determines the angle and distance relationship relative to objects of interest.
Another aspect is a real-time, headset based, tracking mechanism to measure angular orientation of the user's head. This information is then used to determine the angular orientation of the user's head relative to the object(s).
These and other aspects, features and advantages will be more fully understood from the following description of a preferred embodiment taken in conjunction with an appended drawing, in which:
The terminal further includes a CPU or microprocessor 214 linked to the memory via a bus 216, an I/O subsystem 220 and Audio Subsystem 226 for receiving and transmitting signals to the headset 104 and antenna 112. A power supply (not shown) provides the power for the operation of the terminal and the headset.
The terminal detects when the user is moving outside the perimeter defined by the locally cached database and requests a cache update from the server 120. Alternatively, the server can initiate a cache update as required.
Using the locally cached database 215 and current user geometry, the terminal looks up the relevant audio clip, convolves the audio with the impulse response function stored in the terminal and plays the result to the user via the headset, creating a sound perceived as being located at the AEP. As the user's relative positioning changes with respect to the object, either a new audio clip is played or the remainder of the existing audio clip must be re-convolved and then seamlessly spliced into the audio output. The terminal continuously repeats this process in order to maintain a robust user experience.
Constantly monitoring the telemetry of the user enables 3-D sounds to maintain a constant relative position to the user. As a result, the user's experience with the object is animated. By way of example, and returning to
Using the telemetry data to customize the audio presentation material further augments the user's experience. For example, the user is walking through a trade show exhibit hall 116 2 nd looks at a new car 116 3. Based on the geographical coordinates and head orientation, an audio clip is presented, which provides an overview of the car. As the user walks closer to the car, the audio clip is changed and begins to describe the car in detail. Finally, it is possible that when the user walks to the rear left window where the sticker price of the car is located, the user may be presented with financing information for the car in another audio clip.
Multiple HRTF tables 300 can be stored which can improve operation and perceived correctness. For example, unique HRTF tables can be built to mimic the sound characteristics of the environment in which the user is located (e.g. the sound in an auditorium is different from the sound outdoors). Also, since HRTF impulse functions vary slightly based on a person's head shape and auditory pathways, it is possible to store HRTF tables optimized for an individual user's localization. A training sequence in which the user is subject to a set of sounds in specific directions and selects the sound that most accurately matches the direction, would enable the system to use the optimal HRTF tables for the individual.
While the invention has been described with reference to a preferred embodiment, various changes can be made without departing from the spirit and scope of the invention, as defined in the appended claims, in which: