CA2171314C - Electro-optic vision systems which exploit position and attitude - Google Patents

Abstract

The present invention is generally con-cerned with electronic vision devices (1) and methods, and is specifically concerned with image augmentation in combination with nav-igation, position, and attitude devices. In the simplest form, devices (1) of the invention include six major components: a camera, a computer processor, devices to measure the position and attitude of the camera, a data base for storing information associated with various scenes, and a display (1) where an im-age (3) is continuously aligned to a real scene as viewed by a user (2). In the present inven-tion, an image of some real scene is altered by a computer processor to include stored infor-mation of a scene in a storage location iden-tified by real time position and attitude of the vision system (1). The primary function of the invention is to present augmented real images (3), or images that represent a real scene but have deletions, additions, and supplements, and data that is continuously aligned with the real scene as naturally viewed by the user (2) of the system.

Description

WO 95/07526 21~ 4 PCT/US94/06844 , . .
[
. j, s "Electro-Optic Vision Systems which Exploit Position and Attitude"
Specifica~ion for a Leffers Patent Backgrcund of the Invention The present invention is generally concerned with electronic vision devices and methods, and is specific~lly conc~rned with image ~up...t~ ;Qn in coml)il-dlion with navigation, position, and attitude devices.
One may have to look quite far into the annals of history to find the first usesof maps. Maps generally provide information to alert a user to things that are not readily appdl e"l from simple viewing of a real scene from the users location. For example, a user of a city road map may not be able to see a tunnel on Elm street if the user is currently seven miles away on First street and looking in the direction of the Elm street tunnel. However, from the First street location, the user could determine from a road map that there is a tunnel on Elm street. He could learn that the tunnel is 20 three miles long, starts on Eighth street and ends on Eleventh street. There may even be an indication of the size of the tunnel such that it could accommodate four traffic lanes and a bicycle lane.
Unfortunately, it is not always possible to translate the illrolllld~ion from a map to the real scene that the il~",dLion rel,lesenl~ as the scene is actually viewed.
2s It is common for users of maps to attempt to align the map to reality to get a better "feel" of where things are in relation to the real world. Those who are familiar with maps can verify that the fact that maps are drawn with north being generally in the direction ofthe top ofthe map, is of little use when tran.cl~ting the h~mdLion to the scene of interest. Regardless of where north is, one tends to turn the map so that the 30 direction ahead of the user, or in the direction of travel, in a real scene m~t~lles that direction on the map. This may result in the condition of an "upside down" map that is quite difficult to read (the case when the user is traveling south). Althoughtr~n~l~ting the directions ofthe map to reality is a formidable task it is an even greater problem to translate the symbols on the map to those objects in reality which 35 they ,ep,ese"~. The tunnel symbol on the map does not show what the real tunnel actually looks like. The fact that the appea~ ~nce of the tunnel from infinitely many points of view is prohibitively difficult to ~ epl ese"~ on a map accounts for the use of a simple symbol. Furthermore, the map does not have any indication from which point of view the user will first see the tunnel, nor any indication of the path which the user 40 will take to approach the tunnel.

SUBSTITUTE SH~ET (RULE 2~) .~ ' i . ~
wo ss/07s26 2 1 7 1 3 1 ~ PcTrus~/0~8q 1 It is now possible to computerize city road map hL[~l,nalion and display the maps according to the path taken by a user. The map is updated in "real-time" according to the progress of the user through the city streets. It is thelerolt;
possible to relieve the problem of upside-down maps as the computer could re-draw the map with the text in correct orientation relative to the user even when one is traveling in a southerly direction. The computer gel-e.aled map is displayed at a monitor that can be easily refreshed with new h~ll.,alion as the user progressesalong his journey. Maps of this type for automobiles are well known in the art. Even very sophi.ctir,~ted maps with computer generated indicia to assist the user in decision o making are available and described in patents such as DeJong US #5,115,398. This device can display a local scene as it may appear and supelill"~ose onto the scene, symbolic illrolll.~lion that suggests an action to be taken by the user. For example, a left turn as is shown in figure 3 of the disclosure. Even in these advanced systems, a high level of translation is required of the user. The computer generated map does not attempt to present an accurate ~lignmPnt of displayed images to the real object which they rep~esenl. Devices employing image suppl~om~nt~tion are known and include Head Up Displays, HUDs and Helmet Mounted Displays H~Ds. A
HUD is a useful vision system which allows a user to view a real scene, usually through an optical image combiner such as a holographic mirror or a dichroic 20 beamsplitter, and have superimposed thereon, navigational irolll,alion for example symbols of real or im~gin~ry objects, vehicle speed and altitude data, et cetera. It is a primary goal of the Hl~D to . . .~ e the time that the user is looking into the scene of interest. For a fighter pilot, looking at a display device located nearby on an instrument panel, and r.h~nging the focus of ones' eyes to read that device, and to return to the scene of interest, requires a critically long time and could cause a fatal error. A HUD allows a fighter pilot to ...~ continuous concentration on a scene at optical infinity while reading instruments that appear to the eye to also be located at optical infinity and thereby çlimin~ting the need to refocus ones' eyes. A HUD allows a pilot to .n~ a "head-up" position at all times. For the airline industry, HUDs30 have been used to land airplanes in low visibility conditions. HUDs are particularly useful in a landing situation where the boundaries of a runway are obscured in the pilots field of view by fog but artificial boundaries can be projected onto the optical combiner of the HUD system to show where in the user's vision field the real runway boundaries are. The virtual runway projection is positioned in the vision field 35 according to data generated by communication between a colllp~ller with and the airport instrument landing system, ILS which employs a VHF radio beam. The system provides the computer with two data figures. First a glide slope figure, and second, a localizer which is a lateral position figure. With these data, the computer is SUBSTITUTE S~IEET (RULE 26) WO 95/07526 21713 14 PCT/US~1~'0~

~ . . , able to generate an optical image (photon) to ge projected and cGml)hled with the real scene (photon) that passes through the combil-el and thereby enh~nçing certain features of the real scene; for example runway boundaries. The positioning of the overlay depends on the accuracy of the airplane boresight being in ~lignment with the s ILS beam and other physical limit~tions~ The computer is not able to recognizeimages in the real scene and does not attempt to manipulate the real scene except for his~hli~htinp~ parts thereof. HUDs are particularly chara~;Lt;,~ed in that they are an optical co",bil,alion of two photon scenes. The co",bil,alion being a first scene, one that is normally viewed by the users eyes passes through an optical co,l,bhler, and a 10 second, computer gene~led photon image which is coll,l,ined with the real image at an optical element. In a HUD device it is not possible for the computer to address objects of the real scene, for example to alter or delete them. The system only adds enhancem~nt to a feature ofthe real image by drawing ;"leres~illg features ILe,eoll.
Finally, HUDs are very bulky and are typically mounted into an airplane or 15 automobile and require a great deal of space and complex optics incl~ltling holograms and specially dçsi~ned lenses.
Helmet Mounted Displays HMDs are similar to HUDs in that they also combine enhancçmPnt images with real scene photon images but they typically havevery portable components. Micro CRTs and small colllbine~ s make the entire system 20 helmet mountable. It is a complicated matter to align computer g,enel~led images to a real scene in relation to a fast moving helmet. HUDs can align the data generated image that is in~le~ed to the slow moving all~lal-e axis which moves slowly in relation to a runway. For this reason, HMDs generally display data that does not change with the pilots head movements such as altitude and airspeed. E~MDs suffer the same 25 limitation as the HUDs in that they do not provide the capacity to remove or a~l~mçnt elements of the real image.
Another related concept that has resulted in a rapidly developing field of computer ~Qi~ted vision systems is known as virtual reality, VR. Probably best embodied in the fictional television program "Star Trek; The Next Generation", the 30 "Holodeck" is a place where a user can go to have all of his surroundings generated by a computer so as to appear to the user to be another place or another place and time.
Virtual reality systems are useful in particular for a training means. For example in aircra~ .~im~ tion devices. A student pilot can be surrounded by a virtual 35 "cockpit" which is essentially a computer interface whereby the user "feels" the environment that may be present in a real aircraft, in a very real way and perhaps çnh~n~ed with computer generated sounds, images and even m~çh~nic~l stimuli.
Actions taken by the user may be h~Lel~l e~ed by the computer and the computer can SlJBSTlTUTE SHEET (RULE 26) 2 ~ 71 31 4 PCTIUS94/06844 respond to those actions to control the stSimuli that surround the user. VR m~c.hines can create an entire visual scene and there is no effort to superimpose a computer generated scene onto a real scene. A VR device generally does not have any comml-nic~tion between its actual location in reality and the stimuli being presented to 5 the user. The location of the VR m~ ine and the location of the scene being genc~aled generally have no physical relationship.
VR systems can be used to visualize things that do not yet exist. For example, a home can be completely modeled with a computer so that a potential buyer can "walk-through" before it is even built. The buyer could enter the VR atmosphere and o proceed through computer generated images and stimuli that accurately represe~ll what a home would be like once it is built. In this way, one could know if a particular style of home is likable before the large cost of building the home is incurred. The VR m~chine being entirely pro~l~lulled with h~l~ ion from a dçsignlor does not ~ntir.ip~qte things that presently exist and there is no commllnic~tion between the 15 elements presented in the VR system to those elçm~nts e~i~ting in reality.
While the systems and inventions of the prior art are de~igned to achieve particular goals, features, advantages, and objectives, some of those being no less than l~;lllalk~ble, these systems and inventions have limitations and faults that prevent their use in ways that are only possible by way of the present invention. The prior art 20 systems and inventions can not be used to realize the advantages and objectives of the present invention.

Summary of the Invention Comes now, an invention of a vision system in~ lin~ devices and methods of 25 a~lgmçnted reality wherein an image of some real scene is altered by a computer processor to include h~l lll~lion from a data base having stored information of that scene in a storage location that is i~entified by the real time position and attitude of the vision system. It is a plilllaly function of the vision system of the invention, and a contrast to the prior art, to present ~lgm~nted real images and data that is 30 continuously aligned with the real scene as that scene is naturally viewed by the user of the vision system. An ~lgm~ntec~ image is one that lelJlt;sellLs a real scene but has deletions, additions and supplements. The camera of the device has an optical axis which defines the direction of viewing as in a simple "camcorder" type video camera where the image displayed accurately leplesenl~ the real scene as it appears from the 35 point of view of one looking along the optical axis. In this way, one easily orients the h~llllaLion displayed to the world as it exists. A filn~m~nt~l difference between the vision system of the invention and that of a CalllCOI der can be found in the image ~lgm~nt~tion. While a camcorder may present the superposition of an image and SUBSTITUTE SHEET (RULE 26) ~ WO 95/07S26 . PCT/US~tOC8~q ~7 ~3l ~

data such as a "low battery" indicator, et cetera, it has no "knowledge" of the scene that is being viewed. The data displayed usually is related to the vision device or so...~l~.;ng independent ofthe scene such as the time and date. Image fl~lgment~tion of the invention can include i~ ation particular to a scene being viewed with the s invention.
The vision system of the invention can include a data base with prerecorded i. ror~.alion regarding various scenes. The precise position and fltti~-de ofthe vision system indicates to the data base, the scene that is being viewed. A computer processor can receive inrol llla~ion about the particular scene from the data base and 0 can then fl~lgm.ont an image of the scene genelated by the camera of the vision system and present a final image at the display with incllldes a co...bi..ation of h~ro-...~lion from the optical input and illrullllaLion that was stored in the data base. Particularly important, is the possibility of communication belween the data from the data base and the real image. Analyzing and processing routines may include recognition of5 items in the real scene and compa-isons with artifacts ofthe stored data. This could be useful in ali nment of the real images to the recalled data. In a situation where the optical input of a scene is entirely blocked from the camera of the system, for example by dense fog, an image ofthe scene can be generated which incllldes only h ru~n.ation from the data base. Alternatively, the data base being of finite size may not have any 20 inro.---alion about a particular scene. In this case, the image plesenled at the display would be entirely from the optical input of the real scene. This special case reduces the vision system to the equivalent of a simple camcorder or electronic binocular. It is also a very special case where the features of a real scene are selectively removed. If the bright lights of a city-scape obstruct the more subtle navigation lights of a marine 2~ port of entry, then it is possible for the processing routines to disc. i~ a~e between the city lights and the navigation lights. The undesirable city lights could be removed in the processor before the final image is displayed. In the final image, the important navigation lights show clearly and the city lights are not present at all. Ther~rore, a final image of the invention can be comprised of h~rul ...alion from two sources, in 30 various combinations, superimposed together to form a single, high inru~ alion-density image. The inro--ndlion of the two sources are con.paled and combined together to form a single augmented image that is presented at a display and is aligned to the real scene as the scene is viewed by the user.
In the simplest form, devices of the invention can be envisioned to include six 35 major components: A 1) camera to collect optical i.L~....alion about a real scene and present that inrù----alion as an electronic signal to; a 2) computer prOCeSSOr; a 3) device to measure the ~c: ~ ~ ofthe camera; and a 4) device to measure the attitude of the camera (direction of the optic axis), thus uniquely idw~liryil1g the SUBSTITUTE SHEET (RULE 26) WO 9S/07526 ~ PCT/US94/06844 scene being viewed, and thus identifying a location in; a 5) data base where i,~.~.aLion associated with various scenes is stored, the computer processor combines the data from the camera and the data base and perfects a single image to be plese.l~ed at; a 6) display whose image is continuously aligned to the real scene as it s is viewed by the user.
The camera is related to the position measuring device and the attitude measuring device in that the measu. elllell~s of position and attitude are made at the camera with respect to &~ ly references. The position and attitude measurement means are related to the data base in that the values of those measureme..~ specify lo particular data base locations where particular image data are stored. We can think of the position and attitude measurements as d~finin~: the data base pointer of twoorthogonal variables. The camera is related to the computer processor in that the image generated at the camera is an electronic image and is processed by the computer processor. The data base is related to the computer in that the data base S furnishes the processor hlru.~ tion in~lurling images for use in processing routines.
The display is related to the computer as it receives the final processed image and converts the computers electric image signal into an optical image that can be viewed by the user. The display is boresight aligned with the optical axis of the camera such that the information corresponds to reality and appears to the user in a way that 20 allows the user to view the final ~lgm~nted image without needing to translate the image to the orientation of the real scene.
In the simplest form, methods of the invention can be envisioned to include seven major steps: An 1) acquire step whereby the light from a scene is imaged by a lens; a 2) conversion step whereby optical h~llna~ion ofthe acquire step is 2s converted into an electrical signal; a 3) posit ~ determining step in which the position of the camera is measured; an 4) attitude determining step in which the~ttit~lde of the camera is measured; a 5) data recall step where a data location is selected in accordance with the measu. t;...~..L~ in steps 3 and 4 and user impute data, is recalled by a computer processor; a 6) ~. ocr ~ step wherein data from the data 30 store and the electronic image are combined and processed; and a 7) display step wherein a processed final image is displayed.
The product of the acquire step, an optical image, is converted to an electric signal in the conversion step. The electronic image of the conversion step is tran.~mitted to the processor in the processing step. The products of the position 35 d~e.~ g step and attitude detel....l....g step are values that are used in the data recall step. The result of the data recall step is also ~I,.n.~ led to the processor to be combined with the electronic image of the conversion step in the processing step.

SUBSTITUTE SHEET (RULE 26) ~_ Wo 95/07526 ~1~7 ~314. PCT/US94/06844 The product of the processing step, a final electronic l epr~se~ ion of an ~gm~nted image is ll~n~...;lled to, and displayed in, optical format in the display step.The invention will be sulll."~ed further by pres~.l;"g six ~.Y~mples of the invention wherein a description of the devices, methods, and uses thereof, follow.
s In a first sl~mm~ry example ofthe invention, the reader is to im~ine a scenario where a boat is approaching a port of entry and the user of the invention is a navigation officer of that boat. It is quite con.lllon for a navigation officer to require many aids to guide his course through a SLppillg rh~nn~l Charts, a compass, lighted buoys, sonic devices, ranges, radar, binoculars are some of the instruments that one 0 may use to navigate a boat. Recent advances in position detellllinillg technologies, in particular the Global Positioning System, or GPS, have simplified the task of navigation. With the GPS, a navigation officer can rely on knowing the position of the craft to within applux~lla~ely ~t300 feet, north and east; and in some special cases within less. Even with such a good position detelll.in~lion~ the navigation officer must locate where on the chart his position collc;spollds, and identify symbols on the chart to create a mental image of his surro--n-1ing.e Then the navigation officer must look about at the real scene before him for i~ ble objects to determine how whathe sees corresponds to the symbols on the chart. Frequently, visibility is limited by darkness or weather conditions and particular lights must be recognized to identify chart markings. These can be colored fl~chin~ lights and can easily be mict~k~n for the bright lights of a city skyline. In other cases, the l,l&lkel~ may be un-lit and may be impossible to find in the dark. Dangerous objects, for l A&I."~le sunken ships, kelp, and reefs, are generally marked on the chart but can not be seen by the navigation officer because they can be partially or entirely sub.llelged. The navigation officer must im~gine in his mind his position with respect to objects in the real scene and those on the chart and must also im~gine where in the real scene that the chart is warning of dangers. This procedure requires many complex ll ~n~laLions and intel~ alions between the real scene, the markers ofthe chart, the scene as it is viewed by the navigation officer, and the chart as understood by the navigation offlcer. Obviously, there is great potential for mi~t~kes Many very skilled and experienced naval navigators have failed the complicated task of safely navigating into a port resulting in tragic con~eqllences. With the system ofthe invention, a navigation officer can look with cel Laillly at a scene and locate exactly, known marks.
The system of the invention e~ es the need for detelll.h~ing where in a real scene the symbols of a chart COl . e~l,ol1d. The user of the invention can position the display between his eyes and the real scene to see an image of the real scene with the symbols of a chart superimposed thereon. In the navigator's mind it becomes very concrete where the otherwise invisible reefs are located and which lights are the real navigation S~JBSTITUTE SHEET (RULE 26) WO 95/07S26 ~ 1 7 1314 . PCT/US94/06844 Iights and which lights are simply street lights. It is possible for the computer to remove ;,~1 ,.,~lion such as stray lights from the image as it is recorded from the real scene and to present only those lights that are used for navigation in the display. This is possible because the data base of the invention "knows" of all navigation lights and s the processor can e~ e any others. The display that the user views inr.ludes a,~,es~ n of a scene with complic~t~d undesirable objects removed and useful data and objects being added thereto. Whenever the navigator points the device of the invention in some direction, the device records the optical image of the real scene and ~imlllt~neo~l~ly dc;le~ni~les the position and attitude ofthe device and calls on a 0 data base for h~ol ...alion I egal dil.g the scene being observed. The processor analyzes the image and any data recalled and co",l)ines them to form the final displayed image.
In a further summary example of the invention, a city planning co~ ission may wish to know how a proposed building may look in the skyline of the city. Ofcourse it is possible to make a photograph of the skyline and to airbrush the proposed s building into the photograph. This commonly used method has some shortfalls. It shows only a single pt;rs~.ec~ e ofthe proposed building that may be p,t;se"led in the "best light" by a biased developer (or "undesirable light" by a biased opponent/competitor). The building may be presented to appear very handsome nextto the city hall as shown in the developer's rendition. Since only one pe~ ~I,e.;Live is 20 generally shown in a photograph, it may be impossible to determine the full impact the building may have with respect to other points of view. It may not be clear from the plel)ared photograph that the bç~1tifi-l bay view enjoyed by users of city hall would be blocked after the building is constructed. With the current invention, the details of every perspective could be easily vi~ li7e~ Data that accurately represents the 2s proposed building could be entered into a data base of a device of the invention.
When the camera of the invention is pointed in the direction of the new building, the camera portion records the real scene as in appears and transmits that signal to a processor. The device accurately deLe""i"es the position and attitude of the camera with respect to the scene and recalls data from the data base that properly repl ~se,lLs 30 the perspective of the building from that point of view. The processor then combines the real scene with the data of the proposed building to create a final image of the building from that particular pe~ ~e.~ e. It would even be possible for a helicopter to fly in a circle around the location of the building and for a user to see it from all possible points of view. A council member could see what the future structure would 35 be like in real life from any perspective before voting to approve the plan.
In a still further summary example, we choose a scenario where an ~ineer uses products of the invention for analysis and troubleshooting of an ong;~.ee~ g problem. In particular the case where a problem has been detected in the plumbing SUBSTITUTE SHEET (RULE 26) wo 95/075~6 2 1 7 13 i 4 PCT/US~)1tO~8~

aboard a submarine. The complicated works int~llldin~ pipes, tubes, pumps, cables, wires, et cetera, of a submarine may be ~ .nely difficult to understand by looking at a design plan and tr~n~l~ting the ;,~.",aLion from the plan to the real world.
T.. f~JiAIe and positive identification of a particular element may be critical to s survival of the ship in an t;,llel~e"cy. The following illustrative r~ ee~ illg use of the invention provides for a greatly cimrlified way of positive identific~tion of ~ineering ~alul ~S.
An engineer aboard a sub",alille is tasked to work in the torpedo room on a saltwater pump used to pump down the torpedo tubes after use. In pl t;pal ,llion for lo the job, the data base of a portable electro-optic vision device is updated from the ship's central computers with il~llllalion regardi"g the details of the pump in question and of the details of the torpedo room where the pump is located. In the event of a battle damaged ship or in case of limited visibility due to fire or power outages, the vision device can provide ~ An~e to the location of the pump through 5 visual and audio clues. As the various pipes may be routed through b-llkheArls and behind walls, the vision system can be used to see through walls and to "fill-in" the locations of the pipes so that the otherwise hidden pipe can be followed continuously and without ambiguit,v. Upon arriving at the pump in question, the ~ineer pointsthe camera axis of the vision device in the direction of the pump. In the display of the 20 device, a real image of the pump is superimposed with data such as the part number of the pump and clues to features of the pump such as the type of material beingpumped and flow direction. The vision device can receive inputs from the user toselect advance display modes. This input may be made by way of ~ .hAI~ical devices like a "mouse", or a simple button, or may be verbal co.. An~c that are recognized 25 electronically by speech recognition means. If the pump is to be 11icAccemhled~ the user may be able to instruct the device to display seqll~ntiAI rli.cAcsemhly steps by a verbal "step" co~....AI~d Di~csemkly could then be simplified by clues provided by the vision system's display. A particular part can be hi~hli~hted and the motionrequired for its proper removal can be cimlllAted such that the user can learn the correct ~1icAcs~mhly procedure in real time. Similarly, re-assembly can be expedited because prior knowledge ofthe re~ccçmhly sequence rests in the data base and is easily presented to the user in a way that does not require translation from a parts book to reality. A highli~hted part and the motion to assemble that part can be superimposed onto a real image of the workpiece.
In a still further summary example, one can imAgine the complexity of a situation where a structure fire is being fought. It may be typical for a fire captain to be reading the detailed features of the building interior from a blueprint and to ll ans".il that h~ol ...alion by radio to firefighters on the inside of the building. The SUBSTITUTE SHEET (RULE 26) WO 9S/07S26 217~ 1 4 PCT/US94/06844 ~

problem with this method is that it depends on multiple ~ slalions between various media. With each translation, the h~m~ion is distorted according to some l.~nsrer function thus limiting its accuracy. The present invention offers an alternative which allows detailed .1~....alion of a structure to be directly displayed to a firefighter s inside the building.
A video camera and goggle set could be in~t~lled into a sL~dard firefighter helmet, the goggle set having therein, the requisite display. The display unit would be in commllnic~tion~ either by hard wire or radio, to a computer processor that is also in communication with a data store having il~llllalion p-e.ecol-led therein legard;llg 0 features of the structure. A locating system could dele...,i.,c at all times where the firefighter is positioned and where she is looking. A real image from the camera can then be combined with the recorded data to realize a final ~u~m~nte~l image. If the bright light generated by the fire in the real image is blocking important features such as a door handle, the processor can delete the fire from the scene and "draw-in" the 5 door handle. The firefighter could easily navigate her way through a complicated building interior full of smoke and fire that would otherwise prohibit ones progress through that building. First hand i.~llll~lion provided by a vision system of the invention directly to a firewoman can greatly increase the speed and effectiveness of fire fighting tasks. A device mounted into a helmet would necess~ ily be small and 20 compact. It may be a..anged such that the camera, display, and the locating (position and attitude) means are in radio comm--nic~tion with a computer and processing means that may be bulky but located I ~;n,olely. This possibility is considered to be a subset of, and inclllded within, the scope of the invention.
It should be pointed out that in the case of very heavy smoke where no useful 25 real image can be detected by the camera and the displayed image is consists entirely of data from the data store, the system beco...es quite similar to the virtual reality devices. The critical di~tinl,tion is that the images displayed are exactly aligned to reality and oriented to the direction in which the user is looking. If the firefighter pushes on a door, a real door will open. In VR applications, it will appear and feel 30 like a real door is being opened, but the door of a VR system is completely .~iml ~l~ted and there is no real door being opened.
In a still further summary example, to provide a viewing device that allows a tourist to "see" into a previous time. One can im~gine a bus load of tourists arriving at the grounds of Pompeii. Each person equipped with portable vision system could 3~ be free to roam the grounds at will. It is possible to have predetermined points where the user could be directed and to have a sound track that collesl)onds to the scenes as presented to the user. Looking into the display of the vision system a user can see the buildings as they are now and a simulation of how they appeared prior to the eruption SUBSTITUTE SHEET (RULE 26) WO 95107S26 2 ~ ~ ~ 314 PCT/US94/06844 of Mt. Etna. He sees a sim~ tion of people going about their daily lives in ancient rO"~peii. Without warning, the viewer sees the ash falling on the bu~ ing~c and people running for cover. The simul~ted events of the past can be easily viewed in a modern time with provisions of the present invention. In this version, each portable device 5 can be in radio communication with a shared data and proces~ing center that communicates with each individual pe. ~onal device indepçnrlçntly.
In a final su"l".a. ~ example of the invention, we consider the need for surveyors to "see" into the ground to locate pipes and wires that have been buried.
When construction projects require that such pipes and wires be removed or altered, 0 or new projects to be installed such that they do not hl~t,re,~; with eYietin~ works, then surveyors must attempt to locate those works. A surveyor relies on docllm.ont~tion ofthe previous construction projects and a~le",pls to plot onto the real world the locations of the various works as indicated by those maps. Again, the surveyors lransla~ion from maps and other doc~ ;on to the real world and to the S operators of the digging equipment require skill and experience. Even with recent and accurate maps, the task can become a nigl~ e and failures very expensive. Ifone could look at a display aligned to the real scene which inr.lll(led an image ofthe scene and superimposed thereon the positions of the buried works, then it would be a trivial matter to mark the locations for caution. With the system of the invention, the 20 buried pipes, and wires, et cetera, could be recorded in the data base. The locations of construction projects could then be surveyed in a very short time.
It should be noted that most mapping systems of the art, even very sophisticated computerized maps with data and data symbols co~bil-ed with images, require a translation of the presented image to the images of the real world as viewed 25 by a user. Although HUDs combine a real image and data images that are aligned to reality, the invention can be ~ietinglliehed in that it has the capacity for communication between the real image and the data or data images. They cannot provide for recognition, alignment, undesirable feature extraction, and others advanced features of the invention. Only the present invention provides a system of 30 producing ~lgm~nted images aligned to a scene as the scene is naturally viewed.
It is a primary object of the invention to provide a versatile vision system having capabilities completely unknown to systems in the art.
It is a filrther primary object of the invention to introduce to surveying, çngine~ring, touring, navigation and other arts, a device and method of ~ugmented 35 reality whereby a real scene is imaged with ~lgm~nt~tion which provides additional information about the scene to the user.

SUBSTITUTE SHEET (RULE 26) WO 95/07526 PCT/US9 1J'~
~17~31~ .

It is a further object of the invention to provide a vision system for seeing a topography that includes objects that do not exist at the time of use, but are inten~led to be a part of the scene in some future time.
It is a further object of the invention to provide a vision system for seeing a s lopo~phy that includes objects that can not be seen in conventional viewing systems inclutling c~e~as, binoculars, telescopes, video recorders, head up displays, helmet mounted displays, et cetera.
It is a still further object of the invention to provide a vision system that allows undesirable objects of a real scene to be removed and replaced with computer 0 generated objects to leplese~ objects that are important to the user.

Brief Description of the Drawings These and other features, ~cpectc~ and advantages ofthe present invention will become better understood with regard to the follo~,ving description, appended claims 1S and drawings where:
figure one is a drawing of a user of the invention, a scene as it appears in reality, and the scene as it appears in the ~llgmented display ofthe device ofthe invention;
figure t~vo is a drawing of a city street scenario (2a) whereby the invention aDows a surveyor to see otherwise invisible features such as buried pipes and sewer lines (2b);
figure three is a block diagram a~ngen,~;"l of components and their relationship to each other that sets forth the novel co",bi~ ion of the invention;
figure four is a further det~iled block diagram;
2s figure five is a electro-optical sc~ ;c drawing of some of the system components.

Preferred Embodiments of the Invention The invention provides for a vision system device operable for producing an augm~nted image. In drawing figure one, a user 2 is shown to be looking into a canyon through a portable assembly 1 of the invention. An a~lgmp~lted image 3 ispresented to the user and is aligned with the real scene. The ~ugm~nted image may include: elements that do not yet exist, for example a bridge 4, repres~nting information added to a real scene; elemPnt.e that exist in the real scene but do not 3s appear in the final image, for can,ple trees at the right side ofthe bridge, leprese~-l;,~g information that has been deleted; and el~ "e"le that have been changed from their actual appearance. In drawing figure two, the c~n,ple of construction site surveying is illustrated. A scene as viewed normally would look like figure two(a).

SUBSTITUTE SHEET (RULE 26~

wo 95/07526 217 ~ 31~ PCT/US94/06844 In figure two(b), the fresh water pipes 5 leading to homes 7 and fire hydrants 8 and the buried sewer lines can be easily "seen" in the display of the invention.
In a first p.~r~" ed embodiment of the invention, an electro-optic app&~ s cû",p,ising a camera 9, a position dete",-inil,g means 16, an ~ttit~1de dele""ini"g means 1~, a data store 12, a computer 14, and a display 13; the camera being an electro-optic device opelable for converting a photon input from a field of view, into an optical image at an image field and ope~able for converting that photon image into an eICCIIOI~~C image and ope,~ble for II~n!`~ that electronic image to the computer, the positioning means being operable for dete. lnilfing the position of the 0 camera, the ~ttihlde means being operable for det~."l.nillg the camera pointing attitude of the camera defined by the symmetry axis of the camera field of view, the data store being in communication with the position and ~ttitude dt;le, lllining means whereby the values of position and attitude in-lic~te a location or locations in the data store being further in communication 39 with the computer whereby image data canlS be l~ ed thereto, the computer whereby the CleCIIOI~iC image from the camera and the electronic image 4 of the data store are COnlp&l t;d, analyzed, processed and co",bined for display as a single image, the computer being in communication with, the display, whereby camera position and attitude infullllalion particular to the scene are used to augment a real image of a scene that is particular to the position and ~ttitl1~e of the camera, is provided. A detailed description of each of the main device elements follows In plcfe~;d embodiments ofthe invention it is ~nticip~ted that the camera portion of the device will be comprised of four main subsystems integrated together 2s into a portable assembly 1 that is easy for a user 2 to hold. These include: an optical input lens assembly 38, an image stabilization system, at least one charge coupled device, CCD 20, and an optical ranging system 34.
A photon input im~ging lens assembly can be mounted into the portable camera unit 1. Single axis, or "monocular" systems are commercially available and are commonly used in devices such as camcorders and digital photographic cameras. It is pl ;;~1 l c;d that the lens have a zoom and autofocus capability; these systems are well developed in the art. An autofocus system can be computer driven but releasable for user override. Lens zoom operations can also be power driven and controlled m~nll~lly by the user at an electronic control switch, or could be driven by thecomputer based on known features particular to a chosen scene. The physical dimensions of the input lens define the field of view of the device which is variable with the zoom plùpellies ofthe lens. II~ Lion rega,di,lg the field of view can be supplied to the processor to be used in image combination routines.

SUBSTITUTE SHEET (RlJLE 26) 217~31~

Image stabilization systems are also well developed in the electronic im~gin~
arts and the known systems are useful for pl~;r~,led embodiments of this invention.
Image stabilization can be realized by way of solid state piezo-electric devices to drive a distortable prisms. Motion information from sensitive gyros 35 could detect s movements and provide a driving signal to the prisms. The system of the invention can be modified so that output from each of the gyros is made available to the control computer for calculations of the attitude and position of the portable assembly. This can be particularly important for systems not having access to a GPS signal, forexample in a subn,~ine or in a system dçcigned for microscale applications such as o surgery or micro manipulation. As an alternative to meçh~nical stabilization systems, an electronic image stabilization system could be used but there could be a loss of image quality and, in certain situations of rapid motion, a loss of continuity in image input. Also; an electronic image stabilization system could not be readily modified to provide motion i,~, Illa~ion to the control computer.
After the photon optical input from a scene is stabilized by the image stabilization system, it is then focused onto at least one chal~ed coupled device, CCD, that is positioned in the image plane of the input lens. A possible alternative could be a silicon intçn~ified camera or other image-con device but CCDs are p~rel~ed because their output can be compalil)le with electronic computers using digital data.
The light field from the scene is converted to a digital electronic image signal which 1 epl ese"Ls the scene. CCD devices are also quite comrnonly available and used in modern im~ing devices and standard CCDs are sl.ffiçiçnt for use in the present invention without modification or with only minor rh~nges. The camera can be color sensitive and two common methods for producing color images are possible. It is also anticipated that CCD devices that are sensitive to portions of the spectrum other than the human-visible region may be useful. For example some systems may benefit from CCD devices that are tuned for infrared IR im~gjng Use of IR devices may beimportant for applications such as fire fighting and agronomy where the important image i,~",nalion may be in a spectrum other that the visible region.
A ranging system is useful for a plurality of system functions. A first system function is to provide for the focus of the input lens. For images to be properly focused onto the CCD, it is necess~ry to adjust the input lens accol di"g to thedistance of the objects being imaged. As a second system function, range i,~".,~lion is also useful in the processing stage as it gives hints to the proper scaling of the real image with respect to the data from the data store. The camera image and data store inputs will be combined such that scale is an impol l~l consideration for proper apl)ealance ofthe final image. A further use, a third use of optical ranging can be to çlimin~te unwanted image features. For example, if a ship's rigging tends to SUBSTIIIJTE SHEET (RULE 2~) ~ wo 95/07526 ~ 17131~ PCT/US~'1!6Yq~l block the view of the shore, then it is possible to input to the image processor the range that is of interest. The processor can then remove objects that are not within the selectecl range.
A ranging system, for example a sonic or ultrasound device, can measure the s ~ t~n~e from the camera to the objects of a scene; the range. It is also possible to use laser or radar systems but they tend to be more complicated and more expensive.
The computer can receive range i~ alion directly from a ranging system or can receive range i,~",laLion from the focus position and zoom condition 18 of the input lens that can be interrogated by a computer routine. This is possible if the lens is 10 dç~i ned with tr~n~duGer devices that convert the focus and zoom lens conditions into electronic signals that are readable by the computer. As an alternative to active ranging systems or as a supplement to, it is possible to determine the range to objects of a scene from co",l,alisons of image features to known elements in the data base that correspond to those image features. A disclosure of this method is presented in US patent #5,034,812, Rawlings. Once position and attitude ofthe portable camerais established, a real image may be input and analyzed by the computer. By reference to a computerized topographic data base, the position and therefore di~ ce and bearing to user selected features may be established. An example; an object such as a lighthouse whose height was previously stored in the data base may be measured from the real image where the magnification conditions of the lens are known thereby making a range detel"ul,alion possible. Each ofthe ranging measurG",e"~s may be made indepçn~lently or in col~lbhl~Lion with a plurality of other measurements to increase accuracy.
po~;ti~r determining means In plGrel I ed embodiments of the invention it is anticipated that the position de~G,,,un.llg means ofthe device will be a Global Positioning System GPS receiver 16.
There are many alternative positioning systems that could be used effectively for other embod;,-,en~s. These incl~ld~, but are not limited to, Loran, Glonass, and Omega.
Since GPS has very high accuracy and has altitude capability, it is an obvious choice for the invention which benefits from the extra precision of that system. In self contained en~drolullGl~ like subterranean or a SUbln~lille, the positioning means may have to be a simplified version of tri~n~ tion from known locations or other - positioning detelllunalion means. For applications on a microscale, for example semiconductor inspection or microdevice mi1nllf~ctllre, it may be useful to have a 3s laser intelre~ ,lG~Gl position measurement means that has accuracy at the sub-nanometer level. The invention does not depend on the particulars of any positioning means just so that the position can be determined and input into a computer processor.

SUBSTITUTE SHEET (RULE 26) wo 95/07s26 2 ~ 7 ~ 3 1 4 PCT/US94/06844 ~

One objective of the invention is to have a small, lightweight, portable camera.To achieve this, it may be llecçc.c~. y to put the bulk of the computing power in a separate unit that is in communication with the portable camera. In this embodiment, the ~ntçnn~ 36 for the GPS receiver would be prere.ably put within the portable s camera or in close proximity thereto, and the more bulky processing portion of the GPS receiver could be combined with other system computing f~ ties in a b~cl~racor ground unit. For units to be used in obstructed locations where a GPS signal may not be available such as within the bridge of a ship, the ~nt~nn~ can be placed in some known position relative to the portable camera and the known and con~
0 dispiaçem~nt therebetween removed in the processing elecl,ollics.
attitude determining means In plert;"ed embodiments ofthe invention it is anticipated that the ~ttiturle detel ll~h~ g means of the device will be one of three alternatives. Each provides the h~rling, tilt and roll of the portable camera unit. A flux gate compass 37 located 5 within the portable camera provides he~ing hlro~ ~,alion to an accuracy of + 1degree. The output from this flux gate cor"pass can be made available to the computer. Outputs from piezo-electric gyros located in the image stabilization system can also be used by the computer to calculate motion from a datum established by the user. The datum can be from rest in a known level position or can be established by 20 observing the horizon by looking through the portable camera. An alternative is a triaxial m~gnPtQmeter system incorporating a biaxial electrolytic inclinometer. This tr~n~ c~r is located within the camera and is operable for providing the computer with a complete attitude de~elll~in~lion. It is a further alternative to compute attitude (inclllt1ing he~ing) from a known datum using the piezo-electric gyros. Selection of 25 this alternative may be for reasons of lowered cost as well as pe.",ill...~ application where m~gn~o.tic fields may preclude accurate h.o~ing readings ( i.e. close to electric arc welding equipment; large ,.~"~rc.""e,s and etc.) Note that since the image stabilization system already colllains the required piezo-electric gyros, it would be possible and desirable to provide this alternative as an optional method for 30 establishing hç~riing~ in all above alternatives.
computer In pl ere~ ed embodiments of the invention it is anticipated that the computer processor of the device will be a microcomputer with very fast graphic and videoabilities.
35 Because of the advanced state of computer development, many of the nece~s~ry elements of a computer, for example fast frame grabbers 22 and massive cache memories 23, have already been developed. The computers and software used in video games and computer generated animation applications lend themselves to the SUBSTITUTE SHEET (RULE 25) ~ 171314 WO 95/07S26 ~ PCTIUS!)1~

tasks at hand and can be readily be converted to those tasks. As the functions of the present invention are very unique, the l-~cess~,y sonw~e to achieve those functions will also necessarily be unique. It is ~nticirated that the complicated system instruction set design, either imple~ ed in h&rdwa~e such as ~OM or in software,s will be proprietary. It is likely to have task specific instruction sets and to have many devices of the invention each having their own particular instruction set.
data store In pl~re,led embo-lim~ntc ofthe invention it is ~ntit.ir~tecl that the data store means of the device will be a mass memory device; for ,Aa,.,ple a CD-ROM. l'he 10 data store can have pre-prog,~",ed i"ru""aLion ,ega, di~-g particular scenes that are of interest to a specific user. The position dete,l",ni.~, means and the ~ttitl-~e detellllinillg means control the pointer ofthe data store. In the simplest embodiment, the data store has locations defined by an orthogonal array of two variables. The values of position Pn and of attitude An uniquely define a corresponding scene SCENE{Pn, An3. In advanced versions, the range can also be important and a rangevariable, a third orthogonal variable defines a three dimensional array: SCENE{Pn, An~ Rn} The variable defined by R tells the computer in which plane normal to the axis of the camera, lies the il~l ln~lion of interest. In even further advanced versions, a magnification variable Mn can be similarly used to give a four dimensional array:
SCENE{Pn, An~ Rn~ Mn}. Many other factors can be used to control the way that data is stored and recalled. Each of the methods being particular to the task at hand, are subsets of the general case where data I e~ardil,~ a scene is selected from a data store in accordance to identification of the scene being viewed.
display 2s In plt;~"ed embodiments ofthe invention it is anticipated that the display 13 means of the device is a common active matrix LCD 32 device. These are currentlythe brightest, fastest displays available and are quite colllmollly used in conjunction with video applications. Alternatively, it is possible to use plasma display, electrol~ esce,.l or any of several possible displays. The particular type of display is not hl~po, l~ and may be application specific without deviating from the gist of the invention. An impol lalll feature of the display is that an electronic signal generated from a coml)u~el video card 28 can be converted into a optically viewable image that represents an a~lgmented real image of the scene that the camera is addressing. It is also a major feature of the invention to have the display oriented with reality and 3s therefore aligned with respect to the optical axis ofthe camera as is shown in figure 1.
other In prert;"ed embodiments ofthe invention it is ~nti~.ip~ted that the device can also include such appa, ~ ses such as user input control keys 29 to interact with the SUBSTlM~ ~HEET ~Rlll E 2~) -. ',, ~.
computer routines and to specify h~o",.alion that drives further computer routines;
audio tr~n.~ducPrs 31 both input types such as microphones and output types such as speakers also for control of computer routines and for prese"lalion of h~,l"alion to a user. Standard equipment known to couple vision devices to human users and 5 physical conditions such as a tripods, lens shades, eyepiece fixtures 30, et cetera are all considered to be co~p~ )le with the device of the invention.
The invention also provides for methods of producing an ~llgm~nted image.
In p, er~;l, ed embo~impnts of the invention, a vision system method co",~
0 an acquire step, a conversion step, a position del~l"",f,ng step, an attitude d~l~",~i,.."g step, a data recall step, a processing step, and a display step; wherein the im~ ing step the objects of a scene are imaged onto a two dimensional plane as the light e~ g from the objects propagates through at least one lens and forms the image at the plane, v~/hele;ll the digiti7:ing step the two dimensional optical input is lS converted into an electrical signal, wherein the position delel ll,il,ing step the position ofthe camera is dete"""~ed with respect to some a,l,i~,~"~ rt;rt;,t;,lce point, wherein the ~ttittlde d~ ""ining step the attitude of the camera is determined with respect to some albi~laly direction, wherein the data recall step data from a data base is recalled in accol iance with the measu, e",e"ls of the previous two steps, v~L~rein the 20 processing step the image of the ligiti~ing step and the data of the data recall step are combined to form a final image, and wherein the display step the final electronic image is converted to an optical image and displayed such that it is aligned to and co" esponds with the real scene, is provided. A det~iled description of each of the main method steps follows.
25 the acquire step In the acquire step, a camera of the device is pointed in the direction of a scene and a photon image is forrned by a lens. The photon image is formed onto an electronic device that has been placed into the image plane of the camera lens. The desired field of view can be challged with the zoom function of the lens to selectively 30 choose portions of a scene. The acquire step is likely to be continuous in p, efe"ed embo~im~nt~ and pointing at various objects will result in a continuously updated display. Real-time functions such as "p~nning" in which many acquire steps occurs in rapid sllccç~ion, are completely anticipated. The acquire step can be user influenced.
If a pre-selected range is chosen by way of a user input, the computer can 3s accommodate and adjust the lens in accordance with that input. This becomes more clear in the exa",ple where the user wishes to see the features of a distant object but the automatic ranging is keying on some object in the fo,egloLmd. A selected range can el;.. ~ e acquisition of undesirable foreground objects.

SU~3STITUTE SHEET (~ULE 26 WO 95/07526 ~, ~ 7 1~ ~ 4 PCT/US~1~'0~81 the converting step The method ofthe invention distin~.ixhes itselffrom methods such as HUDs and HMDs in that a photon r eprcsenL~lion of a scene is converted into an electronic signal that can be processed by a digital computer. The image of the real scene can be 5 converted into image frames at standard video rates. In this way, çh~n~es are not required to ;xl;.,g video equipment and that equipment can be easily integrated into systems of the invention. The digital signal of a CCD is particularly desirable because a great deal of processing hald~a.c, cc)~ dlilJle with CCDs already exists in the art.
The pixel replesc.lLalion of images of CCD devices is also co...~ le with the pixel 10 leples_lalion of images of LCD displays. The digital image ofthe CCD can be directly l~ ed to the computer processor routines.
the determine position step The position of the camera can be deterrnined in global (macro) applications with GPS methods and in microscopic applications with a laser interferometer. The S positioning step is used to locate the camera with respect to an albill~ly point and to relay the result of that measurement to the computer such that the computer can identify where in the data base the il~ Lion that corresponds to the scene that the camera is addressing is located.
the determine attitude step The attitude of the camera optical axis with respect to all three axis of rotation is measured to identify the direction the camera is pointed thereby further idenliryil.g what scene the camera is addressing and what appears in the camera field of view.
The ~ttit~lde figure typically drives one of the data store pointer variables and thereby, when in colllbinalion with a position figure, uniquely selects data stored therein.
2s Attitude can be realized di~e. cllLly in various applications. In marine applications, "heading" and an implied horizon can uniquely deffne a scene. This is considered a special case of attitude on all three axis of rotation. It many applications, altitude will need to be considered for a scene to be properly identified. The plill.aly function of the position and attitude steps are to give the computer enough i-lru-malion to unambiguously identify a scene.
the recall recorded data step Having made a position and attitude dete--nin~ion, the minimllm amount of - il.rOl IllaLion needed to specify a location in the data base is realized. For some simple applications, this is enough h~l Ill~Lion to recall stored data that can be colllbined with a real image produced by a camera looking into a scene located by the position and attitute measul elllc;llL~. In advanced applications, other factors such as range to object and magnification, may also need to be specified to specify the data to be SUBSTITUTE SHEET (F~ULE 26 W0 95/07526 2 ~7 ~ PCT/U~ ,~

recalled. The data after being recalled is trQn~mitted to the processor to be combined with real image data.
the p. oce~;ng step Real image data are received into the processor from the converting step and s ~u~ ;on data is received into the processor from the data recall step. Feature recognition teçhniques and image manipulation routines are applied to two forms of il~""~ion to finally yield an image that is a desired col"bil-alion of the two. Certain undesirable il~""~lion can be removed from the image of the real scene. Important n~""~lion not appea,i,lg in the image of the real scene, but found in the data can be o ~nh~nced in the final image.
display augmented image A ~ m~nted real image comprised of real image i,~, ~-la~ion and data from the data store is assembled in the processor and ~ n.~ (ed in standard video format to a display device. The final image is then displayed in accurate ~ nm.ont with the lS real scene in a way that allows the viewer to "see" objects and features, otherwise invisible in other vision systems in perfect orientation with the real scene.
mented real images can be generated at video rates giving a display that provides a real-time view of reality with computer generated ~I~gm~nt~tion combined therewith.
Although the present invention has been described in considerable detail with reference to certain prt;r~;"ed versions thereof other versions are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the versions contained herein.
In accordance with each of the pl ere" ~d embodiments of the invention, there is provided an apph-~t-ls for and method of realizing a vision system. It will be appreciated that each ofthe embodiments described include both an app~ s and method, and that the appa ~ s and method of one p.t;r~;.. ed embodiment may be dirrel elll than the appa- ~l~s and method of another embodiment.

SUBST~TUTE SHEET (RULE 26

Claims (5)

Claims What is claimed is:

1) An electro-optic apparatus for producing an augmented image from an image of a real scene and a computer generated image, the augmented image being aligned to the scene such that the directions up, down, left and right of the augmented image correspond directly with up, down, left and right of the real scene and the normal direction of the image plane points in a direction away from the user and towards the scene, said augmented image being comprised of information generated by an imaging means and information recalled from a data store, the information being particular to the position as determined by the apparatus and attitude of the apparatus, which comprises:
an imaging means;
a position determining means;
an attitude determining means;
a data store;
a computer; and a display, said imaging means having an optic axis which is a symmetry axis and defines the direction of viewing, a lens symmetric about that axis which defines an image field, and a charge coupled device in the image field;
said position determining means having a reference point corresponding to said imaging means;
said attitude determining means having a reference direction parallel to the direction of viewing of said imaging means;
said data store having memory locations wherein prerecorded data are stored, a plurality of orthogonal variables which identify those memory locations, and a pointer responsive to values from the position and attitude determining means which selects particular memory locations containing particular prerecorded data corresponding to the real scene;
said computer being in electronic communication with said imaging means, position and attitude determining means, data base, and display, having graphics and image processing capabilities; and said display being an electro-optic emissive display having a normal direction orthogonal to the display plane, the normal direction being colinear with the symmetry axis of the imaging means and being in communication with said computer.

2) The apparatus in claim 1 where the display is integrated into a binocular type viewing device having an optical path which corresponds to each of two eyes.

3) The apparatus of claim 1 further comprising an optics system of lenses to image the display at optical infinity to give the user the appearance and feeling of actually looking at the real scene as opposed to looking at a display device in the near field thereby allowing the user to relax the eye muscles by focusing at infinity.

4) The apparatus of claim 1 said computer being responsive to image features of the real scene as indicated by some predetermined condition.

5) An electro-optic apparatus for producing an augmented image from an image of a real scene and a computer generated image, the augmented image being aligned to the scene such that the directions up, down, left and right of the augmented image correspond directly with up, down, left and right of the real scene and the normal direction of the image plane points in a direction away from the user and towards the scene, said augmented image being comprised of information of generated by an imaging means and information recalled from a data store, the information being particular to the position as determined by the apparatus and attitude of the apparatus, which comprises:
an imaging means;
a position determining means;
an attitude determining means;
a data store;
a computer; and a display, said imaging means being sensitive to photon input and operable for converting light into an electronic signal processable by said computer;
said positioning determining means being operable for determining the location of the imaging means and presents a value which represents that location to the computer;
said attitude determining means being operable for determining the attitude of the imaging means as defined by the axis of the imaging means and presents a value which represents that attitude to the computer;
said data store being operable for storing prerecorded data corresponding to and representing objects known to be in the present field of view of the imaging means as defined by the present position and attitude thereof, recalling that data and transmitting it to the computer;
said computer being operable for receiving signals from imaging means and data store and combining those signals such that a composite image signal is formed and further operable for transmitting the composite image to said display;
said display being an electro-optic emissive display operable for converting the composite image signal of the computer to a physical signal viewable by a user that is aligned to the viewing axis of said imaging means, the electro-optic vision apparatus allows the user to see his environment with computer generated augmented images aligned to the scene as the user would view the scene normally.