WO2012118575A2 - Alignment control in an augmented reality headpiece - Google Patents

Alignment control in an augmented reality headpiece Download PDF

Info

Publication number
WO2012118575A2
WO2012118575A2 PCT/US2012/022568 US2012022568W WO2012118575A2 WO 2012118575 A2 WO2012118575 A2 WO 2012118575A2 US 2012022568 W US2012022568 W US 2012022568W WO 2012118575 A2 WO2012118575 A2 WO 2012118575A2
Authority
WO
WIPO (PCT)
Prior art keywords
image
see
viewer
scene
view
Prior art date
Application number
PCT/US2012/022568
Other languages
French (fr)
Other versions
WO2012118575A3 (en
Inventor
John N. Border
John D. Haddick
Original Assignee
Osterhout Group, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US13/037,324 external-priority patent/US20110214082A1/en
Priority claimed from US13/037,335 external-priority patent/US20110213664A1/en
Application filed by Osterhout Group, Inc. filed Critical Osterhout Group, Inc.
Priority to CA2828413A priority Critical patent/CA2828413A1/en
Priority to DE112012001022T priority patent/DE112012001022T5/en
Publication of WO2012118575A2 publication Critical patent/WO2012118575A2/en
Publication of WO2012118575A3 publication Critical patent/WO2012118575A3/en

Links

Classifications

    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/28Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising
    • G02B27/281Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 for polarising used for attenuating light intensity, e.g. comprising rotatable polarising elements
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01Head-up displays
    • G02B27/017Head mounted
    • G02B27/0172Head mounted characterised by optical features
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/10Beam splitting or combining systems
    • G02B27/1066Beam splitting or combining systems for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01Head-up displays
    • G02B27/0101Head-up displays characterised by optical features
    • G02B2027/0118Head-up displays characterised by optical features comprising devices for improving the contrast of the display / brillance control visibility
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B5/00Optical elements other than lenses
    • G02B5/30Polarising elements
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B6/00Light guides; Structural details of arrangements comprising light guides and other optical elements, e.g. couplings

Definitions

  • the present disclosure pertains to augmented reality imaging with a see- through head mounted display.
  • See-through head mounted displays provide a viewer with a view of the surrounding environment combined with an overlaid displayed image.
  • the overlaid image can be semitransparent so that the overlaid displayed image and the view of the surrounding environment are seen simultaneously.
  • a see-through display can be transparent, semitransparent or opaque. In the transparent mode, the view of the environment is unblocked and an overlaid displayed image can be provided with low contrast. In the semitransparent mode, the view of the environment is partially blocked and an overlaid displayed image can be provided with higher contrast. In the opaque mode, the view of the environment is fully blocked and an overlaid displayed image can be provided with high contrast.
  • augmented reality imaging additional information is provided that relates to the surrounding environment.
  • objects in the surrounding environment are identified in images of the surrounding environment and augmented image content that relates to the objects is provided in an augmented image.
  • augmented image content that can be provided in augmented images includes: address labels for buildings; names for stores; advertising for products; characters for virtual reality gaming and messages for specific people.
  • see-through head mounted displays the view of the surrounding environment is not necessarily aligned with the displayed image.
  • Variations in the location of the display area as manufactured, variations in the way that a viewer wears the see-through head mounted display, and variations in the viewer's eye characteristics can all contribute to misalignments of the displayed image relative to the see-through view.
  • adjustments are needed in see- through head mounted displays to align the displayed image to the see-through view so that augmented image content can be aligned to objects from the surrounding environment in augmented images.
  • a light source is provided with a see-through head mounted display to project a marker onto a calibration screen.
  • the displayed image is adjusted in the see-through head mounted display to align the displayed image to the projected marker. While this technique does provide a method to correct lateral and longitudinal misalignment, it does not correct for differences in image size, also known as magnification, relative to the see-through view.
  • the approach of projecting a marker onto the scene is only practical if the scene is within a few meters of the see-through head mounted display, the projected marker would not be visible on a distant scene.
  • an alignment indicator is generated in the image to be displayed and the indicator is aligned to the see-through view by the viewer manually moving the device relative to the viewer.
  • This invention is directed at a handheld see-through display device which can be moved within the viewer's field of view and is not applicable to a head mounted display where the display is mounted on the viewer's head.
  • All of the points in this two stage approach to alignment are collected by moving the see-through head mounted display to align virtual markers in the displayed image with a single point in the real world and a head tracker is used to determine the relative positions of the see- through head mounted display for each point.
  • a 3D marker is generated in a head mounted stereo see-through display.
  • the 3D marker is visually aligned by the viewer with a designated point in the real world and calibration data is gathered. This process is repeated for several positions within the space that will be used for augmented reality.
  • a model of the augmented reality space is built using the calibration data that has been gathered.
  • One embodiment provides a method for aligning a displayed image in a see-through head mounted display to the see-through view perceived by the viewer.
  • the combined image comprised of the displayed image overlaid on the see-through view provides an augmented reality image to the viewer.
  • the method includes capturing a first image of a scene with a camera included in the see-through head mounted display device wherein the scene has objects.
  • the captured first image is then displayed to a viewer using the see-through head mounted display device so that the displayed image and the see-through view of the scene are both visible.
  • One or more additional image(s) of the scene are captured with the camera in which the viewer indicates a misalignment between the displayed first image and a see-through view of the scene.
  • the captured images are then compared with each other to determine an image adjustment to align corresponding objects in displayed images to objects in the see-through view of the scene.
  • Augmented image information is then provided which includes the determined image adjustments and the augmented image information is displayed to the viewer so that the viewer sees an augmented image comprised of the augmented image information overlaid on the see-through view.
  • FIG. 1 is an illustration of a head mounted see-through display device
  • FIG. 2 is an illustration of a scene and the associated displayed image as seen from the viewer's perspective in both eyes;
  • FIG. 3 is an illustration of a combined view as seen by the viewer's right eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the two images are not aligned;
  • FIG. 4 is an illustration of a combined view of a scene wherein the viewer uses a finger gesture to indicate the perceived location of an object (the window) in the displayed image that is not aligned with the see-through view;
  • FIG. 5 is an illustration of a captured image of the viewer's finger gesture indicating the object (the window) location as shown in FIG. 4;
  • FIG. 6 is an illustration of a see-through view as seen by the viewer including the viewer's finger gesture indicating the location of the object (the window) in the see-through view;
  • FIG. 7 is an illustration of a captured image of the viewer's finger gesture indicating the object (the window) location as shown in FIG. 6;
  • FIG. 8 is an illustration of a combined view as seen by the viewer's right eye wherein the displayed image of the scene is overlaid on the see-through view of the scene and the two images are aligned on an object (the window);
  • FIG. 9 is an illustration of a combined view of a scene and the two images are aligned on an object (the window).
  • the viewer uses a finger gesture to indicate the perceived location of another object (the car tire) in the displayed image that is not aligned with the see-through view;
  • FIG. 10 is an illustration of a captured image of the viewer's finger gesture indicating the another object (the car tire) location as shown in FIG. 9;
  • FIG. 11 is an illustration of a see-through view as seen by the viewer including the viewer's finger gesture indicating the location of the another object (the car tire) in the see-through view;
  • FIG. 12 is an illustration of a captured image of the viewer's finger gesture indicating the another object (the car tire) location as shown in FIG. 11;
  • FIG. 13 is an illustration of a combined view as seen by the viewer's right eye wherein the two images are aligned on the object (the window) and resized to align the another object (the car tire);
  • FIG. 14A is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein a displayed label (the address) is overlaid onto an object (the house) in the see-through view and the label is aligned to the object;
  • FIG 14B is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein augmented image information in the form of displayed objects (the tree and bushes) are overlaid onto objects (the car and house) in the see-through view and the displayed objects are aligned to the objects in the see- through view;
  • FIG. 15 is an illustration of a scene and the associated displayed image as seen from the viewer's perspective in both eyes.
  • a marker is visible in the left eye displayed image which indicates the area for the first alignment between the displayed image and the see-through view;
  • FIG. 16 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the two images are not aligned.
  • a marker indicates a first area for alignment;
  • FIG. 17 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the viewer has moved their head to align objects (the roof) in the two images in the area of the marker;
  • FIG. 18 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the two images have been aligned in one area and a marker indicates a second area for alignment;
  • FIG. 19 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and objects (the car tire) in the two images have been aligned in a second area;
  • FIG. 20 is an illustration of a combined view as seen by a viewer in the left eye wherein the displayed image of the scene is overlaid on the see-through view of the scene and the two images are aligned in the two areas of the markers by shifting and resizing the displayed image;
  • FIG. 21 is a flow chart of the alignment process used to determine image adjustments to align displayed images with the see-through view seen by the viewer;
  • FIG. 22 is a flow chart for using the determined image adjustments to display augmented image information with corresponding object as seen the viewer in the see-through view.
  • a displayed image can be viewed by a viewer at the same time that a see-through view of the surrounding environment can be viewed.
  • the displayed image and the see-through view can be viewed as a combined image where one image is overlaid on the other or the two images can be simultaneously viewed in different portions of the see-through display that is viewable by the viewer.
  • the invention provides a simple and intuitive method for indicating misalignments between displayed images and see-through views along with a method to determine the direction and magnitude of the misalignment so that it can be corrected by changing the way that the displayed image is presented to the viewer.
  • FIG. 1 shows an illustration of a head mounted see-through display device 100.
  • the device includes a frame 105 with lenses 110 that have display areas 115 and clear areas 102.
  • the frame 105 is supported on the viewer's head with arms 130.
  • the arms 130 also contain electronics 125 including a processor to drive the displays and peripheral electronics 127 including batteries and wireless connection to other information sources such as can be obtained on the internet or from localized servers through Wifi, Bluetooth, cellular or other wireless technologies.
  • a camera 120 is included to capture images of the surrounding environment.
  • the head mounted see- through display device 100 can have one or more cameras 120 mounted in the center as shown or in various locations within the frame 105 or the arms 130.
  • the see-through head mounted display device 100 includes a camera 120 capturing images of the surrounding environment. For digital cameras it is typical in the industry to correct for distortions in the image during manufacturing. Rotational alignment of the camera 120 in the frame 105 also typically accomplished during manufacturing.
  • the viewer uses a finger gesture to indicate misalignments between a captured image of the surrounding environment that is displayed on the see-through head mounted display, and the see-through view of the surrounding environment as seen by the viewer.
  • FIG. 2 is an illustration of a scene 250 and the associated displayed images 240 and 245 as seen from behind and slightly above the viewer's perspective in both eyes.
  • the displayed images 240 and 245 as shown in FIG. 2 have been captured by the camera 120 of the scene in front of the viewer.
  • the images 240 and 245 can be the same image, or for the case where the see-through head mounted display device 100 has two cameras 120 (not shown), the images can be of the same scene but with different perspectives as in a stereo image set for three dimensional viewing.
  • FIG. 3 is an illustration of a combined view as seen by the viewer's right eye wherein a displayed image 240 of the scene is overlaid on a see-through view 342 of the scene.
  • the displayed image 240 shown in FIG. 3 has been captured by the camera 120 and is then displayed on the see-through head mounted display device 100 as a combined image where the displayed image 240 appears as a semi- transparent image that is overlaid on the see-through view 342.
  • the displayed image 240 and the see-through view 342 are misaligned as perceived by the viewer.
  • the misalignment between the displayed image 240 and the see-through view 342 can vary with changes in viewer or with changes in the way that the viewer wears the see-through head mounted display device 100 each time the device is used.
  • the invention provides a simple and intuitive method for correcting for misalignments.
  • FIGS. 3-13 A method for determining misalignments is illustrated in FIGS. 3-13, and the flow chart shown in FIG. 21.
  • the camera 120 is used to capture a first image of a scene in front of the viewer.
  • the captured first image is then displayed as a semitransparent image on the see-through head mounted display device 100, so that the viewer sees the displayed image overlaid on the see- through view of the same scene in front of the viewer such as is shown in FIG. 3.
  • the viewer selects a first object in the displayed image to use for determining misalignments.
  • the viewer uses their finger to indicate the perceived location of the selected object in the displayed image as shown in FIG. 4, in this example the viewer is shown indicating the window as the selected first object.
  • the displayed image is overlaid on the see- through view of the scene which includes the viewer's finger 425.
  • a second image is then captured by the camera 120 that includes the finger gesture of the viewer indicating the perceived location of the first object as shown in FIG. 5.
  • the misalignment of the finger to the selected first object as seen in the second image can be different depending on the relative locations and associated perspectives of the scene provided by the camera 120 and the viewer's eye.
  • the displayed image is then turned OFF or removed from the see-through head mounted display 100 so that the viewer only sees the see-through view.
  • the viewer indicates the same selected first object (the window in this example) with the viewer's finger 625 in the see-through view as shown in FIG. 6 and a third image is captured by the camera 120 that includes the scene in front of the viewer and the viewer's finger 725 as shown in FIG. 7.
  • the viewer's finger 725 is not aligned with the selected first object (the window) in the third image due to the combined effects of misalignment of the camera 120 with the see-through view and also due to the different perspective of the scene provided by the camera 120 and the viewer's right eye.
  • FIG. 8 is an illustration of a combined view as seen by the viewer's right eye wherein the displayed first image of the scene is overlaid on the see-through view of the scene and the first image has been aligned on the first object (the window).
  • a second object in this example, the car tire is selected by the viewer and the viewer uses their finger 925 to indicate the location of the object in the displayed image as shown in FIG. 9.
  • a fourth image is then captured as shown in FIG. 10 which includes the scene and the viewer's finger 1025.
  • the displayed image is then turned OFF or removed so that the viewer only sees the see-through view of the scene and the viewer uses their finger 1125 to indicate the perceived location of the second selected object in the see-through view as shown in FIG. 11.
  • a fifth image is then captured as shown in FIG. 12 which includes the scene and the viewer's finger 1225.
  • the fourth and fifth images are then compared to determine the respective locations of the viewer's finger 1025 and 1225 and then to determine the image adjustment needed to align the displayed image to the see-through view at the location of the second selected object (the car tire).
  • the determined image adjustments for the locations of the second selected object are then used, along with the distance in the images between the selected first and second objects, to determine the resizing of the displayed image so that when combined with the previously determined lateral and longitudinal adjustments, the displayed image is substantially aligned over the display area 115 with the see-through view as seen by the viewer.
  • the lateral and longitudinal adjustments are determined in terms of x and y pixels shifts.
  • the resizing is then determined as the relative or percent change in the distance between the locations of the viewer's finger 525 and 1125 in the third and fourth images compared to the distance between the locations of the viewer's finger 525 and 1225 in the third and fifth images respectively.
  • the percent change is applied to the displayed image to resize the displayed image in terms of the number of pixels.
  • the resizing of the displayed image is done before the alignment at a location in the displayed image.
  • FIG. 13 shows an illustration of the displayed image overlaid on the see-through view wherein the displayed image has been aligned on the window object and then resized to align the remaining objects so that the combined image has essentially no perceived misalignments between the displayed image and the see-through view.
  • the timing of the multiple images that are captured in the method of the present invention can be executed automatically or manually.
  • the captures can be executed every two seconds until all the images needed to determine the image adjustments have been captured.
  • the viewer has time to evaluate the misalignment and provide an indication of the misalignment.
  • the viewer can provide a manual indication to the see-through head mounted display device 100 when the viewer is satisfied that the misalignment has been properly indicated.
  • the manual indication can take the form of pushing a button on the see-through head mounted display device 100 for example. Images can be displayed to the viewer with instructions on what to do and when to do it.
  • the methods disclosed herein for determining image adjustments to reduce misalignments between displayed images and see-through views are possible because the misalignments are largely due to angular differences in the locations and sizes of objects in the captured images from the camera 120 and the locations and sizes of corresponding objects in the see-through view. Since both the camera 120 and the viewer's eye perceive images in angular segments within their respective fields of view, angular adjustments on the displayed image can be implemented in terms of pixel shifts and pixel count changes or image size changes of the displayed image. Thus, the image adjustments can take the form of x and y pixel shifts in the displayed image along with upsampling or downsampling of the displayed image to increase or decrease the number of x and y pixels in the displayed image.
  • Rotational misalignments can be determined in the process of determining the resizing needed when comparing the fourth and fifth captured images.
  • the examples provided describe methods for determining image adjustments based on the view from one eye. These determined image adjustments can be applied to the displayed images in both eyes or the image adjustments can be determined independently for each eye.
  • displayed images can be modified to compensate for misalignments.
  • the displayed images can be still images or video. Further images of the scene can be captured to enable objects to be identified and the locations of objects in the further images to be determined. Where methods for identifying objects and determining the locations of objects in images are described for example in U.S. Pat. 7805003.
  • Augmented image information can be displayed relative to the determined locations of the objects such that the augmented image information is aligned with the objects in the see-through view by including the image adjustments in the displayed images.
  • additional further images of the scene are captured only when movement of the viewer or the see-through head mounted display device 100 is detected, as the determined locations of objects in the further images are unchanged when the viewer or the see-through head mounted display device 100 is stationary.
  • the same image adjustments can be used for multiple displays of augmented image information to align the augmented image information with the objects as seen by the viewer in the see-through view.
  • the viewer indicates the misalignment between a displayed image and the see-through view by moving their head. Illustrations of this method are shown in FIGS. 15-20. One or more locations are then chosen in the combined image seen by the viewer where an alignment can be performed. If more than one location is used for the alignment, the locations must be in different portions of the combined image, such as near opposite corners. To aid the viewer in selecting the locations used for performing the alignment, in one embodiment, a marker is provided in the displayed image as shown in FIG. 15 where the marker 1550 is a circle. [0054] The displayed image shown in FIG.
  • FIG. 15 on the see-through head mounted display device 100 is a first image captured of the scene by the camera 120 and the displayed image is shown from behind and slightly above the viewer's perspective so that objects in the scene can be seen as well as the displayed image.
  • FIG. 16 is an illustration of the combined view as seen by the viewer in the left eye wherein the displayed image of the scene is overlaid on the see-through view of the scene and a misalignment can be seen.
  • a marker 1550 indicates a first area for alignment.
  • FIG. 17 illustrates a combined view as seen by a viewer's left eye wherein the viewer has moved his or her head to align objects (the roof) in the displayed image and see- through view in the area of the marker 1550.
  • a second image is then captured by the camera 120.
  • the first captured image is then compared to the second captured image by the electronics 125 including the processor to determine the difference between the two images in the location of the marker 1550.
  • the displayed image and the see-through view would be aligned if the perceived sizes of the displayed image and the see-through view were the same and the determined the difference between the first and second captured images is an image adjustment of an x and y pixel shift on the displayed image. If there are still misalignments between the displayed image and the see- through view after an alignment at the location of the marker 1550 as shown in FIG. 17, then a second alignment is performed at a second marker 1850 as shown in FIG. 18.
  • the two images are aligned at the location where marker 1550 had been located, but the remainder of the image has misalignments due to a mismatch in sizes between the displayed image and the see- through view.
  • the viewer then moves his or her head to align objects in the displayed image to corresponding objects (such as the car tire) in the see-through view in the region of the marker 1850 to indicate the further image adjustment that is a resizing of the displayed image.
  • FIG. 19 shows an illustration of the combined image seen by the viewer after the viewer's head has been moved to align objects in the displayed to corresponding objects in the see-through view.
  • a third image is then captured by the camera 120.
  • the third image is then compared to the second image or the first image by the electronics 125 including the processor to determine the image adjustment needed to align the displayed image to the see-through view in the region of the second marker 1850.
  • the image adjustment determined to align the displayed image to the see-through view at the region of the first marker 1550 is then a pixel shift.
  • the percent change in the distance between the locations of objects in the area of the first and second markers when aligning the displayed image to the see-through view in the region of the second marker 1850 is the image adjustment for resizing the displayed image.
  • FIG. 20 shows the fully aligned displayed image, after applying the pixel shift and the resizing, overlaid on the see-through view as seen by the viewer, where misalignments are not visible.
  • Step 2110 the viewer looks at a scene and the camera 120 captures an image of the scene in Step 2120.
  • the captured image is then displayed on the display areas 115 of the see-through head mounted display device 100 operating in a transparent or semi-transparent mode in Step 2130 so the viewer sees a combined view comprised of the displayed image overlaid on the see-through view.
  • the viewer then provides an indication of the misalignment between objects in the displayed image and corresponding objects in the see-through view in Step 2140.
  • the indication of the misalignments can be done by a series of finger gestures or by moving the viewer's head as described previously.
  • the camera 120 is used to capture additional images of the scene along with the viewer's indication of the misalignments in Step 2150. Then in Step 2160, the captured additional images are compared in the electronics 125 to determine the image adjustments needed to align the displayed images with the see-through view as seen by the viewer.
  • the viewer indicates misalignments between captured images of the scene and the see-through view by a combination of hand gestures and head movement.
  • One or more additional images are captured and compared to determine the image adjustments as previously described.
  • the see-through head mounted display device 100 includes a GPS device or a magnetometer.
  • the GPS device provides data on the current location or previous locations of the see-through head mounted display device 100.
  • the magnetometer provides data on the current direction and previous directions of the viewer's line of sight.
  • the data from the GPS or magnetometer or the combination of data from the GPS and magnetometer can be used to help identify objects in the scene or to determine the addresses or locations of objects in the images captured by the camera 120.
  • augmented image information related to the identified objects can be provided in the combined view that is aligned to the respective objects as perceived by the viewer.
  • augmented image information can be aligned with identified objects in the captured images and identified edges of objects in the captured images.
  • head tracking information can be used to adjust augmented image information and the location of augmented image information relative to objects in the displayed images.
  • FIG. 22 shows a flow chart for using a see-through head mounted display device 100 with a GPS or magnetometer wherein the displayed image has been aligned with the see-through view as perceived by the viewer.
  • the GPS or magnetometer is used to determine the location of the viewer or the direction that the viewer is looking.
  • the camera 120 then captures an image of the scene in Step 2220.
  • the electronics 125 including the processor are then used to analyze the captured image along with the determined location or direction information to identify objects in the scene in Step 2230.
  • the see-through head mounted display device 100 uses the peripheral electronics 127 including a wireless connection to determine whether augmented information is available for the identified objects or the determined location or determined direction in Step 2240.
  • available augmented information is displayed in regions or locations of the displayed image that correspond to the objects locations when aligned to the see-through view.
  • a house can be identified in the captured image by the combination of its shape in the captured image and from the GPS location and the direction, the address of the house can then be determined from a map that is available on the internet and the address can be presented in the displayed image such that it overlays the region of the see-through view that contains the house (see FIG. 14A).
  • an image can be captured of a building. GPS data and magnetometer data can be used to determine the approximate GPS location of the building.
  • Augmented information including the name of the building and ongoing activities in the building can be determined from information available from a server in the building that broadcasts over Bluetooth by matching the GPS location and the direction the viewer is looking.
  • a displayed image is then prepared with the name of the building and a list of ongoing activities located in the region of the displayed image that corresponds to the aligned location of the building in the see-through view.
  • An augmented image is then presented to the viewer as a combined image with the displayed image overlaid on the see-through view.
  • the augmented images produced by the these methods can be used for a variety of applications.
  • the augmented image can be part of a user interface wherein the augmented image information is a virtual keyboard that is operated by the viewer with finger gestures.
  • the virtual keyboard needs to be aligned with the see-through view of the viewer's fingers for the viewer to select the desired keys.
  • the locations of the objects can be determined with the aid of GPS data or magnetometer data and the augmented image information can be advertising or names of objects or addresses of objects.
  • the objects can be buildings, exhibitions or tourist attractions where the viewer uses the augmented image to aid making a decision on where to go or what to do. This information should be aligned with the see-through view of the buildings, exhibitions or tourist attractions.
  • FIG. 14A is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein a displayed label 1470 (the address) is overlaid onto an object (the house) in the see-through view and the displayed label 1470 is aligned to the object.
  • the augmented image includes directions or procedural information related to the objects in the scene and the directions or procedural information needs to be aligned to the objects so the viewer can perform an operation properly.
  • the augmented image can be a modified version of the scene in which objects have been added to form a virtual image of the scene.
  • FIG 14B is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein augmented image information in the form of displayed objects 1475 (the tree and bushes) are overlaid onto objects (the car and house) in the see-through view and the displayed objects 1475 are aligned to the objects in the see-through view.

Abstract

This patent discloses a method for providing an augmented image in a see- through head mounted display. The method includes capturing an image of a scene containing objects and displaying the image to a viewer. The method also includes capturing one or more additional image(s) of the scene in which the viewer indicates a misalignment between the displayed image and a see-through view of the scene. The captured images are then compared to determine an image adjustment to align corresponding objects in displayed images to the objects in the see-through view of the scene. This method provides augmented image information that is displayed in correspondence to the image adjustments so the viewer sees an augmented image comprised of the augmented image information overlaid and aligned to the see- through view.

Description

ALIGNMENT CONTROL IN AN AUGMENTED REALITY HEADPIECE
CROSS REFERENCE TO RELATED APPLICATIONS
[0001] The present application is a continuation-in-part of and claims priority to U.S. Pat. Appl. 13/037,324, filed 28 February 2011, now U.S. Pat. No. , and to U.S. Pat. Appl. 13/037,335, also filed on 28 February 2011, now U.S. Pat. No.
, both of which are hereby incorporated by reference in their entirety.
FIELD OF THE INVENTION
[0002] The present disclosure pertains to augmented reality imaging with a see- through head mounted display.
BACKGROUND
[0003] See-through head mounted displays provide a viewer with a view of the surrounding environment combined with an overlaid displayed image. The overlaid image can be semitransparent so that the overlaid displayed image and the view of the surrounding environment are seen simultaneously. In different modes of operation, a see-through display can be transparent, semitransparent or opaque. In the transparent mode, the view of the environment is unblocked and an overlaid displayed image can be provided with low contrast. In the semitransparent mode, the view of the environment is partially blocked and an overlaid displayed image can be provided with higher contrast. In the opaque mode, the view of the environment is fully blocked and an overlaid displayed image can be provided with high contrast.
[0004] In augmented reality imaging, additional information is provided that relates to the surrounding environment. Typically, in augmented reality imaging, objects in the surrounding environment are identified in images of the surrounding environment and augmented image content that relates to the objects is provided in an augmented image. Examples of augmented image content that can be provided in augmented images includes: address labels for buildings; names for stores; advertising for products; characters for virtual reality gaming and messages for specific people. For augmented reality imaging to be effective, it is important for the augmented image content to be aligned with the objects from the surrounding environment in the augmented images. [0005] However, in see-through head mounted displays, the view of the surrounding environment is not necessarily aligned with the displayed image. Variations in the location of the display area as manufactured, variations in the way that a viewer wears the see-through head mounted display, and variations in the viewer's eye characteristics can all contribute to misalignments of the displayed image relative to the see-through view. As a result, adjustments are needed in see- through head mounted displays to align the displayed image to the see-through view so that augmented image content can be aligned to objects from the surrounding environment in augmented images.
[0006] In U.S. Pat. No. 7369101, a light source is provided with a see-through head mounted display to project a marker onto a calibration screen. The displayed image is adjusted in the see-through head mounted display to align the displayed image to the projected marker. While this technique does provide a method to correct lateral and longitudinal misalignment, it does not correct for differences in image size, also known as magnification, relative to the see-through view. In addition, the approach of projecting a marker onto the scene is only practical if the scene is within a few meters of the see-through head mounted display, the projected marker would not be visible on a distant scene.
[0007] In U.S. Pat. Appl. Publ. 20020167536, an alignment indicator is generated in the image to be displayed and the indicator is aligned to the see-through view by the viewer manually moving the device relative to the viewer. This invention is directed at a handheld see-through display device which can be moved within the viewer's field of view and is not applicable to a head mounted display where the display is mounted on the viewer's head.
[0008] In the article "Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR" by M. Tuceryan, N. Navab, Proceedings of the IEEE and ACM International Symposium on Augmented Reality, pp. 149-158, Munich, Germany October 2000, a method of calibrating a see-through head mounted display to a surrounding environment is presented. The method is for a see-through head mounted display with an inertial tracking device to determine the movement of the viewer's head relative to the surrounding environment. Twelve points are collected wherein the viewer moves their head to align virtual markers in the displayed image with a single point in the surrounding environment. For each point, data is gathered from the inertial tracking device to record the relative position of the viewer's head. A click on an associated mouse is used to indicate the viewer has completed the alignment of each point and to record the inertial tracker data. In the article "Practical solutions for calibration of optical see-through devices", by Y. Gene, M. Tuceryan, N. Navab, Proceedings of International Symposium on Mixed and Augmented Reality (ISMAR'02), 169-175, Darmstadt, Germany, 2002 a two stage approach to alignment of a displayed image in a see-through head mounted display is presented based on the SPAAM technique. The two stage approach includes an 11 point offline calibration and a two point user based calibration. All of the points in this two stage approach to alignment are collected by moving the see-through head mounted display to align virtual markers in the displayed image with a single point in the real world and a head tracker is used to determine the relative positions of the see- through head mounted display for each point.
[0009] In U.S. Pat. 6753828, a 3D marker is generated in a head mounted stereo see-through display. The 3D marker is visually aligned by the viewer with a designated point in the real world and calibration data is gathered. This process is repeated for several positions within the space that will be used for augmented reality. A model of the augmented reality space is built using the calibration data that has been gathered.
SUMMARY
[0010] One embodiment provides a method for aligning a displayed image in a see-through head mounted display to the see-through view perceived by the viewer. The combined image comprised of the displayed image overlaid on the see-through view provides an augmented reality image to the viewer. The method includes capturing a first image of a scene with a camera included in the see-through head mounted display device wherein the scene has objects. The captured first image is then displayed to a viewer using the see-through head mounted display device so that the displayed image and the see-through view of the scene are both visible. One or more additional image(s) of the scene are captured with the camera in which the viewer indicates a misalignment between the displayed first image and a see-through view of the scene. The captured images are then compared with each other to determine an image adjustment to align corresponding objects in displayed images to objects in the see-through view of the scene. Augmented image information is then provided which includes the determined image adjustments and the augmented image information is displayed to the viewer so that the viewer sees an augmented image comprised of the augmented image information overlaid on the see-through view.
BRIEF DESCRIPTION OF THE DRAWINGS
[0011] FIG. 1 is an illustration of a head mounted see-through display device;
[0012] FIG. 2 is an illustration of a scene and the associated displayed image as seen from the viewer's perspective in both eyes;
[0013] FIG. 3 is an illustration of a combined view as seen by the viewer's right eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the two images are not aligned;
[0014] FIG. 4 is an illustration of a combined view of a scene wherein the viewer uses a finger gesture to indicate the perceived location of an object (the window) in the displayed image that is not aligned with the see-through view;
[0015] FIG. 5 is an illustration of a captured image of the viewer's finger gesture indicating the object (the window) location as shown in FIG. 4;
[0016] FIG. 6 is an illustration of a see-through view as seen by the viewer including the viewer's finger gesture indicating the location of the object (the window) in the see-through view;
[0017] FIG. 7 is an illustration of a captured image of the viewer's finger gesture indicating the object (the window) location as shown in FIG. 6;
[0018] FIG. 8 is an illustration of a combined view as seen by the viewer's right eye wherein the displayed image of the scene is overlaid on the see-through view of the scene and the two images are aligned on an object (the window);
[0019] FIG. 9 is an illustration of a combined view of a scene and the two images are aligned on an object (the window). The viewer uses a finger gesture to indicate the perceived location of another object (the car tire) in the displayed image that is not aligned with the see-through view;
[0020] FIG. 10 is an illustration of a captured image of the viewer's finger gesture indicating the another object (the car tire) location as shown in FIG. 9;
[0021] FIG. 11 is an illustration of a see-through view as seen by the viewer including the viewer's finger gesture indicating the location of the another object (the car tire) in the see-through view;
[0022] FIG. 12 is an illustration of a captured image of the viewer's finger gesture indicating the another object (the car tire) location as shown in FIG. 11; [0023] FIG. 13 is an illustration of a combined view as seen by the viewer's right eye wherein the two images are aligned on the object (the window) and resized to align the another object (the car tire);
[0024] FIG. 14A is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein a displayed label (the address) is overlaid onto an object (the house) in the see-through view and the label is aligned to the object;
[0025] FIG 14B is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein augmented image information in the form of displayed objects (the tree and bushes) are overlaid onto objects (the car and house) in the see-through view and the displayed objects are aligned to the objects in the see- through view;
[0026] FIG. 15 is an illustration of a scene and the associated displayed image as seen from the viewer's perspective in both eyes. A marker is visible in the left eye displayed image which indicates the area for the first alignment between the displayed image and the see-through view;
[0027] FIG. 16 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the two images are not aligned. A marker indicates a first area for alignment;
[0028] FIG. 17 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the viewer has moved their head to align objects (the roof) in the two images in the area of the marker;
[0029] FIG. 18 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and the two images have been aligned in one area and a marker indicates a second area for alignment;
[0030] FIG. 19 is an illustration of a combined view as seen by a viewer in the left eye wherein a displayed image of the scene is overlaid on a see-through view of the scene and objects (the car tire) in the two images have been aligned in a second area;
[0031] FIG. 20 is an illustration of a combined view as seen by a viewer in the left eye wherein the displayed image of the scene is overlaid on the see-through view of the scene and the two images are aligned in the two areas of the markers by shifting and resizing the displayed image; [0032] FIG. 21 is a flow chart of the alignment process used to determine image adjustments to align displayed images with the see-through view seen by the viewer; and
[0033] FIG. 22 is a flow chart for using the determined image adjustments to display augmented image information with corresponding object as seen the viewer in the see-through view.
DETAILED DESCRIPTION
[0034] In a see-through display, a displayed image can be viewed by a viewer at the same time that a see-through view of the surrounding environment can be viewed. The displayed image and the see-through view can be viewed as a combined image where one image is overlaid on the other or the two images can be simultaneously viewed in different portions of the see-through display that is viewable by the viewer.
[0035] To provide an effective augmented reality image to a viewer, it is important that the augmented image information is aligned relative to objects in the see-through view so that the viewer can visually associate the augmented image information to the correct object in the see-through view. The invention provides a simple and intuitive method for indicating misalignments between displayed images and see-through views along with a method to determine the direction and magnitude of the misalignment so that it can be corrected by changing the way that the displayed image is presented to the viewer.
[0036] FIG. 1 shows an illustration of a head mounted see-through display device 100. The device includes a frame 105 with lenses 110 that have display areas 115 and clear areas 102. The frame 105 is supported on the viewer's head with arms 130. The arms 130 also contain electronics 125 including a processor to drive the displays and peripheral electronics 127 including batteries and wireless connection to other information sources such as can be obtained on the internet or from localized servers through Wifi, Bluetooth, cellular or other wireless technologies. A camera 120 is included to capture images of the surrounding environment. The head mounted see- through display device 100 can have one or more cameras 120 mounted in the center as shown or in various locations within the frame 105 or the arms 130.
[0037] To align images in a see-through head mounted display, it is necessary to know at least two different points in the images where corresponding objects in the images align. This allows calculations for shifting the images to align at a first point and resizing to align the second point. This assumes that the two images are not rotationally misaligned and the images are not warped or distorted. As shown in FIG. 1, the see-through head mounted display device 100 includes a camera 120 capturing images of the surrounding environment. For digital cameras it is typical in the industry to correct for distortions in the image during manufacturing. Rotational alignment of the camera 120 in the frame 105 also typically accomplished during manufacturing.
[0038] In an embodiment of the invention, the viewer uses a finger gesture to indicate misalignments between a captured image of the surrounding environment that is displayed on the see-through head mounted display, and the see-through view of the surrounding environment as seen by the viewer.
[0039] FIG. 2 is an illustration of a scene 250 and the associated displayed images 240 and 245 as seen from behind and slightly above the viewer's perspective in both eyes. The displayed images 240 and 245 as shown in FIG. 2 have been captured by the camera 120 of the scene in front of the viewer. The images 240 and 245 can be the same image, or for the case where the see-through head mounted display device 100 has two cameras 120 (not shown), the images can be of the same scene but with different perspectives as in a stereo image set for three dimensional viewing.
[0040] FIG. 3 is an illustration of a combined view as seen by the viewer's right eye wherein a displayed image 240 of the scene is overlaid on a see-through view 342 of the scene. The displayed image 240 shown in FIG. 3 has been captured by the camera 120 and is then displayed on the see-through head mounted display device 100 as a combined image where the displayed image 240 appears as a semi- transparent image that is overlaid on the see-through view 342. As can be seen in FIG. 3, the displayed image 240 and the see-through view 342 are misaligned as perceived by the viewer. The misalignment between the displayed image 240 and the see-through view 342 can vary with changes in viewer or with changes in the way that the viewer wears the see-through head mounted display device 100 each time the device is used. As a result, the invention provides a simple and intuitive method for correcting for misalignments.
[0041] A method for determining misalignments is illustrated in FIGS. 3-13, and the flow chart shown in FIG. 21. In an embodiment of the invention, the camera 120 is used to capture a first image of a scene in front of the viewer. The captured first image is then displayed as a semitransparent image on the see-through head mounted display device 100, so that the viewer sees the displayed image overlaid on the see- through view of the same scene in front of the viewer such as is shown in FIG. 3. The viewer then selects a first object in the displayed image to use for determining misalignments. The viewer then uses their finger to indicate the perceived location of the selected object in the displayed image as shown in FIG. 4, in this example the viewer is shown indicating the window as the selected first object.
[0042] As can be seen in FIG. 4, the displayed image is overlaid on the see- through view of the scene which includes the viewer's finger 425. A second image is then captured by the camera 120 that includes the finger gesture of the viewer indicating the perceived location of the first object as shown in FIG. 5. Due to misalignment between the see-through view and images captured by the camera 120 and different perspectives of the scene (also known as parallax) between the camera 120 and the viewer's right eye, there is a misalignment in the second image between the viewer's finger 525 and the selected first object (the window) as shown in FIG. 5. The misalignment of the finger to the selected first object as seen in the second image can be different depending on the relative locations and associated perspectives of the scene provided by the camera 120 and the viewer's eye.
[0043] The displayed image is then turned OFF or removed from the see-through head mounted display 100 so that the viewer only sees the see-through view. The viewer then indicates the same selected first object (the window in this example) with the viewer's finger 625 in the see-through view as shown in FIG. 6 and a third image is captured by the camera 120 that includes the scene in front of the viewer and the viewer's finger 725 as shown in FIG. 7. As with the second image, the viewer's finger 725 is not aligned with the selected first object (the window) in the third image due to the combined effects of misalignment of the camera 120 with the see-through view and also due to the different perspective of the scene provided by the camera 120 and the viewer's right eye. The lateral and longitudinal image adjustments (also known as image shifts) needed to align the displayed image and the see-through view are then determined by comparing the location of the viewer's finger 525 in the second image to the location of the viewer's finger 725 in the third image. Methods for comparing images to align images based on corresponding objects in the images are described for example in United States Patent 7755667. The determined lateral and longitudinal image adjustments are then applied to further displayed images to align the displayed images laterally and longitudinally with the see-through view. [0044] FIG. 8 is an illustration of a combined view as seen by the viewer's right eye wherein the displayed first image of the scene is overlaid on the see-through view of the scene and the first image has been aligned on the first object (the window). However in this case, as can be seen in FIG. 8, objects in the displayed image are not the same size as the see-through view and as a result, objects other than the selected first object are still not aligned. To determine the image adjustments needed to align the rest of the displayed image with the see-through view by resizing the displayed image, a second object (in this example, the car tire) is selected by the viewer and the viewer uses their finger 925 to indicate the location of the object in the displayed image as shown in FIG. 9.
[0045] A fourth image is then captured as shown in FIG. 10 which includes the scene and the viewer's finger 1025. The displayed image is then turned OFF or removed so that the viewer only sees the see-through view of the scene and the viewer uses their finger 1125 to indicate the perceived location of the second selected object in the see-through view as shown in FIG. 11. A fifth image is then captured as shown in FIG. 12 which includes the scene and the viewer's finger 1225. The fourth and fifth images are then compared to determine the respective locations of the viewer's finger 1025 and 1225 and then to determine the image adjustment needed to align the displayed image to the see-through view at the location of the second selected object (the car tire). The determined image adjustments for the locations of the second selected object are then used, along with the distance in the images between the selected first and second objects, to determine the resizing of the displayed image so that when combined with the previously determined lateral and longitudinal adjustments, the displayed image is substantially aligned over the display area 115 with the see-through view as seen by the viewer. The lateral and longitudinal adjustments are determined in terms of x and y pixels shifts.
[0046] The resizing is then determined as the relative or percent change in the distance between the locations of the viewer's finger 525 and 1125 in the third and fourth images compared to the distance between the locations of the viewer's finger 525 and 1225 in the third and fifth images respectively. The percent change is applied to the displayed image to resize the displayed image in terms of the number of pixels. In an alternate method, the resizing of the displayed image is done before the alignment at a location in the displayed image. FIG. 13 shows an illustration of the displayed image overlaid on the see-through view wherein the displayed image has been aligned on the window object and then resized to align the remaining objects so that the combined image has essentially no perceived misalignments between the displayed image and the see-through view.
[0047] The timing of the multiple images that are captured in the method of the present invention can be executed automatically or manually. For example, the captures can be executed every two seconds until all the images needed to determine the image adjustments have been captured. By separating the captures by two seconds, the viewer has time to evaluate the misalignment and provide an indication of the misalignment. Alternately, the viewer can provide a manual indication to the see-through head mounted display device 100 when the viewer is satisfied that the misalignment has been properly indicated. The manual indication can take the form of pushing a button on the see-through head mounted display device 100 for example. Images can be displayed to the viewer with instructions on what to do and when to do it.
[0048] It should be noted that the methods disclosed herein for determining image adjustments to reduce misalignments between displayed images and see-through views are possible because the misalignments are largely due to angular differences in the locations and sizes of objects in the captured images from the camera 120 and the locations and sizes of corresponding objects in the see-through view. Since both the camera 120 and the viewer's eye perceive images in angular segments within their respective fields of view, angular adjustments on the displayed image can be implemented in terms of pixel shifts and pixel count changes or image size changes of the displayed image. Thus, the image adjustments can take the form of x and y pixel shifts in the displayed image along with upsampling or downsampling of the displayed image to increase or decrease the number of x and y pixels in the displayed image.
[0049] While the example described above covers the case where misalignments between the displayed image and the see-through view come from lateral and longitudinal misalignment as well as size differences, more complicated misalignments are possible from distortions or rotations. Rotational misalignments can be determined in the process of determining the resizing needed when comparing the fourth and fifth captured images.
[0050] Determining image adjustments needed to align displayed images to the see-through view when there is a distortion in either the displayed image or the see- through view requires gathering more information. In this case, the viewer would need to select at least one more object in a different location from the first or second object and repeat the process described above.
[0051] The examples provided describe methods for determining image adjustments based on the view from one eye. These determined image adjustments can be applied to the displayed images in both eyes or the image adjustments can be determined independently for each eye.
[0052] After the image adjustments have been determined, displayed images can be modified to compensate for misalignments. The displayed images can be still images or video. Further images of the scene can be captured to enable objects to be identified and the locations of objects in the further images to be determined. Where methods for identifying objects and determining the locations of objects in images are described for example in U.S. Pat. 7805003. Augmented image information can be displayed relative to the determined locations of the objects such that the augmented image information is aligned with the objects in the see-through view by including the image adjustments in the displayed images. In another embodiment, to save power when displaying augmented image information, additional further images of the scene are captured only when movement of the viewer or the see-through head mounted display device 100 is detected, as the determined locations of objects in the further images are unchanged when the viewer or the see-through head mounted display device 100 is stationary. When the viewer or the see-through head mounted display device 100 is stationary, the same image adjustments can be used for multiple displays of augmented image information to align the augmented image information with the objects as seen by the viewer in the see-through view.
[0053] In another method, the viewer indicates the misalignment between a displayed image and the see-through view by moving their head. Illustrations of this method are shown in FIGS. 15-20. One or more locations are then chosen in the combined image seen by the viewer where an alignment can be performed. If more than one location is used for the alignment, the locations must be in different portions of the combined image, such as near opposite corners. To aid the viewer in selecting the locations used for performing the alignment, in one embodiment, a marker is provided in the displayed image as shown in FIG. 15 where the marker 1550 is a circle. [0054] The displayed image shown in FIG. 15 on the see-through head mounted display device 100 is a first image captured of the scene by the camera 120 and the displayed image is shown from behind and slightly above the viewer's perspective so that objects in the scene can be seen as well as the displayed image. FIG. 16 is an illustration of the combined view as seen by the viewer in the left eye wherein the displayed image of the scene is overlaid on the see-through view of the scene and a misalignment can be seen. A marker 1550 indicates a first area for alignment. FIG. 17 illustrates a combined view as seen by a viewer's left eye wherein the viewer has moved his or her head to align objects (the roof) in the displayed image and see- through view in the area of the marker 1550.
[0055] A second image is then captured by the camera 120. The first captured image is then compared to the second captured image by the electronics 125 including the processor to determine the difference between the two images in the location of the marker 1550. At this point, the displayed image and the see-through view would be aligned if the perceived sizes of the displayed image and the see-through view were the same and the determined the difference between the first and second captured images is an image adjustment of an x and y pixel shift on the displayed image. If there are still misalignments between the displayed image and the see- through view after an alignment at the location of the marker 1550 as shown in FIG. 17, then a second alignment is performed at a second marker 1850 as shown in FIG. 18.
[0056] As can be seen in FIG. 18, the two images are aligned at the location where marker 1550 had been located, but the remainder of the image has misalignments due to a mismatch in sizes between the displayed image and the see- through view. The viewer then moves his or her head to align objects in the displayed image to corresponding objects (such as the car tire) in the see-through view in the region of the marker 1850 to indicate the further image adjustment that is a resizing of the displayed image.
[0057] FIG. 19 shows an illustration of the combined image seen by the viewer after the viewer's head has been moved to align objects in the displayed to corresponding objects in the see-through view. A third image is then captured by the camera 120. The third image is then compared to the second image or the first image by the electronics 125 including the processor to determine the image adjustment needed to align the displayed image to the see-through view in the region of the second marker 1850. The image adjustment determined to align the displayed image to the see-through view at the region of the first marker 1550 is then a pixel shift. The percent change in the distance between the locations of objects in the area of the first and second markers when aligning the displayed image to the see-through view in the region of the second marker 1850 is the image adjustment for resizing the displayed image. FIG. 20 then shows the fully aligned displayed image, after applying the pixel shift and the resizing, overlaid on the see-through view as seen by the viewer, where misalignments are not visible.
[0058] The method of alignment can be further described in relation to the flow chart shown in FIG. 21. In Step 2110, the viewer looks at a scene and the camera 120 captures an image of the scene in Step 2120. The captured image is then displayed on the display areas 115 of the see-through head mounted display device 100 operating in a transparent or semi-transparent mode in Step 2130 so the viewer sees a combined view comprised of the displayed image overlaid on the see-through view. The viewer then provides an indication of the misalignment between objects in the displayed image and corresponding objects in the see-through view in Step 2140. The indication of the misalignments can be done by a series of finger gestures or by moving the viewer's head as described previously. The camera 120 is used to capture additional images of the scene along with the viewer's indication of the misalignments in Step 2150. Then in Step 2160, the captured additional images are compared in the electronics 125 to determine the image adjustments needed to align the displayed images with the see-through view as seen by the viewer.
[0059] In a further embodiment, the viewer indicates misalignments between captured images of the scene and the see-through view by a combination of hand gestures and head movement. One or more additional images are captured and compared to determine the image adjustments as previously described.
[0060] In another embodiment, the see-through head mounted display device 100 includes a GPS device or a magnetometer. The GPS device provides data on the current location or previous locations of the see-through head mounted display device 100. The magnetometer provides data on the current direction and previous directions of the viewer's line of sight. The data from the GPS or magnetometer or the combination of data from the GPS and magnetometer can be used to help identify objects in the scene or to determine the addresses or locations of objects in the images captured by the camera 120. By aligning the displayed image to the see-through view, augmented image information related to the identified objects can be provided in the combined view that is aligned to the respective objects as perceived by the viewer.
[0061] After alignment augmented image information can be aligned with identified objects in the captured images and identified edges of objects in the captured images. In addition, in see-through head mounted display devices 100 that include head tracking devices, such as gyros or accelerometers, head tracking information can be used to adjust augmented image information and the location of augmented image information relative to objects in the displayed images.
[0062] FIG. 22 shows a flow chart for using a see-through head mounted display device 100 with a GPS or magnetometer wherein the displayed image has been aligned with the see-through view as perceived by the viewer. In Step 2210, the GPS or magnetometer is used to determine the location of the viewer or the direction that the viewer is looking. The camera 120 then captures an image of the scene in Step 2220. The electronics 125 including the processor are then used to analyze the captured image along with the determined location or direction information to identify objects in the scene in Step 2230. The see-through head mounted display device 100 then uses the peripheral electronics 127 including a wireless connection to determine whether augmented information is available for the identified objects or the determined location or determined direction in Step 2240. In Step 2250, available augmented information is displayed in regions or locations of the displayed image that correspond to the objects locations when aligned to the see-through view.
[0063] For example, a house can be identified in the captured image by the combination of its shape in the captured image and from the GPS location and the direction, the address of the house can then be determined from a map that is available on the internet and the address can be presented in the displayed image such that it overlays the region of the see-through view that contains the house (see FIG. 14A). In a further example, an image can be captured of a building. GPS data and magnetometer data can be used to determine the approximate GPS location of the building. Augmented information including the name of the building and ongoing activities in the building can be determined from information available from a server in the building that broadcasts over Bluetooth by matching the GPS location and the direction the viewer is looking. A displayed image is then prepared with the name of the building and a list of ongoing activities located in the region of the displayed image that corresponds to the aligned location of the building in the see-through view. An augmented image is then presented to the viewer as a combined image with the displayed image overlaid on the see-through view.
[0064] The augmented images produced by the these methods can be used for a variety of applications. In an embodiment, the augmented image can be part of a user interface wherein the augmented image information is a virtual keyboard that is operated by the viewer with finger gestures. In this example, the virtual keyboard needs to be aligned with the see-through view of the viewer's fingers for the viewer to select the desired keys. In another embodiment, the locations of the objects can be determined with the aid of GPS data or magnetometer data and the augmented image information can be advertising or names of objects or addresses of objects. The objects can be buildings, exhibitions or tourist attractions where the viewer uses the augmented image to aid making a decision on where to go or what to do. This information should be aligned with the see-through view of the buildings, exhibitions or tourist attractions.
[0065] FIG. 14A is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein a displayed label 1470 (the address) is overlaid onto an object (the house) in the see-through view and the displayed label 1470 is aligned to the object. In a further embodiment, the augmented image includes directions or procedural information related to the objects in the scene and the directions or procedural information needs to be aligned to the objects so the viewer can perform an operation properly. In yet another embodiment, the augmented image can be a modified version of the scene in which objects have been added to form a virtual image of the scene. FIG 14B is an illustration of a combined view augmented reality image as seen by the viewer's right eye wherein augmented image information in the form of displayed objects 1475 (the tree and bushes) are overlaid onto objects (the car and house) in the see-through view and the displayed objects 1475 are aligned to the objects in the see-through view. [0066] Table of numerals for fig
Figure imgf000017_0001
[0067] This disclosure has been made in detail with particular reference to certain embodiments, but it will be understood that variations and modifications can be effected within the spirit and scope of the invention.

Claims

WHAT IS CLAIMED IS:
1. A method for providing an augmented image in a see-through head mounted display which includes a camera comprising:
capturing a first image of a scene with the camera wherein the scene contains objects;
displaying the first image to a viewer;
capturing one or more additional image(s) of the scene with the camera in which the viewer indicates a misalignment between the displayed first image and a see-through view of the scene;
comparing the captured images to determine an image adjustment to align corresponding objects in the first image to the objects in the see-through view of the scene;
providing augmented image information;
applying the determined image adjustments to the augmented image information; and
displaying the augmented image information so the viewer sees an augmented image comprised of the augmented image information overlaid on the see-through view.
2. The method of claim 1 wherein the image adjustment comprises a lateral shift, a longitudinal shift or a resizing.
3. The method of claim 1 wherein the viewer indicates the misalignment by a hand gesture that is captured in the one or more additional image(s) of the scene.
4. The method of claim 1 wherein the viewer indicates the misalignment by moving his or her head between captured images.
5. The method of claim 1 wherein the viewer indicates misalignments at two or more different locations in the see-through view of the scene.
6. The method of claim 1 further comprising:
capturing another image of a scene with the camera; analyzing the another image to identify the locations of objects in the scene; and
providing augmented image information using the determined image adjustments so the augmented image information is aligned to objects in the scene.
7. The method of claim 6, further comprising identifying the objects.
8. The method of claim 7 wherein the augmented image information is related to the objects in the scene.
9. The method of claim 1 further comprising:
capturing another image of the scene;
analyzing the another image and using the determined image adjustment to determine the locations of objects in the see-through view;
providing augmented image information;
applying the determined image adjustment to the augmented image information;
displaying the augmented image information so the viewer sees another augmented image comprised of the augmented image information overlaid on the see- through view; and
repeating these steps for additional other images to provide an augmented video.
10. The method of claim 9 wherein the see-through head mounted display further includes a GPS sensor or a magnetometer; and the locations of objects are further determined by using data provided by the GPS sensor or the magnetometer.
11. The method of claim 9 wherein the see-through head mounted display further includes a gyro or an accelerometer; and the locations of objects are further determined by using data provided by the gyro or accelerometer.
12. The method of claim 1 wherein the augmented image is part of user interface.
13. The method of claim 8 wherein the augmented image includes instructions.
14. The method of claim 8 wherein the augmented image includes names or addresses of objects.
15. The method of claim 1 wherein the viewer indicates the misalignment by a combination of hand gesture and head movement.
16. The method of claim 9 wherein the capturing of additional another images of the scene and analyzing of the additional another images is done when movement of the viewer is detected.
17. The method of claim 1 wherein instructions are displayed to the viewer.
PCT/US2012/022568 2011-02-28 2012-01-25 Alignment control in an augmented reality headpiece WO2012118575A2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CA2828413A CA2828413A1 (en) 2011-02-28 2012-01-25 Alignment control in an augmented reality headpiece
DE112012001022T DE112012001022T5 (en) 2011-02-28 2012-01-25 Alignment control in a head-worn augmented reality device

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US13/037,335 2011-02-28
US13/037,324 2011-02-28
US13/037,324 US20110214082A1 (en) 2010-02-28 2011-02-28 Projection triggering through an external marker in an augmented reality eyepiece
US13/037,335 US20110213664A1 (en) 2010-02-28 2011-02-28 Local advertising content on an interactive head-mounted eyepiece

Publications (2)

Publication Number Publication Date
WO2012118575A2 true WO2012118575A2 (en) 2012-09-07
WO2012118575A3 WO2012118575A3 (en) 2013-03-14

Family

ID=46758533

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/US2012/022492 WO2012118573A1 (en) 2011-02-28 2012-01-25 Light control in head mounted displays
PCT/US2012/022568 WO2012118575A2 (en) 2011-02-28 2012-01-25 Alignment control in an augmented reality headpiece

Family Applications Before (1)

Application Number Title Priority Date Filing Date
PCT/US2012/022492 WO2012118573A1 (en) 2011-02-28 2012-01-25 Light control in head mounted displays

Country Status (3)

Country Link
CA (2) CA2828407A1 (en)
DE (2) DE112012001032T5 (en)
WO (2) WO2012118573A1 (en)

Cited By (88)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8467133B2 (en) 2010-02-28 2013-06-18 Osterhout Group, Inc. See-through display with an optical assembly including a wedge-shaped illumination system
US8472120B2 (en) 2010-02-28 2013-06-25 Osterhout Group, Inc. See-through near-eye display glasses with a small scale image source
US8477425B2 (en) 2010-02-28 2013-07-02 Osterhout Group, Inc. See-through near-eye display glasses including a partially reflective, partially transmitting optical element
US8482859B2 (en) 2010-02-28 2013-07-09 Osterhout Group, Inc. See-through near-eye display glasses wherein image light is transmitted to and reflected from an optically flat film
US8488246B2 (en) 2010-02-28 2013-07-16 Osterhout Group, Inc. See-through near-eye display glasses including a curved polarizing film in the image source, a partially reflective, partially transmitting optical element and an optically flat film
WO2014109430A1 (en) * 2013-01-09 2014-07-17 Lg Electronics Inc. Head mount display device providing eye gaze calibration and control method thereof
US8814691B2 (en) 2010-02-28 2014-08-26 Microsoft Corporation System and method for social networking gaming with an augmented reality
US9091851B2 (en) 2010-02-28 2015-07-28 Microsoft Technology Licensing, Llc Light control in head mounted displays
US9097891B2 (en) 2010-02-28 2015-08-04 Microsoft Technology Licensing, Llc See-through near-eye display glasses including an auto-brightness control for the display brightness based on the brightness in the environment
US9097890B2 (en) 2010-02-28 2015-08-04 Microsoft Technology Licensing, Llc Grating in a light transmissive illumination system for see-through near-eye display glasses
US9122054B2 (en) 2014-01-24 2015-09-01 Osterhout Group, Inc. Stray light suppression for head worn computing
US9128281B2 (en) 2010-09-14 2015-09-08 Microsoft Technology Licensing, Llc Eyepiece with uniformly illuminated reflective display
US9129295B2 (en) 2010-02-28 2015-09-08 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a fast response photochromic film system for quick transition from dark to clear
US9134534B2 (en) 2010-02-28 2015-09-15 Microsoft Technology Licensing, Llc See-through near-eye display glasses including a modular image source
US9158116B1 (en) 2014-04-25 2015-10-13 Osterhout Group, Inc. Temple and ear horn assembly for headworn computer
US9182596B2 (en) 2010-02-28 2015-11-10 Microsoft Technology Licensing, Llc See-through near-eye display glasses with the optical assembly including absorptive polarizers or anti-reflective coatings to reduce stray light
USD743963S1 (en) 2014-12-22 2015-11-24 Osterhout Group, Inc. Air mouse
US9223134B2 (en) 2010-02-28 2015-12-29 Microsoft Technology Licensing, Llc Optical imperfections in a light transmissive illumination system for see-through near-eye display glasses
US9229227B2 (en) 2010-02-28 2016-01-05 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a light transmissive wedge shaped illumination system
US9285589B2 (en) 2010-02-28 2016-03-15 Microsoft Technology Licensing, Llc AR glasses with event and sensor triggered control of AR eyepiece applications
USD751552S1 (en) 2014-12-31 2016-03-15 Osterhout Group, Inc. Computer glasses
US9298007B2 (en) 2014-01-21 2016-03-29 Osterhout Group, Inc. Eye imaging in head worn computing
US9298002B2 (en) 2014-01-21 2016-03-29 Osterhout Group, Inc. Optical configurations for head worn computing
USD753114S1 (en) 2015-01-05 2016-04-05 Osterhout Group, Inc. Air mouse
US9310610B2 (en) 2014-01-21 2016-04-12 Osterhout Group, Inc. See-through computer display systems
US9316833B2 (en) 2014-01-21 2016-04-19 Osterhout Group, Inc. Optical configurations for head worn computing
US9329387B2 (en) 2014-01-21 2016-05-03 Osterhout Group, Inc. See-through computer display systems
US9341843B2 (en) 2010-02-28 2016-05-17 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a small scale image source
US9366862B2 (en) 2010-02-28 2016-06-14 Microsoft Technology Licensing, Llc System and method for delivering content to a group of see-through near eye display eyepieces
US9366868B2 (en) 2014-09-26 2016-06-14 Osterhout Group, Inc. See-through computer display systems
US9366867B2 (en) 2014-07-08 2016-06-14 Osterhout Group, Inc. Optical systems for see-through displays
US9401540B2 (en) 2014-02-11 2016-07-26 Osterhout Group, Inc. Spatial location presentation in head worn computing
US9423612B2 (en) 2014-03-28 2016-08-23 Osterhout Group, Inc. Sensor dependent content position in head worn computing
US9423842B2 (en) 2014-09-18 2016-08-23 Osterhout Group, Inc. Thermal management for head-worn computer
US9448409B2 (en) 2014-11-26 2016-09-20 Osterhout Group, Inc. See-through computer display systems
US9494800B2 (en) 2014-01-21 2016-11-15 Osterhout Group, Inc. See-through computer display systems
US9523856B2 (en) 2014-01-21 2016-12-20 Osterhout Group, Inc. See-through computer display systems
US9529195B2 (en) 2014-01-21 2016-12-27 Osterhout Group, Inc. See-through computer display systems
US9529442B2 (en) 2013-01-09 2016-12-27 Lg Electronics Inc. Head mounted display providing eye gaze calibration and control method thereof
US9532715B2 (en) 2014-01-21 2017-01-03 Osterhout Group, Inc. Eye imaging in head worn computing
US9547465B2 (en) 2014-02-14 2017-01-17 Osterhout Group, Inc. Object shadowing in head worn computing
US9575321B2 (en) 2014-06-09 2017-02-21 Osterhout Group, Inc. Content presentation in head worn computing
US9619021B2 (en) 2013-01-09 2017-04-11 Lg Electronics Inc. Head mounted display providing eye gaze calibration and control method thereof
US9651787B2 (en) 2014-04-25 2017-05-16 Osterhout Group, Inc. Speaker assembly for headworn computer
US9651784B2 (en) 2014-01-21 2017-05-16 Osterhout Group, Inc. See-through computer display systems
US9672210B2 (en) 2014-04-25 2017-06-06 Osterhout Group, Inc. Language translation with head-worn computing
US9671613B2 (en) 2014-09-26 2017-06-06 Osterhout Group, Inc. See-through computer display systems
US9684172B2 (en) 2014-12-03 2017-06-20 Osterhout Group, Inc. Head worn computer display systems
US9715112B2 (en) 2014-01-21 2017-07-25 Osterhout Group, Inc. Suppression of stray light in head worn computing
US9720234B2 (en) 2014-01-21 2017-08-01 Osterhout Group, Inc. See-through computer display systems
US9740280B2 (en) 2014-01-21 2017-08-22 Osterhout Group, Inc. Eye imaging in head worn computing
US9746686B2 (en) 2014-05-19 2017-08-29 Osterhout Group, Inc. Content position calibration in head worn computing
US9753288B2 (en) 2014-01-21 2017-09-05 Osterhout Group, Inc. See-through computer display systems
US9759917B2 (en) 2010-02-28 2017-09-12 Microsoft Technology Licensing, Llc AR glasses with event and sensor triggered AR eyepiece interface to external devices
US9766463B2 (en) 2014-01-21 2017-09-19 Osterhout Group, Inc. See-through computer display systems
US9784973B2 (en) 2014-02-11 2017-10-10 Osterhout Group, Inc. Micro doppler presentations in head worn computing
US9811153B2 (en) 2014-01-21 2017-11-07 Osterhout Group, Inc. Eye imaging in head worn computing
US9810906B2 (en) 2014-06-17 2017-11-07 Osterhout Group, Inc. External user interface for head worn computing
US9829707B2 (en) 2014-08-12 2017-11-28 Osterhout Group, Inc. Measuring content brightness in head worn computing
US9836122B2 (en) 2014-01-21 2017-12-05 Osterhout Group, Inc. Eye glint imaging in see-through computer display systems
US9841599B2 (en) 2014-06-05 2017-12-12 Osterhout Group, Inc. Optical configurations for head-worn see-through displays
US9910284B1 (en) 2016-09-08 2018-03-06 Osterhout Group, Inc. Optical systems for head-worn computers
US9939934B2 (en) 2014-01-17 2018-04-10 Osterhout Group, Inc. External user interface for head worn computing
US9952664B2 (en) 2014-01-21 2018-04-24 Osterhout Group, Inc. Eye imaging in head worn computing
US9965681B2 (en) 2008-12-16 2018-05-08 Osterhout Group, Inc. Eye imaging in head worn computing
US10062182B2 (en) 2015-02-17 2018-08-28 Osterhout Group, Inc. See-through computer display systems
US10180572B2 (en) 2010-02-28 2019-01-15 Microsoft Technology Licensing, Llc AR glasses with event and user action control of external applications
US10191279B2 (en) 2014-03-17 2019-01-29 Osterhout Group, Inc. Eye imaging in head worn computing
US10254856B2 (en) 2014-01-17 2019-04-09 Osterhout Group, Inc. External user interface for head worn computing
US10422995B2 (en) 2017-07-24 2019-09-24 Mentor Acquisition One, Llc See-through computer display systems with stray light management
US10539787B2 (en) 2010-02-28 2020-01-21 Microsoft Technology Licensing, Llc Head-worn adaptive display
US10558050B2 (en) 2014-01-24 2020-02-11 Mentor Acquisition One, Llc Haptic systems for head-worn computers
US10578869B2 (en) 2017-07-24 2020-03-03 Mentor Acquisition One, Llc See-through computer display systems with adjustable zoom cameras
US10649220B2 (en) 2014-06-09 2020-05-12 Mentor Acquisition One, Llc Content presentation in head worn computing
US10663740B2 (en) 2014-06-09 2020-05-26 Mentor Acquisition One, Llc Content presentation in head worn computing
US10684687B2 (en) 2014-12-03 2020-06-16 Mentor Acquisition One, Llc See-through computer display systems
US10853589B2 (en) 2014-04-25 2020-12-01 Mentor Acquisition One, Llc Language translation with head-worn computing
US10860100B2 (en) 2010-02-28 2020-12-08 Microsoft Technology Licensing, Llc AR glasses with predictive control of external device based on event input
US10969584B2 (en) 2017-08-04 2021-04-06 Mentor Acquisition One, Llc Image expansion optic for head-worn computer
US11103122B2 (en) 2014-07-15 2021-08-31 Mentor Acquisition One, Llc Content presentation in head worn computing
US11104272B2 (en) 2014-03-28 2021-08-31 Mentor Acquisition One, Llc System for assisted operator safety using an HMD
US11227294B2 (en) 2014-04-03 2022-01-18 Mentor Acquisition One, Llc Sight information collection in head worn computing
US11269182B2 (en) 2014-07-15 2022-03-08 Mentor Acquisition One, Llc Content presentation in head worn computing
US11409105B2 (en) 2017-07-24 2022-08-09 Mentor Acquisition One, Llc See-through computer display systems
US11487110B2 (en) 2014-01-21 2022-11-01 Mentor Acquisition One, Llc Eye imaging in head worn computing
US11669163B2 (en) 2014-01-21 2023-06-06 Mentor Acquisition One, Llc Eye glint imaging in see-through computer display systems
US11737666B2 (en) 2014-01-21 2023-08-29 Mentor Acquisition One, Llc Eye imaging in head worn computing
US11892644B2 (en) 2014-01-21 2024-02-06 Mentor Acquisition One, Llc See-through computer display systems

Families Citing this family (36)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8511827B2 (en) 2008-01-22 2013-08-20 The Arizona Board Of Regents On Behalf Of The University Of Arizona Head-mounted projection display using reflective microdisplays
US20120081800A1 (en) 2009-04-20 2012-04-05 Dewen Cheng Optical see-through free-form head-mounted display
US20110075257A1 (en) 2009-09-14 2011-03-31 The Arizona Board Of Regents On Behalf Of The University Of Arizona 3-Dimensional electro-optical see-through displays
US9244277B2 (en) 2010-04-30 2016-01-26 The Arizona Board Of Regents On Behalf Of The University Of Arizona Wide angle and high resolution tiled head-mounted display device
EP3270194B1 (en) 2012-01-24 2020-10-14 The Arizona Board of Regents on behalf of The University of Arizona Compact eye-tracked head-mounted display
EP2910022B1 (en) 2012-10-18 2023-07-12 The Arizona Board Of Regents On Behalf Of The University Of Arizona Stereoscopic displays with addressable focus cues
US9448404B2 (en) 2012-11-13 2016-09-20 Qualcomm Incorporated Modifying virtual object display properties to increase power performance of augmented reality devices
NZ724518A (en) 2014-03-05 2021-12-24 Univ Arizona Wearable 3d augmented reality display
WO2015142654A1 (en) * 2014-03-18 2015-09-24 3M Innovative Properties Company Low profile image combiner for near-eye displays
US9965030B2 (en) * 2014-07-31 2018-05-08 Samsung Electronics Co., Ltd. Wearable glasses and method of displaying image via the wearable glasses
US10176961B2 (en) 2015-02-09 2019-01-08 The Arizona Board Of Regents On Behalf Of The University Of Arizona Small portable night vision system
WO2016133886A1 (en) * 2015-02-17 2016-08-25 Osterhout Group, Inc. See-through computer display systems
NZ773844A (en) 2015-03-16 2022-07-01 Magic Leap Inc Methods and systems for diagnosing and treating health ailments
JP6892213B2 (en) * 2015-04-30 2021-06-23 ソニーグループ株式会社 Display device and initial setting method of display device
CA3033651C (en) 2016-08-12 2023-09-05 Arizona Board Of Regents On Behalf Of The University Of Arizona High-resolution freeform eyepiece design with a large exit pupil
CN114690429A (en) * 2017-02-23 2022-07-01 奇跃公司 Variable-focus virtual image device based on polarization conversion
US10495895B2 (en) 2017-06-14 2019-12-03 Varjo Technologies Oy Display apparatus and method of displaying using polarizers
CN107422481A (en) * 2017-08-07 2017-12-01 杭州太若科技有限公司 Apparatus and method for realizing augmented reality
DE102017217923A1 (en) 2017-10-09 2019-04-11 Audi Ag Method for operating a display device in a motor vehicle
CN107589546A (en) * 2017-10-23 2018-01-16 北京小米移动软件有限公司 Optical system and augmented reality glasses
CN109991744B (en) * 2018-01-02 2020-12-01 京东方科技集团股份有限公司 Display device, display method, and head-up display device
CN110573933A (en) * 2018-02-12 2019-12-13 优奈柯恩(北京)科技有限公司 Wearable AR system, AR display equipment and projection source module thereof
CN116149062A (en) * 2018-02-12 2023-05-23 优奈柯恩(北京)科技有限公司 AR display device and wearable AR equipment
CN110146978A (en) * 2018-02-12 2019-08-20 杭州太若科技有限公司 AR display device and wearable AR equipment
EP3769512A4 (en) 2018-03-22 2021-03-24 Arizona Board of Regents on Behalf of the University of Arizona Methods of rendering light field images for integral-imaging-based light field display
CN109387942A (en) * 2018-03-28 2019-02-26 深圳惠牛科技有限公司 A kind of optical system and augmented reality equipment
US11204501B2 (en) 2018-04-24 2021-12-21 Mentor Acquisition One, Llc See-through computer display systems with vision correction and increased content density
CN108594441A (en) * 2018-07-04 2018-09-28 王锐 A kind of optical system
CN108572457A (en) * 2018-07-12 2018-09-25 王锐 A kind of optical presentation system
CN108681073A (en) * 2018-07-17 2018-10-19 王锐 A kind of augmented reality optical presentation system
WO2020114582A1 (en) * 2018-12-04 2020-06-11 Telefonaktiebolaget Lm Ericsson (Publ) Improved optical see-through viewing device and method for providing virtual content overlapping visual objects
CN112051671B (en) * 2019-06-06 2023-10-13 舜宇光学(浙江)研究院有限公司 Near-eye display optical machine, near-eye display method and near-eye display equipment
CN111290125A (en) * 2020-03-31 2020-06-16 优奈柯恩(北京)科技有限公司 Optical device and head-mounted apparatus
CN111474715A (en) * 2020-04-23 2020-07-31 歌尔股份有限公司 Optical system and augmented reality device
CN116964508A (en) * 2020-12-09 2023-10-27 伊诺托尼克斯股份有限公司 Improved optical performance of head mounted displays inside laser safety eyewear
WO2023131980A1 (en) * 2022-01-07 2023-07-13 Tesseract Imaging Limited Apparatus for viewing optical images and method thereof

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5625765A (en) * 1993-09-03 1997-04-29 Criticom Corp. Vision systems including devices and methods for combining images for extended magnification schemes
US20060152434A1 (en) * 2003-06-12 2006-07-13 Frank Sauer Calibrating real and virtual views
US20080246694A1 (en) * 2007-04-06 2008-10-09 Ronald Fischer Personal theater display

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5151722A (en) * 1990-11-05 1992-09-29 The Johns Hopkins University Video display on spectacle-like frame
JPH08136852A (en) * 1994-11-02 1996-05-31 Olympus Optical Co Ltd Optical device having eyepiece optical system
JPH09219832A (en) * 1996-02-13 1997-08-19 Olympus Optical Co Ltd Image display
US6011653A (en) * 1996-03-11 2000-01-04 Seiko Epson Corporation Head-mounted display
US6753828B2 (en) 2000-09-25 2004-06-22 Siemens Corporated Research, Inc. System and method for calibrating a stereo optical see-through head-mounted display system for augmented reality
US7805003B1 (en) 2003-11-18 2010-09-28 Adobe Systems Incorporated Identifying one or more objects within an image
US7755667B2 (en) 2005-05-17 2010-07-13 Eastman Kodak Company Image sequence stabilization method and camera having dual path image sequence stabilization

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5625765A (en) * 1993-09-03 1997-04-29 Criticom Corp. Vision systems including devices and methods for combining images for extended magnification schemes
US20060152434A1 (en) * 2003-06-12 2006-07-13 Frank Sauer Calibrating real and virtual views
US20080246694A1 (en) * 2007-04-06 2008-10-09 Ronald Fischer Personal theater display

Cited By (196)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9965681B2 (en) 2008-12-16 2018-05-08 Osterhout Group, Inc. Eye imaging in head worn computing
US9229227B2 (en) 2010-02-28 2016-01-05 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a light transmissive wedge shaped illumination system
US8472120B2 (en) 2010-02-28 2013-06-25 Osterhout Group, Inc. See-through near-eye display glasses with a small scale image source
US8482859B2 (en) 2010-02-28 2013-07-09 Osterhout Group, Inc. See-through near-eye display glasses wherein image light is transmitted to and reflected from an optically flat film
US9223134B2 (en) 2010-02-28 2015-12-29 Microsoft Technology Licensing, Llc Optical imperfections in a light transmissive illumination system for see-through near-eye display glasses
US9759917B2 (en) 2010-02-28 2017-09-12 Microsoft Technology Licensing, Llc AR glasses with event and sensor triggered AR eyepiece interface to external devices
US9285589B2 (en) 2010-02-28 2016-03-15 Microsoft Technology Licensing, Llc AR glasses with event and sensor triggered control of AR eyepiece applications
US9091851B2 (en) 2010-02-28 2015-07-28 Microsoft Technology Licensing, Llc Light control in head mounted displays
US9097891B2 (en) 2010-02-28 2015-08-04 Microsoft Technology Licensing, Llc See-through near-eye display glasses including an auto-brightness control for the display brightness based on the brightness in the environment
US9097890B2 (en) 2010-02-28 2015-08-04 Microsoft Technology Licensing, Llc Grating in a light transmissive illumination system for see-through near-eye display glasses
US9875406B2 (en) 2010-02-28 2018-01-23 Microsoft Technology Licensing, Llc Adjustable extension for temple arm
US8467133B2 (en) 2010-02-28 2013-06-18 Osterhout Group, Inc. See-through display with an optical assembly including a wedge-shaped illumination system
US9129295B2 (en) 2010-02-28 2015-09-08 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a fast response photochromic film system for quick transition from dark to clear
US9134534B2 (en) 2010-02-28 2015-09-15 Microsoft Technology Licensing, Llc See-through near-eye display glasses including a modular image source
US9366862B2 (en) 2010-02-28 2016-06-14 Microsoft Technology Licensing, Llc System and method for delivering content to a group of see-through near eye display eyepieces
US9182596B2 (en) 2010-02-28 2015-11-10 Microsoft Technology Licensing, Llc See-through near-eye display glasses with the optical assembly including absorptive polarizers or anti-reflective coatings to reduce stray light
US9341843B2 (en) 2010-02-28 2016-05-17 Microsoft Technology Licensing, Llc See-through near-eye display glasses with a small scale image source
US9329689B2 (en) 2010-02-28 2016-05-03 Microsoft Technology Licensing, Llc Method and apparatus for biometric data capture
US8488246B2 (en) 2010-02-28 2013-07-16 Osterhout Group, Inc. See-through near-eye display glasses including a curved polarizing film in the image source, a partially reflective, partially transmitting optical element and an optically flat film
US8814691B2 (en) 2010-02-28 2014-08-26 Microsoft Corporation System and method for social networking gaming with an augmented reality
US8477425B2 (en) 2010-02-28 2013-07-02 Osterhout Group, Inc. See-through near-eye display glasses including a partially reflective, partially transmitting optical element
US10860100B2 (en) 2010-02-28 2020-12-08 Microsoft Technology Licensing, Llc AR glasses with predictive control of external device based on event input
US10539787B2 (en) 2010-02-28 2020-01-21 Microsoft Technology Licensing, Llc Head-worn adaptive display
US10268888B2 (en) 2010-02-28 2019-04-23 Microsoft Technology Licensing, Llc Method and apparatus for biometric data capture
US10180572B2 (en) 2010-02-28 2019-01-15 Microsoft Technology Licensing, Llc AR glasses with event and user action control of external applications
US9128281B2 (en) 2010-09-14 2015-09-08 Microsoft Technology Licensing, Llc Eyepiece with uniformly illuminated reflective display
US9619021B2 (en) 2013-01-09 2017-04-11 Lg Electronics Inc. Head mounted display providing eye gaze calibration and control method thereof
WO2014109430A1 (en) * 2013-01-09 2014-07-17 Lg Electronics Inc. Head mount display device providing eye gaze calibration and control method thereof
EP2943836A4 (en) * 2013-01-09 2016-10-26 Lg Electronics Inc Head mount display device providing eye gaze calibration and control method thereof
US9529442B2 (en) 2013-01-09 2016-12-27 Lg Electronics Inc. Head mounted display providing eye gaze calibration and control method thereof
US10254856B2 (en) 2014-01-17 2019-04-09 Osterhout Group, Inc. External user interface for head worn computing
US11169623B2 (en) 2014-01-17 2021-11-09 Mentor Acquisition One, Llc External user interface for head worn computing
US9939934B2 (en) 2014-01-17 2018-04-10 Osterhout Group, Inc. External user interface for head worn computing
US11231817B2 (en) 2014-01-17 2022-01-25 Mentor Acquisition One, Llc External user interface for head worn computing
US11507208B2 (en) 2014-01-17 2022-11-22 Mentor Acquisition One, Llc External user interface for head worn computing
US11782529B2 (en) 2014-01-17 2023-10-10 Mentor Acquisition One, Llc External user interface for head worn computing
US9933622B2 (en) 2014-01-21 2018-04-03 Osterhout Group, Inc. See-through computer display systems
US10191284B2 (en) 2014-01-21 2019-01-29 Osterhout Group, Inc. See-through computer display systems
US9436006B2 (en) 2014-01-21 2016-09-06 Osterhout Group, Inc. See-through computer display systems
US11892644B2 (en) 2014-01-21 2024-02-06 Mentor Acquisition One, Llc See-through computer display systems
US11796799B2 (en) 2014-01-21 2023-10-24 Mentor Acquisition One, Llc See-through computer display systems
US9494800B2 (en) 2014-01-21 2016-11-15 Osterhout Group, Inc. See-through computer display systems
US9523856B2 (en) 2014-01-21 2016-12-20 Osterhout Group, Inc. See-through computer display systems
US9529192B2 (en) 2014-01-21 2016-12-27 Osterhout Group, Inc. Eye imaging in head worn computing
US9529199B2 (en) 2014-01-21 2016-12-27 Osterhout Group, Inc. See-through computer display systems
US9529195B2 (en) 2014-01-21 2016-12-27 Osterhout Group, Inc. See-through computer display systems
US11796805B2 (en) 2014-01-21 2023-10-24 Mentor Acquisition One, Llc Eye imaging in head worn computing
US9532715B2 (en) 2014-01-21 2017-01-03 Osterhout Group, Inc. Eye imaging in head worn computing
US9532714B2 (en) 2014-01-21 2017-01-03 Osterhout Group, Inc. Eye imaging in head worn computing
US9538915B2 (en) 2014-01-21 2017-01-10 Osterhout Group, Inc. Eye imaging in head worn computing
US11737666B2 (en) 2014-01-21 2023-08-29 Mentor Acquisition One, Llc Eye imaging in head worn computing
US11669163B2 (en) 2014-01-21 2023-06-06 Mentor Acquisition One, Llc Eye glint imaging in see-through computer display systems
US9594246B2 (en) 2014-01-21 2017-03-14 Osterhout Group, Inc. See-through computer display systems
US11650416B2 (en) 2014-01-21 2023-05-16 Mentor Acquisition One, Llc See-through computer display systems
US9615742B2 (en) 2014-01-21 2017-04-11 Osterhout Group, Inc. Eye imaging in head worn computing
US11619820B2 (en) 2014-01-21 2023-04-04 Mentor Acquisition One, Llc See-through computer display systems
US9651783B2 (en) 2014-01-21 2017-05-16 Osterhout Group, Inc. See-through computer display systems
US9651789B2 (en) 2014-01-21 2017-05-16 Osterhout Group, Inc. See-Through computer display systems
US9651784B2 (en) 2014-01-21 2017-05-16 Osterhout Group, Inc. See-through computer display systems
US9651788B2 (en) 2014-01-21 2017-05-16 Osterhout Group, Inc. See-through computer display systems
US9658458B2 (en) 2014-01-21 2017-05-23 Osterhout Group, Inc. See-through computer display systems
US9658457B2 (en) 2014-01-21 2017-05-23 Osterhout Group, Inc. See-through computer display systems
US11622426B2 (en) 2014-01-21 2023-04-04 Mentor Acquisition One, Llc See-through computer display systems
US11487110B2 (en) 2014-01-21 2022-11-01 Mentor Acquisition One, Llc Eye imaging in head worn computing
US9684171B2 (en) 2014-01-21 2017-06-20 Osterhout Group, Inc. See-through computer display systems
US9684165B2 (en) 2014-01-21 2017-06-20 Osterhout Group, Inc. Eye imaging in head worn computing
US11353957B2 (en) 2014-01-21 2022-06-07 Mentor Acquisition One, Llc Eye glint imaging in see-through computer display systems
US11126003B2 (en) 2014-01-21 2021-09-21 Mentor Acquisition One, Llc See-through computer display systems
US9715112B2 (en) 2014-01-21 2017-07-25 Osterhout Group, Inc. Suppression of stray light in head worn computing
US9720234B2 (en) 2014-01-21 2017-08-01 Osterhout Group, Inc. See-through computer display systems
US11103132B2 (en) 2014-01-21 2021-08-31 Mentor Acquisition One, Llc Eye imaging in head worn computing
US9720235B2 (en) 2014-01-21 2017-08-01 Osterhout Group, Inc. See-through computer display systems
US9720227B2 (en) 2014-01-21 2017-08-01 Osterhout Group, Inc. See-through computer display systems
US11099380B2 (en) 2014-01-21 2021-08-24 Mentor Acquisition One, Llc Eye imaging in head worn computing
US9740280B2 (en) 2014-01-21 2017-08-22 Osterhout Group, Inc. Eye imaging in head worn computing
US9740012B2 (en) 2014-01-21 2017-08-22 Osterhout Group, Inc. See-through computer display systems
US9746676B2 (en) 2014-01-21 2017-08-29 Osterhout Group, Inc. See-through computer display systems
US11054902B2 (en) 2014-01-21 2021-07-06 Mentor Acquisition One, Llc Eye glint imaging in see-through computer display systems
US9753288B2 (en) 2014-01-21 2017-09-05 Osterhout Group, Inc. See-through computer display systems
US9377625B2 (en) 2014-01-21 2016-06-28 Osterhout Group, Inc. Optical configurations for head worn computing
US9766463B2 (en) 2014-01-21 2017-09-19 Osterhout Group, Inc. See-through computer display systems
US9772492B2 (en) 2014-01-21 2017-09-26 Osterhout Group, Inc. Eye imaging in head worn computing
US11002961B2 (en) 2014-01-21 2021-05-11 Mentor Acquisition One, Llc See-through computer display systems
US10890760B2 (en) 2014-01-21 2021-01-12 Mentor Acquisition One, Llc See-through computer display systems
US9811153B2 (en) 2014-01-21 2017-11-07 Osterhout Group, Inc. Eye imaging in head worn computing
US9811159B2 (en) 2014-01-21 2017-11-07 Osterhout Group, Inc. Eye imaging in head worn computing
US10866420B2 (en) 2014-01-21 2020-12-15 Mentor Acquisition One, Llc See-through computer display systems
US9811152B2 (en) 2014-01-21 2017-11-07 Osterhout Group, Inc. Eye imaging in head worn computing
US9298007B2 (en) 2014-01-21 2016-03-29 Osterhout Group, Inc. Eye imaging in head worn computing
US9829703B2 (en) 2014-01-21 2017-11-28 Osterhout Group, Inc. Eye imaging in head worn computing
US9836122B2 (en) 2014-01-21 2017-12-05 Osterhout Group, Inc. Eye glint imaging in see-through computer display systems
US10698223B2 (en) 2014-01-21 2020-06-30 Mentor Acquisition One, Llc See-through computer display systems
US10579140B2 (en) 2014-01-21 2020-03-03 Mentor Acquisition One, Llc Eye glint imaging in see-through computer display systems
US9298002B2 (en) 2014-01-21 2016-03-29 Osterhout Group, Inc. Optical configurations for head worn computing
US10481393B2 (en) 2014-01-21 2019-11-19 Mentor Acquisition One, Llc See-through computer display systems
US9885868B2 (en) 2014-01-21 2018-02-06 Osterhout Group, Inc. Eye imaging in head worn computing
US9298001B2 (en) 2014-01-21 2016-03-29 Osterhout Group, Inc. Optical configurations for head worn computing
US10222618B2 (en) 2014-01-21 2019-03-05 Osterhout Group, Inc. Compact optics with reduced chromatic aberrations
US9927612B2 (en) 2014-01-21 2018-03-27 Osterhout Group, Inc. See-through computer display systems
US9310610B2 (en) 2014-01-21 2016-04-12 Osterhout Group, Inc. See-through computer display systems
US9329387B2 (en) 2014-01-21 2016-05-03 Osterhout Group, Inc. See-through computer display systems
US10139632B2 (en) 2014-01-21 2018-11-27 Osterhout Group, Inc. See-through computer display systems
US9952664B2 (en) 2014-01-21 2018-04-24 Osterhout Group, Inc. Eye imaging in head worn computing
US9958674B2 (en) 2014-01-21 2018-05-01 Osterhout Group, Inc. Eye imaging in head worn computing
US9316833B2 (en) 2014-01-21 2016-04-19 Osterhout Group, Inc. Optical configurations for head worn computing
US9971156B2 (en) 2014-01-21 2018-05-15 Osterhout Group, Inc. See-through computer display systems
US10001644B2 (en) 2014-01-21 2018-06-19 Osterhout Group, Inc. See-through computer display systems
US10007118B2 (en) 2014-01-21 2018-06-26 Osterhout Group, Inc. Compact optical system with improved illumination
US10012838B2 (en) 2014-01-21 2018-07-03 Osterhout Group, Inc. Compact optical system with improved contrast uniformity
US10012840B2 (en) 2014-01-21 2018-07-03 Osterhout Group, Inc. See-through computer display systems
US11822090B2 (en) 2014-01-24 2023-11-21 Mentor Acquisition One, Llc Haptic systems for head-worn computers
US9122054B2 (en) 2014-01-24 2015-09-01 Osterhout Group, Inc. Stray light suppression for head worn computing
US10558050B2 (en) 2014-01-24 2020-02-11 Mentor Acquisition One, Llc Haptic systems for head-worn computers
US9939646B2 (en) 2014-01-24 2018-04-10 Osterhout Group, Inc. Stray light suppression for head worn computing
US9400390B2 (en) 2014-01-24 2016-07-26 Osterhout Group, Inc. Peripheral lighting for head worn computing
US9784973B2 (en) 2014-02-11 2017-10-10 Osterhout Group, Inc. Micro doppler presentations in head worn computing
US9841602B2 (en) 2014-02-11 2017-12-12 Osterhout Group, Inc. Location indicating avatar in head worn computing
US9401540B2 (en) 2014-02-11 2016-07-26 Osterhout Group, Inc. Spatial location presentation in head worn computing
US9843093B2 (en) 2014-02-11 2017-12-12 Osterhout Group, Inc. Spatial location presentation in head worn computing
US9928019B2 (en) 2014-02-14 2018-03-27 Osterhout Group, Inc. Object shadowing in head worn computing
US9547465B2 (en) 2014-02-14 2017-01-17 Osterhout Group, Inc. Object shadowing in head worn computing
US10191279B2 (en) 2014-03-17 2019-01-29 Osterhout Group, Inc. Eye imaging in head worn computing
US11104272B2 (en) 2014-03-28 2021-08-31 Mentor Acquisition One, Llc System for assisted operator safety using an HMD
US9423612B2 (en) 2014-03-28 2016-08-23 Osterhout Group, Inc. Sensor dependent content position in head worn computing
US11227294B2 (en) 2014-04-03 2022-01-18 Mentor Acquisition One, Llc Sight information collection in head worn computing
US9651787B2 (en) 2014-04-25 2017-05-16 Osterhout Group, Inc. Speaker assembly for headworn computer
US11474360B2 (en) 2014-04-25 2022-10-18 Mentor Acquisition One, Llc Speaker assembly for headworn computer
US10634922B2 (en) 2014-04-25 2020-04-28 Mentor Acquisition One, Llc Speaker assembly for headworn computer
US9672210B2 (en) 2014-04-25 2017-06-06 Osterhout Group, Inc. Language translation with head-worn computing
US9158116B1 (en) 2014-04-25 2015-10-13 Osterhout Group, Inc. Temple and ear horn assembly for headworn computer
US11727223B2 (en) 2014-04-25 2023-08-15 Mentor Acquisition One, Llc Language translation with head-worn computing
US11880041B2 (en) 2014-04-25 2024-01-23 Mentor Acquisition One, Llc Speaker assembly for headworn computer
US10853589B2 (en) 2014-04-25 2020-12-01 Mentor Acquisition One, Llc Language translation with head-worn computing
US9746686B2 (en) 2014-05-19 2017-08-29 Osterhout Group, Inc. Content position calibration in head worn computing
US10877270B2 (en) 2014-06-05 2020-12-29 Mentor Acquisition One, Llc Optical configurations for head-worn see-through displays
US11402639B2 (en) 2014-06-05 2022-08-02 Mentor Acquisition One, Llc Optical configurations for head-worn see-through displays
US9841599B2 (en) 2014-06-05 2017-12-12 Osterhout Group, Inc. Optical configurations for head-worn see-through displays
US11327323B2 (en) 2014-06-09 2022-05-10 Mentor Acquisition One, Llc Content presentation in head worn computing
US9720241B2 (en) 2014-06-09 2017-08-01 Osterhout Group, Inc. Content presentation in head worn computing
US11360318B2 (en) 2014-06-09 2022-06-14 Mentor Acquisition One, Llc Content presentation in head worn computing
US11663794B2 (en) 2014-06-09 2023-05-30 Mentor Acquisition One, Llc Content presentation in head worn computing
US9575321B2 (en) 2014-06-09 2017-02-21 Osterhout Group, Inc. Content presentation in head worn computing
US10976559B2 (en) 2014-06-09 2021-04-13 Mentor Acquisition One, Llc Content presentation in head worn computing
US11790617B2 (en) 2014-06-09 2023-10-17 Mentor Acquisition One, Llc Content presentation in head worn computing
US11022810B2 (en) 2014-06-09 2021-06-01 Mentor Acquisition One, Llc Content presentation in head worn computing
US11887265B2 (en) 2014-06-09 2024-01-30 Mentor Acquisition One, Llc Content presentation in head worn computing
US10649220B2 (en) 2014-06-09 2020-05-12 Mentor Acquisition One, Llc Content presentation in head worn computing
US10139635B2 (en) 2014-06-09 2018-11-27 Osterhout Group, Inc. Content presentation in head worn computing
US10663740B2 (en) 2014-06-09 2020-05-26 Mentor Acquisition One, Llc Content presentation in head worn computing
US11054645B2 (en) 2014-06-17 2021-07-06 Mentor Acquisition One, Llc External user interface for head worn computing
US11294180B2 (en) 2014-06-17 2022-04-05 Mentor Acquisition One, Llc External user interface for head worn computing
US11789267B2 (en) 2014-06-17 2023-10-17 Mentor Acquisition One, Llc External user interface for head worn computing
US9810906B2 (en) 2014-06-17 2017-11-07 Osterhout Group, Inc. External user interface for head worn computing
US10698212B2 (en) 2014-06-17 2020-06-30 Mentor Acquisition One, Llc External user interface for head worn computing
US10564426B2 (en) 2014-07-08 2020-02-18 Mentor Acquisition One, Llc Optical configurations for head-worn see-through displays
US9366867B2 (en) 2014-07-08 2016-06-14 Osterhout Group, Inc. Optical systems for see-through displays
US11409110B2 (en) 2014-07-08 2022-08-09 Mentor Acquisition One, Llc Optical configurations for head-worn see-through displays
US10775630B2 (en) 2014-07-08 2020-09-15 Mentor Acquisition One, Llc Optical configurations for head-worn see-through displays
US9798148B2 (en) 2014-07-08 2017-10-24 Osterhout Group, Inc. Optical configurations for head-worn see-through displays
US11786105B2 (en) 2014-07-15 2023-10-17 Mentor Acquisition One, Llc Content presentation in head worn computing
US11269182B2 (en) 2014-07-15 2022-03-08 Mentor Acquisition One, Llc Content presentation in head worn computing
US11103122B2 (en) 2014-07-15 2021-08-31 Mentor Acquisition One, Llc Content presentation in head worn computing
US10908422B2 (en) 2014-08-12 2021-02-02 Mentor Acquisition One, Llc Measuring content brightness in head worn computing
US11360314B2 (en) 2014-08-12 2022-06-14 Mentor Acquisition One, Llc Measuring content brightness in head worn computing
US9829707B2 (en) 2014-08-12 2017-11-28 Osterhout Group, Inc. Measuring content brightness in head worn computing
US11630315B2 (en) 2014-08-12 2023-04-18 Mentor Acquisition One, Llc Measuring content brightness in head worn computing
US9423842B2 (en) 2014-09-18 2016-08-23 Osterhout Group, Inc. Thermal management for head-worn computer
US10078224B2 (en) 2014-09-26 2018-09-18 Osterhout Group, Inc. See-through computer display systems
US9366868B2 (en) 2014-09-26 2016-06-14 Osterhout Group, Inc. See-through computer display systems
US9671613B2 (en) 2014-09-26 2017-06-06 Osterhout Group, Inc. See-through computer display systems
US9448409B2 (en) 2014-11-26 2016-09-20 Osterhout Group, Inc. See-through computer display systems
US11809628B2 (en) 2014-12-03 2023-11-07 Mentor Acquisition One, Llc See-through computer display systems
US10684687B2 (en) 2014-12-03 2020-06-16 Mentor Acquisition One, Llc See-through computer display systems
US11262846B2 (en) 2014-12-03 2022-03-01 Mentor Acquisition One, Llc See-through computer display systems
US9684172B2 (en) 2014-12-03 2017-06-20 Osterhout Group, Inc. Head worn computer display systems
USD743963S1 (en) 2014-12-22 2015-11-24 Osterhout Group, Inc. Air mouse
USD792400S1 (en) 2014-12-31 2017-07-18 Osterhout Group, Inc. Computer glasses
USD751552S1 (en) 2014-12-31 2016-03-15 Osterhout Group, Inc. Computer glasses
USD753114S1 (en) 2015-01-05 2016-04-05 Osterhout Group, Inc. Air mouse
USD794637S1 (en) 2015-01-05 2017-08-15 Osterhout Group, Inc. Air mouse
US10062182B2 (en) 2015-02-17 2018-08-28 Osterhout Group, Inc. See-through computer display systems
US9910284B1 (en) 2016-09-08 2018-03-06 Osterhout Group, Inc. Optical systems for head-worn computers
US10534180B2 (en) 2016-09-08 2020-01-14 Mentor Acquisition One, Llc Optical systems for head-worn computers
US11604358B2 (en) 2016-09-08 2023-03-14 Mentor Acquisition One, Llc Optical systems for head-worn computers
US11366320B2 (en) 2016-09-08 2022-06-21 Mentor Acquisition One, Llc Optical systems for head-worn computers
US11668939B2 (en) 2017-07-24 2023-06-06 Mentor Acquisition One, Llc See-through computer display systems with stray light management
US11789269B2 (en) 2017-07-24 2023-10-17 Mentor Acquisition One, Llc See-through computer display systems
US11226489B2 (en) 2017-07-24 2022-01-18 Mentor Acquisition One, Llc See-through computer display systems with stray light management
US11409105B2 (en) 2017-07-24 2022-08-09 Mentor Acquisition One, Llc See-through computer display systems
US11042035B2 (en) 2017-07-24 2021-06-22 Mentor Acquisition One, Llc See-through computer display systems with adjustable zoom cameras
US10578869B2 (en) 2017-07-24 2020-03-03 Mentor Acquisition One, Llc See-through computer display systems with adjustable zoom cameras
US10422995B2 (en) 2017-07-24 2019-09-24 Mentor Acquisition One, Llc See-through computer display systems with stray light management
US11550157B2 (en) 2017-07-24 2023-01-10 Mentor Acquisition One, Llc See-through computer display systems
US11567328B2 (en) 2017-07-24 2023-01-31 Mentor Acquisition One, Llc See-through computer display systems with adjustable zoom cameras
US10969584B2 (en) 2017-08-04 2021-04-06 Mentor Acquisition One, Llc Image expansion optic for head-worn computer
US11500207B2 (en) 2017-08-04 2022-11-15 Mentor Acquisition One, Llc Image expansion optic for head-worn computer

Also Published As

Publication number Publication date
CA2828407A1 (en) 2012-09-07
WO2012118575A3 (en) 2013-03-14
WO2012118573A1 (en) 2012-09-07
DE112012001022T5 (en) 2013-12-19
DE112012001032T5 (en) 2014-01-30
CA2828413A1 (en) 2012-09-07

Similar Documents

Publication Publication Date Title
US20120120103A1 (en) Alignment control in an augmented reality headpiece
WO2012118575A2 (en) Alignment control in an augmented reality headpiece
CN104205175B (en) Information processor, information processing system and information processing method
EP3149698B1 (en) Method and system for image georegistration
US10198865B2 (en) HMD calibration with direct geometric modeling
JP6609929B2 (en) Depth-parallax calibration of binocular optical augmented reality system
JP6659924B2 (en) Adjusting the presentation of the head-mounted display
KR101266198B1 (en) Display apparatus and display method that heighten visibility of augmented reality object
CN204465706U (en) Terminal installation
CN104731337B (en) Method for representing virtual information in true environment
EP1404126B1 (en) Video combining apparatus and method
US20160267720A1 (en) Pleasant and Realistic Virtual/Augmented/Mixed Reality Experience
CN101833896B (en) Geographic information guide method and system based on augment reality
US9375639B2 (en) Image display system and head-mounted display device
US20100287500A1 (en) Method and system for displaying conformal symbology on a see-through display
US20140285523A1 (en) Method for Integrating Virtual Object into Vehicle Displays
US8847850B1 (en) Head mounted display device for displaying augmented reality image capture guide and control method for the same
US20160307374A1 (en) Method and system for providing information associated with a view of a real environment superimposed with a virtual object
US20160019722A1 (en) System and Method for Defining an Augmented Reality View in a Specific Location
CN104995583A (en) Direct interaction system for mixed reality environments
JP2009267729A (en) Image processing apparatus, image processing method, program, and recording medium
JP2010072477A (en) Image display apparatus, image display method, and program
JP2012128779A (en) Virtual object display device
WO2015048905A1 (en) System and method for incorporating a physical image stream in a head mounted display
US9918066B2 (en) Methods and systems for producing a magnified 3D image

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12752340

Country of ref document: EP

Kind code of ref document: A2

ENP Entry into the national phase in:

Ref document number: 2828413

Country of ref document: CA

WWE Wipo information: entry into national phase

Ref document number: 1120120010221

Country of ref document: DE

Ref document number: 112012001022

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12752340

Country of ref document: EP

Kind code of ref document: A2