US20110026787A1 - Method for dealing with singularities in gravity referenced endoscopic imaging - Google Patents

Method for dealing with singularities in gravity referenced endoscopic imaging Download PDF

Info

Publication number
US20110026787A1
US20110026787A1 US12/902,998 US90299810A US2011026787A1 US 20110026787 A1 US20110026787 A1 US 20110026787A1 US 90299810 A US90299810 A US 90299810A US 2011026787 A1 US2011026787 A1 US 2011026787A1
Authority
US
United States
Prior art keywords
neighborhood
endoscope
displaying
user
view vector
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/902,998
Inventor
Eric Lawrence Hale
Hans David Hoeg
Nathan Jon Schara
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
EndActive Inc
Karl Storz Imaging Inc
Original Assignee
EndActive Inc
Karl Storz Imaging Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by EndActive Inc, Karl Storz Imaging Inc filed Critical EndActive Inc
Priority to US12/902,998 priority Critical patent/US20110026787A1/en
Assigned to ENDACTIVE, INC. reassignment ENDACTIVE, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HALE, ERIC L., HOEG, HANS DAVID, SCHARA, NATHAN JON
Assigned to KARL STORZ DEVELOPMENT CORP. reassignment KARL STORZ DEVELOPMENT CORP. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ENDACTIVE, INC.
Assigned to KARL STORZ IMAGING, INC. reassignment KARL STORZ IMAGING, INC. NUNC PRO TUNC ASSIGNMENT (SEE DOCUMENT FOR DETAILS). Assignors: KARL STORZ DEVELOPMENT CORP.
Publication of US20110026787A1 publication Critical patent/US20110026787A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/06Devices, other than using radiation, for detecting or locating foreign bodies ; determining position of probes within or on the body of the patient
    • A61B5/065Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002Operational features of endoscopes
    • A61B1/00043Operational features of endoscopes provided with output arrangements
    • A61B1/00045Display arrangement
    • A61B1/0005Display arrangement combining images e.g. side-by-side, superimposed or tiled
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002Operational features of endoscopes
    • A61B1/00043Operational features of endoscopes provided with output arrangements
    • A61B1/00055Operational features of endoscopes provided with output arrangements for alerting the user
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/05Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances characterised by the image sensor, e.g. camera, being in the distal end portion
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B23/00Telescopes, e.g. binoculars; Periscopes; Instruments for viewing the inside of hollow bodies; Viewfinders; Optical aiming or sighting devices
    • G02B23/24Instruments or systems for viewing the inside of hollow bodies, e.g. fibrescopes
    • G02B23/2476Non-optical details, e.g. housings, mountings, supports
    • G02B23/2484Arrangements in relation to a camera or imaging device

Definitions

  • the present invention relates to endoscopic imaging, and in particular, to endoscopic image orientation and its relationship to the direction of gravity and the viewer's reference frame.
  • the surgeon's viewpoint is different from the viewpoint of the endoscope, and the surgeon must continuously try to correlate his own mental picture of the anatomy with the endoscopic picture on the display.
  • the need to know what is up and down inside the endoscopic cavity is so strong that it has become common for surgeons to observe the flow direction of fluid droplets on the endoscope cover window or search for pooling blood in order to get a sense of direction inside the cavity.
  • knowing the up direction helps in understanding the endoscope's position relative to the surrounding anatomy.
  • the surgeon would be able to relate to the endoscopic cavity as if his own eyes were actually inside the cavity.
  • a warning flag is shown in a gravity leveled endoscopic imaging system.
  • the current image orientation is also maintained when the view vector is in the neighborhood of a singular configuration, thus avoiding sudden flipping or spinning of the image.
  • an option for the user to interactively control the flip or spin of the image when in such a neighborhood is provided.
  • What is claimed is a method for avoiding rapid or sudden image rotation in a gravity leveled endoscopic imaging system, comprising monitoring of the attitude of said view vector; specifying a neighborhood of a viewing singularity; relating said view vector attitude to said neighborhood; and providing a certain image orientation whenever said view vector is within said neighborhood.
  • FIGS. 1A and 1B illustrate viewing singularities
  • FIGS. 2A , 2 B, 2 C, and 2 D illustrate the concept of singular viewing configurations as they relate to humans
  • FIGS. 3A , 3 B, 3 C, and 3 D show a theoretical endoscopic viewing sphere and the issue of how to define the view-up vector in an endoscopic view
  • FIG. 4 shows view vector trajectories passing ear a pole
  • FIG. 5 shows a neighborhood around a pole
  • FIGS. 6A , 6 B, 6 C, and 6 D show maintaining a specific up-vector within a neighborhood and the indication of a pole and the maintained up-vector as the endoscopic view swings towards a pole and through a pole to its other side;
  • FIGS. 7A and 7B illustrate the concept of maintaining an image orientation as the endoscopic view swings through a pole and exits in a direction different from the entry direction;
  • FIG. 8 shows how the endoscopic information is displayed.
  • FIG. 9 illustrates movement of the endoscope in multiple degrees of freedom.
  • FIGS. 1A and 1B schematically show singular configurations for endoscopes 10 which include the sensing means required for leveling the endoscopic image.
  • the sensing means could be housed anywhere in an endoscope depending on the type of scope; for a rigid endoscope it is typically more practical to put the sensing means in the proximal portion, while for flexible endoscopes it is generally necessary to put the sensing means near the tip.
  • the endoscopic line of sight is represented by a view vector 12 . When the view vector 12 is parallel to the direction of gravity 14 , whether pointing up or down, it is in a singular configuration where the endoscopic image has no inherent up-direction.
  • the singular configuration occurs when the endoscope 10 is vertical ( FIG. 1A ).
  • the attitude of the endoscope 10 when in a singular configuration depends on the angular offset 18 between the scope longitudinal axis 16 and the view vector 12 ( FIG. 1B ). This is also true for flexible and rigid variable direction of view endoscopes which have a variable offset 18 .
  • FIGS. 2A , 2 B, 2 C, and 2 D illustrate the human viewing process in relation to the vertical orientation of our view field.
  • an imagined equator (not shown) centered on a viewer's 26 head
  • the up-direction is obvious and the image 27 seen by the viewer 26 is correctly oriented ( FIG. 2A ).
  • It is indicated by an up-vector 28 which is a projection of the gravity direction 16 onto the view field.
  • the up-vector 28 is normal to the line of sight.
  • she tilts her head further upward along an imaginary arc 29 , she finally reaches a configuration where her line of sight is parallel to the direction of gravity 16 and is looking straight up 30 .
  • a viewing pole 31 can be defined as the intersection between the arc 29 and the direction of gravity 16 .
  • the up-vector 28 has been reflected about the gravity axis 16 .
  • the brain and equilibrium senses help with the interpretation of such a situation such that the viewer 26 is never confused as to her orientation in space.
  • Our environment also gives us strong visual clues like vertical and horizontal lines to aid our orientational understanding.
  • an endoscope does not provide information processing like the human brain, and a typical endoscopic environment does not have inherent directional clues.
  • Displaying the endoscopic image reversed up to down and left to right is consistent with the human viewing configuration 32 , but arguably the image should be displayed is if viewed from the configuration of FIG. 2C where instead of having tilted her head beyond the singular configuration 30 , the viewer 26 has turned her entire body around and tilted her head up 34 .
  • FIG. 2D the viewer 26 swings her line of sight through the downward singular configuration 36 ( FIG. 2D ) to another backwards viewing configuration 38 .
  • the present invention lets the user interactively select how she would like the endoscopic counterpart to the image 27 displayed.
  • FIGS. 3A , 3 B, 3 C, and 3 D illustrate these orientation issues further.
  • a theoretical viewing sphere 40 (a three dimensional version of the viewing arc 29 in FIG. 2 ) is centered on the tip of an endoscope 10 , as shown in FIG. 3A (The orientation of the endoscope 10 is not relevant).
  • the up-directions of points within the endoscopic view field 42 are continuously changing.
  • FIG. 3B When looking below the equator 44 lines of longitude 29 , 46 , 47 diverge ( FIG. 3B ), and when looking above the equator 44 lines of longitude 29 , 46 , 47 ( FIG. 3C ) converge.
  • the up-direction 28 varies throughout the view field 42 .
  • the severity of this variation depends on the position of the view vector 12 in the viewing sphere 40 and the size of the view field 42 . As the poles 31 are approached, this variation gets worse, and different points within a view field 42 can have diametrically opposite up-vectors. At the poles 31 there is mathematically no up-direction and in the neighborhood of the poles 31 the up-vector varies rapidly because the lines of longitude radially diverge or converge ( FIG. 3D ).
  • Equation 2 defines a neighborhood 52 around a singularity.
  • the view vector angle .delta. satisfies this equation, where .epsilon. can be selected according to preference, the view vector 12 is said to be in the neighborhood 52 of a singularity.
  • This neighborhood 52 can be thought of as a cone centered on the direction of gravity 16 . Any time the view vector 12 falls within this cone 52 , Equation 2 is satisfied (This applies to both the south and north poles. Depending on implementations and definitions, the neighborhood around the north pole could be defined with (180-.epsilon.) substituted for .epsilon.). Other definitions of a neighborhood of a singularity can also be used.
  • the attitude of the view vector is continuously monitored by the rotation pick-up sensors and a processor compares its configuration to the set of configurations contained within a neighborhood of a singularity.
  • a processor compares its configuration to the set of configurations contained within a neighborhood of a singularity.
  • the user can either specify a desired image up-vector, or the processor can instruct the automatic leveling system to maintain the former up-vector 28 which existed when the view vector moved into the neighborhood 52 .
  • the endoscopic image would thus be kept in an artificially imposed orientation 28 instead of following the natural up-direction 54 which would exist at any point within the neighborhood 52 (It would be the same for the north pole except that the up-direction would be towards rather than away from the pole).
  • FIGS. 7A and 7B show the result of the present invention as the view vector enters a neighborhood 52 along some trajectory 56 .
  • the image leveling system maintains the up-direction which existed at the time the view vector crossed over the boundary of the neighborhood 52 .
  • Alternate view vector trajectories 57 , 58 , and 59 associated with various scope manipulations are shown in FIG. 7B . It should be noted that as the view vector leaves the neighborhood 52 , the image will again be leveled according to the gravity leveling system, and for certain trajectories the image orientation may suddenly adjust itself a significant amount depending on the definition of the neighborhood 52 .
  • the neighborhood 52 has a definite boundary, there will be a discontinuity at the boundary, but if it is defined with a soft boundary, the image orientation may follow a smooth and slow varying function, as in trajectory 57 .
  • this mask can be defined by any appropriate mathematical function.
  • a neighborhood can be set up such that the endoscopic view is gravity leveled in certain regions and traditional in other regions.
  • the neighborhood can also have a directionality. The directionality of a neighborhood determines the endoscopic image orientation (and up-vector) when the view vector is within the neighborhood, and this directionality can be specified by a rule or a mathematical function.
  • the neighborhood can be tailored according to the endoscopic procedure. For instance, if the user simply expects to generally swing the endoscope past a pole 31 and then back out in a direction generally parallel to the entry direction (trajectories 56 , 58 ), then the neighborhood 52 could be small because the exit path is close to the entry path and thus will not cause a drastic change in image orientation as the view vector leaves the neighborhood 52 .
  • the neighborhood 52 can also be changed or disabled by the user. In this way the user can dynamically select how she wants to view the environment according to personal preference. This is important because different users have different ways of thinking about the endoscopic space (For example, when a scope is swung back towards the user ( FIG. 4A ) some users may think of this as looking down and backwards, while others may think of this as standing on the other side and looking down and forwards.)
  • FIG. 8 shows a display 62 , with an endoscopic image 64 of a certain anatomical feature 66 .
  • the boundary of a neighborhood is indicated by a line of latitude 68 .
  • An indicator of the image center 70 can be selectively enabled to provide information about the relative location of a pole, indicated by a dot 72 .
  • the display 62 includes a graphical model 74 of the global viewing configuration, including graphical models of the endoscope 76 , view vector 78 , gravity direction 80 , singularity neighborhood boundary 82 , and viewing sphere 84 . These graphical models aid the viewing process by providing the user with information about the relative arrangement of the view vector and the singular configuration.
  • Buttons 88 are provided for the user to interactively manage image orientation and neighborhood settings.
  • the user may select to toggle the image orientation between the current upright image orientation maintained within a neighborhood and the natural upright orientation which would exist without the neighborhood setting.
  • the user can select to display the pole and alternately a full coordinate grid.
  • the interface also features an optional warning flag 90 which shows up when the endoscopic view vector is in the vicinity a pole.

Abstract

A method for avoiding rapid or sudden image rotation or reversal in a gravity leveled endoscopic imaging system is disclosed. A mathematical neighborhood of a singular viewing configuration is defined and within this neighborhood the endoscopic image orientation follows specified rules.

Description

    CROSS-REFERENCE TO RELATED APPLICATIONS
  • This application is a continuation of U.S. patent application Ser. No. 11/055,445, filed Feb. 10, 2005, which application is currently pending and also claims the benefit under 35 U.S.C. §119(e) of the U.S. Provisional Patent Application Ser. No. 60/626,122 filed on Nov. 9, 2004, the content of each of which is incorporated by reference herein in its entirety.
  • FIELD OF THE INVENTION
  • The present invention relates to endoscopic imaging, and in particular, to endoscopic image orientation and its relationship to the direction of gravity and the viewer's reference frame.
  • BACKGROUND OF THE INVENTION
  • As a surgeon or an assistant manipulates an endoscope with an attached camera, the camera faithfully relates what it sees, with its own upright axis displayed as the upright axis of the image on the display. This often results in rotation of the viewed image. As the image rotates, the surgeon loses track of what is actually up and down inside the endoscopic cavity. This disorientation is one of endoscopy's greatest enemies and has lead to severe mistakes such as the snipping of optical nerves which during a procedure were believed to be a different part of the anatomy. In open procedures, the surgeon can see the anatomy directly and therefore does not have a disorientation problem. However, during an endoscopic procedure the surgeon's viewpoint is different from the viewpoint of the endoscope, and the surgeon must continuously try to correlate his own mental picture of the anatomy with the endoscopic picture on the display. In doing this, the need to know what is up and down inside the endoscopic cavity is so strong that it has become common for surgeons to observe the flow direction of fluid droplets on the endoscope cover window or search for pooling blood in order to get a sense of direction inside the cavity. Aside from being important for distinguishing anatomical features which may look similar, knowing the up direction helps in understanding the endoscope's position relative to the surrounding anatomy. Ideally, the surgeon would be able to relate to the endoscopic cavity as if his own eyes were actually inside the cavity.
  • Attempted solutions to this problem have been proposed in U.S. Pat. No. 5,307,804 to Bonnet (1994), U.S. Pat. No. 5,899,851 to Koninckx (1999), U.S. Pat. No. 6,097,423 to Mattsson-Boze, et al. (2000), U.S. Pat. No. 6,471,637 to Green, et al. (2002), U.S. patent application Ser. No. 10/093,650 by Chatenever, et al. (2002), and U.S. patent application Ser. Nos. 10/829,767 and 60/560,172 by Schara et al. (2004), which are incorporated herein by reference in their entireties. The objects of these inventions are to provide schemes which can maintain the proper upright gravity-leveled orientation of the endoscopic image regardless of how the endoscope is being manipulated.
  • None of these solutions address the problem of so-called viewing singularities (poles). In a singular viewing configuration there is no unique upright image orientation. This occurs when the viewing direction (described as a view vector) is parallel to the direction of gravity. Although a mathematical discontinuity exists only at a singularity itself, the effect of the singularity is nearly everywhere and decreases as one moves away from it.
  • A viewing singularity is similar to standing on the North Pole and having to define which direction is south. In gravity-leveled endoscopic systems singularities cause the endoscopic image to suddenly flip or spin rapidly. This is obviously confusing and annoying to the user. Until now, it has not been clear how one should deal with situations where there is no defined up or down in the endoscopic image.
  • Thus, it is an object of this invention to provide a method for dealing with singularities in gravity-leveled endoscopic imaging systems such that the endoscopic image does not unexpectedly flip or spin during the endoscopic viewing process. It is an additional object of this invention to be applicable to any axial, oblique, side, or retro viewing endoscope as well as any endoscope with a variable direction of view.
  • SUMMARY OF THE INVENTION
  • In a gravity leveled endoscopic imaging system, when the user moves the view vector towards a singular configuration, a warning flag is shown. The current image orientation is also maintained when the view vector is in the neighborhood of a singular configuration, thus avoiding sudden flipping or spinning of the image. Also, an option for the user to interactively control the flip or spin of the image when in such a neighborhood is provided. What is claimed is a method for avoiding rapid or sudden image rotation in a gravity leveled endoscopic imaging system, comprising monitoring of the attitude of said view vector; specifying a neighborhood of a viewing singularity; relating said view vector attitude to said neighborhood; and providing a certain image orientation whenever said view vector is within said neighborhood.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIGS. 1A and 1B illustrate viewing singularities;
  • FIGS. 2A, 2B, 2C, and 2D illustrate the concept of singular viewing configurations as they relate to humans;
  • FIGS. 3A, 3B, 3C, and 3D show a theoretical endoscopic viewing sphere and the issue of how to define the view-up vector in an endoscopic view;
  • FIG. 4 shows view vector trajectories passing ear a pole;
  • FIG. 5 shows a neighborhood around a pole;
  • FIGS. 6A, 6B, 6C, and 6D show maintaining a specific up-vector within a neighborhood and the indication of a pole and the maintained up-vector as the endoscopic view swings towards a pole and through a pole to its other side;
  • FIGS. 7A and 7B illustrate the concept of maintaining an image orientation as the endoscopic view swings through a pole and exits in a direction different from the entry direction; and
  • FIG. 8 shows how the endoscopic information is displayed.
  • FIG. 9 illustrates movement of the endoscope in multiple degrees of freedom.
  • DETAILED DESCRIPTION OF THE INVENTION
  • FIGS. 1A and 1B schematically show singular configurations for endoscopes 10 which include the sensing means required for leveling the endoscopic image. The sensing means could be housed anywhere in an endoscope depending on the type of scope; for a rigid endoscope it is typically more practical to put the sensing means in the proximal portion, while for flexible endoscopes it is generally necessary to put the sensing means near the tip. The endoscopic line of sight is represented by a view vector 12. When the view vector 12 is parallel to the direction of gravity 14, whether pointing up or down, it is in a singular configuration where the endoscopic image has no inherent up-direction. For rigid straight-viewing endoscopes where the view vector 12 is aligned with the longitudinal axis 16 of the endoscope 10, the singular configuration occurs when the endoscope 10 is vertical (FIG. 1A). For oblique viewing endoscopes the attitude of the endoscope 10 when in a singular configuration depends on the angular offset 18 between the scope longitudinal axis 16 and the view vector 12 (FIG. 1B). This is also true for flexible and rigid variable direction of view endoscopes which have a variable offset 18.
  • FIGS. 2A, 2B, 2C, and 2D illustrate the human viewing process in relation to the vertical orientation of our view field. When looking straight ahead 20 or slightly above 22 or below 24 an imagined equator (not shown) centered on a viewer's 26 head, the up-direction is obvious and the image 27 seen by the viewer 26 is correctly oriented (FIG. 2A). It is indicated by an up-vector 28 which is a projection of the gravity direction 16 onto the view field. The up-vector 28 is normal to the line of sight. As the viewer 26 tilts her head further upward along an imaginary arc 29, she finally reaches a configuration where her line of sight is parallel to the direction of gravity 16 and is looking straight up 30. In this configuration the up-vector is horizontal and there no longer exists an obvious up-direction. This is very much like standing at the North Pole and being asked to decide which way is south. A viewing pole 31 can be defined as the intersection between the arc 29 and the direction of gravity 16. As the viewer 26 tilts her head further she actually starts looking behind 32 herself and the image she now sees 27 is flipped both up to down and left to right (FIG. 2B) with the up-vector 28 having been reflected about the gravity axis 16. For a human the brain and equilibrium senses help with the interpretation of such a situation such that the viewer 26 is never confused as to her orientation in space. Our environment also gives us strong visual clues like vertical and horizontal lines to aid our orientational understanding. However, an endoscope does not provide information processing like the human brain, and a typical endoscopic environment does not have inherent directional clues. Thus, as an endoscopic view vector is swung past a pole (and into a different hemisphere), it is not clear how the image should be displayed. Displaying the endoscopic image reversed up to down and left to right is consistent with the human viewing configuration 32, but arguably the image should be displayed is if viewed from the configuration of FIG. 2C where instead of having tilted her head beyond the singular configuration 30, the viewer 26 has turned her entire body around and tilted her head up 34. A similar discussion is relevant when the viewer 26 swings her line of sight through the downward singular configuration 36 (FIG. 2D) to another backwards viewing configuration 38. The present invention lets the user interactively select how she would like the endoscopic counterpart to the image 27 displayed.
  • FIGS. 3A, 3B, 3C, and 3D illustrate these orientation issues further. A theoretical viewing sphere 40 (a three dimensional version of the viewing arc 29 in FIG. 2) is centered on the tip of an endoscope 10, as shown in FIG. 3A (The orientation of the endoscope 10 is not relevant). As the tip of the view vector 12 is moved along a longitudinal arc 29 the up-directions of points within the endoscopic view field 42 are continuously changing. When looking below the equator 44 lines of longitude 29, 46, 47 diverge (FIG. 3B), and when looking above the equator 44 lines of longitude 29, 46, 47 (FIG. 3C) converge. Thus, the up-direction 28 varies throughout the view field 42. The severity of this variation depends on the position of the view vector 12 in the viewing sphere 40 and the size of the view field 42. As the poles 31 are approached, this variation gets worse, and different points within a view field 42 can have diametrically opposite up-vectors. At the poles 31 there is mathematically no up-direction and in the neighborhood of the poles 31 the up-vector varies rapidly because the lines of longitude radially diverge or converge (FIG. 3D).
  • This rapid up-vector variation at the poles and in the neighborhood of the poles is what causes problems for the current gravity-leveled endoscopic systems. When the user manipulates the endoscope and moves the view vector in the vicinity of a pole 31, the image rotates rapidly. This is illustrated in FIG. 4, which shows a set of trajectories 48, 49, 50 swept out by the tip of the endoscopic view vector (not shown). The closer a trajectory is to the pole 31, the more severe the variation in the up-vector 28. Thus, near the pole 31 even small movements of the endoscope can cause a rapid image reversal. At the pole 31 itself the image will suddenly flip without warning. This problem is annoying to the surgeon even when using rigid straight viewing endoscopes where the user has a chance of predicting image rotations and flips based on the attitude of the endoscope. It is worse for oblique or variable direction endoscopes where the relationship between the endoscope axis and the hidden view vector direction is not obvious such that image reversals are unpredictable.
  • These singularities are inherent to the physical universe and can not be removed. They can however be artificially masked or altered with mathematics and electronic processing. As shown in FIG. 5A, the most effective way to deal with the image spin is to mathematically define a neighborhood 52 around a singular configuration in which the image spin is reduced or eliminated by electronic processing. Using the mathematical framework disclosed in U.S. patent application Ser. No. 10/829,767 Schara et al., the angle that the view vector 12 makes with the direction of gravity 16, .delta., is given by equation 1: .delta.=arctan .function. [cos .times..times. .alpha..times..times. cos .times..times..PHI.−sin .times..times..alpha. .times..times. sin .times..times..PHI..times..times. sin .function. (.beta.+.theta.) cos 2 .function. (.beta.+.theta.).times. sin 2 .times..alpha.+(cos .times..times..alpha..times..times. sin .times..times..PHI.+cos .times. .times..PHI..times..times. sin .times..times..alpha..times..times. sin .function. (.beta.+.theta.))2] (1).delta..itoreq. (2) Equation 2 defines a neighborhood 52 around a singularity. When the position of the endoscope is such that the view vector angle .delta. satisfies this equation, where .epsilon. can be selected according to preference, the view vector 12 is said to be in the neighborhood 52 of a singularity. This neighborhood 52 can be thought of as a cone centered on the direction of gravity 16. Any time the view vector 12 falls within this cone 52, Equation 2 is satisfied (This applies to both the south and north poles. Depending on implementations and definitions, the neighborhood around the north pole could be defined with (180-.epsilon.) substituted for .epsilon.). Other definitions of a neighborhood of a singularity can also be used.
  • The attitude of the view vector is continuously monitored by the rotation pick-up sensors and a processor compares its configuration to the set of configurations contained within a neighborhood of a singularity. As shown in FIG. 6A, when the view vector is within the neighborhood 52, the user can either specify a desired image up-vector, or the processor can instruct the automatic leveling system to maintain the former up-vector 28 which existed when the view vector moved into the neighborhood 52. For a particular viewing position on the surface of the theoretical viewing sphere near the south pole 31 the endoscopic image would thus be kept in an artificially imposed orientation 28 instead of following the natural up-direction 54 which would exist at any point within the neighborhood 52 (It would be the same for the north pole except that the up-direction would be towards rather than away from the pole). For example, if a user pivots an endoscope 10 about its patient entry point 56 such that the view vector 12 and view 42 swing to the other side of the pole 31 (FIGS. 6B, 6C, 6D), it will often be desirable to maintain the up-direction 28 that existed before the pole 31 was traversed rather than using the theoretically correct up-direction 54 for the new viewing configuration. This scenario is analogous to the situation in FIG. 2D. When the viewer 26 looks down past her feet and behind herself, she may still be inclined to think of up as the direction in which her feet are pointing.
  • FIGS. 7A and 7B show the result of the present invention as the view vector enters a neighborhood 52 along some trajectory 56. The image leveling system maintains the up-direction which existed at the time the view vector crossed over the boundary of the neighborhood 52. Alternate view vector trajectories 57, 58, and 59 associated with various scope manipulations are shown in FIG. 7B. It should be noted that as the view vector leaves the neighborhood 52, the image will again be leveled according to the gravity leveling system, and for certain trajectories the image orientation may suddenly adjust itself a significant amount depending on the definition of the neighborhood 52. If the neighborhood 52 has a definite boundary, there will be a discontinuity at the boundary, but if it is defined with a soft boundary, the image orientation may follow a smooth and slow varying function, as in trajectory 57. In essence, what the neighborhood 52 does is artificially mask the singularity, and this mask can be defined by any appropriate mathematical function. For example, a neighborhood can be set up such that the endoscopic view is gravity leveled in certain regions and traditional in other regions. In addition to the size of a neighborhood, as specified by a relation such as Equation 2, the neighborhood can also have a directionality. The directionality of a neighborhood determines the endoscopic image orientation (and up-vector) when the view vector is within the neighborhood, and this directionality can be specified by a rule or a mathematical function. The neighborhood can be tailored according to the endoscopic procedure. For instance, if the user simply expects to generally swing the endoscope past a pole 31 and then back out in a direction generally parallel to the entry direction (trajectories 56, 58), then the neighborhood 52 could be small because the exit path is close to the entry path and thus will not cause a drastic change in image orientation as the view vector leaves the neighborhood 52. The neighborhood 52 can also be changed or disabled by the user. In this way the user can dynamically select how she wants to view the environment according to personal preference. This is important because different users have different ways of thinking about the endoscopic space (For example, when a scope is swung back towards the user (FIG. 4A) some users may think of this as looking down and backwards, while others may think of this as standing on the other side and looking down and forwards.)
  • FIG. 8 shows a display 62, with an endoscopic image 64 of a certain anatomical feature 66. The boundary of a neighborhood is indicated by a line of latitude 68. An indicator of the image center 70 can be selectively enabled to provide information about the relative location of a pole, indicated by a dot 72. The display 62 includes a graphical model 74 of the global viewing configuration, including graphical models of the endoscope 76, view vector 78, gravity direction 80, singularity neighborhood boundary 82, and viewing sphere 84. These graphical models aid the viewing process by providing the user with information about the relative arrangement of the view vector and the singular configuration. Buttons 88 are provided for the user to interactively manage image orientation and neighborhood settings. For example, the user may select to toggle the image orientation between the current upright image orientation maintained within a neighborhood and the natural upright orientation which would exist without the neighborhood setting. Also, the user can select to display the pole and alternately a full coordinate grid. The interface also features an optional warning flag 90 which shows up when the endoscopic view vector is in the vicinity a pole.
  • Methods for leveling the endoscopic image are described in the above disclosures, but the details of these methods are not necessary for an understanding of this invention.
  • The present invention has been described above in terms of a presently preferred embodiment so that an understanding of the present invention can be conveyed. However, there are many alternative arrangements for a method for providing gravity referenced endoscopic imaging not specifically described herein but for which the present invention is applicable. For example, an alternative mathematical framework describing the endoscope and its configurations would lead to an alternative formula for the view vector orientation and the neighborhood around a singularity. Also, there are many different ways to display the imaging information. In addition, while the examples were given with respect to endoscopes for use in surgical procedures, the present invention is equally applicable with respect to borescopes or the like for use within various mechanical structures. Therefore, the term “endoscope” as used herein, refers to an endoscope or any similar device such as a borescope, a fiberscope, etc.
  • This invention is not to be limited by the embodiments shown in the drawings and described in the description, which are given by way of example and not of limitation, but only in accordance with the scope of the appended claims.

Claims (17)

1. A method for helping a user predict rapid or sudden image rotation in a gravity leveled endoscopic imaging system, the method comprising:
monitoring the attitude of the view vector of an endoscope having an longitudinal axis, said monitoring comprising accounting for endoscope pitch, endoscope roll, and view vector angle relative to the longitudinal axis;
specifying a neighborhood of a viewing singularity; and
displaying a graphical representation of said view vector and a graphical representation of said neighborhood.
2. The method of claim 1, further comprising receiving an input specifying a neighborhood size and directionality from a user, wherein the step of specifying a neighborhood of a viewing singularity is based on the user input.
3. The method of claim 2, wherein the input specifying a neighborhood size and directionality is received from a user via a graphical user interface.
4. The method of claim 1, further comprising displaying a graphical representation of the endoscope.
5. The method of claim 1, further comprising displaying the direction of gravity.
6. The method of claim 1, further comprising displaying a viewing sphere.
7. The method of claim 6, further comprising displaying a pole.
8. The method of claim 6, further comprising displaying a full coordinate grid.
9. The method of claim 1, further comprising displaying a warning when the view vector is in the vicinity of a pole.
10. A method for helping a user predict rapid or sudden image rotation in a gravity leveled endoscopic imaging system, the method comprising:
receiving an input during a medical procedure from a user manipulating an endoscope during the medical procedure that specifies a neighborhood size and directionality from the user, the endoscope having a longitudinal axis;
monitoring the attitude of the view vector of the endoscope, said monitoring comprising accounting for endoscope pitch, endoscope roll, and view vector angle relative to the longitudinal axis;
specifying a neighborhood of a viewing singularity based on said user input; and
displaying a graphical representation of said view vector and a graphical representation of said neighborhood.
11. The method of claim 10, wherein the input specifying a neighborhood size and directionality is received from a user via a graphical user interface.
12. The method of claim 10, further comprising displaying a graphical representation of the endoscope.
13. The method of claim 10, further comprising displaying the direction of gravity.
14. The method of claim 10, further comprising displaying a viewing sphere.
15. The method of claim 14, further comprising displaying a pole.
16. The method of claim 14, further comprising displaying a full coordinate grid.
17. The method of claim 10, further comprising displaying a warning when the view vector is in the vicinity of a pole.
US12/902,998 2004-11-09 2010-10-12 Method for dealing with singularities in gravity referenced endoscopic imaging Abandoned US20110026787A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/902,998 US20110026787A1 (en) 2004-11-09 2010-10-12 Method for dealing with singularities in gravity referenced endoscopic imaging

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US62612204P 2004-11-09 2004-11-09
US11/055,445 US7811224B2 (en) 2004-11-09 2005-02-10 Method for dealing with singularities in gravity referenced endoscopic imaging
US12/902,998 US20110026787A1 (en) 2004-11-09 2010-10-12 Method for dealing with singularities in gravity referenced endoscopic imaging

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/055,445 Continuation US7811224B2 (en) 2004-11-09 2005-02-10 Method for dealing with singularities in gravity referenced endoscopic imaging

Publications (1)

Publication Number Publication Date
US20110026787A1 true US20110026787A1 (en) 2011-02-03

Family

ID=35744764

Family Applications (2)

Application Number Title Priority Date Filing Date
US11/055,445 Active 2028-11-10 US7811224B2 (en) 2004-11-09 2005-02-10 Method for dealing with singularities in gravity referenced endoscopic imaging
US12/902,998 Abandoned US20110026787A1 (en) 2004-11-09 2010-10-12 Method for dealing with singularities in gravity referenced endoscopic imaging

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US11/055,445 Active 2028-11-10 US7811224B2 (en) 2004-11-09 2005-02-10 Method for dealing with singularities in gravity referenced endoscopic imaging

Country Status (5)

Country Link
US (2) US7811224B2 (en)
EP (1) EP1654978B1 (en)
JP (1) JP2006136722A (en)
CA (1) CA2526287C (en)
DE (1) DE602005022394D1 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9907457B2 (en) 2013-02-01 2018-03-06 Deka Products Limited Partnership Endoscope with pannable camera
US10616491B2 (en) 2013-02-01 2020-04-07 Deka Products Limited Partnership Endoscope with pannable camera and related method
US11497382B1 (en) 2020-04-27 2022-11-15 Canon U.S.A., Inc. Apparatus and method for endoscopic image orientation control
EP4223204A1 (en) 2022-02-02 2023-08-09 Canon U.S.A. Inc. Antitwist mechanism for robotic endoscope camera
US11786106B2 (en) 2020-05-26 2023-10-17 Canon U.S.A., Inc. Robotic endoscope probe having orientation reference markers
US11925321B2 (en) 2020-08-06 2024-03-12 Canon U.S.A., Inc. Anti-twist tip for steerable catheter

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11412998B2 (en) 2011-02-10 2022-08-16 Karl Storz Imaging, Inc. Multi-source medical display
US10674968B2 (en) 2011-02-10 2020-06-09 Karl Storz Imaging, Inc. Adjustable overlay patterns for medical display
US10631712B2 (en) * 2011-02-10 2020-04-28 Karl Storz Imaging, Inc. Surgeon's aid for medical display
US9230223B2 (en) * 2011-12-05 2016-01-05 Houzz, Inc. Consistent presentation of content and passive relevance determination of content relationship in an on-line commerce system

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4902129A (en) * 1988-09-06 1990-02-20 Schott Fiber Optics Orientation indicator for a flexible fiberscope or endoscope including method of manufacture
US5899851A (en) * 1993-07-09 1999-05-04 Saturnus A.G. TV camera with rotational orientation correction
US6434507B1 (en) * 1997-09-05 2002-08-13 Surgical Navigation Technologies, Inc. Medical instrument and method for use with computer-assisted image guided surgery
US20020161280A1 (en) * 1999-09-24 2002-10-31 David Chatenever Image orientation for endoscopic video displays
US20030032878A1 (en) * 1996-06-28 2003-02-13 The Board Of Trustees Of The Leland Stanford Junior University Method and apparatus for volumetric image navigation
US20030220541A1 (en) * 1998-12-08 2003-11-27 Intuitive Surgical, Inc. Devices and methods for presenting and regulating auxiliary information on an image display of a telesurgical system to assist an operator in performing a surgical procedure
US20040127769A1 (en) * 2001-12-14 2004-07-01 Hale Eric L. Interface for a variable direction-of-view endoscope
US20040210105A1 (en) * 2003-04-21 2004-10-21 Hale Eric Lawrence Method for capturing and displaying endoscopic maps
US20050020878A1 (en) * 2002-07-31 2005-01-27 Junichi Ohnishi Endoscope
US7122001B2 (en) * 2002-04-08 2006-10-17 Olympus Corporation Encapsulated endoscope system in which endoscope moves in lumen by itself and rotation of image of region to be observed is ceased
US7479106B2 (en) * 2004-09-30 2009-01-20 Boston Scientific Scimed, Inc. Automated control of irrigation and aspiration in a single-use endoscope

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06269403A (en) 1993-03-19 1994-09-27 Olympus Optical Co Ltd Electronic endoscope apparatus

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4902129A (en) * 1988-09-06 1990-02-20 Schott Fiber Optics Orientation indicator for a flexible fiberscope or endoscope including method of manufacture
US5899851A (en) * 1993-07-09 1999-05-04 Saturnus A.G. TV camera with rotational orientation correction
US20030032878A1 (en) * 1996-06-28 2003-02-13 The Board Of Trustees Of The Leland Stanford Junior University Method and apparatus for volumetric image navigation
US6434507B1 (en) * 1997-09-05 2002-08-13 Surgical Navigation Technologies, Inc. Medical instrument and method for use with computer-assisted image guided surgery
US20030220541A1 (en) * 1998-12-08 2003-11-27 Intuitive Surgical, Inc. Devices and methods for presenting and regulating auxiliary information on an image display of a telesurgical system to assist an operator in performing a surgical procedure
US20020161280A1 (en) * 1999-09-24 2002-10-31 David Chatenever Image orientation for endoscopic video displays
US7211042B2 (en) * 1999-09-24 2007-05-01 Karl Storz Imaging, Inc. Image orientation for endoscopic video displays
US20040127769A1 (en) * 2001-12-14 2004-07-01 Hale Eric L. Interface for a variable direction-of-view endoscope
US7122001B2 (en) * 2002-04-08 2006-10-17 Olympus Corporation Encapsulated endoscope system in which endoscope moves in lumen by itself and rotation of image of region to be observed is ceased
US20050020878A1 (en) * 2002-07-31 2005-01-27 Junichi Ohnishi Endoscope
US20040210105A1 (en) * 2003-04-21 2004-10-21 Hale Eric Lawrence Method for capturing and displaying endoscopic maps
US7479106B2 (en) * 2004-09-30 2009-01-20 Boston Scientific Scimed, Inc. Automated control of irrigation and aspiration in a single-use endoscope

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9907457B2 (en) 2013-02-01 2018-03-06 Deka Products Limited Partnership Endoscope with pannable camera
US10362927B2 (en) 2013-02-01 2019-07-30 Deka Products Limited Partnership Endoscope with pannable camera
US10616491B2 (en) 2013-02-01 2020-04-07 Deka Products Limited Partnership Endoscope with pannable camera and related method
US10863888B2 (en) 2013-02-01 2020-12-15 Deka Products Limited Partnership Endoscope with pannable camera
US11497382B1 (en) 2020-04-27 2022-11-15 Canon U.S.A., Inc. Apparatus and method for endoscopic image orientation control
US11786106B2 (en) 2020-05-26 2023-10-17 Canon U.S.A., Inc. Robotic endoscope probe having orientation reference markers
US11925321B2 (en) 2020-08-06 2024-03-12 Canon U.S.A., Inc. Anti-twist tip for steerable catheter
EP4223204A1 (en) 2022-02-02 2023-08-09 Canon U.S.A. Inc. Antitwist mechanism for robotic endoscope camera

Also Published As

Publication number Publication date
US20060100482A1 (en) 2006-05-11
JP2006136722A (en) 2006-06-01
CA2526287C (en) 2008-09-02
US7811224B2 (en) 2010-10-12
CA2526287A1 (en) 2006-05-09
EP1654978A1 (en) 2006-05-10
DE602005022394D1 (en) 2010-09-02
EP1654978B1 (en) 2010-07-21

Similar Documents

Publication Publication Date Title
US20110026787A1 (en) Method for dealing with singularities in gravity referenced endoscopic imaging
US11461983B2 (en) Surgeon head-mounted display apparatuses
EP3834768B1 (en) Augmented reality headset with varied opacity for navigated robotic surgery
JP6751456B2 (en) Augmented reality navigation system for use in a robotic surgical system and method of use thereof
US9615772B2 (en) Global endoscopic viewing indicator
US7517314B2 (en) Endoscopic imaging with indication of gravity direction
US10092165B2 (en) System and method for enhancing picture-in-picture display for imaging devices used for surgical procedures
US9033871B2 (en) Gravity referenced endoscopic image orientation
CN107079097B (en) Endoscopic system, image processing apparatus, image processing method and program
CN105078580B (en) Surgical robot system and its laparoscopic procedure method and human body temperature type operation image processing apparatus and its method
WO2018025483A1 (en) Video outputting system, video outputting method, and control apparatus
CN114630611A (en) System and method for changing visual direction during video-guided clinical surgery using real-time image processing
US11678791B2 (en) Imaging system and observation method
CN107844190A (en) Image presentation method and device based on Virtual Reality equipment
EP1844696B1 (en) Endoscopic imaging with indication of gravity direction
CN106455917B (en) Encapsulated medical device guiding system
US20070010710A1 (en) Biological imaging systems
US20200093545A1 (en) Control apparatus, control method, and medical observation system
JP2007275257A (en) Method for forming endoscopic image with display in gravity direction
WO2019073689A1 (en) Information processing device, information processing method, and program
CA2541759C (en) Endoscopic imaging with indication of gravity direction
KR20210089261A (en) Cannula and proximity camera with imaging control system for rotating images
US10973391B1 (en) Mixed reality viewing of a surgical procedure
WO2021009401A1 (en) Spatial-orientation endoscopic system

Legal Events

Date Code Title Description
AS Assignment

Owner name: KARL STORZ DEVELOPMENT CORP., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ENDACTIVE, INC.;REEL/FRAME:025349/0450

Effective date: 20050701

Owner name: KARL STORZ IMAGING, INC., CALIFORNIA

Free format text: NUNC PRO TUNC ASSIGNMENT;ASSIGNOR:KARL STORZ DEVELOPMENT CORP.;REEL/FRAME:025349/0496

Effective date: 20101004

Owner name: ENDACTIVE, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HALE, ERIC L.;HOEG, HANS DAVID;SCHARA, NATHAN JON;REEL/FRAME:025349/0332

Effective date: 20061023

STCB Information on status: application discontinuation

Free format text: ABANDONED -- AFTER EXAMINER'S ANSWER OR BOARD OF APPEALS DECISION