Search Images Maps Play YouTube News Gmail Drive More »
Sign in
Screen reader users: click this link for accessible mode. Accessible mode has the same essential features but works better with your reader.

Patents

  1. Advanced Patent Search
Publication numberUS20040083085 A1
Publication typeApplication
Application numberUS 10/620,016
Publication dateApr 29, 2004
Filing dateJul 14, 2003
Priority dateJun 1, 1998
Also published asUS8036868, US20060276974, US20100073472, US20110187846, WO2005008297A2, WO2005008297A3, WO2005008297A9
Publication number10620016, 620016, US 2004/0083085 A1, US 2004/083085 A1, US 20040083085 A1, US 20040083085A1, US 2004083085 A1, US 2004083085A1, US-A1-20040083085, US-A1-2004083085, US2004/0083085A1, US2004/083085A1, US20040083085 A1, US20040083085A1, US2004083085 A1, US2004083085A1
InventorsJack Zeineh, Usman Rashid, Rui-Tao Dong
Original AssigneeZeineh Jack A., Usman Rashid, Rui-Tao Dong
Export CitationBiBTeX, EndNote, RefMan
External Links: USPTO, USPTO Assignment, Espacenet
Integrated virtual slide and live microscope system
US 20040083085 A1
Abstract
A method for creating a virtual slide is provided. A virtual slide is a digital representation of an area of interest of a microscopic slide. One method is to use a motorized microscope that can move a specimen with respect to a microscopic objective. With such a system, one can capture one or more images through a microscopic objective, such that a region of interest is imaged. Each image is then joined together to form a composite or “virtual image.” In one embodiment, after a virtual slide is created, a user may fully utilize the full capabilities of the remote microscope. Among these capabilities is a set of “optical objectives” and :virtual objectives.” Optical objectives are images created by digitizing an image through a microscopic objective in real time. Virtual objectives are digitally created magnifications created by utilizing the existing virtual slide data to digitally create a field of view.
Images(2)
Previous page
Next page
Claims(1)
We claim:
1. A method for simultaneously viewing remote microscope images comprising virtual and live images and producing a seamless view of live and virtual images comprising:
providing a virtual slide
providing a live microscope slide
automatically and sequentially shifting regions overlapping regions of images from the slides to obtain the optimal image
Description
    BACKGROUND OF THE INVENTION
  • [0001]
    This invention relates to a method and apparatus for viewing remote microscope images.
  • [0002]
    Currently there is increasing demand for pathologist review of samples at remote locations. There exist multiple systems to address this need. They typically fall within one of two categories: live remote microscopy and virtual slide imaging.
  • [0003]
    In live remote control microscopy, a user receives images that are taken from a slide on a microscope. In virtual slide imaging, a user receives images previously captured. Virtual slide systems take one or more images of an area of interest and assemble them together (if there is more than one image) to form a virtual slide. Each of these techniques has its advantages. Live remote imaging provides users with the closest approximation to manual manipulation. Virtual slides allow faster image viewing, since images are already captured.
  • [0004]
    Virtual slide systems take one or more images and assemble them to form a “virtual slide.”
  • [0005]
    However, users in the past were limited in their ability to integrate these technologies. One could only view and manipulate live and virtual images independently of one another. A user would have to clumsily go back and forth between these two modes of operation to separately look at the virtual slides and live microscope slides. We present a new method that integrates these ideas into one seamless operating environment.
  • SUMMARY OF THE INVENTION
  • [0006]
    In accordance with the present invention, a method and apparatus are provided for the analysis of remote slides in a hybrid live and virtual medium. Users obtain benefits of each technique in a unified environment.
  • DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT
  • [0007]
    A virtual slide is a digital representation of an area of interest of a microscopic slide. A virtual slide can be created multiple ways.
  • [0008]
    One method is to use a motorized microscope that can move a specimen with respect to a microscope objective (e.g., a microscope with a motorized stage). With such system, one can capture one or more images through a microscope objective, such that a region of interest (all or part of the microscopic slide) is imaged. Each image is then joined together to form a composite or “virtual image.” Multiple methods of joining images together are known in the art. One example is when images are simply abutted one next to another. However, this method does not generally produce virtual slides without seams, because errors such as camera rotation relative to the axis of motion are difficult to correct. Even with submicron accuracy stages it is, in practice, difficult to obtain consistent positioning. Another method is to utilize overlap between adjacent images to edge align images for maximum seamlessness. This can be done by sequentially shifting overlapping regions in the x and/or y axis, for example, by a stepping motor, and calculating a correlation value (or measure of goodness of overlap). The shift which results in the best correlation value is then used to join the images together (FIG. 1). While this method can be computationally expensive, it reduces reliance on difficult-to-attain mechanical positioning,requirements and ultimately produces the best images in the sense of seamlessness.
  • [0009]
    In another method, the virtual slide is made simply by utilizing an imaging device with optics suitable to take a an image of the area of interest on the slide in one snapshot. This method is embodied in the form of a conventional digital or analog camera with a macro lens.
  • [0010]
    This virtual slide can then be used to create a thumbnail view of the slide. To create the thumbnail view, the virtual slide is shrunk in resolution from its original, base resolution to a target resolution. If the target resolution is the same as the base resolution, then the image is unchanged. However, typically the resolution of the thumbnail desired is several times smaller than the base resolution.
  • [0011]
    With the virtual slide created, the user may fully utilize the full capabilities of the remote microscope. The user is presented with an image window and a set of control features (FIG. 2). Among these control features is a set of “optical objectives” and “virtual objectives.”
  • [0012]
    Optical objectives are images that are created by a camera digitizing an image through a microscope objective (e.g., 10, 20, or 40) in real time (i.e., an image is captured at the time the user requests the image). Virtual objectives are digitally created magnifications created not by digitizing in real time, but rather by utilizing the existing virtual slide data to digitally create a field of view.
  • [0013]
    When a user selects one of the optical objectives, a “change objective” command is sent to the microscope. This change objective command can also, specify additional qualifying information, such as microscope x, y, z positions, exposure setting, compression type and level, and image dimensions. If additional qualifying information is not sent, then the implicit qualifying information is the current state of the microscope or the last specified state. When the microscope receives the command, actions are taken to change the objective lens and to change the state of the microscope commensurate with the command (e.g., change the relative position of the objective lens relative to the microscopic slide, change exposure, etc.). An image is then digitized, compressed if so specified, and then transmitted to the user for display.
  • [0014]
    When a user selects a virtual objective, a virtual objective command is sent to the microscope. Similar to an optical objective command, this virtual objective command can also specify additional qualifying information, such as microscope x, y, and/or z position. If additional qualifying information is not sent, then the implicit qualifying information is current microscope state or the last specified state. A region of interest is defined by the virtual request—it is the area on the microscope slide included in the field specified by the coordinates of the stage x, y and magnification of the command. This region of interest may optionally be trimmed such that image information already residing at the requesting user's, view is not retransmitted to the user.
  • [0015]
    An image of the region of interest can be created from the virtual slide in multiple ways. If the virtual slide is not compressed, the retrieval of image information corresponding to the region of interest can be done by simply copying data from the virtual slide. If the virtual slide is compressed, a region corresponding to at least the region of interest can be decompressed to a raw bitmap from the main compressed image. If the virtual slide was stored as multiple compressed images rather than one large image, additional efficiencies are possible. For example, only those images that contain the desired region of interest need be accessed for decompression, rather than the entire area of the virtual slide. This enhances performance.
  • [0016]
    The decompression itself can also be performed in various ways. Strategies such as scaled decoding, as in the case of jpeg type compression, can be employed to improve the speed of the decompression by coupling a resolution reduction process with the decompression to speed up decompression when resolution reduction is required. Once the region of interest is decompressed, it can then be recompressed using a variety of strategies known to those skilled in the imaging field, which need not be the same as the method by which the virtual slide was compressed.
  • [0017]
    An alternative type of decompression/recompression step can also be used involving partial decompression. Partial decompression, such as decoding of Huffman-encoded data, as in the case of jpeg, can be performed to produce raw coefficients, rather than full decompression, which produces a raw bitmap. The raw coefficients corresponding to the area of the region of interest can then be selected. These coefficients which correspond to the region of interest are then re-encoded. In the case of jpeg compression, this would involve re Huffman encoding of the coefficients, rather than in normal full compression, where a dct must be performed followed by quantization and then Huffman coding.
  • [0018]
    Whichever the technique, the result is a compressed region of interest. The compressed region of interest is then transmitted to the user for viewing. The described method is more advantageous than sending the entire virtual slide, as one efficiently sends only that information required by the user.
  • [0019]
    However, direct transfer of the compressed image without decompression is feasible when the virtual slide is stored as multiple compressed images. The compressed images that include the area specified by the region of interest can be directly transferred to the user, rather than going through a decompression/recompression step. The disadvantage is that one may transfer more information than is needed if, for example, the compressed images are at a higher resolution that the requested resolution. This can be partially solved by creation of multiple resolution versions of the virtual slide. There are also compression strategies available that allow only portions of the compressed images to be sent, such that a given resolution can be attained depending on which portions of the compressed image one chooses to send (e.g., progressive encoding). However, there is still the issue that the region of interest only partially covers the area of the compressed image. In this case, direct transfer of the image results in inefficiently sending data including both the region of interest and data outside the region of interest to the user.
  • [0020]
    With this invention, users are afforded a streamlined method of utilizing the features of virtual and live microscopy techniques.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0021]
    [0021]FIG. 1 is a drawing showing the overlap between adjacent images during optimization of overlap.
  • [0022]
    [0022]FIG. 2 is a photograph of a user's view of the remote microscope, showing a thumbnail view, microscope imaging window, and a set of microscope controls.
  • [0023]
    While the apparatus and methods of the present invention have been illustrated in terms of certain embodiments, the invention claimed herein is not limited to embodiments disclosed in this application. Rather, the scope of the invention is defined by the claims attached hereto.
  • [0024]
    While the invention has been illustrated and explained herein in terms of certain embodiments the invention is not limited to the specific embodiments disclosed. Rather, the invention is defined by the scope of the claims appended hereto.
Patent Citations
Cited PatentFiling datePublication dateApplicantTitle
US4922909 *Jul 17, 1987May 8, 1990Little James HVideo monitoring and reapposition monitoring apparatus and methods
US5329616 *Aug 13, 1991Jul 12, 1994Canon Kabushiki KaishaCompressed image stores for high resolution computer graphics
US5499097 *Sep 19, 1994Mar 12, 1996Neopath, Inc.Method and apparatus for checking automated optical system performance repeatability
US5602674 *Jun 16, 1995Feb 11, 1997Compucyte Corp.Computerized specimen encoder
US5619032 *Jan 18, 1995Apr 8, 1997International Remote Imaging Systems, Inc.Method and apparatus for automatically selecting the best focal position from a plurality of focal positions for a focusing apparatus
US5626144 *Dec 22, 1995May 6, 1997Enact Health Management SystemsSystem for monitoring and reporting medical measurements
US5655029 *Oct 30, 1996Aug 5, 1997Neuromedical Systems, Inc.Device and method for facilitating inspection of a specimen
US5790710 *Jan 17, 1997Aug 4, 1998Jeffrey H. PriceAutofocus system for scanning microscopy
US5793969 *Jan 11, 1996Aug 11, 1998Neopath, Inc.Network review and analysis of computer encoded slides
US5836877 *Feb 24, 1997Nov 17, 1998Lucid IncSystem for facilitating pathological examination of a lesion in tissue
US5883982 *Mar 13, 1997Mar 16, 1999Neopath, Inc.Astigmatism measurement apparatus and method based on a focal plane separation measurement
US5920657 *Aug 8, 1997Jul 6, 1999Massachusetts Institute Of TechnologyMethod of creating a high resolution still image using a plurality of images and apparatus for practice of the method
US5991729 *Jun 28, 1997Nov 23, 1999Barry; James T.Methods for generating patient-specific medical reports
US6031930 *Aug 23, 1996Feb 29, 2000Bacus Research Laboratories, Inc.Method and apparatus for testing a progression of neoplasia including cancer chemoprevention testing
US6043475 *Apr 15, 1997Mar 28, 2000Olympus Optical Co., Ltd.Focal point adjustment apparatus and method applied to microscopes
US6101265 *Mar 3, 1997Aug 8, 2000Bacus Research Laboratories, Inc.Method and apparatus for acquiring and reconstructing magnified specimen images from a computer-controlled microscope
US6122396 *Aug 17, 1999Sep 19, 2000Bio-Tech Imaging, Inc.Method of and apparatus for automating detection of microorganisms
US6226392 *Sep 13, 1999May 1, 2001Bacus Research Laboratories, Inc.Method and apparatus for acquiring and reconstructing magnified specimen images from a computer-controlled microscope
US6259080 *Mar 15, 1999Jul 10, 2001Olympus Optical Co. Ltd.Autofocus device for microscope
US6272235 *Feb 27, 1998Aug 7, 2001Bacus Research Laboratories, Inc.Method and apparatus for creating a virtual microscope slide
US6396941 *Jun 12, 2000May 28, 2002Bacus Research Laboratories, Inc.Method and apparatus for internet, intranet, and local viewing of virtual microscope slides
US6404906 *Dec 28, 2000Jun 11, 2002Bacus Research Laboratories,Inc.Method and apparatus for acquiring and reconstructing magnified specimen images from a computer-controlled microscope
US6466690 *Dec 19, 2000Oct 15, 2002Bacus Research Laboratories, Inc.Method and apparatus for processing an image of a tissue sample microarray
US6522774 *May 19, 2000Feb 18, 2003Bacus Research Laboratories, Inc.Method and apparatus for creating a virtual microscope slide
US6606413 *Jun 1, 1999Aug 12, 2003Trestle Acquisition Corp.Compression packaged image transmission for telemicroscopy
US6674881 *Dec 14, 2001Jan 6, 2004Bacus Laboratories, Inc.Method and apparatus for internet, intranet, and local viewing of virtual microscope slides
US6674884 *Nov 13, 2001Jan 6, 2004Bacus Laboratories, Inc.Apparatus for remote control of a microscope
US6711283 *May 3, 2000Mar 23, 2004Aperio Technologies, Inc.Fully automatic rapid microscope slide scanner
US6775402 *Feb 12, 2003Aug 10, 2004Bacus Laboratories, Inc.Method and apparatus for creating a virtual microscope slide
US20020149628 *Dec 22, 2000Oct 17, 2002Smith Jeffrey C.Positioning an item in three dimensions via a graphical representation
US20030012420 *Jun 6, 2002Jan 16, 2003Applied Imaging CorporationAutomated scanning method for pathology samples
US20030039384 *Oct 15, 2002Feb 27, 2003Bacus Research Laboratories, Inc.Method and apparatus for processing an image of a tissue sample microarray
US20030090127 *Nov 4, 2002May 15, 2003Nissan Motor Co., Ltd.Front body structure for vehicle
US20030112330 *Dec 16, 2002Jun 19, 2003Olympus Optical Co., Ltd.Microscopic image capture apparatus
US20030123717 *Feb 12, 2003Jul 3, 2003Bacus Research Laboratories, Inc.Method and apparatus for creating a virtual microscope slide
US20030138139 *Dec 18, 2002Jul 24, 2003Strom John T.Dual-axis scanning system and method
US20030184730 *Jan 23, 2003Oct 2, 2003The Regents Of The University Of CaliforniaFast 3D cytometry for information in tissue engineering
US20030210262 *May 10, 2002Nov 13, 2003Tripath Imaging, Inc.Video microscopy system and multi-view virtual slide viewer capable of simultaneously acquiring and displaying various digital views of an area of interest located on a microscopic slide
US20030228038 *Mar 31, 2003Dec 11, 2003Chroma Vision Medical Systems, Inc., A California CorporationMethod and apparatus for automated image analysis of biological specimens
US20030228053 *May 5, 2003Dec 11, 2003Creatv Microtech, Inc.Apparatus and method for three-dimensional image reconstruction
US20030231791 *Jun 12, 2003Dec 18, 2003Torre-Bueno Jose De LaAutomated system for combining bright field and fluorescent microscopy
US20040004614 *Feb 24, 2003Jan 8, 2004Bacus Laboratories, Inc.Focusable virtual microscopy apparatus and method
US20040047033 *Sep 3, 2003Mar 11, 2004Olympus Optical Co., Ltd.Microscopic image capture apparatus and microscopic image capturing method
US20040136582 *Jan 5, 2004Jul 15, 2004Bacus Laboratories, Inc.Apparatus for remote control of a microscope
US20040141637 *Jan 5, 2004Jul 22, 2004Bacus Laboratories, Inc.Method and apparatus for internet, intranet, and local viewing of virtual microscope slides
US20040236773 *Jun 16, 2004Nov 25, 2004Bacus Laboratories, Inc.Method and apparatus for creating a virtual microscope slide
Referenced by
Citing PatentFiling datePublication dateApplicantTitle
US7027628 *Nov 13, 2001Apr 11, 2006The United States Of America As Represented By The Department Of Health And Human ServicesAutomated microscopic image acquisition, compositing, and display
US7305109Feb 1, 2006Dec 4, 2007The Government of the United States of America as represented by the Secretary of Health and Human Services, Centers for Disease Control and PreventionAutomated microscopic image acquisition compositing, and display
US7391894Jul 12, 2006Jun 24, 2008Carl Zeiss Microimaging Ais, Inc.System and method for remote navigation of a specimen
US7505616Jul 12, 2006Mar 17, 2009Carl Zeiss Microimaging Ais, Inc.System and method for reconstructing a diagnostic trajectory
US7916913Jun 26, 2009Mar 29, 2011Carl Zeiss Microimaging GmbhSystem and method for reconstructing a diagnostic trajectory
US7916916Sep 3, 2009Mar 29, 2011Carl Zeiss Microimaging GmbhSystem and method for remote navigation of a specimen
US8036868Sep 11, 2009Oct 11, 2011Carl Zeiss Microimaging GmbhIntergrated systems and methods of virtual or live microscope slide presentation
US20060251309 *Jul 12, 2006Nov 9, 2006Trestle Acquisition Corp.System and method for remote navigation of a specimen
US20060251331 *Jul 12, 2006Nov 9, 2006Trestle Acquisition Corp.System and method for reconstructing a diagnostic trajectory
US20060276974 *Aug 11, 2006Dec 7, 2006Trestle Acquisition Corp.System for remote viewing and display of a slide
US20100034437 *Jun 26, 2009Feb 11, 2010Zeineh Jack ASystem and method for reconstructing a diagnostic trajectory
US20100067759 *Sep 3, 2009Mar 18, 2010Zeineh Jack ASystem and Method for Remote Navigation of a Specimen
US20100073472 *Sep 11, 2009Mar 25, 2010Carl Zeiss Microlmaging Ais, Inc.Intergrated virtual slide and live microscope system
US20110187846 *Dec 2, 2010Aug 4, 2011Carl Zeiss Microimaging GmbhSystem for remote viewing and display of a slide
Classifications
U.S. Classification703/11, 725/1
International ClassificationG02B, G01N33/48, G06T9/00
Cooperative ClassificationG02B21/367, G06T9/007
European ClassificationG02B21/36V1, G06T9/00T
Legal Events
DateCodeEventDescription
Jan 8, 2004ASAssignment
Owner name: TRESTLE CORPORATION, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZEINEH, JACK A.;RASHID, USMAN;DONG, RUI-TAO;REEL/FRAME:014957/0626
Effective date: 20031229
Feb 21, 2006ASAssignment
Owner name: TRESTLE ACQUISITION CORP., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TRESTLE CORPORATION;REEL/FRAME:017278/0294
Effective date: 20060221
Feb 28, 2006ASAssignment
Owner name: CLARIENT, INC., A DELAWARE CORPORATION, CALIFORNIA
Free format text: SECURITY AGREEMENT;ASSIGNOR:TRESTLE ACQUISITION CORP., A DELAWARE CORPORATION;REEL/FRAME:017223/0757
Effective date: 20060227
Jun 20, 2006ASAssignment
Owner name: CLARIENT, INC., A DELAWARE CORPORATION, CALIFORNIA
Free format text: SECURITY AGREEMENT;ASSIGNOR:TRESTLE ACQUISITION CORP., A DELAWARE CORPORATION;REEL/FRAME:017811/0685
Effective date: 20060619
Sep 27, 2006ASAssignment
Owner name: TRESTLE ACQUISITION CORP., A WHOLLY-OWNED SUBSIDIA
Free format text: TERMINATION OF PATENT SECURITY AGREEMENT RECORDED AT REEL/FRAME NO. 017223/0757;ASSIGNOR:CLARIENT, INC.;REEL/FRAME:018313/0364
Effective date: 20060922
Sep 28, 2006ASAssignment
Owner name: TRESTLE ACQUISITION CORP., A WHOLLY OWNED SUBSIDIA
Free format text: TERMINATION OF PATENT SECURITY AGREEMENT RECORDED AT REEL FRAME NO. 017811/0685;ASSIGNOR:CLARIENT, INC.;REEL/FRAME:018313/0808
Effective date: 20060922
Owner name: CLRT ACQUISITION LLC, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TRESTLE ACQUISITION CORP.;REEL/FRAME:018322/0644
Effective date: 20060922
Jan 22, 2007ASAssignment
Owner name: CLARIENT, INC., CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CLRT ACQUISITION LLC;REEL/FRAME:018787/0870
Effective date: 20070105
Nov 7, 2007ASAssignment
Owner name: CARL ZEISS MICROIMAGING AIS, INC., NEW YORK
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:CLARIENT, INC.;REEL/FRAME:020072/0662
Effective date: 20071016