FIELD OF THE INVENTION
This invention relates to a method of and apparatus for acquiring and analyzing digital images of an image viewed through a computer-controlled automated microscope and more particularly, to using the latter in a quantitative analysis of plant or biological specimens.
BACKGROUND OF THE INVENTION
In the image analysis and quantification of DNA from tissue sections as disclosed in U.S. Pat. No. 4,741,031, and also especially in the immunohisto-chemistry assays on the kinds of cell analysis systems disclosed in U.S. Pat. Nos. 5,086,476; 5,202,931; and 5,252,487 issued to Bacus, there is a problem of first locating the cancer regions for analysis under low power and then remembering them when performing the analysis under higher power. Specifically, the problem is that once the microscope is set up for quantitation by image analysis under, e.g. 40×, where all of the diaphragms are set and light adjusted, etc., if the operator needs to move to another tissue area, it is first desirable to locate it at e.g. 10×. In fact, often regions can only be located at this power. In order to do so however, all of the settings (diaphragms, light levels, wavelengths of light, etc.) have to be changed to view the tissue at this magnification. Currently, there is no way to ensure that one could go back to the settings at the previous 40× magnification and continue on with the quantitative image analysis of that same specimen. This necessitates finding those areas under 40×, without changing objectives, which is very slow and time-consuming, and often important cancer areas can be missed.
Also, another problem with tissue analysis, at its current state-of-the-art, is that it is not completely automated, for example, with regard to finding structural regions such as glands, basal layers or other important diagnostic regions. However, as set forth in my co-pending patent application Ser. No. 701,974, filed Aug. 23, 1996, if these regions are located, important very sensitive diagnostic measurements can be performed, which patent application is incorporated by reference as if fully reproduced herein. For example, as disclosed in the aforesaid patent application, assays are made of a variety of tissue types, both human and animal for analysis of neoplasia in tissue, for pre-invasive cancer in tissue, and the effects on the tissue of chemopreventive agents. A quantitative analysis by image processing techniques is performed on tissue types, having various architectural features, such as breast tissue, colon tissue, prostate tissue, esophageal tissue, skin tissue, cervix tissue, etc. These tissues have different morphologies, and they undergo different neoplasias usually resulting from a cellular mutation, as may be enhanced by a carcinogen, or resulting from a cellular proliferation rate enhanced by hormones, growth factors, or other inducers of abnormal tissue growth. Often it is desired to quantify small changes in the neoplasia when it is incipient or through a series of analyses performed at close time intervals to measure whether the neoplasia progression is increasing or has been slowed, stopped or regressed.
Usually, the tissue specimens are cut to expose the basal layer for review under the microscope. Typically, the quantitative measurements are performed at 40× to obtain 100 to 400 tissue images. The 40× objective provides a narrow field of view of a very small portion of the entire basal layer. Often, the basal layer is somewhat elongated and generally linear such as a basal layer in a rat esophagus; and the analysis of the basal layer requires examining it along its length. The basal layer in a mouse colon is more in the form of an irregular, circular shape; and the analysis of this basal layer requires traveling about this circular shape. In breast tissue samples, suspected tumor areas may be at widely-spaced points in the stained tissue; and one wants to be able to navigate and travel to these specific suspected areas and to do the 40× analysis at these areas in an efficient manner. There is a need to allow an experienced operator to interact with the analysis to locate and identify such regions in an interactive manner. Especially, an interactive manner that would be familiar and consistent with the use of a microscope manually, with high power magnification and low power magnification simultaneously available, but performed on a computer terminal. Such a level of interaction is different than the interaction with the system disclosed in the above-listed Bacus patents. There is a need to take the level of interaction to a higher level and let each component, the human and the computer, perform the part that it does best, in the most cost-effective manner.
There are available on the market computer-controlled, automated microscopes such as those sold by Carl Zeiss, Inc., Thornwood, N.J., under the name Axioplan 2 for taking photographic images of a specimen in the microscopic field of view. Those particular microscopes have computer-controlled and automatically adjusted subsystems, such as an illumination subsystem, a focusing subsystem, a diaphragm or optical stops subsystem, an objective lens subsystem, or a filtering subsystem. As an operator selects changes from one objective lens, such as one providing low magnification, e.g., 4×, to a higher magnification, e.g., 40×, the computer-automated system will turn the lens turret to switch in the high magnification automatically and adjusts the lens and also automatically adjusts the illumination to eliminate glare and to provide the proper light illumination including light density. Further, the focus is adjusted, and the proper diaphragm openings are automatically reset. Thus, the computer-controlled, automated subsystems automatically reset to values stored and predetermined for each selected objective lens and the analysis being done.
Those particular microscopes can be used to view various objects or specimens, but are most typically used to view and to take still photographs of biological specimens, such as tissues and cells. Those particular microscopes lack a computer-controlled X and Y stage for translating a specimen-carrying slide with respect to the field of view of the selected objective lens. Currently, pathologists and others who use such microscopes want to view the specimen images in full color or in enhanced colors using fluorescent illumination and/or monochromatic images using the automated filter subsystems on the microscopes. Currently trained pathologists or clinicians are accustomed to manually adjust and have a microscope available to them to view larger areas of the specimen at low magnification, and then to momentarily switch in a new higher magnification lens to obtain a more highly magnified image of a portion of the specimen viewed at low magnification. Pathologists and those working in this area have created in themselves a desire to view suspect tissue through a microscope and appear to resist analysis systems that do not provide them this ability.
The microscopic field of view reduces very substantially as the magnification increases. The skill level of the clinician and/or pathologist is important to locate viewing the most suspicious areas or points of interest on the specimen. Sometimes, a technician will do a first assay and analysis. A pathologist will return to the selected points of interest or other points of interest for review and analysis. One concern with respect to a quantitative analysis of breast cancer tissue or prostate biopsy tissue samples to pap smears or other tests for various cancers or the like is that a particularly suspicious point in the tissue may be overlooked and missed during the visual assay or for selection for an automated review analysis. When observing at high magnifications, the field of view is limited to very small area of the specimen. Hence, the observer has difficulty in knowing and remembering the actual, exact location of this small periscopic view within the very large whole specimen.
Often, also the problem is finding or locating the tissue or cells for view at high magnification so that artifacts and/or blank spaces on the slide are not viewed. A number of approaches have been proposed to prescreen and locate by an X and Y address the cells or small points of interest from a very large number of potential points of interest.
There are currently available commercial services for prescreening pap smears where one can mail in slides and the service will do a microscopic prescan at high magnification for suspected or suspicious areas of interest which are marked and given address locations, and also a video tape of the slide specimen is returned by this service to the sender. The sender then reviews the areas of interest located during the prescreening and/or the video tape to complete the analysis.
In an attempt to locate and allow review of specified points of interest, U.S. Pat. No. 5,428,690 to Bacus discloses a system for prescreening of a field of cells on a specimen slide at low magnification before the viewer. When seeing a point of interest to be viewed at high magnification, the viewer will operate a switch or the like to select and record the address of these selected prescreened points of interest. Later, these prescreened points of interest are then brought into position to be analyzed at high magnification. This system is fairly or too slow for many uses.
A very expensive system is currently in use in which a pathologist located at a diagnostic center is able to make a diagnostic opinion with respect to specimens under a microscope at a remote center. The pathologist at the diagnostic center manipulates robotic controls to send telepathy signals via a special, dedicated satellite or other large bandwidth channel to control the microscope at the remote site in approximately real time. The pathologist then can maneuver the remote microscope to shift the microscope's field of view and to send, by telepathy, a highly magnified, very small image back to the pathologist. This system requires each subscriber to have a special microscope operable by manipulation of the robotic controls at the diagnostic center and a dedicated or large bandwidth channel to convey real time video signals and hence results in a very high cost for the assay being done. To assist the pathologist in staying within the specimen at the remote site, a peripheral edge or map of the specimen is made using a second video camera and a light box or by using computerized scanning equipment to trace the outline or peripheral edge of the specimen boundaries. A small circle of light is displayed within the map of the specimen so that the pathologist at the diagnostic center knows the location of the field of view of the highly magnified image within the specimen. In a sense, the pathologist is operating in real time in the same way that he would use his own microscope at his diagnostic center except for a slight transmission delay used to transmit the video signals of the highly magnified image over large bandwidth channel. Although the pathologist has a small map or peripheral outline of the specimen, the pathologist's field of view of the actual specimen is only the small circle of view that is coming through the microscope objective lens. This does not help the pathologist locate suspicious areas of interest as in a prescreening of the entire tissue. The pathologist may switch to the lowest magnification to get the largest field of view of a small section of the specimen, but he never views the entire specimen at any magnification. Also, there is no image analysis quantitative testing from the received images at the diagnostic center; and no quantitative assaying is done with these images at the diagnostic center.
There is a particular interest today in using the Internet system because it is so readily accessible by users at a low cost and using a computer and viewing screen connected to the computer. One problem with trying to do any transmission of digitized, microscopic, highly magnified images over the Internet is that the bandwidth is too narrow to accommodate the tremendous amount of stored data which needs to be transmitted. There is a need for a system which would allow a pathologist or another person, to be able to perform tissue analysis or quantitative assays using a standard computer terminal from a location remote from the automated microscope.
SUMMARY OF THE INVENTION
In accordance with the present invention, a person such as a pathologist at a computer terminal may control an automated microscope to acquire on a computer screen or window images of the specimen at different magnifications as selected by the person. Further, the person may receive on the screen a low magnification, reconstructed image of the entire specimen to aid the person in interactively selecting points of interest on the specimen, such as along a basal layer of a tissue specimen.
More specifically, and in accordance with the present invention, the microscope's small field of view limitation of a specimen is overcome by providing to the viewer a reconstructed, digitized and magnified image of the entire specimen (or a very large portion of the specimen) for performing a visual analysis of the entire tissue in full color to aid in the selection of points of interest to be viewed at a higher magnification. This is achieved by acquiring a large number of low magnification images of the specimen through a microscopic scanning system, e.g. , 35 image tiles of the specimen at 1.25×, and then assembling and coordinating the tiles to form an overall, low magnified image of the specimen, i.e., a macro image of the specimen. Preferably, the digitized macro image is reduced in size by a software system to a smaller size, e.g., a ¼ size image that is displayed on a local screen or is sent over a low band width or a high band width channel to a remote screen. Thus, the pathologist not only does not need to have others do a slow laborious prescreening to locate suspicious areas for analysis or for viewing at high magnification, he can use his own experiences to go directly to the most suspicious areas which he sees on the macro image. He can, on a priority basis, do the most suspicious area first, followed by lower priority areas of interest.
In accordance with the present invention, there is provided a new and improved automated, computer-controlled microscope that displays the low magnification composite image of the specimen to allow the user to view and to interactively select points of interest, each of which may be displayed at high magnification. This is achieved by providing the user with a marker, such as a cursor or the like, to select the defined area of interest; and to acquire reproduced, spatially adjacent high magnification, digitized images of the selected area of interest. More specifically, the specimen, when it was first scanned at low magnification to provide a macro view of the specimen, the addresses or locations of the tile images and/or pixels for the composite image were acquired. Therefore, any selected region of interest in the macro image has locations to which the microscopic stage may be automatically repositioned under a changed, higher magnification lens to acquire higher magnification, digitized image tiles that can be assembled into a micro image. Herein, both the macro and micro images are formed of adjacent digitized image tiles which have been coordinated to reproduce spatially the original image that was scanned.
It is the high magnification images, usually at 40×, that were analyzed using image processing techniques as disclosed in the aforesaid patent application, to provide an assay or numerical histological data for the specimen.
In accordance with the preferred embodiment of the invention, the pathologist may select a larger region for analysis at high resolution than can be accommodated at this magnification on his high magnification viewing screen. He can, though, view all of the adjacent, highly magnified, digitized image tiles on this high magnification screen by scrolling up or down or right to left to shift these digitized, adjacent image tiles into view on the screen. Thus, even at a higher magnification of a region the pathologist is able to obtain a much larger view than the small field for the objective lens in use of adjacent tissue or cells to give him a broader, overall perspective of what is happening or what has happened in specific section of a specimen. For instance, a pathologist may want to see at high magnification and to assay at this high magnification, a suspicious area, the pathologist can draw a mark about the area and cause it to then be assayed and displayed.
By having displayed on a low magnification screen or split screen of the full composite area, by having the high magnification region being marked on the low image screen, and by having the region being scrolled to view adjacent high magnification images on the high magnification screen, the pathologist has available information to guide him in the sense of helping to navigate intelligently within the specimen in his search for cancerous tissue or the like for further inspection or measurement of malignant characteristics. Often, when restricted to a field of view of an objective microscope, the pathologist has a difficult time, in the words of a cliche, of seeing the forest for the trees. In the present invention, the pathologist has a full, magnified, reduced in size specimen view with the higher image area marked thereon to guide him and to let him see the forest. He can also see a region of the forest at higher magnification by scrolling adjacent tree images onto the high magnification screen.
In accordance with a further aspect of the invention, the user may elect to change to an intermediate magnification being viewed by either switching automatically to a new objective lens and acquiring new digitized image tiles at the intermediate magnification or by using software to reconstruct from the existing high and low magnification, digitized images a new intermediate digitized image.
The preferred, low magnification image, which is reduced in size, can be transmitted over narrow band width channels such as a local area network or over the Internet through various servers and computers. Likewise, the reconstructed, high magnification images can be transmitted over such narrow band width channels. Because the microscope is fully computer controlled, a pathologist or other person having a split screen computer such as a PC, can be connected to the microscope and operate it from a remote location to obtain the macro image and to navigate to points of interest and obtain the desire micro images. With the present invention, there is no need for a specialized microscope at each remote location nor for a broad band channel to send video signals in real time between the diagnostic center and the remote location.