Multifunction Autofocus System and Method for Automated Microscopy
Multifunction autofocus for automated microscopy includes automatic coarse focusing of an automated microscope by reflective positioning, followed by automatic image-based autofocusing of the automated microscope performed in reference to a coarse focus position. In some aspects, the image-based autofocusing utilizes astigmatism in microscope optics for multi-planar image acquisition along a Z axis direction of a microscope. In some other aspects, the image-based autofocusing utilizes astigmatism in microscope optics in combination with chromatic aberration for multi-planar image acquisition along the Z axis direction.
Latest Sanford-Burnham Medical Research Institute Patents:
- METHODS FOR REVERSING HEPATIC STEATOSIS
- Method for improving digestive health
- Methods and compositions related to peptides and proteins with C-terminal elements
- Methods and compositions using peptides and proteins with C-terminal elements
- Compounds and methods for activating the apoptotic arm of the unfolded protein response
This application is a divisional of U.S. Ser. No. 13/317,727, filed Oct. 25, 2011, to be issued as U.S. Ser. No. 10/001,622 on Jun. 19, 2018. This application contains subject matter related to the subject matter of U.S. patent application Ser. No. 12/587,923, filed Oct. 14, 2009.
STATEMENT OF GOVERNMENT INTERESTThe invention or inventions described herein were made in part with United States government support under NIBIB Grant 5 R01 EB006200, NHGRI Grant 5 U54 HG003916, and NHGRI Grant HG005033-CDP3, all awarded by the National Institutes of Health. The United States government has certain rights in this invention.
BACKGROUNDThe field includes automated microscopy. More particularly, the field includes automatically focusing a microscope using a reflection from a reference surface positioned with respect to an objective of the microscope and information acquired from magnified images obtained by the microscope. In particular, the field includes a multifunction autofocus mechanism for automated microscopy that combines reflective positioning with use of the image characteristics obtained from a sequence of focal planes.
Automated microscopy refers to a microscope system including a microscope equipped with one or more motor-actuated components and a programmable control mechanization that governs the functions of the system and operates one or more motor-actuators to set and/or change operational conditions of the microscope. Such conditions include, for example, focus, illumination, and specimen position. The microscope includes an objective and a stage. In particular, but without limitation, the control mechanization operates to automatically focus the microscope and to scan a specimen via the microscope by generating control signals that cause relative movement between the objective and the stage. In some aspects, the stage is motorized and the control mechanization, using autofocus and positioning algorithms, processes images to generate control signals operative to cause the motorized stage to be moved and/or positioned in three dimensions with reference to the objective. See, for example, http://www.microscopyu.com/articles/livecellimaging/automaticmicroscope.html.
Image-based autofocus of a microscope uses a value obtained by transformation of a magnified image to indicate a degree of focus or sharpness. For example, but without limitation, presume an automated microscope stage supports a sample plate with one or more samples on a surface thereof. (The term “sample plate” is used herein to denote a microscope component such as a microscope slide, a multiwall plate, a microtiter plate, or equivalent). The stage is operated to place the sample plate at an X-Y location where the stage is moved vertically through a succession of vertical (Z) positions. At a first Z position, a magnified image is obtained from the sample, and the magnified image is transformed to obtain a first value of an image characteristic such as focus. A second value is obtained by the same transformation of another magnified image obtained at a second Z position. More values are obtained at additional vertical positions. The values are compared and an automated stage controller is operated to move the stage to a vertical position where the values indicate the best degree of focus is obtained. In automated scanning microscopy, best focus is maintained in this manner as the sample plate is scanned by systematically moving the stage in an X-Y scan pattern. See, for example, U.S. Pat. Nos. 5,548,661; 5,790,710; 5,790,710; 5,995,143; 6,640,014; and 6,839,469. Other mechanisms can be utilized to change focus to different focal planes of a specimen, including movement of the objective relative to the stage. If a camera is used for image acquisition, the camera can be moved relative to the objective, and/or optical elements between the camera and objective can be moved.
Reflective positioning autofocus does not use a magnified image obtained from a sample to determine best focus; rather, it uses a light signal that is generated by the microscope's optical system, projected through the objective toward the sample plate, and reflected off of a surface of the sample plate, back through the objective. A desired focus location for the sample is obtained by a value entered by a user that vertically offsets the focal plane from the Z position where the reflected light signal is in focus. See, for example, U.S. Pat. No. 7,071,451; see also the Nikon® perfect focus system description at http://www.microscopyu.com/tutorials/flash/focusdrift/perfectfocus/index.html.
Automated digital imaging with a high resolution microscope requires autofocus that performs with submicron accuracy and precision. Both image-based autofocus and reflective positioning exhibit shortcomings that compromise performance in virtually all modes of high resolution whole-slide and whole-microtiter plate automated imaging. Image-based autofocus does find a best focus location in a three-dimensional specimen, but it is slow. Image-based autofocus finds the best focus as indicated, for example, by a best resolution of the sample itself; but it can be fooled by empty fields of view and artifacts (e.g., dumps of cells or debris). It has been found that the speed of image-based autofocus can be increased by taking advantage of chromatic aberrations to increase the number of planes per image at which sharpness/resolution measurements are made simultaneously, thereby also reducing the complexity of the multi-planar imaging optics. See, in this regard, US 2010/0172020, published Jul. 8, 2010, for “Automated Scanning Cytometry using Chromatic Aberration for Multiplanar Image Acquisition”. Reflective positioning is designed to speed up total internal reflective (TIRF) microscopy where the goal is to find a single image plane—the surface at which the image is created. However, it cannot find the best focal planes of cells and tissue sections that inevitably reside at different distances from the reflective surface. Thus, although reflective positioning is fast, it requires an estimate of the axial offset from the reflecting surface that can exhibit substantial variability during a scan, thereby often deviating from best focus.
The combination of automated microscopy with image cytometry has become an essential biological research tool that is incorporated into a wide variety of applications. Such applications include: high content screening (HCS) instruments use in compound (drug discovery and toxicity) and genomic (RNAi and cDNA) automated screening, where libraries of hundreds to tens of thousands can be screened by individual investigators and whole genomes and compound libraries of hundreds of thousands can be screened by large laboratories; high throughput microscopy (HTM) where screening per se is not necessarily the experimental goal; and image (or scanning) cytometers where the emphasis is on automated spatially dependent measurements of large numbers of cells as a superset of the measurements acquired by flow cytometry instruments. The latter application tends to be used more in analysis of tissue sections and the former more often for cell monolayers, but the principles are analogous. In the last decade, HCS/automated image cytometry has grown rapidly not only for large screening efforts but also for routine use in individual labs enabling automated parallel analysis of from one to a few 96-, 384-well microtiter plates where months or years of experiments can be distilled into a few weeks (aka, HC analysis or HCA). Advances in automatic acquisition, measurement, comparison and pattern classification tools for cellular images continue to add sophisticated new automation for mining statistically relevant quantitative data from the inherently heterogeneous cell populations.
Obtaining a sharply focused image automatically is a key requirement for automated microscopy. It is therefore desirable to provide autofocus systems and methods that improve the speed and image sharpness of an automated microscopy system while still providing a best focus for each magnified image acquired.
Measurement of Image-Based Autofocus AccuracyBest focus of a microscope is the Z (axial) position with the best resolution of an image. Autofocus error is, in turn, defined by an axial distance from best focus and a degree of defocus is reported as a percent loss in resolution.
To find best focus as a standard against which to compare autofocus methods, measurements of the slope of the change in brightness of the images of a “knife edge” were made on through-focus stacks of reflection images centered on the sharp edge between 100% transmitting and 100% reflective regions on a microscope resolution target. The derivatives of 35 lines perpendicular to the knife edge and near the center of the through-focus stacks of the knife edge were fit with Gaussians. The resolution was computed as 2.77*(mean of standard deviations (“SDs”) of the 35 Gaussian fits of each optical section), which, in some respects, corresponds to translating this experimental knife-edge resolution measurement to the theoretical Rayleigh resolution criterion. The results for two different 20× objectives (0.75 NA 20× Nikon® Plan Apo VC and Nikon® 0.5 NA 20× Plan Fluor) are plotted in
In
-
- 1. Through-focus resolution measurements can objectively identify best focus,
- 2. Image-based autofocus is accurate (resolution losses of 0.2-6% are likely within resolution measurement noise—see also Table 2); it is a resolution measure that is designed to be insensitive to contrast reversals (See Bravo-Zanoguera, M. E., Laris, C. A., Nguyen, L. K., Oliva, M. & Price, J. H. Dynamic autofocus for continuous-scanning time-delay-and-integration image acquisition in automated microscopy. Journal of Biomedical Optics 12, 34011/34011-34016 (2007)),
- 3. Decreased sampling (1.0× instead of 1.5× tube lens) resulted in decreased peak resolution without always decreasing the sensitivity of resolution to defocus far from focus,
- 4. Lower NA objectives have both lower maximum resolution and greater depth of field than objectives with higher NA.
This approach provides a way to grade the severity of reflective positioning and autofocus errors. A standard criterion to define “out of focus” for an image detector is the depth of field obtained per Equation (1):
where λ0 is the wavelength of light, n is the index of refraction of the immersion medium, NA is the numerical aperture, M is the magnification, and e is smallest distance that can be resolved by the detector (From Inoue, S. & Spring K. R., Video Microscopy, The Fundamentals, (Plenum, New York, 1997). The depth of field and corresponding focus error data are summarized in Table 2.
For comparison with detection of defocus by eye, see the enlarged through-focus sequences of DAPI-labeled NIH-3T3 cells in
Both image-based autofocus and reflective positioning exhibit limitations that compromise performance for some fields of view encountered in both whole-slide and whole-microtiter plate scanning. One solution, as in the PerkinElmer Opera) imaging reader, is to remove some of the out-of-focus information with confocal optics; but this is an expensive approach that may also be less optimal than imaging at best focus. Reflective positioning can take <0.1 s and autofocus can take as long as 1-2 s using previous technology. Reflective positioning locates a surface and collects the image at a preset user-defined offset from the surface.
Autofocus is an important component of microscope automation: if autofocus is poor, the primary output—the image, as well as the data derived from it—are compromised. As opposed to macroscopic digital imaging (photography), where depths of field even with close-up low F/# lenses are measured in millimeters, the microscopic depth of field with a 20× 0.75 NA objective and 0.52 NA condenser is 1.23 μm with a 1.0× tube lens (see Table 2). This makes microscopy autofocus substantially more challenging than autofocus in macroscopic photography. At about NA 0.5 and higher, depths of field are often comparable to the thicknesses of commonly cultured, tightly adherent cells (rounded, less adherent cells, such as T-lymphocyte cell lines, exceed the depth of field at even lower NAs). Many reflective systems sense only the first surface, and most specimen holders (coverslips, slides, and membrane-, glass-, and plastic-bottom microtiter plates) vary in thickness. Reflective system characteristics include: absolute (open loop) displacement sensors, error-minimizing (closed loop) positioning, and through the lens (TTL) and non-TTL (or beside the lens, BTL) configurations. By observing the digital images produced by reflective positioning HCS instruments as they scan, out-of-focus images are visible to the eye. Image-based autofocus optimizes the quality, sharpness, and or resolution of the image itself depending on the algorithm. But autofocus can also be compromised by a poor specimen (e.g., thick piles of dead cells or debris and lint or fibers occupying large axial ranges), out-of-focus contrast-generating artifacts (e.g., cells, debris, scratches, condensation, dried water droplets and lint on the other surfaces of the specimen holder—coverglass, slide or plate bottom), and by mismatches in image sampling that alter the system modulation transfer function and violate the assumptions about the native resolution of the image upon which the sharpness measurements rely. There are many other things that can compromise image quality, including not properly aligning the condenser and lamp. Relative merits of the two approaches are summarized in Table 3.
We observed that with cell monolayers cultured on a coverslip, reflective positioning can provide reasonably focused images, whereas with cells cultured on the slide or tissue sections mounted on the slide, the degree of defocus appeared greater.
Image-based autofocus, on the other hand, finds the sharpest image directly from a stack of images. For example, autofocus functions based on image resolution find the axial position where the most cells are in focus if the tissue is thicker than the depth of field. (Bravo-Zanoguera, M. E., Lads, C. A., Nguyen, L. K., Oliva, M. & Price, J. H. Dynamic autofocus for continuous-scanning time-delay-and-integration image acquisition in automated microscopy. Journal of Biomedical Optics 12, 34011/34011-34016 (2007)) However, if the field is blank or if there is debris out of the normal plane of focus, autofocus can lose track of focus.
To understand these errors systematically, we used both image-based autofocus and reflective positioning to scan three specimens:
1) NIH 3T3 cells cultured on a coverslip,
2) A histological section of human prostate cancer, and
3) Adipocytes differentiated from 3T3-L1 cells cultured in a 96-well microtiter plate
A test of the differences between image-based autofocus and reflective positioning on a prostate cancer tissue section using the same experimental setup are shown in
For all three specimens (
NIH-3T3 cells are adherent and flat, and are thus expected to provide a straightforward test for reflective positioning with an offset. Nevertheless, over 20% of the images are out of focus as judged by a 50% loss in resolution and 40% are out of focus as judged by the depth of field criterion.
In the examples of
These problems are solved by an auto-focusing sequence in which a reflective positioning procedure is performed to obtain an initial “focus guess” (or coarse focus), followed by an image-based autofocus procedure that obtains a fine focus with reference to the initial focus. Desirably, this sequence increases image quality and the robustness of autofocus. Because it integrates multiple focus functions, such a sequence is referred to as a “multifunction autofocus sequence”. An autofocus method including a multifunction autofocus sequence is a multifunction autofocus method, and a mechanism that enables an automated microscopy system to perform such a sequence is referred to as a “multifunction autofocus mechanism.” An automated microscopy system equipped with a multifunction autofocus mechanism has a multifunction autofocus construction and performs a multifunction autofocus method.
In an objective, the depth of field decreases inversely proportional to NA2 (Eq. 1) and fluorescence brightness increases proportional to NA4 and inversely proportional to magnification2. (Inoue, S. & Spring, K. R. Video Microscopy, The Fundamentals, (Plenum, New York, 1997). In high content assays and screens (HCA/S), high resolution is not always required. But large scale primary HCS motivates scanning as fast as images of sufficient quality can be collected. So the ability to use the highest NA objective at a given magnification is important. Instruments that cannot reliably provide in-focus images, however, motivate use of low NA objectives with greater depth of field to obtain consistent measurements, which collect less light, thereby requiring longer camera integration times and slowing screening. By providing accurate autofocus on every field, the multifunction autofocus sequence enables use of the best possible set of optics for each imaging application.
In some aspects it is desirable to exploit chromatic aberration in microscope optics wherein light of different colors is focused to different points along the Z axis due to the wavelength-dependence of the refractive indices of lenses. The dispersion of the focal planes for the different colors enables multi-planar image acquisition along the Z axis. In some other aspects, the number of color-dependent focal planes is increased by introduction of astigmatism into the microscope optics to create at least two different focal planes for each color.
Preferably, a multifunction autofocus system for automated microscopy includes a coarse focusing mechanism equipped with a reflective positioning system, in combination with a fine focusing mechanism equipped with an image-based autofocus system. In some aspects, the image-based autofocus system is equipped with a chromatic aberration optical mechanism in combination with an astigmatic optical mechanism for multi-planar image acquisition along the Z axis.
Preferably, a multifunction autofocus method for automated microscopy includes a coarse focusing procedure employing reflective positioning step, in combination with a fine focusing procedure employing image-based autofocus. In some aspects, an image-based autofocus procedure uses a plurality of images along the Z axis that have been generated by chromatic aberration in combination with astigmatism.
An automated microscopy system equipped with a multifunction autofocus system having a multifunction autofocus construction and performing a multifunction autofocus method is now described with reference to certain preferred embodiments. Although the embodiments are set forth in terms of specific constructions, these constructions are intended to illustrate principles of multifunction autofocus, but are not intended to limit the application of those principles to the examples shown.
Multi-Focal-Plane Z-Positional Optics:
Image-based autofocus includes the established principle of placing camera sensors at various Z-axial positions to simultaneously collect images on an automated microscope at different focal planes. At any X-Y scan location, a specimen is imaged at a plurality of Z positions, thereby providing images of varying sharpness that provide information useful for determining a Z position providing a best-focused image. For example, U.S. Pat. No. 4,636,051 describes imaging three focal planes—a final focus position, one in front of it and one behind it—to perform image-based autofocus. U.S. Pat. No. 6,640,014 describes a multi-focal plane system based on fiber optic imaging bundles that collects nine different focal planes simultaneously for image-based autofocus in continuous scanning. This design scans images by using time-delay-and-integrate (TDI) cameras to image a constantly-moving specimen, but is fragile and difficult to calibrate. U.S. Pat. No. 6,839,469 describes a system based on beam-splitters that divide the light into eight paths to relay the images to eight different CCD cameras that are axially positioned by micrometers. This system maintains autofocus during continuous scanning and is easier to calibrate and use than the system described in U.S. Pat. No. 6,640,014. Prabhat et al. (Simultaneous imaging of several focal planes in fluorescence microscopy for the study of cellular dynamics in 3D, Proc. SPIE 6090, 60900L-60901-60907 (2006)) use a 4-camera multi-focal-plane imaging system to image fast events in live cells in 3D, including endosome dynamics to exocytosis at the membrane by direct imaging. While each of these implementations accomplishes the goal of imaging multiple focal planes simultaneously, they are cumbersome to implement, relatively expensive and the substantial glass and long light paths can degrade image quality.
Multi-Planar Image Acquisition Using Chromatic Aberration:
Another image-based autofocus method uses chromatic aberration to calculate a best focus position from images acquired at a set of Z-axially displaced focal planes. This method is described in cross-referenced U.S. patent application Ser. No. 12/587,923, now published as US 2010/0172020, which is incorporated herein by reference. (Counterpart PCT application PCT/US2009/005633 has been published as WO/US2010/044870). According to this method, light of different colors is focused to different points along the Z axis due to the wavelength-dependence of the refractive indices (dispersions) of lenses.
Chromatic aberration (or achromatism) is the focusing of light of different colors to different points due to the wavelength-dependence of the refractive indices (dispersions) of lenses. In this regard, see the diagrams of
US 2010/0172020 describes a use of chromatic aberration to image multiple focal planes and speed up autofocus. According to the publication, differences in foci are controlled by planar glass samples inserted into the optical path in front of a CCD camera to alter chromatic aberration. In focus function calculations, the differences in best foci for each color with planar glass samples inserted were 0.35 mm (B), 0.9 mm (R), and 0.6 mm (G). To better detail chromatic aberrations, we have used the OneLight Spectra™ lamp (http://www.onelightcorp.com) for computer control of and mapped chromatic aberration for different optics.
Autofocus Performance Using Chromatic Aberration:
We performed two autofocus experiments with NIH 3T3 cells cultured on a coverglass, using methods described in Bravo-Zanoguera, M., Massenbach, B., Kellner, A. & Price, J. High-performance autofocus circuit for biological microscopy, Review of Scientific Instruments 69, 3966-3977 (1998), via incremental scanning with a 3-chip RGB camera operating at 30 fps and the relay lens of
Analogous experiments were carried out with an IMAGINGSOURCE® DB21AF04 firewire Bayer camera at 60 fps. The same experimental parameters were used, except the sample was imaged at 3 different mechanical positions 0.75 μm apart for a total sampling range of 6 μm, and empty fields containing <4% of the image content as measured by the upper half of the spatial frequency spectrum were eliminated. Two scans yielded combined SDs of 22.9 nm and 25.2 nm, respectively.
Oblique (Reflected) White Light Illumination for Autofocus on Microtiter Plates:
In some aspects, transmitted illumination using phase contrast of unstained cells performs better than fluorescence microscopy for autofocus. But the meniscus in the well of a microtiter plate acts as a lens and degrades the phase image to such an extent that it cannot be used for autofocus. Further, in the pharmaceutical industry, drug discovery screening often involves covering each plate with an opaque film. For these reasons, HCS instruments using image-based autofocus typically focus in fluorescence.
Doubling the Number of Focal Planes Using Astigmatism:
An optical system with astigmatism is one where rays that propagate in two perpendicular planes have different foci (see
Refer now to
The data of
Accordingly, a multifunction autofocus system or method for automated microscopy makes multi-planar image sharpness/resolution measurements using multi-planar image acquisition equipped with astigmatic aberration, in combination with reflective positioning to reduce the complexity of automated microscopy without adversely impacting its optical performance.
Implementations of Multifunction Autofocus:
Various implementations of multifunction autofocus include, without limitation, the following combinations:
Reflective positioning for coarse focus, in combination with image-based autofocus for fine focus as described elsewhere (examples include: Groen F C A, Young I T. Ligthart G: A comparison of different focus functions for use in autofocus algorithms. Cytometry 6:81-91, 1985; J H Price, D A Gough, “Comparison of Digital Autofocus Functions for Phase-Contrast and Fluorescence Scanning Microscopy,” Cytometry, 16(4): 283-297, 1994; M Bravo-Zanoguera, B von Massenbach, A L Kellner and J H Price, “High Performance Autofocus Circuit for Biological Microscopy,” Review of Scientific Instruments, 69(11):3966-3977, 1998);
Reflective positioning for coarse focus, in combination with image-based autofocus using chromatic aberration for fine focus based on multi-planar image acquisition using the lens configuration of
Reflective positioning for coarse focus, in combination with image-based autofocus using astigmatic aberration for fine focus based on multi-planar image acquisition using the lens configuration of
Reflective positioning for coarse focus, in combination with image-based autofocus using chromatic and astigmatic aberrations for fine focus based on multi-planar image acquisition using the lens configuration of
Reflective positioning for coarse focus, in combination with image-based autofocus using chromatic aberration for fine focus based on multi-planar image acquisition using the optical configuration of
Reflective positioning for coarse focus, in combination with image-based autofocus using chromatic and astigmatic aberrations for fine focus based on multi-planar image acquisition using the lens configuration of
The nine-color configuration of
To transmit the light for image-based autofocus measurements in a multifunction autofocus for fluorescent microscopy, the wavelengths are chosen to be in the emission bands of the fluorescent colors so that they are passed through the microscope to one or more autofocus cameras. In one construction, a strobed illuminator is used to perform multiplane image acquisition for autofocus when a scientific grade imaging camera is not collecting the fluorescent image. When autofocus is complete (the fluorescent camera is in focus), the fluorescent light source is turned on (either electronically or via a shutter). There are various implementations that split out the autofocus light to the autofocus cameras and pass the fluorescent light to the fluorescent camera(s). For example, a dichroic mirror can be used to reflect the autofocus wavelengths and transmit the fluorescence wavelengths, or a movable mirror can be used. A “scraper” (
A dichroic mirror can be used to reflect light out of the light path for autofocus. To avoid light losses when using the dichroic mirror, a movable mirror can be used that swings into place to reflect light for autofocus and swings out of the way to allow unimpeded transmission to the scientific camera(s) that acquire images for cytometric analysis. Alternatively, a “scraper” mirror can be used that images a field of view adjacent to the field of view imaged by the scientific camera(s). The scraper mirror location is shown in a diagram of the microscope optical system of
Representative Embodiment of Multifunction Autofocus:
A representative embodiment of an automated microscopy system 200 with multifunction autofocus is shown in
In
As per
Continuing with the description of
With further reference to
With reference to
With further reference to
As per
Astigmatic aberration is introduced into light reaching the CCD cameras 245a and 245b by the cylindrical lens 297. Preferably, the cylindrical lens 297 is one of a piano-convex, piano-concave, biconvex, or biconcave lens. The effect of the astigmatism is to cause light received by the multi-planar optics to propagate differently in two perpendicular (X and Y) planes to at least one, but preferably to both of the CCD cameras 245a and 245b.
Chromatic aberration is introduced into light reaching the CCD cameras by lens relationships corresponding to the lens relationship illustrated in
An alternative embodiment of the second optical system 240 illustrated in
Illumination for the second optical system 240 of
As per
The principles of reflective dark field illumination are basically the same as in transmissive dark field illumination except the light now comes from the same direction. Accordingly, the beam splitter 292 is used to divide the illumination and imaging paths; for this, we prefer a 45/55 (50/50 ideal, if available) pellicle beam splitter. It is desirable to minimize optics on the objective side of this beam splitter because back reflections off of optical elements can interfere with the imaging. We prefer the pellicle beam splitter because both plate and cube beam splitters have back reflections and ghosting that interferes with the imaging and alignment of masks/stops.
External Fine Focus:
With reference to
Multifunction Autofocus Operations:
Image-based autofocus involves determining a focus index, and, from the focus index, selecting the point, usually the maximum, that corresponds to best focus. Preferably, the focus index f(z) is determined by calculating the relative amount of mid to high frequency content present in the image of the sample for various z-positions of the sample. In this regard, see, for example, Bravo-Zanoguera, M. E., Lars, C. A., Nguyen, L. K., Oliva, M. & Price, J. H. Dynamic autofocus for continuous-scanning time-delay-and-integration image acquisition in automated microscopy. Journal of Biomedical Optics 12, 34011/34011-34016 (2007). A power-weighted average is then used to calculate best focus
where, for example, α=8 is used as the weight for best focus here. The calculation of the focus index is achieved by using a band pass convolution filter h(x) along the horizontal dimension x of each image iz(x, y) at each axial position z
Calibration:
In previous work, a 31-tap bandpass filter was designed that worked well for many samples given that the camera was sampling at 1.45 times the Nyquist cutoff frequency of the imaging system. (Oliva, M. A., Bravo-Zanoguera, M. & Price, J. H. Filtering out contrast reversals for microscopy autofocus. Applied Optics 38, 638-646 (1999)). But since the choice of objective, relay lens, and cameras creates different magnifications and sampling for different experimental setups, we implement a fast calibration step that includes optimization of the upper frequency band for the sharpness measurements. First, the power spectrum is calculated across one dimension of an image, and the mid to high frequencies are summed to give a focus index (fz):
Where fy,n is the power spectrum of each line of the image along one dimension, N is the number of pixels in the x-direction, and wn is the weighting used for each frequency n (in this example wn=1 for all n).
Using the power spectrum of each line, we expand on the principal concept of summing the mid to high frequencies by implementing rapid tuning of the filtering characteristics for different samples and optical configurations. The frequencies used in the sum of Eq. 4 are fine tuned differently than the upper-half of the frequency spectrum for each optical setup based upon two criteria: 1. the focus function is unimodal (with only one peak); and, 2. the mathematically determined focus agrees with the best focus determined by eye.
The power weighted averaging is the same as in Eq. 2. This technique allows tuning of the filter conditions rapidly for different samples, objectives, magnifications, and pixel size of the camera. This is especially convenient when a change in objective or camera results in a substantially different sampling rate relative to the Nyquist frequency.
In order to utilize the information from different colors to determine best focus a calibration is performed to relate the focus information from the different colors relative to one another. In this regard, see
The shift is the subtraction of the minimum and the scale is division by the maximum minus minimum. The shift and scale for different colors can have largely different values, thus without some form of calibration, a larger focus index value at a given sample z position for one color does not necessarily imply it is in better focus than another color.
This calibration is carried out for each focal plane produced by astigmatism (fx and fy in
To calibrate for a multi-field scan, a full through-focus function curve is acquired for the three wavelengths at the first field. Then the shift and scale from these curves are applied to the focus indices for the corresponding wavelengths in all subsequent fields. Similarly, the calibration also serves to determine the relative focal shift of the different colors. This calibration makes three assumptions: 1. the content of each field is comparable to the calibration field; 2. the image content at each wavelength is proportional to the number of cells being imaged; and, 3. the shape of each focus function curve at each wavelength is similar.
With respect to the second and third conditions, if the information in a given wavelength channel is not strongly correlated with the information in another channel, the calibration can fail, in which case chromatic aberration cannot be used to find focus in subsequent fields. This could occur if the sample is inhomogeneous in a manner that creates wavelength-dependent absorptions and those inhomogeneities vary substantially from field-to-field.
During the scan, occasional fields are encountered that contained few or no cells to focus on. If the sum of the high frequencies used in Eq. 8 is <4% of the sum of the high frequencies in the best-focus image of the calibration field, the field is skipped and marked as blank. This also prevents the scan from deviating too far from the contour of focus positions on the slide when encountering empty fields.
Multifunction Autofocus Methods:
Per
(1) Obtaining coarse focus. The reflective positioning system 230 is activated to obtain a degree of coarse focus with respect to a reflected object observed through the objective 203. For example the object can be reflected from a surface of the specimen carrier 212 such as the lower surface 214s of the coverslip 214, which is immediately adjacent the specimen 202. In operation, the illumination source for the reflective positioning system 230 is turned on, and the RP optics forms an image from the light via the objective 203. Using light reflected back from the surface 214s, through the objective 203, to the RP optics, the Z drive is operated to move stage 207 along the optical axis until the reflected mage is in focus. The control of the Z drive is closed loop and this focusing step is performed without contribution from the image-based autofocus system 240.
(2) Moving the stage 207 via X-Y scanning to a new field of view of the specimen.
(3) Obtaining magnified images for fine focus. The image-based autofocus system 240 is operated to determine a best focus with respect to one or more sequences of magnified images of the specimen 202 obtained from a stack of focal planes along the optical axis, including the focal plane at the coarse-focus Z-position determined by the reflective positioning system 230. In this regard, autofocus illumination is initiated. For example, the reflective dark field illumination system 310 of
(4) Obtaining fine focus. The sequence of magnified images is acquired and provided to the autofocus programming 261 of
(5) Control of the stage's position along the optical axis is assumed by a fine focus function of the autofocus programming 261, and
(6) fine focus is obtained by moving the stage 207 along the optical axis from the coarse focus position to the calculated best focus position, and the calculated best focus position replaces the coarse focus position. Presuming fluorescence imaging, for each fluorescence channel, the stage 207 is moved along the optical axis to a calculated best focus value for the channel, the microscope filter/illumination setting is changed for the channel and, (7) a fluorescence image of the specimen is obtained from the channel via the imaging system 220. Alternatively, the coarse focus position is left alone and external fine focus is used to carry out fine focus for each fluorescent camera; a separate EFF system for each fluorescent camera utilizes a pre-calibrated offset to correct for differences in focus due to chromatic aberration or differences in axial position of the respective cellular components. Thus, with EFF, fine focusing operates independently of coarse focusing, and each EFF/filter/camera combination is adjusted in parallel to speed multi-channel (multi-color) fluorescence image acquisition.
(8) If the scan is not complete, the stage 207 is moved to the next field of view and the preceding sequence is repeated. Otherwise, if there is not another specimen, the procedure exits. Or else if there is another specimen, load and commence the preceding sequence.
With reference to
Per
(370, 371) Calibration values are initialized per equations (4) and (5). Preferably, but not necessarily, raw (unprocessed) focus values are obtained from a “through-focus z-stack”, that is to say, a sequence of magnified images from a sequence of focal planes extending along the optical axis and including a preselected focus position. The focus position can be selected, for example by the eye of a user. A representative through-focus z-stack relative to the preselected focus position is: −925, −800, −675, −550, −475, −400, −325, −275, −250, −225, −200, −192, −184, . . . (steps of 8) . . . , 184, 192, 200, 225, 250, 275, 325, 400, 475, 550, 675, 800, 925. When these steps are expressed in microscope z-position units of 25 nm/unit, the total through-focus Z-stack is ˜46 um. These positions are merely for example and were chosen for a 5 um tissue section. The wide range provided an accurate estimate of a baseline and the smallest steps of 8 units=0.2 um were chosen based upon the width of each focus function. Smaller ranges and smaller steps could be used for well-plates. For the fluorescent mode example, while the through-focus Z-stack is collected, the focus index is calculated for each color channel and each axis. Thus, for example, with 2 CCD cameras for autofocus, 3 colors, and 2 astigmatism axes, 12 focus functions are calculated for the stacks. The focus functions are the raw focus functions used for calibration according to equation (6).
(372) Reflective positioning is enabled and coarse focus is acquired; it is assumed that this places the stage close to where best focus is. Image-based autofocus using chromatic and astigmatic aberrations provides a correction to this position. Upon a subsequent activation of reflective positioning (such as when a new specimen is placed on the stage or the stage is moved to a new X-Y field), a certain amount of time may be required to settle the reflective positioning system.
(373) Presuming a scan mode operation, the stage is moved to a new X-Y field.
(374) At the new field, the dark field illumination is strobed and a single image is acquired on each autofocus camera 245a and 245b.
(375) For each color from each camera calculate the focus index for both astigmatism axes. Following the example of
(376) Calibrate the focal indices using calibration information obtained at 370, 371. For example, using equation (6), for each calibrated focus index, shift and scale the raw (uncalibrated) value according to the values recorded during the calibration initialization through-focus z-series. This should be f=(f_raw−shift)/scale, where the shift and scale are unique for each wavelength, camera, and axis.
(377) Check the observed field for image content. In this regard, if none of the focus indices are greater than a focus index threshold value (for example, 0.1, in which there is less than 10% of the image information in the focus spectral band compared to the calibration field) the field is deemed to contain low content and, at 380, the focus defaults to the reflective positioning (coarse) focus position, and the reflective positioning system is disabled. Fluorescent images are acquired at predetermined offsets from the coarse focus position, the reflective positioning system is re-enabled (372), and the stage is moved to the next field position (373). If at least one focus index is greater than 0.1, a new best focus is calculated and the stage is moved along the optical axis to the best focus location (378), and the coarse function value is updated to the best focus value while the method proceeds through 380, 381, et seq. It should be noted that the focus index threshold value of 0.1 is arbitrary; other values may be determined by trial and error, or by other methods.
With alternative external fine focus (EFF) 223, which operates independently of coarse focus, step 380 (Coarse focus is disabled) is skipped and fine focus is carried out by the EFF system 223, and EFF is adjusted for each fluorescent channel (color). If the field is deemed to contain low content, EFF is not used for fine focus, rather only to adjust for the precalibrated differences in focus between the various fluorescent channels.
Industrial Application:
Referring to
The open frame microscope is intended for use with a broad range of commercially-available, or custom-built, automated microscopy systems. For purposes of illustration, we describe an open frame microscope designed with reference to a commercially-available Nikon® CFI60 optical system (“the microscope system”), although this is not intended to limit the application of the principles to be discussed and illustrated. In the microscope system, objectives are coupled with a tube lens thereby establishing a common optical design that minimizes aberrations while obtaining diffraction-limited performance. All objectives are parfocal, with the front focal plane located 60 mm from the rear shoulder of the objective. There is an “infinity” corrected space between the objective and the tube lens. This enables components, particularly angle-sensitive components such as dichroic mirrors and interference filters, to be placed in this space. Vignetting caused by the aperture of the tube lens is negligible if the distance between the objective and the tube lens is less than 200 mm. The tube lens (Nikon® MXA22018) creates an image 200 mm displaced from its rear principal plane. A rectangular aperture is placed in this plane to define the object field-of-view and to reduce stray light.
The image created by the tube lens is relayed to the three-color TDI (time delay integration) camera array and to the autofocus system using a Schneider® Macro Varon™ MVR4.5/85 relay lens (85 mm focal length, f/4.5). The TDI camera is used for continuous scanning as previously described, for example in U.S. Pat. No. 6,839,469. For incremental scanning, the TDI camera is replaced with a 3-color 2D scientific grade camera. The MVR4.5/85 was originally designed for ultra-high resolution industrial line scan applications; in fact, its performance is well-suited for microscopy. The measured polychromatic modulation transfer function indicates that it performs well as a relay lens. We modeled the polychromatic MTF (modulation transfer function) of the MVR4.5/85 at four different field locations (0.0 mm, 17.2 mm, 25.2 mm, 31.0 mm) for a magnification (β) of −1.5×. Near diffraction-limited performance was obtained, with modulation indices of better than 0.25 for spatial frequencies of 100 cycles per mm The modulation indices were modeled as a function of transverse field locations for several different spatial frequencies (10, 20, 30, 40, 50, 60 cycles per mm) and similar excellent near-diffraction limited results were obtained. The fluorescence excitation light from the LDLS (laser diode light source) is delivered via relay optics mounted on the open-frame system as shown in
Preferably, the objective is a Nikon® CFI Plan Apochromat VC 20× NA0.75 lens. In tests using a Nikon® Ti-E inverted microscope we obtain diffraction-limited performance with extremely low chromatic aberration (see
Mechanical framework of the open frame microscope. As per
A microtiter plate is scanned over the objective using a motorized stage or microtiter plate robotic. The framework for the external casing for environmental containment is fabricated using extruded 6105-T5 aluminum framing (80/20 Inc., T-slot framing). Residual thermal drift of the mechanical and opto-mechanical systems are compensated by the chromatic aberration image-based autofocus.
The open frame design has several options in addition to the conventional motorized stage typically used on a microscope such as the Nikon Ti-E inverted microscope for specimen scanning. For example, two robotic alternatives, an Aerotech® gantry robot or a Denso® SCARA robot with voice coil actuators for both length-of-well continuous scanning. The SCARA robot can move well-well and the feedback-controlled voice coil provides precision-velocity scanning across each well. Fine focus will serve three functions: plate loader, plate scanner, and objective selector. For axial specimen positioning (focus), alternatively a piezoelectric microtiter plate axial positioner can be integrated into the chosen plate translation robot. The objective and filter cube are kinematically located using a combination of precision ball bearings and precision rods and fixed using high-strength NdFeB magnets. In this regard,
An aberration autofocus (AAF) unit (comprised of 240 in
The position sensing system for the multifunction autofocus element of an automated microscopy system according to this specification can be provided as a TTL configuration (see, for example, the Nikon® perfect focus reflective positioning system), or it can be in a BTL configuration as shown in
Increasing scanning speed is important where larger numbers of samples are to be screened. For incremental stage motion, scanning speed can be increased by carrying out steps in parallel as demonstrated in a scanning process that includes:
-
- 1. Calibrate the reflective sensing position (TTL or BTL) to the best focus of the image acquisition camera.
- 2. Place the first sample on the microscope manually or robotically.
- 3. Define the scanning area(s) or load them from a file for scanning.
- 4. Calibrate the multi-planar image-based autofocus system as described above and defined by equations (4)-(6).
- 5 Define the scanning area(s) or load them from a file for scanning.
- 6. Initiate (or continue) automatic scanning, which includes:
- a. Initiate the stage movement to the next field of view (BTL-profile the surface via reflective position sensing while moving and record the profile).
- b. Coarse focus: Move the Z-position (focus) to the position recorded by the reflective position sensing device (BTL) or closed-loop RP to adjust focus while moving to the field of view (TTL).
- c. Stop at the field of view and obtain a sequence of images for autofocus.
- d. Calculate best focus using equations (2) and (3).
- e. Move to best focus (fine focus).
- f. Acquire the in focus image and:
- i. Save the image to the hard drive and
- ii. Update the reflective positioning offset position for the next field of view to compensate for changing thickness of the substrate and specimen.
- g. If scan is not complete:
- i. Repeat 5.
- h. else if scan is complete:
- i. If there is not another specimen, exit the scanning program,
- ii. Else signal the robot to unload the specimen and load the next specimen, begin moving to the first field of view, and go to 5
- 7. Perform image analysis either after scanning or in parallel as the images are acquired and stored.
Alternatively, a look-ahead image-based autofocus system can be used with BTL or TTL reflective positioning. With the scraper mirror optical configuration of
-
- 1. Calibrate the reflective sensing position to the best focus of the image acquisition camera.
- 2. Place the first sample on the stage manually or robotically.
- 3. Define the scanning area(s) or load them from a file for scanning.
- 4. Calibrate the image-based autofocus system as described above and defined by equations (4)-(6).
- 5. Perform (or continue) automatic scanning, which includes:
- a. If the next field of view is not adjacent to the previous field of view (e.g., true for the first field of view of the specimen, the beginning of a scanning row or the beginning of a new well):
- i. Initiate stage movement to the lateral position just before the field of view (the position where the autofocus cameras are centered in the field of view), profile the surface via reflective position sensing while moving and record the profile.
- ii. Move the axial position (focus) to the position processed from the reflective position sensing device while moving to the first field of view (look ahead).
- iii. Complete the lateral and axial moves of 5.a.i and 5.a.ii.
- iv. Strobe to produce a sequence of images on the autofocus cameras and carry out the following in parallel:
- 1. Initiate lateral stage movement to the imaging field of view.
- 2. Download the sequence of images from the autofocus cameras to the controller and
- a. Then calculate best focus using equations (2) and (3), and
- b. Then initiate axial movement to the final best focus position.
- 3. Complete the lateral and axial moves.
- 4. Strobe to produce a sequence of images on the autofocus cameras
- v. Initiate acquisition of the in-focus image and
- 1. Download the sequence of images from the autofocus cameras,
- a. Then calculate best focus using equations (2) and (3), and
- b. Store the best focus position.
- vi. Complete image acquisition of the in focus image and:
- 1. Save the image to the hard drive and
- 2. Update the reflective positioning offset position for the next field of view to compensate for changing thickness of the specimen holder and specimen.
- b. Else if the next field of view is adjacent to the current field of view:
- i. Initiate stage movement to the next field of view
- ii. Initiate axial movement to the best focus position
- iii. Complete the lateral stage and axial focus moves.
- iv. Strobe to produce a sequence of images of the adjacent field of view on the autofocus cameras.
- v. Initiate acquisition of the in focus image.
- vi. Download the sequence of images from the autofocus cameras,
- 1. Then calculate best focus using equations (2) and (3), and
- 2. Then store the best focus position.
- vii. Complete image acquisition of the in focus image and:
- 1. Save the image to the hard drive and
- 2. Update the reflective positioning offset position for the next field of view to compensate for changing thickness of the substrate and specimen.
- c. If scan is not complete:
- i. Repeat 5.
- d. Else if scan is complete:
- i. If there is not another specimen, exit the scanning program,
- ii. Else signal the robot to unload the specimen and load the next specimen, and
- a. Repeat 5.
- a. If the next field of view is not adjacent to the previous field of view (e.g., true for the first field of view of the specimen, the beginning of a scanning row or the beginning of a new well):
- 6. Perform image analysis either after scanning or in parallel as the images are acquired and stored.
Although autofocus systems and methods for automated microscopy have been described with reference to preferred embodiments, constructions, and procedures, it should be understood that various modifications can be made without departing from the spirit of the principles represented thereby. Accordingly, the scope of patent protection afforded those principles is limited only by the following claims.
Claims
1. An automated microscopy system for simultaneously measuring image characteristics of a plurality of focal planes in a magnified image with an image-based autofocus system, in which the image-based autofocus system includes at least one optical pathway to transfer light from which the magnified image is formed and the optical pathway includes an astigmatic optical device that introduces astigmatism into the light in the at least one optical pathway.
2. The automated microscopy system of claim 1, in which the astigmatic optical device includes a cylindrical lens.
3. The automated microscopy system of claim 1, in which the astigmatic optical device includes two half cylindrical lenses.
4. The automated microscopy system of claim 1, in which the image-based autofocus system further includes means for calculating a best focus based on the image characteristics.
5. The automated microscopy system of claim 1, in which the optical pathway introduces chromatic aberration into the light that causes different colors of light to be focused at different focal planes.
6. The automated microscopy system of claim 5, in which the astigmatic optical device is a cylindrical lens.
7. The automated microscopy system of claim 5, in which the astigmatic optical device includes two half cylindrical lenses.
8. The automated microscopy system of claim 1, in which the image-based autofocus system acquires a sequence of magnified images simultaneously and each magnified image is acquired from a respective focal plane of the plurality of focal planes, and a controller calculates image characteristics for each magnified image and determines a best focus from the image characteristics.
9. The automated microscopy system of claim 1, in which the image-based autofocus system acquires a sequence of magnified images simultaneously and each magnified image is acquired from a respective focal plane of the plurality of focal planes, and a controller calculates a focus index for autofocus.
10. The automated microscopy system of claim 1, in which the image-based autofocus system acquires a sequence of magnified images simultaneously and each magnified image is of a particular color of light, each color of light provides an image focused at a different focal plane due to chromatic aberration, and a controller calculates image characteristics from more than one focal plane for each image.
11. The automated microscopy system of claim 1, in which the optical pathway transfers the light to respective multi-planar optical subsystems, each including a camera, the cameras are positioned at different focal planes, and a beam splitter divides the light between the multi-planar optical subsystems.
12. The automated microscopy system of claim 11, in which each camera generates a sequence of image characteristics for the plurality of focal planes and a controller calculates a focus index for each of the plurality of focal planes using the image characteristics.
13. The automated microscopy system of claim 1, in which the plurality of focal planes are distributed along an optical axis, and the image-based autofocus system simultaneously acquires a plurality of magnified images at the plurality of focal planes.
14. The automated microscopy system of claim 1, including a means for estimating a coarse focus corresponding to a predetermined distance from an object surface, a means for estimating a fine focus position corresponding to a focal plane of the plurality of focal planes, and a programmable controller for combining the two estimates to provide a best focus position and adjusting focus to the best focus position.
15. The automated microscopy system of claim 14, including means for scanning a specimen and multifunction autofocus means for updating the best focus position with respect to the specimen at each scan location of a plurality of scan locations.
16. The automated microscopy system of claim 15, including means for making measurements of the specimen.
17. A method of focusing an automated microscopy system including an objective, a stage, and an optical axis passing through the objective and the stage, by:
- obtaining a coarse focus value with a reflective positioning system;
- obtaining a coarse focus corresponding to the coarse focus value;
- obtaining a fine focus value with an image-based autofocus system;
- updating the coarse focus value with the fine focus value; and,
- causing relative movement between the objective and the stage to obtain a fine focus corresponding to the updated coarse focus value.
18. The method of claim 17 in which obtaining a fine focus value with an image-based autofocus system includes directing light from a specimen to the Image-based autofocus system, introducing astigmatism into the light when the light is transferred to the image-based autofocus system, simultaneously generating a stack of focal plane images of the specimen from the light and obtaining a best focus value from the stack, wherein the best focus value is the fine focus value.
19. The method of claim 18, in which simultaneously generating a stack of focal plane images of the specimen from the light includes introducing chromatic aberration into the light along with the astigmatism.
20. The method of claim 18, in which directing light from a specimen to the Image-based autofocus system includes directing oblique or dark field reflected illumination through the objective to the specimen, directing the oblique or dark field illumination scattered by the specimen through the objective to the image-based autofocus system.
21. The method of claim 20, in which simultaneously generating a stack of focal plane images of the specimen from the light includes introducing chromatic aberration into the dark field illumination scattered by the specimen.
22. The method of claim 21, in which obtaining a coarse focus value with a reflective positioning system includes sensing a reflected image through one of the objective and a sensor beside the objective.
23. A method of operating an automated microscopy system including an objective, a stage, and an optical axis passing through the objective and the stage, by the automated-microscopy-system-executed steps of:
- obtaining a coarse focus position in response to an image reflected from an object plane;
- and then, adjusting the coarse focus position by a best focus value associated with a focal plane of a stack of focal planes extending along the optical axis simultaneously in an image by a means that introduces astigmatism into an optical system; followed by,
- positioning one of the objective, the stage, and an external fine focus mechanism to a focus position corresponding to the best focus value.
24. The method of claim 23 performed during the automated-microscopy-system-executed X-Y scanning of the stage.
25. The method of claim 24 in which the X-Y scanning is one of incremental and continuous scanning.
26. The method of claim 25, a respective image is obtained from each focal plane of the stack of focal planes, further including the automated-microscopy-system-executed steps of:
- obtaining a focus index for each respective image;
- calibrate each focus index;
- determining presence of image content for each calibrated focus index; and,
- calculating the best focus if sufficient image content is present for at least one focus index.
Type: Application
Filed: Jun 18, 2018
Publication Date: Oct 25, 2018
Applicants: Sanford-Burnham Medical Research Institute (La Jolla, CA), Vala Sciences, Inc. (San Diego, CA)
Inventors: Jeffrey H. Price (San Diego, CA), Derek N. Fuller (San Diego, CA), Albert L. Kellner (San Diego, CA), Behrad Azimi (San Diego, CA)
Application Number: 16/011,610