|Publication number||USRE42715 E1|
|Application number||US 12/775,597|
|Publication date||Sep 20, 2011|
|Filing date||May 7, 2010|
|Priority date||May 19, 2004|
|Also published as||US7394937, US20050259867|
|Publication number||12775597, 775597, US RE42715 E1, US RE42715E1, US-E1-RE42715, USRE42715 E1, USRE42715E1|
|Inventors||Richard A. Sones|
|Original Assignee||Applied Vision Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (15), Non-Patent Citations (1), Referenced by (2), Classifications (21), Legal Events (2)|
|External Links: USPTO, USPTO Assignment, Espacenet|
U.S. patent application Ser. No. 10/404,027, filed on Apr. 1, 2003, is incorporated herein by reference in its entirety. Also, U.S. patent application Ser. No. 10/411,741, filed on Apr. 10, 2003, is incorporated herein by reference in its entirety.
Certain embodiments of the present invention relate to process monitoring and correction. More particularly, certain embodiments of the present invention relate to a vision system and method for monitoring a process in order to control certain characteristic parameters of monitored objects such as, for example, absolute color.
Manufacturers of products that are produced in high volume as part of a process using, for example, a process line, employ quality assurance methods to ensure that certain features of the product (e.g., color, pattern, alignment, texture) are consistent and match a production reference standard. For example, in the soda can industry, the patterns and colors on the outer surface of the cans should be monitored somehow as the cans proceed down a process line to ensure that the process of printing the outer surface of the cans does not result in out of tolerance conditions (e.g., color drift, pattern alignment drift, etc.). The product moving down a process line is often spatially oriented in a random manner along the process line. For example, soda cans having a specific pattern printed on the cylindrical outer surface are typically oriented randomly about the vertical axis of rotation of the predominantly cylindrical can.
These methods can be as simple as a production floor operator performing a set-up of a product run by making visual comparison of a finished set-up part to a standard reference chart or reference part. Based on this comparison the operator makes adjustments to the process. Then another set-up part is created and compared, more adjustments made until acceptable results are achieved, and the product run is initiated. This subjective method may lead to errors because of differences in the ambient light conditions, positions of the inspection light source, and differences in surface textures between the reference part and the finished part, different people conducting the comparisons, and other factors. While such a subjective comparison may be appropriate for some manufacturing processes, other more sophisticated processes (e.g., multi-color processes) may require more objective techniques.
Examples of such processes include package printing processes, soda can printing processes, and other processes which may employ more complex color schemes that are repeated or are placed next to each other in use. Besides merely color concerns, these complex color schemes may have spatial or pattern defects. A trained quality assurance color inspector using a standard illuminant may be able to catch many of these defects by using a subjective comparison with a standard reference part, however, many of such defects may not be discernible to the naked eye. In such applications, manufacturers have typically used a color densitometer, a tristimulus calorimeter, or a reflectance spectrophotometer to provide more precise color matching by utilizing colorimetry, discussed in more detail below.
The process of quantitative color analysis is generally referred to as colorimetry. Since the introduction of the CIE (Commission International de l'Eclairage) color measurement system in the early 1930's, many different measurement systems have been proposed for different applications. One such measurement system is the CIE XYZ color space. The CIE XYZ color space characterizes colors by a luminance parameter Y and two color coordinates X and Z which specify the point on the chromaticity diagram. The XYZ parameters are based on the spectral power distribution of the light emitted from a colored object and are factored by sensitivity curves which have been measured for the human eye. The human eye has three different types of color-sensitive cones. Accordingly, the XYZ functions were intended to correspond to the average sensitivity of the human eye and provide a device-independent representation of color. Therefore, the spectral responses of the XYZ functions are known as “tristimulus” functions and make up the coordinate system to quantify a color image or color space.
The apparent color of an object depends not only on its intrinsic spectral reflectivity, but also on the spectrum of the light used to illuminate it. The CIE also has defined a number of standard illuminants which are defined, theoretically, in terms of their spectral content. To completely specify the color of an object, one must measure the XYZ values of the light emanating from the object when it is illuminated by a standard illuminant.
Another CIE color space which is frequently used is the L*a*b* color space. The values of L*, a*, and b* are derived mathematically from the tristimulus values of X, Y, and Z:
where the values with the subscript “n” are found in published tables and correspond to a chosen standard illuminant. The value of L* is proportional to the brightness (luminosity) of the color. The value of a* describes the red/green composition of the color. The value of b* describes the yellow/blue composition of the color.
The goal of the L*a*b* color space is to provide a color space where the Euclidean distance between color 1 and color 2
is a “perceptually uniform” measure of the difference between color 1 and color 2. A value of ΔE=1 corresponds to a color difference which is very subtle—so subtle that it would take a trained color observer working under ideal lighting conditions to notice the difference. A value of ΔE=2 corresponds to a difference in color which is twice as noticeable as ΔE=1, and so on. The “perceptual distance” denoted by a given value of ΔE is intended to be independent of the location in color space (that is, independent of hue, saturation, and brightness), but this independence is actually only an approximation. Regardless, ΔE has been accepted in the color industry to quantify color differences.
As stated above, manufacturers typically have used a tristimulus colorimeter, a reflectance spectrophotometer, or a color densitometer to provide more precise color matching by utilizing one or more color measurement systems. These instruments provide quantitative and objective feedback, but are slow and inconvenient, and only measure color at one small spot (typically 5 mm in diameter) at a time, making it inconvenient to impossible to use them to compare all the colors on a complex multi-color pattern. In addition, these devices tend to be expensive due to the manufacturing care necessary to construct a device capable of providing precise color measurements suitable for laboratory use. These disadvantages make these devices particularly unsuitable for the production floor for use in process control.
Another disadvantage with densitometers is that they do not provide absolute color metrics (such as XYZ tristimulous values). Instead, they report the overall reflectivity of a surface for red, green, and blue light. Color densitometers are only suited for relative (as opposed to absolute) measurements. These relative measurements are often sufficient when the goal is simply to determine if the color on one object “matches” the color on another object.
Therefore there remains a need in the art for a fast and convenient way to efficiently monitor a production process with respect to a standard reference, where the production objects being monitored may have a random spatial orientation, at least around one axis.
Further limitations and disadvantages of conventional, traditional, and proposed approaches will become apparent to one of skill in the art, through comparison of such systems and methods with the present invention as set forth in the remainder of the present application with reference to the drawings.
An embodiment of the present invention comprises a method for monitoring a process. The method includes generating a set of reference images of at least one reference object using a vision training system. Each reference image of the set of reference images corresponds to a unique and precise spatial orientation of the reference object with respect to the training system. The method further includes acquiring one monitored image from each of a plurality of randomly oriented monitored objects over a period of time using a process monitoring system. Each of the plurality of monitored objects is substantially identical to the reference object. However, there may be differences from the reference object due to process variations. The method also includes comparing at least one acquired monitored image of the plurality of monitored objects to the set of reference images of the reference object to form at least one comparison image of comparison values. The method further includes generating at least one process deviation value in response to the comparing step.
Another embodiment of the present invention comprises a vision training system for characterizing a reference object. The training system includes a rotatable staging platform to stage a reference object and a source of illumination to illuminate the reference object on the staging platform. The training system further includes a color camera to collect a set of reference images of the reference object as the reference object rotates on the rotatable staging platform. Each reference image of the set of reference images corresponds to a unique and precise rotational position of the reference object with respect to the color camera. The training system also includes a computer-based platform connected to the color camera to store and process the set of reference images.
A further embodiment of the present invention comprises a vision monitoring system for monitoring a process. The monitoring system includes a source of illumination positioned to illuminate objects to be monitored as the objects move along a process line. The monitoring system further includes a color camera positioned on the process line to capture at least one image from each illuminated object, forming a plurality of monitored images, as each illuminated object passes through a field-of-view of the color camera. The monitoring system also includes a computer-based platform storing a set of reference images and being connected to the color camera to store the plurality of monitored images and to generate at least one process deviation value by comparing at least one monitored image of the plurality of monitored images to the set of reference images.
These and other advantages and novel features of the present invention, as well as details of an illustrated embodiment thereof, will be more fully understood from the following description and drawings.
The training system 100 also comprises an imaging assembly 120 which includes a color camera 121 and a source of illumination 122. The imaging assembly 120 is attachable to the rotatable staging platform 110 such that, as the reference object 111 rotates, the reference object 111 is within the field of view 125 of the color camera 121 and is illuminated by the source of illumination 122. In accordance with an embodiment of the present invention, the source of illumination 122 comprises an array of light emitting diodes (LEDs) having a white light spectrum. Such a white light spectrum is useful for determining the true colors of the reference object.
The training system 100 also includes a computer based-platform 130 connected to the color camera 121 in order to store and process image data of the reference object 111 collected by the color camera 121. In accordance with an embodiment of the present invention, the computer-based platform 130 comprises a standard, commercial, off-the-shelf personal computer running a general purpose operating system. However, the computer-based platform 130 also includes image processing software tools which may be commercially available and/or customized software. In accordance with an embodiment of the present invention, the computer-based platform 130 also provides control signals to the color camera 121 in order to control certain functionality of the color camera 121 (e.g., image capture rate).
In accordance with an embodiment of the present invention, the color camera 121 outputs analog imaging signals and the computer-based platform 130 includes a frame grabber 131 to convert the analog imaging signals to frames of digital imaging data. In accordance with another embodiment of the present invention, the color camera 121 outputs digital imaging signals and the frame grabber 131 is not used. The color camera 121 comprises a three-color camera providing RGB (red, green, blue) color imaging signals. Examples of three-color cameras include: CCD video cameras—still cameras, CID video cameras—still cameras, etc. In accordance with an alternative embodiment of the present invention, the camera 121 comprises a gray scale or monochrome camera.
In use, the training system 100 captures color images of the reference object 111 (e.g., a largely cylindrical can with printed graphics and text on its outer surface) as the reference object 111 rotates on the turn-table 115. The reference object 111 represents an ideal standard of the objects to be monitored on a production process line.
In accordance with one embodiment of the present invention, sixteen reference images are collected from the reference object using the color camera 121 such that each of the sixteen reference images correspond to one unique, precise vertical segment or strip of the outer surface of the reference object 111. That is, each of the sixteen reference images correspond to a unique rotational position of the reference object 111 with respect to the field-of-view 125 of the color camera 121 as the reference object 111 is rotated about the axis 112. The sixteen images cover all 360 degrees about the circumference of the reference object 111. These reference images are transferred and digitally stored in the computer-based platform 130 as arrays of pixel data (e.g., RGB color values).
In accordance with an embodiment of the present invention, the reference images are processed by the computer-based platform 130 to correct for lighting non-uniformity due to variations in the source of illumination 122 and/or due to the spatial relationship between the source of illumination 122, the reference object 111, and the color camera 121. A white reference image may initially be used with a brightness reference strip to calibrate the training system 100 such that reference images of true reference objects (e.g., soda cans) can be subsequently corrected for lighting non-uniformity. The reference images will subsequently be used in a process monitoring system to compare monitored images from monitored objects on a processing line to the reference images in order to ensure process control.
The training system 100 may also include a user interface including a display which may be used by an operator to view images and to control the training system 100 via, for example, a menu-driven touch-screen display. The user interface connects to the computer-based platform 130.
In accordance with an embodiment of the present invention, the computer-based platform 620 comprises a standard, commercial, off-the-shelf personal computer (PC) running a general purpose operating system. However, the computer-based platform 620 also includes image processing software tools which may be commercially available and/or customized software. In accordance with an embodiment of the present invention, the computer-based platform 620 also provides control signals to the color camera 612 in order to control certain functionality of the color camera 612 (e.g., focusing).
In accordance with an embodiment of the present invention, the color camera 612 outputs analog imaging signals and the computer-based platform 620 includes a frame grabber 621 to convert the analog imaging signals to frames of digital imaging data. In accordance with another embodiment of the present invention, the color camera 612 outputs digital imaging signals and the frame grabber 621 is not used. The color camera 612 comprises a three-color camera providing RGB (red, green, blue) color imaging signals. In accordance with an alternative embodiment of the present invention, the camera 612 comprises a gray scale or monochrome camera.
The imaging assembly 610 is positioned (i.e., mounted) on a product processing line such that a portion of the objects (e.g., printed soda cans) moving past the imaging assembly 610 on the processing line and facing the color camera 612 may be imaged by the color camera 612 while being illuminated by the source of illumination 611 (e.g., an array of white LEDs). In accordance with an embodiment of the present invention, the imaging assembly 120 of
If the imaging part of the monitoring system 600 is different than the imaging part of the training system 100, then the methods employed in U.S. patent application Ser. No. 10/404,027, filed on Apr. 1, 2003, which is incorporated herein by reference in its entirety, may be used to convert and transfer the reference images from the training system 100 to the monitoring system 600.
The monitoring system 600 may also include a user interface including a display which may be used by an operator to view images and to control the monitoring system 600 via, for example, a menu-driven touch-screen display. The user interface connects to the computer-based platform 620.
In accordance with an alternative embodiment of the present invention, any given reference image corresponding to a particular spatial orientation (e.g., a precise rotational position on a turn-table with respect to a color camera) comprises a composite reference image which is a composite of a plurality of images, each being from a different reference object but having the same spatial orientation. For example, such a composite reference image could be a simple average of several images from different reference objects.
In accordance with an alternative embodiment of the present invention, only a subset of pixels corresponding to a region-of-interest (ROI) of the matched images may be compared, thus reducing the amount of computational operations required to complete the comparison, or simply to focus on a part of the object requiring more sensitive inspection. The ROI may comprise, for example, a disk-shape area, a square area, a rectangular area, or some other shaped area.
During monitoring, about ⅛th of the soda can is acquired as an image to increase the probability of determining a match since each reference image corresponds to 1/16 th of the soda can, in accordance with an embodiment of the present invention.
The characteristic parameter may comprise, for example, RGB color data for qualitative comparisons (e.g., looking for printing flaws on a soda can), or absolute colorimetric data such as XYZ color data or L*a*b* color data for true color comparisons. As a result, the comparison image of difference values may comprise, for example, ΔRΔGΔB values, ΔXΔYΔZ colorimetric values, ΔL*Δa*Δb* colorimetric values, or ΔE colorimetric values. Typically, for qualitative colorimetric comparisons, only selected regions-of-interest (ROI) of the images are compared to cut down on the amount of processing.
As an example, the pixel data may comprise RGB values from the three-color camera 612. These RGB values may be used directly to form the comparison image by subtracting the RGB pixel values of the matched reference image from the RGB pixel values of the matched monitored image. In this way, qualitative differences can be ascertained from the comparison image.
For colorimetric comparisons, RGB pixel values (corrected for lighting variations) are averaged and converted to XYZ or L*a*b* colorimetric values for a predetermined ROI of the matched pair of reference and monitored images. Multiple RGB pixels are averaged and then the colorimetric values are generated using a color transformation algorithm. The averaging helps reduce noise that is present in the original RGB data. As a result, a single colorimetric value is determined for the ROI of the reference image and a single colorimetric value is determined for the corresponding aligned ROI of the monitored image.
Next, the single colorimetric value for the predetermined region-of-interest (ROI) of the matched reference image is subtracted from the single colorimetric value for the corresponding aligned ROI of the matched monitored image, forming a ΔXΔYΔZ or ΔL*Δa*Δb* colorimetric difference value. This difference value is used for quantitative comparisons of absolute color in the ROI. As an alternative, the entire matched images can be averaged, converted to colorimetric values, and subtracted, instead of just the ROI. However, this requires more processing.
As an example,
In step 740, at least one process deviation signal is generated from the comparison image data. For example, the values ΔRΔGΔB of the comparison image 1000 of
Each ΔE value for each pixel is compared to a predetermined threshold. A count value is generated corresponding to the number of comparison image pixels whose ΔE values are greater than (or, alternatively, less than) the predetermined threshold. This count value is output as the process deviation signal 625 (see
In accordance with an embodiment of the present invention, the ΔRΔGΔB values are scaled such that comparison image difference values of zero correspond to a value of 128 on a RGB color scale of 0-255. As a result, both positive (greater than 128) and negative (less than 128) ΔRΔGΔB values are accommodated on the 0-255 RGB color scale, which represents 256 distinct color differences.
For quantitative colorimetric evaluation, ΔXΔYΔZ or ΔL*Δa*Δb* colorimetric data can be used to calculate a Euclidean distance for averaged pixels in a region of interest as, for example,
The pixels in a ROI are averaged, forming an average characteristic parameter value (e.g., average RGB color value), and converted to a L*a*b* colorimetric value, for example. This is done for both the reference image and the corresponding aligned monitored image. The difference is taken and constitutes a process deviation signal or value (e.g., ΔL*Δa*Δb* value) which can be used as an input to an adaptive process control system 630 to adjust color.
As a matter of practicality, not every soda can passing by the monitoring system 600 on the process line may be properly imaged (i.e., some images may be of poor quality and need to be discarded) or matched to a reference image (e.g., when the confidence measure is low). In such a process monitoring situation, it is not important to check and characterize every monitored object (e.g., every soda can). Instead, it is important to obtain a good sampling of the soda cans as they go by such that the process deviation signal 625 is monitored over time to make sure the process (e.g., color printing process of the soda cans) is not getting out of control. For example, in accordance with an embodiment of the present invention, the process deviation signal 625 may be a running average of the count value described above. Other process deviation value signals are possible as well, in accordance with various embodiments of the present invention.
When reference images are captured by the training system 100, the reference images are corrected for lighting non-uniformity by the correct color image tool 1110. Similarly, when monitored images are captured by the monitoring system 600, the monitored images are corrected for lighting non-uniformity by the correct color image tool 1110. As a pre-processing step to pattern location, the reference images and a monitored image to be compared may be converted from color to monochrome (e.g., to a gray scale pattern or a simple edge pattern), using the color-to-mono tool 1120 to make the pattern locator process simpler.
The pattern locator tool 1130 takes the monitored image and tries to match its pattern to that of one of the reference images.
The aligner tool 1140 is used to align the monitored image 1220 to the reference image 1210 based on the skew parameters (3.54, 7.11, −1.19°) calculated as part of the pattern matching process.
Whether doing qualitative comparisons on RGB data or quantitative comparisons on colorimetric data, an aligned image is used for comparison to the reference image or a region of interest of the reference image.
Alternatively, a comparison image 1410 of pixels is formed in step 733 of the method 700 by, for example, first converting the RGB pixel values of the matched pair of images to XYZ or L*a*b* colorimetric data using a color transformation algorithm. The resultant comparison image comprises ΔXΔYΔZ or ΔL*Δa*Δb* colorimetric data as previously described.
Also, a ROI of RGB data can be averaged for the reference image and the matched monitored image, converted to colorimetric data, and then subtracted to form a colorimetric difference value (i.e., a process deviation signal).
In practical applications, both a ΔRΔGΔB comparison image and a ΔXΔYΔZ or ΔL*Δa*Δb* comparison value, based on a region of interest (ROI), are generated. The ΔRΔGΔB comparison image is used for a qualitative assessment of the process and the ΔXΔYΔZ or ΔL*Δa*Δb* comparison value is used for quantitative assessment of color.
As described before, a process deviation signal 625 may be generated using thresholding and counting techniques, or other techniques as well, in accordance with various embodiments of the present invention. Again, the process deviation signal 625 may be used as an input to an adaptive process control system 630 to bring the process back into control. Alternatively, the process deviation signal may be used by an operator to manually adjust the process.
In summary, a method and system to monitor randomly oriented objects on a production process line are disclosed. A color camera is used initially to collect a set of reference images of at least one reference object. The reference images represent various spatial orientations of the reference object. The reference object serves as the standard for the process. The reference images are stored in a computer-based platform. The color camera is then used to capture images of monitored objects as the monitored objects pass by the color camera on a process line. The monitored objects may have a random spatial orientation with respect to the color camera as the monitored objects pass through the field-of-view of the color camera. The captured images of the monitored objects are processed by the computer-based platform and compared to the reference images in order to determine if certain characteristic parameters of the monitored objects have deviated from those same characteristic parameters of the reference object. If so, the process may be adjusted to correct for the deviations in order to bring the process back into tolerance.
While the invention has been described with reference to certain embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from its scope. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5245399||May 28, 1992||Sep 14, 1993||Ball Corporation||System for non-contact colored label identification and inspection and method therefor|
|US5544240||Jan 31, 1995||Aug 6, 1996||International Business Machines Corporation||Image processing apparatus with improved data conversion testing|
|US5754448||Jul 12, 1995||May 19, 1998||Minnesota Mining And Manufacturing Company||System and method for color characterization and transformation|
|US5818443||May 6, 1996||Oct 6, 1998||Cognex Corporation||Single step coarse registration and inspection of circular objects|
|US5835244||Oct 7, 1996||Nov 10, 1998||Linotype-Hell Ag||Method and apparatus for the conversion of color values|
|US5911003||Apr 26, 1996||Jun 8, 1999||Pressco Technology Inc.||Color pattern evaluation system for randomly oriented articles|
|US6134343 *||Apr 7, 1999||Oct 17, 2000||Cognex Corporation||System or method for detecting defect within a semi-opaque enclosure|
|US6249600||Nov 26, 1997||Jun 19, 2001||The Trustees Of Columbia University In The City Of New York||System and method for generation of a three-dimensional solid model|
|US6340976||Apr 15, 1998||Jan 22, 2002||Mitsubishi Denki Kabushiki Kaisha||Multivision system, color calibration method and display|
|US6459425||Aug 25, 1998||Oct 1, 2002||Richard A. Holub||System for automatic color calibration|
|US6501850||Jun 19, 1998||Dec 31, 2002||Eastman Kodak Company||Method for including traditional photographic calibration into digital color management|
|US6701001||Jun 20, 2000||Mar 2, 2004||Dunkley International, Inc.||Automated part sorting system|
|US7187472||Sep 3, 2002||Mar 6, 2007||Innolutions, Inc.||Active color control for a printing press|
|US20030179920||Mar 10, 2003||Sep 25, 2003||Intelligent Machine Concepts, L.L.C.||Inspection system for determining object orientation and defects|
|US20040247193 *||Mar 12, 2004||Dec 9, 2004||Qualtrough Paul Thomas||Method and apparatus for article inspection|
|1||Hardeberg, et al., "Multispectral image capture using a tunable filter," Ecole Nationale Superieure des Telecommunications (Paris, France).|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US9615862||Nov 20, 2015||Apr 11, 2017||Spinal Llc||Modular head inserter|
|US9649142||Mar 10, 2015||May 16, 2017||Spinal Llc||Modular head assembly|
|U.S. Classification||382/218, 382/159, 382/141|
|International Classification||G06K9/00, G06K9/68, G06K9/62|
|Cooperative Classification||G01J3/463, G01N21/8851, G01N21/8806, G06T7/001, G06T2207/30164, G01N21/909, G01N21/25, G01J3/46|
|European Classification||G01N21/88P, G06T7/00B1R, G01J3/46, G01N21/90Q, G01N21/88K, G01N21/25, G01J3/46D|
|Nov 21, 2011||FPAY||Fee payment|
Year of fee payment: 4
|Dec 28, 2015||FPAY||Fee payment|
Year of fee payment: 8