|Publication number||US4486842 A|
|Application number||US 06/347,080|
|Publication date||Dec 4, 1984|
|Filing date||Feb 9, 1982|
|Priority date||Feb 12, 1981|
|Also published as||EP0059121A1|
|Publication number||06347080, 347080, US 4486842 A, US 4486842A, US-A-4486842, US4486842 A, US4486842A|
|Original Assignee||Regie Nationale Des Usines Renault|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (12), Referenced by (21), Classifications (9), Legal Events (5)|
|External Links: USPTO, USPTO Assignment, Espacenet|
1. Field of the Invention
This invention concerns an apparatus as well as a procedure for locating three-dimensional objects packed in bulk for purposes of controlling a gripping terminal.
2. Description of the Prior Art
High-speed automatic feeding of an assembly line with three-dimensional parts which are picked up in succession by a robot from a container in which they are packed in bulk and jumbled together is complicated by the problems of the robot in recognizing and grasping the bulk-packed parts one by one with a high rate of success.
The solutions proposed previously involve the use of complex devices generally supported by powerful computing devices.
It has therefore been suggested that a topological survey be carried out of the protruberant areas at the surface of a bulk load, using methods involving the analysis of stereoscopic views or by projecting a luminous grid and analyzing the trace thereof, also by a stereoscopic procedure, in order to reconstruct the relief of the surface of the bulk load.
These methods, however, have the drawback of requiring complicated computations and calling for equipment which is difficult to deploy in an industrial setting.
Accordingly, the objects of this invention are to provide a novel device which overcomes the above-mentioned drawbacks and which in particular, makes it possible to process the data on a real time basis using instrumentation which has become conventional, simple algorithms which may be rapidly executed, and equipment suited for industrial application.
These and other objects are achieved according to the invention by producing a new and improved apparatus for locating three-dimensional objects packed in bulk for purposes of controlling a gripping terminal, including at least one optical projection system associated with drive mechanisms commanded by an interface connected to a central processing unit, a light sensor such as a TV camera plumb with the surface, for example a horizontal surface, receiving the bulk load and connected to processing and analysis devices of the central processing unit, wherein the optical system provides a substantially planar light bundle which sweeps the surface of the bulk load thanks to drive mechanisms which cause the angle θ between the bundle and the aforesaid surface to vary, and the value of that angle, measured by a coder, is transmitted to the central processing unit, which analyzes the image of the trace of the light bundle sensed by the camera and has processing circuits which select the point(s) whose values correspond to predetermined conditions.
In accordance with a first embodiment of the invention, the camera sweep is carried out perpendicularly to the axis of rotation of the projection system, and, for each sweep line xi, the analysis devices note the ordinate yi with respect to a plane (usually horizontal) reference of its intersection with the image of the bundle trace, this for each value [θj].
In a second embodiment of the invention, the camera sweep is parallel to the axis of rotation of the projection system and, for each value θj, the analysis devices note the ordinate yi of the first illuminated sweep line and the abscissa xi of the first illuminated point of the trace image, this with respect to a plane (usually horizontal) reference.
Yet another feature of the invention resides in the provision of processing devices which calculate the value (zi) of the points (xi, yi) on the basis of the θj values transmitted by the angular coder.
The invention further includes a procedure for locating three-dimensional objects packed in bulk for purposes of controlling a gripping terminal using an apparatus including at least one optical system connected with drive mechanisms commanded by an interface connected to a central processing unit, a light sensor such as a television camera plumb with a reference surface, a horizontal surface for example, receiving the bulk load and connected to the processing and analysis devices of the central processing unit, including delivering by means of the optical system a plane light bundle at an angle θ with respect to the reference surface, causing the optical system to rotate so as to cause the bundle to sweep the surface of the bulk load, analyzing the light trace from the light bundle picked up by the camera, and by using processing mechanisms, selecting the point(s) of the bulk load whose values correspond to predetermined conditions.
Furthermore, the procedure according to the invention includes drawing up, with respect to a reference, the cartesian coordinates (xi, yi, zi) of the points on the surface of the bulk load, this on the basis of the angular values θj and the coordinates (xi, yi) of the points of intersection of the sweep line and the first illuminated point of the image of the luminous trace picked up by the camera.
A more complete appreciation of the invention and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein:
FIG. 1 is a schematic block diagram illustrating the geometry of the invention;
FIG. 2 is a block diagram of a locating device according to the invention;
FIG. 3 is a schematic diagram of a second embodiment of the invention;
FIG. 4 is a flow chart of the sampling cycle of the apparatus of the invention;
FIG. 5 is a flow chart of the operating cycle of the apparatus represented in FIG. 2;
FIG. 6 is a flow chart of the operating cycle of the apparatus represented in FIG. 3; and
FIG. 7 is a schematic diagram of an embodiment for implementing the optical system of the apparatus of the invention.
Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, and more particularly to FIG. 1 thereof, there is shown a basic block diagram illustrating the procedure according to the invention. As shown in FIG. 1 a crate 1 containing the three-dimensional bulkloaded parts 3 intended to be grasped by the gripping terminal of a robot is situated on a plane (e.g., horizontal) surface with trace P in such a way as to be swept is by a substantially planar light bundle 5. Light bundle 5 has a width which is sufficiently great to cover the side 7 adjacent to the bulk load. The bundle is provided with an optical projection system 9 placed above a certain distance to the left of the crate 1 along a longitudinal axis x' oriented, for greater simplicity, parallel to the side 7 and connected to a three-dimensional orthogonal reference with horizontal axis y parallel to the trace P and vertical axis z directed upwards.
A fixed television camera 11 is mounted plumb with the center of the bulk load at a height such that its field covers the complete surface of the bulk load and in such a way that pictures are not affected by parallax effects. The camera is connected on the one hand to a TV monitor 13 and, on the other hand, to a central processing unit 15 which consists principally of analysis and processing circuits.
With horizontal plane x'y, the plane light bundle 5 makes an angle θ which is made variable by rotating the projection system about its axis x'. The light bundle 5 produces a trace on the bulk load, which is an irregular curve because of the fact that the surface of the bulk load is not planar. The resultant is picked up by the television camera and is shown on the screen of the monitor in the form of an irregular curve 17, continuous or otherwise, which is analyzed in horizontal plane x'y, for example, in the following manner:
If the line sweep is carried out in direction y, for each sweep line xi note is taken of the ordinate yi with respect to a reference plane xy of the point of intersection B of the respective line xi and of the dark/light edge of the image of the trace, for which the coordinates with respect to xyz are (xi,yi,zi) where the figure zi is defined by the relationship zi=-(yi+a)·tan θj. The origin of the ordinates coincides with the edge of the field of the TV camera and (-a) is the ordinate of the light source.
By varying θ, for each value θj it is possible to record the coordinates of points (xi, yi, zi), so as to retain after later processing with all these points only those points for which the zi values correspond to a predetermined condition, in particular, those for which the zi values are between two given figures H and (H-h), where H is generally the highest figure for the bulk load and h is a predetermined parameter; the points thus selected then constitute the image of the protruberant part of the bulk load.
The procedure according to the invention makes it possible to simplify considerably the image analysis by eliminating the objects situated at low points, which are generally of little interest anyway inasmuch as they are inaccessible to the gripping terminal of the robot.
FIGS. 2 and 7 schematically represent an apparatus whereby it is possible to implement the procedure according to the invention. In FIG. 2, there is shown the crate 1 of bulk loaded items above which is situated an optical projection system 9 for forming a planar light bundle 5. As shown in FIG. 7, optical projection system 9 includes a light source 19, for example a rectilinear filament incandescent lamp situated at or near the focal point S of a cylindrical elliptical mirror 21. Mirror 21 deines a focus point F at which is located an aperture 23. Light from the light source 19 is reflected off the mirror 21 and passed through the aperture 23 and to a lens 25. Lens 25 gives off a planar light bundle 5 whose width is substantially equal to that of the side 7 of the bulk load it sweeps; and providing substantially more illumination than ambient lighting, with the sweeping provided thanks to a drive motor 27 for rotating the projection system about axis x'. The foregoing, as well as an angular coder 29, is connected to command (interface) circuits tied into the central processing unit 15.
If the bulk load is contained in a crate with opaque walls, it is necessary to use a second optical system 31 on ordinate b, symmetrical to the first system with respect to the optical axis of the camera, with the two systems in turn each lighting half the surface of the bulk load and thereby overcoming any problems associated with shadows from the crate edges; in this case, figure zi is then given by the formula zi=-(b-yi)·tan θj with respect to the second system, which also has its own drive motor and angular coder, both of which are connected to the command (interface) circuits and hence to the central processing unit.
As described previously, the fixed television camera 11 is mounted plumb with the center of the bulk load at a height such that its field of view covers the complete surface of that load. The camera is connected on the one hand to the TV monitor 13 and, on the other hand, to the central processing unit, which inter alia contains electronic circuits for controlling the synchronization of the TV camera, the amplification and analog/digital conversion of the video signal, the validation of the measurements as a function of the motor command, and the listing of the values xi, yi and θj as well as the processing mechanisms.
The command (interface) circuits induce the motor(s) to rotate the light bundle(s) between θ1 and θN about the axis from the light source, with a speed of rotation on the order of 5 degress per second so that the sweep would produce 250 images for an angular range of about 30 degrees, for the case where the image of the bulk load is within a square measuring 250×250 pixels.
For each value of θj (θ1, θN), the device records the plane coordinates (xi, yi) and calculates the value of zi--(yi+a)·tan θj of the image points of the bundle trace; determination of the cartesian coordinates (xi, yi, zi) is followed by a processing phase which consists of retaining only those points for which zi corresponds to a predetermined condition.
The flow chart in FIG. 4 represents the sampling operation of the device by virtue of calculation of parameter "a" defined above, while the flow chart in FIG. 5 describes the image analysis operating cycle and the operating cycle for later processing so as to retain in memory only those points for which zi is located between two values H and (H-h).
To carry out the sampling, the bulk load is replaced by a horizontal plane at Ho, and after using a leveling staff to determine the relationship between the coordinates of the points within the scene in the field of the camera and the xy values are recorded for the different angular positions θj in the interval (θ1, θN) so as subsequently to calculate the average value of "a" on the basis of a set of equations Ho=-(yj+a)·tan θj.
The operating cycle represented in FIG. 5 begins at θ1 for j=0 followed by issuing a command to the motor of the projection system positioning θ over θj for j=j+1, following which TV analysis is initiated over 250 lines with the triplet (xi, yi, θj) being listed for each line for each value of θ between θ1 and θN. The central processing unit then computes the Zk=-(yi+a)·tan θj, finds the value of the highest point in the bulk load Zk Max=H, and then retains only those points whose values are situated in the interval between H and (H-h), where h is a predetermined parameter. In this flow chart, P designates the number of triplets (xyz) and K the current value of P.
In the event heights H and H-h are known in advance, as in the case of a semi-arranged bulk load, acquisition is simplified and is limited to the area defined by
a-H/tan θj≦Y≦-a-(H-h)/tan θj·
This considerably simplifies the image analysis by eliminating the objects situated in the low spots, which are generally of little interest as they are out of reach of the robot's gripping terminal.
Analysis and processing of the image defined above, carried out using standard technical means, make it possible to determine the location and orientation of the objects situated at the surface of the bulk load for purposes of controlling the gripping terminal of an industrial robot designed to grasp them, and to do so at a high rate of speed.
The apparatus and the operational mode described above also make it possible to determine, for example with respect to reference ox yz, the coordinates xi, yi, zi of the highest point on the surface of the bulk load.
However, this determination may be simplified by adopting the embodiment of the device represented in FIG. 3, in which the camera is turned 90 degrees so that the sweep lines are parallel to x. Indeed, supposing yi is the ordinate of the first line illuminated, when θ=θj, and xi the abscissa of the first point illuminated; and supposing the coordinates point (xi, yi) is the highest of all those illuminated at that time and its value, computed according to the formula zi-a-(yi+a)·tan θj, is compared to the highest figure previously encountered in the course of that sweep; if the value is higher, the triplet (xi, yi, zi) is entered in the memory and the previous triplet is erased. At this point, the following value of θ according to the operating cycle defined by the flow chart of FIG. 6 is checked, so that by the end of the processing of one sweep of the bundle between θ1 and θN, it is possible to retain in the memory only the triplet XYH corresponding to the three coordinates of the highest point.
This FIG. 3 embodiment has the advantage of requiring only a very small storage memory, and its purpose is to identify the highest point in the bulk load for purposes of using a simple gripping terminal to grasp the corresponding area, in particular an electromagnet, grab or suction mechanism.
Of course, the means of implementing the procedure have been provided by way of nonrestrictive example only, and may be replaced by equivalent means; accordingly, the optical system described may be replaced by a laser beam. Moreover, the operating cycles defined by FIGS. 4, 5 and 6, readily understood by a specialist, have not been described in minute detail but nevertheless are an integral part of this description. Finally, xi, yi, zi may be calculated with respect to any other reference.
Obviously, numerous additional modifications and variations of the present invention are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims, the invention may be praticed otherwise than as specifically described herein.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US3804270 *||Jan 12, 1973||Apr 16, 1974||Bendix Corp||Bulk storage and automatic presenter system with image producing means|
|US4017721 *||May 16, 1974||Apr 12, 1977||The Bendix Corporation||Method and apparatus for determining the position of a body|
|US4120403 *||Jun 28, 1976||Oct 17, 1978||George E. Lauer||Photoelectric apparatus for sorting variegated articles according to size|
|US4146924 *||Sep 22, 1975||Mar 27, 1979||Board Of Regents For Education Of The State Of Rhode Island||System for visually determining position in space and/or orientation in space and apparatus employing same|
|US4188544 *||Aug 22, 1977||Feb 12, 1980||Weyerhaeuser Company||Method and apparatus for automatically processing a workpiece employing calibrated scanning|
|US4254433 *||Jun 25, 1979||Mar 3, 1981||General Motors Corporation||Visual motion tracking system|
|US4305130 *||May 29, 1979||Dec 8, 1981||University Of Rhode Island||Apparatus and method to enable a robot with vision to acquire, orient and transport workpieces|
|US4315771 *||Jan 30, 1980||Feb 16, 1982||Institut De Recherches De La Siderurgie Francaise||Process to continuously determine the profile of a charge fed into a blast furnace|
|US4318081 *||Dec 19, 1979||Mar 2, 1982||Hajime Industries Ltd.||Object inspection system|
|US4330779 *||Sep 8, 1980||May 18, 1982||The Bendix Corporation||Display analyzer having angular degrees of freedom|
|US4402053 *||Sep 25, 1980||Aug 30, 1983||Board Of Regents For Education For The State Of Rhode Island||Estimating workpiece pose using the feature points method|
|GB2018422A *||Title not available|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US4573073 *||Jun 2, 1983||Feb 25, 1986||General Electric Company||Integrated lighting and camera system|
|US4611292 *||Sep 29, 1983||Sep 9, 1986||Hitachi, Ltd.||Robot vision system|
|US4613269 *||Feb 28, 1984||Sep 23, 1986||Object Recognition Systems, Inc.||Robotic acquisition of objects by means including histogram techniques|
|US4731853 *||Mar 21, 1985||Mar 15, 1988||Hitachi, Ltd.||Three-dimensional vision system|
|US4744713 *||May 21, 1986||May 17, 1988||Texas Instruments Incorporated||Misalignment sensor for a wafer feeder assembly|
|US4748571 *||Jan 20, 1987||May 31, 1988||Fmc Corporation||Line-up vision system|
|US4753357 *||Dec 15, 1986||Jun 28, 1988||Ishikawajima-Harima Heavy Industries Co., Ltd.||Container crane|
|US4786848 *||Jul 27, 1987||Nov 22, 1988||Davidson Textron Inc.||Water jet trim head simulator|
|US4813125 *||Jun 8, 1987||Mar 21, 1989||Utica Enterprises, Inc.||Method and apparatus for establishing the position of a datum reference from an object having dimensional variations within a tolerance range|
|US4870336 *||Sep 16, 1988||Sep 26, 1989||Davidson Textron Inc.||Water jet trim head simulator|
|US4956775 *||Oct 1, 1985||Sep 11, 1990||Klamer R B||Object sensor for detecting characteristics such as color for games|
|US5033011 *||Aug 25, 1989||Jul 16, 1991||Kabushiki Kaisha Toshiba||Apparatus and method of controlling a refuse disposal crane|
|US5129010 *||Jun 15, 1990||Jul 7, 1992||Kabushiki Kaisha Toyoto Chuo Kenkyusho||System for measuring shapes and dimensions of gaps and flushnesses on three dimensional surfaces of objects|
|US6334773 *||Feb 24, 1998||Jan 1, 2002||Dentronic Ab||Method and arrangement for making artificial teeth|
|US7342668 *||Sep 17, 2004||Mar 11, 2008||D4D Technologies, Llc||High speed multiple line three-dimensional digitalization|
|US9302391 *||Dec 6, 2011||Apr 5, 2016||Canon Kabushiki Kaisha||Object gripping apparatus, method of controlling the same and storage medium|
|US20080253612 *||Oct 16, 2006||Oct 16, 2008||Morphic Technologies Aktiebolag||Method and an Arrangement for Locating and Picking Up Objects From a Carrier|
|US20120158180 *||Dec 6, 2011||Jun 21, 2012||Canon Kabushiki Kaisha||Object gripping apparatus, method of controlling the same and storage medium|
|DE102011084353A1||Oct 12, 2011||Apr 18, 2013||Trumpf Laser- Und Systemtechnik Gmbh||Method for setting up gripper robot for transport of series of workpiece to processing apparatus, involves calibrating gripper robot arranged sensor based on deviations of actual position sensor on defined reference position|
|DE102011084353B4 *||Oct 12, 2011||May 7, 2015||Trumpf Laser- Und Systemtechnik Gmbh||Verfahren und Computerprogramm zum Einrichten eines Greifroboters, sowie Greifroboter|
|WO2007046763A1 *||Oct 16, 2006||Apr 26, 2007||Morphic Technologies Aktiebolag (Publ.)||A method and an arrangement for locating and picking up objects from a carrier|
|U.S. Classification||700/259, 348/94, 414/730|
|International Classification||G01B11/00, G06T1/00, G05D3/12, B25J19/04|
|Aug 15, 1984||AS||Assignment|
Owner name: REGIE NATIONALE DES USINES RENAULT 10, AVE. EMILE
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST.;ASSIGNOR:HERMANN, JEAN-PAUL;REEL/FRAME:004289/0127
Effective date: 19820128
Owner name: REGIE NATIONALE DES USINES,FRANCE
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HERMANN, JEAN-PAUL;REEL/FRAME:004289/0127
Effective date: 19820128
|May 24, 1988||FPAY||Fee payment|
Year of fee payment: 4
|Jul 7, 1992||REMI||Maintenance fee reminder mailed|
|Dec 6, 1992||LAPS||Lapse for failure to pay maintenance fees|
|Feb 16, 1993||FP||Expired due to failure to pay maintenance fee|
Effective date: 19921208