CA2491554A1 - Camera-pen-tip mapping and calibration - Google Patents

Camera-pen-tip mapping and calibration Download PDF

Info

Publication number
CA2491554A1
CA2491554A1 CA002491554A CA2491554A CA2491554A1 CA 2491554 A1 CA2491554 A1 CA 2491554A1 CA 002491554 A CA002491554 A CA 002491554A CA 2491554 A CA2491554 A CA 2491554A CA 2491554 A1 CA2491554 A1 CA 2491554A1
Authority
CA
Canada
Prior art keywords
pentip
pen
virtual
location
tip
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CA002491554A
Other languages
French (fr)
Other versions
CA2491554C (en
Inventor
Jian Wang
Liyong Chen
Xiaoxu Ma
Yingnong Dang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Corp
Original Assignee
Microsoft Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Corp filed Critical Microsoft Corp
Publication of CA2491554A1 publication Critical patent/CA2491554A1/en
Application granted granted Critical
Publication of CA2491554C publication Critical patent/CA2491554C/en
Expired - Fee Related legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0354Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
    • G06F3/03545Pens or stylus
    • FMECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F21LIGHTING
    • F21VFUNCTIONAL FEATURES OR DETAILS OF LIGHTING DEVICES OR SYSTEMS THEREOF; STRUCTURAL COMBINATIONS OF LIGHTING DEVICES WITH OTHER ARTICLES, NOT OTHERWISE PROVIDED FOR
    • F21V14/00Controlling the distribution of the light emitted by adjustment of elements
    • F21V14/08Controlling the distribution of the light emitted by adjustment of elements by movement of the screens or filters
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304Detection arrangements using opto-electronic means
    • G06F3/0317Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
    • G06F3/0321Detection arrangements using opto-electronic means in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet

Abstract

X-y positions of a digital pen's tip may be determined by using a calibration parameter to map the x-y positions of the respective centers of images captured by the pen's camera. The calibration parameter may be generated by iteratively calculating estimates of the calibration parameter. Calibration input data may be produced by a user placing the pen tip in a fixed location on a surface, which may be a positionally encoded medium, such as paper, and then rotating the pen and/or moving the opposite end of the pen in various directions to capture multiple images for use in generating the calibration parameter. A user may perform such a calibration procedure without the need for complicated calibration equipment typically used in connection with conventional calibration techniques.

Claims (25)

1. A method of calibrating a digital pen, which includes a pen tip and a camera, and of mapping a location of a center of a camera-captured image to a location of the pen tip, the method comprising:
generating a calibration parameter based on a plurality of images captured by the digital pen's camera by placing the pen in a plurality of different orientations while keeping the pen tip at a substantially fixed location L pentip on a positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter; and using the calibration parameter to map the location of the center of the camera-captured image to the location of the pen tip so that a stroke written by a user on the positionally encoded surface can be recovered at a location within an electronic copy of the positionally encoded surface, wherein the position of the recovered stroke within the electronic copy corresponds to a plurality of locations of the pen tip as the user was writing the stroke on the positionally encoded surface.
2. The method of claim 1, further comprising: calculating a transform F
S.fwdarw.P and an inverse transform F P.fwdarw.S = by m-array decoding and image matching, for each image of the plurality of captured images and then calculating:
i = 1,2,.multidot.,N , and .cndot.(L pentip + .DELTA.L i), i = 1,2,.multidot.,N where N is a number of captured images used for generating the calibration parameter and .DELTA.P i is an offset between an actual pen tip location in an ith frame and L pentip .
3. The method of claim 2, further comprising: initializing ~virtual-pentip as (0, 0), where ~virtual-pentip is an estimated value of L virtual-pentip.
4. The method of claim 3, further comprising: in the equation .cndot. L virtual-pentip, i = 1,2,.multidot., N , setting L
virtual-pentip as ~virtual-pentip to get i = 1,2,.multidot.,N and, by averaging estimating L pentip as:
~pentip =
5. The method of claim 4, further comprising: in the equation .cndot.(L pentip + .DELTA.L i), i = 1,2,.multidot.,N , setting L pentip as ~pentip to get i = 1,2,.multidot.,N and, by averaging estimating L virtual-pentip as:
~virtual-pentip =
6. The method of claim 5, further comprising: repeating the steps of claims 4 and a plurality of times such that the estimate of L virtual-pentip converges to a substantially more accurate result L~pentip and the estimate of L pentip converges to a substantially more accurate result L~.
7. The method of claim 6, further comprising: using L~pentip as the calibration parameter for mapping the location of the center of the camera-captured image to the location of the pen tip.
8. The method of claim 1, further comprising: using a virtual pen tip to map a recovered camera-captured stroke to a recovered pen-tip stroke, wherein a location of the virtual pen-tip on an image-sensing plane of the digital pen depends on a predetermined relationship between the tip of the pen and the pen's camera, wherein the predetermined relationship is based on the configuration of the pen.
9. A computer-readable medium containing computer-executable instructions that generate a calibration parameter for a digital pen, which includes a pen tip and a camera, wherein mapping, the location of the image center to the location of the tip of the pen on a positionally encoded surface, depends on the calibration parameter, by performing steps comprising: generating the calibration parameter based on a plurality of images captured by the digital pen's camera by placing the pen in a plurality of different orientations while keeping the pen tip at a fixed location L pentip on the positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter.
10. The computer-readable medium of claim 9, containing further computer-executable instructions that calculate a transform F S.fwdarw.P and an inverse transform by image matching, for each image of the plurality of captured images and that calculate:
and i = 1,2,.multidot., N where N is a number of captured images used for generating the calibration parameter and .DELTA.P i is an offset between an actual pen tip location in an ith frame and L pentip.
11. The computer-readable medium of claim 10, containing further computer-executable instructions that initialize ~virtual-pentip as (0, 0), where ~virtual-pentip is an estimated value of L virtual-pentip.
12. The computer-readable medium of claim 11, containing further computer-executable instructions for, in the equation i = 1,2,.multidot.,N, setting L virtual-pentip as ~virtual-pentip to get i = 1,2,.multidot., N
and, by averaging estimating L pentip as:
13. The computer-readable medium of claim 12, containing further computer-executable instructions for, in the equation i = 1,2,.multidot.N, setting L pentip as ~pentip to get i =1,2,.multidot.,N and, by averaging estimating L virtual-pentip as: ~virtual-pentip =
14. The computer-readable medium of claim 13, containing further computer-executable instructions for repeating the calculations of claims 16 and 17 a plurality of times such that the estimate of L virtual-pentip converges to a substantially more accurate result and the estimate of L pentip converges to a substantially more accurate result
15. The computer-readable medium of claim 14, containing further computer-executable instructions for using as the calibration parameter L virtual-pentip for mapping the location of the image center to the location of the tip of the pen.
16. A computer-readable medium containing computer-executable instructions that calibrate a digital pen, which includes a pen tip and a camera, and that map a location of a center of a camera-captured image to a location of the pen tip, by performing steps comprising:
generating a calibration parameter based on a plurality of images captured by the digital pen's camera by placing the pen in a plurality of different orientations while keeping the pen tip at a substantially fixed location L pentip on a positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter; and using the calibration parameter to map the location of the center of the camera-captured image to the location of the pen tip so that a stroke written by a user on the positionally encoded surface can be recovered at a location within an electronic copy of the positionally encoded surface, wherein the position of the recovered stroke within the electronic copy corresponds to a plurality of locations of the pen tip as the user was writing the stroke on the positionally encoded surface.
17. The computer-readable medium of claim 16, containing further computer-executable instructions for using a virtual pen tip to map a recovered camera-captured stroke to a recovered pen-tip stroke, wherein a location of the virtual pen-tip on an image-sensing plane of the digital pen depends on a predetermined relationship between the tip of the pen and the pen's camera, wherein the predetermined relationship is based on the configuration of the pen.
18. A system that calibrates a digital pen, which includes a pen tip and a camera, and that maps a location of a center of a camera-captured image to a location of the pen tip, the system comprising:
a calibration module that generates a calibration parameter based on a plurality of images captured by the digital pen's camera with the pen in a plurality of different orientations while the pen tip is at a substantially fixed location L pentip on the positionally encoded surface, wherein the calibration parameter is generated based on a plurality of iteratively generated estimates of the calibration parameter; and a mapping module that uses the calibration parameter to map the location of the center of the camera-captured image to the location of the pen tip so that a stroke written by a user on the positionally encoded surface can be recovered at a location within an electronic copy of the positionally encoded surface, wherein the position of the recovered stroke within the electronic copy corresponds to a plurality of locations of the pen tip as the user was writing the stroke on the positionally encoded surface.
19. The system of claim 18, wherein the calibration module calculates a transform F S.fwdarw.P and an inverse transform F P.fwdarw.S = by image matching, for each image of the plurality of captured images and then calculates:
.cndot. L virtual-pentip i = 1,2,.multidot.,N , and .cndot. (L pentip + .DELTA.L i), i =1,2,.multidot.,N where N is a number of captured images used for generating the calibration parameter and .DELTA.L i is an offset between an actual pen tip location in an ith frame and L pentip.
20. The system of claim 19, wherein the calibration module initializes ~virtual-pentip as (0, 0), where ~virtual-pentip is an estimated value of L virtual-pentip.
21. The system of claim 20 wherein, in the equation i = 1,2,.multidot., N , L virtual-pentip is set as ~virtual-pentip to get i = 1,2,.multidot.,N and, L pentip is estimated, by averaging as:
22. The system of claim 21, wherein, in the equation , i = 1,2,.multidot.,N , L pentip is set as ~pentip to get i = 1,2,.multidot.,N and L virtual-pentip is estimated, by averaging as:
23. The system of claim 22, wherein, the calibration module repeats the calculations of claims 29 and 30 a plurality of times such that the estimate of L virtual-pentip converges to a substantially more accurate result and the estimate of L pentip converges to a substantially more accurate result
24. The system of claim 23, wherein, the calibration module outputs from the calibration module as the calibration parameter for use by the mapping module to map the recovered image-center stroke to the recovered pen-tip stroke.
25. The system of claim 18, wherein the mapping module uses a virtual pen tip to map a recovered camera-captured stroke to a recovered pen-tip stroke, wherein a location of the virtual pen-tip on an image-sensing plane of the digital pen depends on a predetermined relationship between the tip of the pen and the pen's camera, wherein the predetermined relationship is based on the configuration of the pen.
CA2491554A 2004-01-06 2005-01-05 Camera-pen-tip mapping and calibration Expired - Fee Related CA2491554C (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/753,023 US7136054B2 (en) 2004-01-06 2004-01-06 Camera-pen-tip mapping and calibration
US10/753,023 2004-01-06

Publications (2)

Publication Number Publication Date
CA2491554A1 true CA2491554A1 (en) 2005-07-06
CA2491554C CA2491554C (en) 2012-03-13

Family

ID=34592570

Family Applications (1)

Application Number Title Priority Date Filing Date
CA2491554A Expired - Fee Related CA2491554C (en) 2004-01-06 2005-01-05 Camera-pen-tip mapping and calibration

Country Status (10)

Country Link
US (1) US7136054B2 (en)
EP (1) EP1553482A1 (en)
JP (1) JP4750425B2 (en)
KR (1) KR101037232B1 (en)
CN (1) CN100472554C (en)
AU (1) AU2004242471A1 (en)
BR (1) BRPI0500015A (en)
CA (1) CA2491554C (en)
MX (1) MXPA05000368A (en)
RU (1) RU2363032C2 (en)

Families Citing this family (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7133563B2 (en) 2002-10-31 2006-11-07 Microsoft Corporation Passive embedded interaction code
US7263224B2 (en) * 2004-01-16 2007-08-28 Microsoft Corporation Strokes localization by m-array decoding and fast image matching
US8054495B2 (en) * 2004-04-07 2011-11-08 Hewlett-Packard Development Company, L.P. Digital documents, apparatus, methods and software relating to associating an identity of paper printed with digital pattern with equivalent digital documents
US20060084039A1 (en) * 2004-10-19 2006-04-20 Massachusetts Institute Of Technology Drawing tool for capturing and rendering colors, surface images and movement
US7505982B2 (en) * 2004-12-03 2009-03-17 Microsoft Corporation Local metadata embedding solution
US7536051B2 (en) * 2005-02-17 2009-05-19 Microsoft Corporation Digital pen calibration by local linearization
US7607076B2 (en) * 2005-02-18 2009-10-20 Microsoft Corporation Embedded interaction code document
US7826074B1 (en) 2005-02-25 2010-11-02 Microsoft Corporation Fast embedded interaction code printing with custom postscript commands
US7532366B1 (en) 2005-02-25 2009-05-12 Microsoft Corporation Embedded interaction code printing with Microsoft Office documents
US7421439B2 (en) 2005-04-22 2008-09-02 Microsoft Corporation Global metadata embedding and decoding
US7400777B2 (en) 2005-05-25 2008-07-15 Microsoft Corporation Preprocessing for information pattern analysis
US7729539B2 (en) 2005-05-31 2010-06-01 Microsoft Corporation Fast error-correcting of embedded interaction codes
US7528848B2 (en) * 2005-06-30 2009-05-05 Microsoft Corporation Embedded interaction code decoding for a liquid crystal display
WO2007003682A1 (en) * 2005-06-30 2007-01-11 Nokia Corporation Camera control means to allow operating of a destined location of the information surface of a presentation and information system
US7619607B2 (en) * 2005-06-30 2009-11-17 Microsoft Corporation Embedding a pattern design onto a liquid crystal display
US7817816B2 (en) 2005-08-17 2010-10-19 Microsoft Corporation Embedded interaction code enabled surface type identification
EP1909162A1 (en) * 2006-10-02 2008-04-09 Koninklijke Philips Electronics N.V. System for virtually drawing on a physical surface
ES2323399B2 (en) * 2007-06-19 2010-02-26 Gat Microencapsulation Ag SUSPENSIONS IN AGRICULTURAL SULFONILE AND COMBINATIONS OILS.
US8760401B2 (en) * 2008-04-21 2014-06-24 Ron Kimmel System and method for user object selection in geographic relation to a video display
US20090309854A1 (en) * 2008-06-13 2009-12-17 Polyvision Corporation Input devices with multiple operating modes
US9354725B2 (en) 2012-06-01 2016-05-31 New York University Tracking movement of a writing instrument on a general surface
WO2014022905A1 (en) * 2012-08-10 2014-02-13 Research In Motion Limited Stacked device position identification
KR101410990B1 (en) * 2012-09-04 2014-06-23 이승희 Input system with electronic pen
SG10201901111UA (en) * 2013-10-25 2019-03-28 Wacom Co Ltd Dynamic handwriting verification, handwriting-baseduser authentication, handwriting data generation, and handwriting data preservation
KR101589707B1 (en) * 2014-09-25 2016-01-28 (주) 씨앤피 Potable Magnifying Apparatus having the pen-type camera
TWI564773B (en) * 2015-08-05 2017-01-01 緯創資通股份有限公司 Optical touch system and optical touch apparatus thereof
US10838551B2 (en) * 2017-02-08 2020-11-17 Hewlett-Packard Development Company, L.P. Calibration of displays
US11301063B2 (en) 2018-07-18 2022-04-12 John J. King, JR. Smart pen device and method of implementing a smart pen device

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ATE90129T1 (en) * 1989-10-24 1993-06-15 Geberit Ag INSTALLATION SPACE COVER FOR FOAMED INSTALLATION COMPONENTS.
US5146552A (en) 1990-02-28 1992-09-08 International Business Machines Corporation Method for associating annotation with electronically published material
DE69202975T2 (en) * 1991-04-03 1996-02-15 Hewlett Packard Co POSITIONING DEVICE.
CA2045907C (en) * 1991-06-28 1998-12-15 Gerald B. Anderson A method for storing and retrieving annotations and redactions in final form documents
US5414227A (en) * 1993-04-29 1995-05-09 International Business Machines Corporation Stylus tilt detection apparatus for communication with a remote digitizing display
JPH08237407A (en) * 1994-12-09 1996-09-13 Xerox Corp Method of positioning relative alignment of picture tile andcorrecting penetrative distortion
US5626620A (en) * 1995-02-21 1997-05-06 Medtronic, Inc. Dual chamber pacing system and method with continual adjustment of the AV escape interval so as to maintain optimized ventricular pacing for treating cardiomyopathy
US6208894B1 (en) * 1997-02-26 2001-03-27 Alfred E. Mann Foundation For Scientific Research And Advanced Bionics System of implantable devices for monitoring and/or affecting body parameters
US5855594A (en) * 1997-08-08 1999-01-05 Cardiac Pacemakers, Inc. Self-calibration system for capture verification in pacing devices
US6181329B1 (en) * 1997-12-23 2001-01-30 Ricoh Company, Ltd. Method and apparatus for tracking a hand-held writing instrument with multiple sensors that are calibrated by placing the writing instrument in predetermined positions with respect to the writing surface
US6070102A (en) * 1998-04-29 2000-05-30 Medtronic, Inc. Audible sound confirmation of programming an implantable medical device
CN1303494A (en) 1998-04-30 2001-07-11 C技术公司 Control device and method of controlling object
US6551357B1 (en) * 1999-02-12 2003-04-22 International Business Machines Corporation Method, system, and program for storing and retrieving markings for display to an electronic media file
US6522928B2 (en) * 2000-04-27 2003-02-18 Advanced Bionics Corporation Physiologically based adjustment of stimulation parameters to an implantable electronic stimulator to reduce data transmission rate
JP4419305B2 (en) * 2000-09-29 2010-02-24 ブラザー工業株式会社 Coordinate reading device
JP3771831B2 (en) * 2001-11-01 2006-04-26 インターナショナル・ビジネス・マシーンズ・コーポレーション Computer system and program for sharing annotation information added to digital content
US20030117378A1 (en) * 2001-12-21 2003-06-26 International Business Machines Corporation Device and system for retrieving and displaying handwritten annotations

Also Published As

Publication number Publication date
AU2004242471A1 (en) 2005-07-21
KR20050072691A (en) 2005-07-12
MXPA05000368A (en) 2005-08-19
US7136054B2 (en) 2006-11-14
JP4750425B2 (en) 2011-08-17
US20050146518A1 (en) 2005-07-07
RU2363032C2 (en) 2009-07-27
RU2004139188A (en) 2006-06-10
JP2005202953A (en) 2005-07-28
KR101037232B1 (en) 2011-05-25
CA2491554C (en) 2012-03-13
CN1637774A (en) 2005-07-13
BRPI0500015A (en) 2005-08-09
CN100472554C (en) 2009-03-25
EP1553482A1 (en) 2005-07-13

Similar Documents

Publication Publication Date Title
CA2491554A1 (en) Camera-pen-tip mapping and calibration
JP2005202953A5 (en)
Joshi et al. Image deblurring using inertial measurement sensors
US8264553B2 (en) Hardware assisted image deblurring
TWI479881B (en) System, method and computer program product for 3d video stabilization by fusing orientation sensor readings and image alignment estimates
KR100947405B1 (en) Implement for Optically Inferring Information from a Planar Jotting Surface
US9153025B2 (en) Plane detection and tracking for structure from motion
US8290294B2 (en) Dehazing an image using a three-dimensional reference model
CN106780576A (en) A kind of camera position and orientation estimation method towards RGBD data flows
CN105069753B (en) A kind of shake Restoration method of blurred image of facing moving terminal
KR101863744B1 (en) Simulation apparatus and simulation method for evaluation of performance of underwater video mosaicking algorithm
US11049313B2 (en) Rendering an object
KR100951309B1 (en) New Calibration Method of Multi-view Camera for a Optical Motion Capture System
US10521892B2 (en) Image lighting transfer via multi-dimensional histogram matching
CN104680544B (en) Variation scene flows method of estimation based on the regularization of 3-dimensional flow field
CN115564842A (en) Parameter calibration method, device, equipment and storage medium for binocular fisheye camera
CN114494388A (en) Three-dimensional image reconstruction method, device, equipment and medium in large-view-field environment
JP2005049999A (en) Image input system, image input method, program described to execute method on information processing device, and storage medium with program stored
CN109215128B (en) Object motion attitude image synthesis method and system
CN104504691A (en) Camera position and posture measuring method on basis of low-rank textures
CN111145267B (en) 360-degree panoramic view multi-camera calibration method based on IMU assistance
JP5988364B2 (en) Image processing apparatus and method
JP5439343B2 (en) 3D information acquisition apparatus and 3D information acquisition program
US8872832B2 (en) System and method for mesh stabilization of facial motion capture data
CN105229706B (en) Image processing apparatus, image processing method and program

Legal Events

Date Code Title Description
EEER Examination request
MKLA Lapsed

Effective date: 20200106