|Publication number||US7843575 B1|
|Application number||US 11/969,628|
|Publication date||Nov 30, 2010|
|Filing date||Jan 4, 2008|
|Priority date||Jan 15, 2007|
|Also published as||US8013234, US8159683|
|Publication number||11969628, 969628, US 7843575 B1, US 7843575B1, US-B1-7843575, US7843575 B1, US7843575B1|
|Inventors||David T. Starkey|
|Original Assignee||Midi9 LLC|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (15), Referenced by (7), Classifications (7), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This patent application claims priority of provisional patent application Ser. No. 60/884,995 filed Jan. 15, 2007, the entirety of which is incorporated herein by reference. The present invention relates to measurement of key movement on a piano keyboard. In particular, the invention relates to the detection of displacement and velocity of a moving key.
Through the years, many systems have been devised to provide a musical note level or MIDI language recording of a player's performance of a piano or other keyboard musical instrument. These devices have been used to provide real-time accompaniment or sound reinforcement for the performer. Some of these devices are complex or bulky, and require an intricate, invasive installation, for example those mounted external to the piano case, such as U.S. Pat. No. 4,768,412 and Pub 2002/0003708. There are many known ways of detecting a displacement and a displacement speed, or other note expression data of a keyboard for a musical instrument. The earliest known methods were mechanical switch structures. However, these had significant disadvantages, such as uniformity of response. Additionally, these types of switches affected key touch while playing, required time-consuming mounting, needed very tight vertical and horizontal adjustment, and generally had a high cost. For a further discussion of the problems of electro-mechanical switches see U.S. Pat. No. 4,628,786 issued to Buchla. The apparatus, according to Buchla, overcomes some of the problems and disadvantages of these prior art mechanical switch structures by providing a pickup sensor for each key that is moved within an electric field formed between a pair of stationary electrodes. Although still a mechanical system, since the pickup never contacts either of the electrodes, Buchla avoids many of the problems inherent in previous mechanical contact switches. A major advantage of the Buchla apparatus is that the voltage impressed on the electric field pickup sensor varies as a function of the position of the pickup within the electric field (i.e. voltage varies according to amount of key depression). The Buchla invention is thus a continuous position sensor providing the capability to capture greater note expression data, and provide for better velocity resolution. However, this apparatus is complex, costly, and requires time consuming installation and calibration. Additionally, the circuitry required to impress the AC voltages of equal amplitude but opposite phase on the two electrodes and the detection circuitry are complex and costly.
In response to the above inherent limitations of mechanical switch structures, various attempts have utilized types of opto-electronic switches. One of the first was U.S. Pat. No. 4,351,221 issued to Starnes, et al. This system however requires two optical LED sensors per key and utilizes the sensors in a manner that creates a double contact system, thus eliminating the advantages of continuous linear position sensor. Additionally, this apparatus requires elaborate and delicate installation of photo sensors by permanent attachment to the underside of the piano keys.
Another opto-electronic switch mechanism utilizing a stationary opto-isolator in conjunction with a light path paddle mounted on a moving key is known from U.S. Pat. No. 4,362,934 issued to McLey. However, this apparatus has the same limitations as the system issued to Starnes, et al discussed above and further, is more commonly limited to electronic keyboards.
Another optical sensing means is known from U.S. Pat. No. 4,736,662 issued to Yamamoto. In order to reduce the number of displacement speed electrical signal converting elements and provide a design for use in a limited space, this apparatus utilizes stationary optical sensor elements which are coupled to optical detector using optical fibers, and the optical sensors are positioned adjacent to moving key paddles of the key striker mechanism, where the moving paddles obstruct the optical path. Another invention is known from U.S. Pat. No. 4,768,412 issued to Sanderson. This system uses a single optical sensor per key comprising an optical transmission sensor whose light path is interrupted by a paddle connected to the keys, where an adapter plate rests on top of the keyboard to translate key motion to the sensors. U.S. Pat. App. 2004/0003708 by Buschla describes a similar system with sensors placed over the tops of the keyboard, where a reflectivity measurement is made from the surfaces of the white keys and the black keys act as photo path interrupters. Velocity measurement is made by using vertically displaced black key sensors and measuring the time interval from first to second sensor path opening as the key is depressed. The white keys and black keys photodiodes are read by an A/D converter. One drawback of this system is that the basis of measurement is completely different between white keys (declining reflectivity) and black keys (key movement time from first to second sensor).
The use of optical interrupters with paddles mounted under a keyboard to measure continuous key position is also described in U.S. Pat. Nos. 5,824,930 and 6,229,081, and 6,297,437 by Ura et al.
Another invention is known from U.S. Pat. No. 5,567,902 by Kimble et al. This device uses non-multiplexed optical encoders mounted underneath keys to determine key position, but does not address crosstalk between keys or ambient light correction, and the problem of non-linearity of reflectivity response is handled by limiting the usable range of the sensor of a 25% section that is linear with position. As all of the LEDs within an octave are simultaneously enabled, optical pollution from illumination of an adjacent key into the sensor for the key to be measured must be handled using baffles and the like. Additionally, mechanical responses from adjacent keys to the key to be measured are not considered.
Another keyboard sensor is described in U.S. Pat. No. 5,231,283 by Starkey and Williams, which utilizes an under key paddle which pivots on a hinge below the keyboard, thereby moving the paddle across an optical detector which measure the degree of path interruption. The paddle is shaped to provide a linear response with key movement, and a hinge and spring mechanism removes the requirement to attach the paddles directly to the keys, as described earlier for U.S. Pat. Nos. 4,362,934 and 4,351,221. U.S. Pat. No. 5,231,283 also describes a method to convert position and time data into accurate MIDI information.
U.S. Pat. No. 5,524,521 by Clift et al describes a sensor system for use on the hammers of a piano, where a sensor for the hammer measures surface reflectivity of the hammer, thereby extracting a velocity and duration measurement.
One object of the invention is to provide a position transducer and compensation system that can provide highly accurate data in both the time and position domains.
Another object of the invention to be minimize the physical size of the sensor, preferably arranging the sensors on a linear strip or strips, so it can fit inside a new or existing acoustic piano, electronic organ, or any keyboard device requiring accurate measurement of individual key positions.
Another object of the invention is the measurement of key position using reflected light from a sensor having an optical source and optical detector which measures light reflecting from a surface of the key to be measured. Anon-contact mounting method eliminates any interference with keyboard motion.
Another object of the invention is the provision of continuous stream of samples of key position information to provide closed loop operation for player piano systems.
Another object of the invention is the reduction of key position estimation errors caused by either ambient light or reflection crosstalk from adjacent keys when estimating key position by measuring the optical power reflected from a particular key.
Another object of the invention is the elimination of key position errors from the motion of adjacent keys. Another object of the invention is to provide reflectivity measurements for use in estimating key position that are impervious to the variations of black and white keys.
Another object of the invention is to provide positional measurement of sufficient accuracy and frequency so as to provide accurate key velocity information.
Another object of the invention is a reflectivity-based measurement of key position which allows for the sensors to be placed in a visually inconspicuous region with respect to a performer and audience, and functionally unobtrusive location with respect to the piano key and hammer mechanisms.
The present invention provides methods and devices for providing data from the performance of a musical instrument. The instrument can be any musical instrument having a keyboard, such as a piano, organ or accordion, including their electronic versions. The invention can be applied to keys, pedals, or buttons or switches that provide position or velocity information. The invention can also be applied to any device having keys where accurate information of the position of individual keys is important, such as a video game controller.
More specifically, the invention provides an accurate measurement of the position of a key of an instrument, which can be further used with time data to provide velocity or acceleration information. This information provided can then be converted to any standard format for musical information, such as a MIDI format. The process of generating MIDI velocity data is well described in U.S. Pat. No. 5,231,283 by Starkey et al. Then this data can be transmitted to one or more musical instruments or storage devices for real time performance. In addition, the invention can generate positional data that can be further interpreted for more complex music synthesis, reproduction or performance archival purposes.
The present invention measures (samples) the position of the keys at discrete time points. The position is measured using a reflective scanning system, such as one positioned under the front portion of the piano key. The device can be mounted in several different positions and configurations behind the fulcrum: above the key looking down, behind the key looking at the back of the key.
In the reflective scanning system, a light source (such as an LED transmitter) provides light, which is reflected off the surface of a key. The reflected light is detected and measured by a device such as a photo-transistor. The measurement of light can involve the relative amplitude of light or changes in angle of reflected light. Improved position information is provided by one or any combination of the following elements, which can be performed in any order:
A. Reduction of cross-talk (interference from light reflected from other keys or from light sources directed to other keys)
B. Reduction of interference due to (varying levels of) ambient light
C. Providing position and velocity information by a linearization algorithm
In one embodiment, the invention provides a no-contact, inexpensive and unobtrusive opto-electronic sensor for acoustic pianos. The invention uses low cost components, so there is no need for focused light beams or any attachments to piano keys. The invention further uses independent microprocessor systems using a synchronization system. It requires no mechanical adjustments after installation under the keys of the piano keyboard. After installation the device learns the environment inside the piano and uses several novel techniques to compensate for parasitic effects found under the piano keys. These techniques allow for large latitude in installation ensuring that no further mechanical adjustments will be required. Immediately following installation the system learns its environment including individual key travel and cross talk between adjacent keys. During operation it samples (measures) the position of each key nearly 1000 times per second, compensating for ambient light, adjacent key crosstalk and the nonlinearity of the position curve. Thus, the invention allows the system to produce uncompromised MIDI velocity accuracy at a low cost.
Each sensor comprises an infrared LED transmitter such as LED1 which illuminates the bottom of a piano key, and an adjacent photo-transistor such as PD1 which converts the reflected photons into an electrical current. The other sensor LEDs and photo-transistors are similarly arranged in pairs. In each individual n-way mux, The current of the photo-transistors PD1, PD2, PD3 is summed and converted 120 into a voltage 122 that is digitized into a digital representation by an analog to digital converter (ADC) optionally located in the microprocessor 124 of each key decoder 102, 104, and 106, and the remainder of the digital processing is performed by a CPU and other resources in the microprocessor unit 124. The micro-processor 124 accepts each photo-transistor measurement after digitization, and each digital value is de-noised and then adjusted to compensate for ambient light. The digital value is then compensated for cross talk from adjacent keys. The value can be further adjusted to compensate for electrical gain of the LED and the gain variations of the photo-transistor and mechanical variations due to the piano design and manufacturing variations and natural vs. sharp notes. This numerical representation is then sampled to a microprocessor that completes the calculation to create the data to be transmitted.
The present invention can use any available microprocessor 124, preferably having at least 30 MHz clock rate, including RISC or DSP architectural features. Prior art microprocessors are small enough to easily fit on a compact circuit board mounted under the pianos keys. These microprocessors also contain ADCs, and one suitable device is a digital signal processor (DSP) from Freescale semiconductor, part number 56F8014. This part includes 8 internal ADCs and sufficient communication ports, a small power requirement, and enough memory and processing capability to perform the required functions.
The present invention can use any means for emitting light (including visible, infrared and UV light), such as an LED or laser diode. The invention can use any means for detecting and measuring light such as a photodiode or photo-transistor. The optical emitter and detector can be combined in a single device or separated into individual devices.
In the embodiment shown in
In the embodiment shown in
The present sensor system signal processing addresses several problems of the prior art through the particular measurement technique. These problems are:
1) Ambient light which erroneously adds to the reflected light from the same-key LED.
2) Light reflected from different keys which adds to the same-key LED detected light, also known as cross-talk.
3) Varying sensor time response, including the slow speed of the photo-transistor under low light level conditions, which results in a limitation in sampling rate.
4) Non-linear relationship between key position and measured reflected light, which non-linearity also has the effect of amplifying error when estimating key velocity and acceleration, which are first and second derivatives with respect to time of position.
The reflected light contribution from a particular enabled LED which couples to photo-transistors such as PD2 of
Ignoring for now mechanical crosstalk, the first order of multiplexing is to individually enable the LEDs and sample the current in the associated photo-transistor receiving reflected light from the key to be measured. In the present invention, the sample time is related to the optical detector settling time. The consequence of this relationship is that greater accuracy in the form of additional bits of digitizing requires longer sample times. For example, the sample time for the phototransistors is on the order of 10 microsecond (uS) settling time for 8 bit digitizing accuracy, and on the order of 15 uS for 12 bit accuracy, and 12 bit resolution is used because of the additional dynamic range requirement associated with sensing ambient light levels and reflections over required distance ranges.
The present invention reduces interference arising from ambient light by measuring the ambient light during each measurement cycle. A typical measurement cycle duration T1 of
Thus, the invention provides an effective correction for ambient light, even when ambient light conditions for each key may change during, and indeed throughout, the course of performance.
Although each key has its own set of LED transmitter and photo-transistor, each set does not necessarily operate independently; rather there can be interaction between them. Accordingly, the invention also provides a method to essentially isolate each of the sensing transistors to eliminate current due to crosstalk. As was shown in
The present invention reduces the undesirable effects of crosstalk for a measured key by detecting light from neighboring keys and removing it from the light measurement of the measured key, as shown in
The adjustment procedure, which is performed for each key, and shown for current key (key x) in
1) Measure the position of each key when all of the keys are in rest (up) position, thereby getting a LinRest.x value for each key.
2) In sequence, press each key fully down individually, measuring its
3) Store the results (such as in nonvolatile memory)
4) When the adjustment procedure is completed, switch to normal operation mode.
In this manner, for each key, a dataset of 4 adjustment values per key depression are collected, two of which are stored for the present key x (LinRest.x and LinDown.x), and two of which are stored for adjacent keys (BassErr.x+1 and TrebErr.x−1).
In the below equations, the notation used is as follows:
LghtCmp.x represents the value from the present key x associated ADC output (with range 0 to 4095, or 0x0 to 0xfff in hex notation), corrected for ambient light, as described above.
LinPos.x represents the linearized value of reflected light, which compensates for the 1/r2 variation of reflected light with key position.
LinRest.x and LinDown.x are LinPos.x at the rest and down positions, respectively.
KeyComp.x represents a scaling term which is applied to generate the normalized linear value NormLinPos.x
BassErr.x represents the value of NormLinPos.(x−1) when key x only is depressed. BassErr.x is initialized to 1 greater than LinRest.x.
TrebErr.x represents the value of NormLinPos.(x+1) when key x is depressed exclusively
BassComp.x is an interim calculation derived in part from BassErr
TrebComp.x is an interim calculation derived in part from TrebErr
Pos.x is the linearized position, corrected for adjacent reflection crosstalk, and calculated from the above variables and having a range of 0 to 16.
As used below, LghtCmp.x is used to indicate the reflected light for key number x, where adjacent keys are indicated by x−1 (below) and x+1 (above).
Using the matrix obtained during the initialization phase, subsequent light measurements for each key can be corrected to provide position data (Pos.x) as follows:
LghtCmp.x=LightOn.x−LightOff.x (Eqn 1)
NormLinPos.x=(LinPos.x−LinRest.x)*KeyComp.x (Eqn 4)
BassErr.x=NormLinPos(x−1) when key x depressed exclusively (Eqn 5)
TrebErr.x=NormLinPos.(x+1) when key x depressed exclusively (Eqn 6)
BassComp.x=KeyComp(x−1)*(BassErr.x−LinRest.(x−1)) (Eqn 7)
TrebComp.x=KeyComp(x+1)*(TrebErr.x−LinRest.(x+1)) (Eqn 8)
Pos.x=NormLinPos.x−BassComp.x*((NormLinPos.x−1)−NormLinPos.x))−TrebComp.x*((NormLinPos(x+1)−(NormLinPos.x)) (Eqn 9)
where in one embodiment:
LghtCmp range: 0x0 to 0xfff
LinPos range 0x0 to 0x800
KeyComp range: 0x0 to 0xff
Pos.x range: 0x0 to 0xff
In the above equations, equation 1 articulates the measurement of ambient plus reflected light during the T2 interval of
As used in the example calculation above, Pos.x represents a scaled distance between the key and the sensor, where 0=depressed to the bottom and 16=rest position.
It should also be noted that because the invention provides corrections for each key individually, it is equally applicable to white and black keys, regardless of whether the adjacent key is black or white. Moreover, the invention adapts to surfaces of keys with different finishes or having different reflective properties due to variations in manufacture or design.
One embodiment of the invention further provides a method for quickly computing the square root required in equation 2 to convert each of the values LghtCmp.x 516 of
Solving this equation for X:
X=K2/(Y 0.5), where K2=K10.5
The preceding table shows empirical measurements of the response curve compared to an inverse square law. The first column is depression of a key as was measured in inches approx 2 inches from the front of a white key or equivalent on a black key. The second column is the numerical out of the 12 bit ADC sampling a sensor mounted under the key as shown in
A scaled inverse square model that can be used is
Where Y is the value linearly related to the current flowing in the transistor due to reflected energy from the key being measured by the 12-bit ADC, ranging from 0 to 4096
Where X is represents the position of the key.
K is a constant to provide correct scaling of the linear position and can be determined empirically by skilled engineers for the desired application and the DSP chip being used. The formula can be implemented in commercially available chips. However, if a desired DSP chip does not have hardware divide or square root capability, the equation can be transformed to:
And can be implemented with the following exemplary C code (as known in the prior art and described in many references, including http://www-ccs.ucsd.edu/c/): unsigned int LinearizeSensorData(unsigned int input)
return (0x800−AntiLogVel(0x0E00−(log 256(input)>>1)));
unsigned int ExpoizeSensorData(unsigned int input)
return (AntiLogVel(7168−(log 256(0x800−input)<<1)));
unsigned int log 256(unsigned int arg)
unsigned int argg,arggx;
More specific information about MIDI can be found in U.S. Pat. No. 5,231,283, which is hereby incorporated by reference.
Thus, the present invention provides position information. When time information is also considered, the invention can provide velocity and acceleration information for use in electronic musical data formats according to protocols and formats known in the art.
In summary, the advantages provided by the present system over the prior art are:
1) The ability to accurately measure key motion in time and velocity, in particular meeting the following requirements:
2) Sufficient linearity correction to compensate for the mechanical variations within the keyboard, particularly with regard to the problem of:
3) A scan rate sufficient to accurately determine velocity, for which 3-way multiplexing is believed to be the best mode.
4) The ability to trade off between scan rate and positional resolution, whereby longer settling times result in greater measurement accuracy but reduced scan rate. The timings T1, T2, and T3 of
5) Accurate measurement of key position is critical to accurate estimate of acceleration. If the key is being accelerated linearly a 10% error is position equates to a 29% error in velocity. It is important to accurately provide positional information for both solenoid player mechanisms which generate distinctly different velocity profiles than keys being played by humans, as well as accommodate the wide techniques of humans.
6) Crosstalk or interference from adjacent keys is fully compensated in the present system. This is critical for useful operation of a reflective system.
7) The sample rate of the system must be high, however the photo-transistors have a slow response time if not electrically biased properly. Using a photo-transistor with a simple resistor pullup can cause response time in the milliseconds leading to expensive multiplexing systems. This invention describes a very low cost, low part count, and high speed multiplexing system
8) Noise on the ADC is a critical problem. The preferred embodiment of the present invention uses a 12 bit ADC where none of the electrical signal travel farther that 8 inches and the voltage are all below 5 volts and current requirements is very low, typically below 200 ma.
9) In prior art systems the mechanics of the LED/Photo-transistor pair was not tightly controlled. The prior art devices must be precisely aligned to insure consistent transfer curves, in contrast with the simple adjustment procedure of the present invention, sequentially pressing each individual key down which provides a complete calibration of the particular keys being measured.
10) The present system measures background ambient light with every sample (shown as T3 of
11) There are other aspects that have been ignored by previous scanning systems, and are incorporated into the adjustment or operation of the present system, such as:
The description of the embodiments of the present invention are provided for understanding of the invention, and do not limit the invention. In particular, the n-way multiplexer is shown for n=3, however smaller or larger numbers of multiplexers are possible, and the key measurement function such as 114-1 of
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US4022097 *||Jul 15, 1974||May 10, 1977||Strangio Christopher E||Computer-aided musical apparatus and method|
|US4351221||Jun 15, 1979||Sep 28, 1982||Teledyne Industries, Incorporated||Player piano recording system|
|US4362934||Apr 8, 1981||Dec 7, 1982||Syntronics Music Corporation||Velocity sensing device using opto-electronic switches|
|US4628786||Jun 18, 1985||Dec 16, 1986||Kimball International, Inc.||Velocity responsive musical instrument keyboard|
|US4736662||Nov 6, 1986||Apr 12, 1988||Nippon Gakki Seizo Kabushiki Kaisha||Optical sensor for sensing displacement speed or displacement of a moveable element in musical instrument|
|US4768412||May 9, 1986||Sep 6, 1988||Sanderson Stephen N||Low profile keyboard device and system for recording and scoring music|
|US5231283||Nov 26, 1990||Jul 27, 1993||Gulbransen, Inc.||Opto-electronic sensing method and device for an acoustic piano|
|US5524521||Feb 27, 1995||Jun 11, 1996||Burgett, Inc.||Method and apparatus for optically determining note characteristics in a keyboard operated musical instrument|
|US5567902 *||Jan 6, 1995||Oct 22, 1996||Baldwin Piano And Organ Company||Method and apparatus for optically sensing the position and velocity of piano keys|
|US5824930||Jun 5, 1996||Oct 20, 1998||Yamaha Corporation||Keyboard musical instrument having key monitor exactly discriminating key motion|
|US6229081||Sep 14, 1999||May 8, 2001||Yamaha Corporation||Keyboard musical instrument, position sensing device and light-emitting controller both incorporated therein|
|US6297437||Sep 20, 1999||Oct 2, 2001||Yamaha Corporation||Keyboard musical instrument and information processing system incorporated therein for discriminating different kinds of key motion|
|US6420642 *||Jul 14, 2000||Jul 16, 2002||Yamaha Corporation||Automatic player incorporated in keyboard musical instrument and plunger sensor for detecting motion of plunger|
|US7285718||Dec 21, 2004||Oct 23, 2007||Yamaha Corporation||Keyboard musical instrument and other-type musical instrument, and method for generating tone-generation instructing information|
|US20040003708 *||Jun 20, 2003||Jan 8, 2004||Buchla Donald Frederick||Device and method for sensing positions and/or velocities of keys and pedals of a piano|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8013234 *||Sep 23, 2010||Sep 6, 2011||Midi9 LLC||Reflective piano keyboard scanner|
|US8138402 *||Mar 5, 2010||Mar 20, 2012||Yamaha Corporation||Keyboard musical instrument and solenoid drive mechanism|
|US9099068 *||Jul 8, 2014||Aug 4, 2015||Light4Sound||Optoelectronic pickup for musical instruments|
|US9524708||Jun 1, 2015||Dec 20, 2016||Light4Sound||Optoelectronic pickup for musical instruments|
|US20100229708 *||Mar 5, 2010||Sep 16, 2010||Yamaha Corporation||Keyboard musical instrument and solenoid drive mechanism|
|US20140318349 *||Jul 8, 2014||Oct 30, 2014||Light4Sound||Optoelectronic Pickup for Musical Instruments|
|US20160098975 *||Sep 28, 2015||Apr 7, 2016||Richard Shepherd||Hammer velocity measurement system|
|Cooperative Classification||G10H2220/305, G10H1/0553, G01B11/026|
|European Classification||G01B11/02D, G10H1/055L|
|Jan 25, 2011||AS||Assignment|
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:STARKEY, DAVID T.;REEL/FRAME:025692/0371
Effective date: 20101210
Owner name: MIDI9 LLC, CALIFORNIA
|Aug 3, 2012||AS||Assignment|
Owner name: QRS MUSIC TECHNOLOGIES, INC., FLORIDA
Effective date: 20120321
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MIDI9 LLC;REEL/FRAME:028717/0394
|May 28, 2014||FPAY||Fee payment|
Year of fee payment: 4