« PreviousContinue »
METHOD AND APPARATUS FOR REMOTE REAL TIME COLLABORATIVE MUSIC PERFORMANCE
FIELD OF THE INVENTION
 The present invention relates generally to a system for electronic music performance. More particular still, the invention relates to a system for permitting participants to collaborate in the performance of music, i.e. to jam, where any performer may be remote from any others.
CROSS REFERENCE TO RELATED
 Not Applicable
STATEMENT REGARDING FEDERALLY
SPONSORED RESEARCH OR DEVELOPMENT
 Not Applicable
REFERENCE TO COMPUTER PROGRAM
 The following two appendices are comprised of the respectively listed files, all of which are included on the compact disc filed with this application, and incorporated herein by reference:
 APPENDIX A—exemplary application implemented in Java Studio, by Microsoft Corporation, using calls to SERIALIO, a serial interface class library by Solutions Consulting, Inc., of Santa Barbara, Calif, and the ActiveX Seer Music Player by Seer Systems of Los Altos, Calif.
programs. Private music lessons are expensive and many individuals lack the interest that book learning alone would require.
 Personal computers and video game machines are found in most households. A growing fraction of these machines are able to interconnect using modems or the Internet.
 Software has long been available to allow a computer to become a musical instrument and to provide music theory instruction. Practica Musica (1987), by Ars Nova Software, Kirkland, Wash, is an example of such. This program was sometimes bundled with a plastic keyboard overlay that would temporarily convert a computer keyboard into a miniature white-and-black-keys piano keyboard.
 A drawback of such music programs is that they only admit one person at a time. It is desirable to allow students to receive music education using their computers, but to allow multiple students to play together. Further, by allowing the multiple students to be at remote locations (e.g. each in their own home), geography and transportation cost and time cease to be a barrier. Such a capability would allow an online community of music students to interact and collaborate. One should anticipate the formation of "Virtual Garage Bands" and the creation of songs by composers and lyricists who have never met in person.
 Historically, computer games only operated for a single player at a time, or for multiple players only at a single location, sharing a single computer. However, there is now a burgeoning market for multi-player games. Individuals with computers or video game machines at separate locations can connect via phone lines or the Internet and cooperate or compete in a computer game. One example of such a game is Mechwarrior (1995) by Activision, which allows players' computers to connect via phone lines. Another example is EverQuest (2001) by Sony Digital Entertainment, where many hundreds of players, each with a computer, connect via the Internet, to a game server owned and operated by the publisher, to play in the same game.
 A key difficulty in designing multi-player computer games is the communication latency that occurs between the players' computers. This results in a computerized version of the children's argument, "I tagged you first.""No you didn't, I tagged you first." Two separated computers each accept their own (local) user's input. The computers then communicate those inputs to each other, and finally use both users' inputs to perform a game calculation. However, unless latency (delay) in the communication channel is managed in some way, each computer gives its local user a reaction-time advantage because the other (remote) user is always penalized by the communication channel delay. Eventually, this can result in a disagreement between the two computers—"I tagged you first."
 A number of methodologies, each having various virtues and drawbacks, have been developed to solve the communication latency issue for multi-player gaming.
 Matheson, in U.S. Pat. No. 4,570,930 teaches a method for synchronizing two computer games. Applicable only to games having discretely calculated "generations", Matheson provides that each generation is numbered, and each generation calculation uses the users' inputs gathered during the prior generation. Matheson's generations may, at
best, each be %o of a second long, i.e. at the game's video update rate. However, generations can become arbitrarily long if one or another user's input is not communicated in a timely manner, or needs to be retransmitted. Unfortunately, such a behavior is not conducive to musical performance.
 Hochstein, et al, in U.S. Pat. No. 5,292,125, unlike Matheson, shows a technique for continuous play, not requiring Matheson's "generations". Hochstein measures the roundtrip communications transport time between two game stations. Subsequently, each user's input to their local game station is delayed by half the round trip time, but is transmitted to their opponent's station immediately. This meets a "fair game" criteria that Hochstein proposes, by which neither player enjoys a speed advantage over the other.
 While the "generation-less" technique is more conducive to musical performance, Hochstein does not address three issues. First, Hochstein does not account for unreliability of the communication channel. Second, simultaneity of players' input is necessary but not sufficient to ensure that game stations remain synchronized. There is the asynchrony of the game station's main loop which will cause divergence in the game state. Matheson understood this. Third, the "fair game" criteria calls for system performance to be degraded to the lowest common denominator. In U.S. Pat. No. 5,350, 176, Hochstein, et al. provides synchronization codes which addresses only the first of these. In doing so, he has nearly reverted to Matheson's generations. Bakoglu, et al., in U.S. Pat. No. 5,685,775 provide an alternative synchronization, but at the expense of incurring the entire roundtrip communication transport delay, rather than only a portion.
 O'Callaghan is the first to provide the "fair game" criteria for more than two remote stations. Under his U.S. Pat. No. 5,820,463, a collection of two or more stations algorithmically selects a master. All inputs from all stations are sent to the master station, and all are subsequently sent out to each of the other stations for processing. Key drawbacks here, are just as above: performance is degraded to the least common denominator, and a latency of the full roundtrip communication transport delay is incurred.
 In U.S. Pat. No. 6,067,566, Moline teaches a method whereby a live musical performance, preferably encoded as well known Musical Instrument Digital Interface (MIDI) commands, can be sent over a network to many stations. The live performance can be selectively recorded or mixed with other pre-recorded tracks. The mechanism is a timestamp that is attached to each musical event (e.g. a MIDI Note-On command). By sequencing the timestamps from separate tracks, the tracks can be mixed. By delaying the mixing for at least the maximum expected delay of the communication channel, the (almost) live musical performance can be added to the pre-recorded tracks at a remote location. Further, a station receiving this performance can play along with the (almost) live performance. Moline is limited, however, in that the "play along" performance is not bi-directional. That is, a true jam session is not taking place. Moline suggests that a repetitive musical pattern could be established and enforced, and that jamming could take place by having each participant hear and play along with the others' performance from one or more prior cycles of the pattern. That play along performance is what would subse
quently be heard by the others, during the next (or later) cycle. Such a constraint severely limits the range of artistic expression.
 Rocket Network, Inc, of San Francisco, Calif., (www.rocketnetwork.com) allows a similar collaboration model, but without a true real time component. Through their Res Rocket 1.4 MIDI Collaboration software, a player can retrieve a MIDI sequence from a central server, subsequently play along with it and add or modify selected parts, and upload the additions or changes to the server for other collaboration partners to download in turn.
 Tonos Entertainment, Inc, of Culver City, Calif., (www.tonos.com) provides a similar capability, but is based on MP3 files, rather than MIDI.
 Neumann, et al. in U.S. Pat. No. 6,175,872 does not have such a limitation. By requiring synchronization to a single clock, time stamping of MIDI packets, the streams of MIDI data generated at remote locations can be sent to the local station, sequenced into the correct musical order, and played aloud. The participant playing on the local machine similarly transmits his musical performance, with timestamp, to the other stations. By further requiring that communication transport delay shall be less than twenty milliseconds, Neumann provides real time, remote collaboration and jamming. However, the twenty-millisecond constraint is not met for dial-up users to the Internet, nor even with a direct connection between callers on most local telephone exchanges. Further, the physical limits of the finite speed of light in fiber or cable accumulates at roughly 1 millisecond per 100 miles. Such a requirement, Neumann admits, limits collaborative jamming to a campus-sized WAN. Still, Neumann's contribution allows the merger of multiple remote MIDI streams in another play along environment.
 Thus, there is a need for a system that allows multiple musical players, especially music students, to collaborate and jam in real time and over useful distances, such as across neighborhoods, cities, states, continents, and even across the globe.
 Because of the delays inherent in communication over significant distances, a technique is needed which does not compound that delay.
 Further, there needs to be a way of limiting the adverse effects of excessive delay, and to allow each station to achieve an acceptable level of responsiveness.
 The present invention satisfies these and other needs and provides further related advantages.
OBJECTS AND SUMMARY OF THE
 The present invention relates to a system and method for playing music with one or more other musicians, that is, jamming, where some of the other people are at remote locations.
 Each musician has a station, typically including a keyboard, computer, synthesizer, and a communication channel The communication channel might be a modem connected to a telephone line, a DSL connection, or other local, wide, or Internet network connection.
 When musicians desire a jam session, their respective station computers communicate with each other, or
perhaps with a designated host computer, and determine the communication delays to and from each other station in the jam.
 Subsequently, each musician's performance is immediately transmitted to every other musician's station. However, the performance is delayed before being played locally.
 Upon receipt, remote performances are also delayed, with the exception of the performance coming from the station having the greatest associated network delay, which can be played immediately.
 The local performance is played locally after undergoing a delay equal to that of the greatest associated network delay.
 By this method, each musician's local performance is kept in time with every other musician's performance. The added delay between the musician's performance and the time it is played, becomes an artifact of the instrument. As such, the musician is able to compensate for it and "play ahead" or "on top of" the jam beat.
 Sometimes, some of the stations may have a low (good) communication delay between them, while others may have a high (bad) delay. In such a case, each musician can choose to have his station disregard high delay stations during live jamming, and to allow performance with only low delays.
 In addition, a "groove" can be distributed to the stations. The groove is a track that provides a framework for the jam session. In its simplest form, it might be a metronome. But it could also be a MIDI sequence, a WAV file (instrumental or lyrical), or an MP3. Regardless of the communication delays, the groove plays in synchrony on all machines, as the command to start the groove is delayed appropriately by each station receiving the play command, and the station issuing the play command.
 It is the object of this invention to make it possible for a plurality of musicians to perform and collaborate in real time, even at remote locations.
 In addition to the above, it is an object of this invention to limit delay to the minimum necessary.
 It is an object of this invention to incorporate the artifacts of communication delay into the local performance in a manner which can be intuitively compensated for by the local musician.
 It is a further object to permit each musician to further limit delay artifacts, to taste.
 It is a further object of this invention to provide a groove track, against which all musicians can perform.
 These and other features and advantages of the invention will be more readily apparent upon reading the following description of a preferred exemplified embodiment of the invention and upon reference to the accompanying drawings wherein:
BRIEF DESCRIPTION OF THE DRAWINGS
 The aspects of the present invention will be apparent upon consideration of the following detailed description
taken in conjunction with the accompanying drawings, in which like referenced characters refer to like parts throughout, and in which:
 FIG. 1 is a detailed block diagram of multiple musical performance stations configured to jam over a communications channel, and including an optional server;
 FIG. 2A is a schematic of multiple musical stations connected over a simplified communications channel topology to illustrate the effect of communications delay;
 FIG. 2B is a schematic like that of FIG. 2A, but
illustrating the added effect of a server;
 FIG. 3 shows the preferred graphical user interface for remote jamming;
 FIG. 4 depicts an instrument picker; and,
 FIG. 5 is a flow chart for handling musical events.
 While the invention will be described and disclosed in connection with certain preferred embodiments and procedures, it is not intended to limit the invention to those specific embodiments. Rather it is intended to cover all such alternative embodiments and modifications as fall within the spirit and scope of the invention.
DETAILED DESCRIPTION OF THE
 Referring to FIG. 1, a plurality of performance stations represented by stations 10,12, and 14 are interconnected by the communication channel 150. The invention is operable with as few as two, or a large number of stations. This allows collaborations as modest as a duet played by a song writing team, up to complete orchestras, or larger. Because of the difficult logistics of managing large numbers of remote players, this invention will be used most frequently by small bands of two to five musicians.
 Note that while the term "musician" is used throughout, what is meant is simply the user of the invention, though it may be that the user is a skilled musical artist, a talented amateur, or musical student.
 For some implementations, a fanout server 18 is used. Each performance station 10, 12, 14 communicates over communication channel 150 directly with fanout server 18. The fanout server is responsible for forwarding all pertinent communications from any of the performance stations to each of the others.
 Communications channel 150 may be a telephone network, a local or wide area Ethernet, the Internet, or any other communications medium.
 In FIG. 1, each of remote performance stations 12 and 14 mirror the elements of local performance station 10. Each of performance stations 12 and 14 have keyboard and controls 100,100', 100", event interpretation 110,110', 110", event formatting for jam partners 120, 120', 120", transmit module 130, 130', 130", communication channel interface 140, 140', 140", receive module 160, 160', 160", delay 170, 170', 170", instrument synthesizer 180,180', 180", and audio output 190, 190', 190", respectively.
 Each performance station is preferably comprised of a personal computer having a keyboard and controls 100. Other common graphical user interface (GUI) controls, such