CA2417666C

CA2417666C - Protected ethernet backplane communication

Info

Publication number: CA2417666C
Application number: CA002417666A
Authority: CA
Inventors: Claude Dubreuil
Original assignee: Marconi Intellectual Property Ringfence Inc
Current assignee: Ericsson AB
Priority date: 2000-07-28
Filing date: 2001-07-02
Publication date: 2006-05-09
Anticipated expiration: 2021-07-02
Also published as: DE60133747T2; US6804193B1; CA2417666A1; US20050044137A1; EP1305912B1; WO2002010930A3; AU2001273659A1; US7240127B2; EP1305912A2; WO2002010930A2; DE60133747D1

Abstract

A multiprocessor system is provided that has a plurality of processor module s coupled together via a backplane. The system comprises a first processor module having a first processor and a first switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O ports of the first switch, the first switch being operable to route data packets. The system further comprises a second processor module having a second processor and a second switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O ports of the second switch, the secon d switch being operable to route data packets. The system also comprises a thi rd processor module having a third processor and a first communication device that is operable to communicate with the first switch via a first communication path on the backplane and operable to communicate with the second switch via a second communication path on the backplane. In addition, the system comprises a fourth processor module having a fourth processor and a second communication device that is operable to communicate with the first switch via a third communication path on the backplane and operable to communicate with the second switch via a fourth communication path on the backplane. The first switch is operable to route data packets from one of th e first, second, third or fourth processors to another of the first, second, third or fourth processors. The second switch is also operable to route data packets from one of the first, second, third or fourth processors to another of the first, second, third or fourth processors.

Description

PROTECTED ETHERNET BACKPLANE COMMUNICATION
FIELD OF THE INVENTION
The present invention relates in general to communication between multiple data processors and, more particularly, to communication between multiprocessors using a switch protocol.
BACKGROUND OF THE INVENTION
Communication between computers has become an important aspect of to everyday life in both private and business environments. Computers converse with each other based upon a physical medium for transmitting the messages back and forth, and upon a set of rules implemented by electronic hardware attached to and programs running on the computers. These rules, often called protocols, define the orderly transmission and receipt of messages in a network of 15 connected computers.
The use of multiple processors in a single system is well-known in the feld of data processing systems, and the resulting systems are called multiprocessor systems. As data processing systems have expanded to incorporate multiprocessors, communication systems for allowing 2o communication between the multiple processors have been proposed. The multiprocessor communication systems must be continually improved to allow for greater data processing capacity and faster speeds the multiprocessor environment is capable of delivering.

To improve upon the presently known multiprocessor communication systems, the present invention has been proposed. The present invention provides a system for inter-processor communication in a backplane based multiprocessor system. According to one aspect of the invention, provided is a 3o communication system that implements the Ethernet MAC protocol over a backplane physical media that can take advantage of the Ethernet MAC that is built into many processors. According to another aspect of the present invention, provided is a protected communication system that provides redundancy at all levels within the system.
The present invention provides many advantages over the presently known commmlication systems for multiprocessors. Not all of these advantages are simultaneously required to practice the invention as claimed, and the following list is merely illustrative of the types of benefits that may be provided, alone or in combination, by the present invention. These advantages include:
(1) to the use of a standard Ethernet protocol to take advantage of the design simplification but without the cost, space, and power that normal Ethernet physical media requires; (2) providing a redundant communication system wherein each processor has a redundant communication path and wherein the redundancy is transparent to the processors; (3) commuiucating over a low power 15 low voltage differential signal (LVDS) channel without the need for signal shaping, the use of an analog phase locked loop (PLL), or magnetic components typical of Ethernet systems; (4) providing a soft reset mechanism; (5) providing a mechanism for monitoring the linlc status of both connections, simultaneously;
and (6) providing each processor with a full 10 Mbs comlectivity to a non-2o blocking Ethernet switch.
In accordance with one aspect of the present invention, a multiprocessor system is provided that has a plurality of processor modules coupled together via a backplane. The system comprises a first processor module having a first processor and a switch with a plurality of I/O ports and a plurality of 25 communication paths coupled to the I/O ports of the switch, the switch being operable to route data packets formatted according to an Ethernet MAC
protocol.
The system further comprises a second processor module having a second processor and a first communication device that is operable to communicate with the switch via a first communication path on the backplane. The system also 3o comprises a third processor module having a third processor and a second communication device that is operable to communicate with the switch via a second communication path on the backplane. The switch is operable to route Ethernet MAC protocol data packets from one of the first, second or third processors to another of said first, second or third processors.
In accordance with another aspect of the present invention a multiprocessor system is provided that has a plurality of processor modules coupled together via a backplane. The system comprises a first processor module having a first processor and a first switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O ports of the first switch, the l0 first switch being operable to route data packets. The system further comprises a second processor module having a second processor and a second switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O
ports of the second switch, the second switch being operable to route data packets. The system also comprises a third processor module having a third 15 processor and a first communication device that is operable to cormnunicate with the first switch via a first communication path on the backplane and operable to communicate with the second switch via a second communication path on the backplane. In addition, the system comprises a fourth processor module having a fourth processor and a second communication device that is operable to 2o communicate with the first switch via a third communication path on the backplane and operable to communicate with the second switch via a fourth communication path on the backplane. The first switch is operable to route data packets from one of the first, second, third or fourth processors to another of the first, second, third or fourth processors. The second switch is also operable to 25 route data packets from one of the first, second, third or fourth processors to another of the first, second, third or fourth processors.
BRIEF DESCRIPTION OF DRAWINGS
The present invention will become more apparent from the following 3o description when read in conjunction with the accompanying drawings wherein:

FIG. 1 is a front view of an exemplary backplane base multiprocessor system in which the present invention is useful;
FIG. 2 is a schematic view of an exemplary backplane based multiprocessor system;
FIG. 3 is a block diagram of a ring network in which the present invention is useful;
FIG. 4 is a block diagram showing a preferred coupling arrangement for the communication system according to the present invention;
FIG. 5 is a block diagram of the preferred system processor modules that l0 include the switches of the present invention;
FIG. 6 is a block diagram of a preferred line processor module and its communication device;
FIG. 7 is a block diagram of a preferred communication device;
FIG. 8 is a block diagram of a preferred I/O port;
FIG. 9 is a block diagram of a preferred transmitter block; and FIG. 10 is a block diagram of a preferred receiver block.
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
Refernng now to the figures, FIG. 1 shows an exemplary backplane based multiprocessor system 2 comprising a plurality of processor modules 10, 12,14, 16, 18, and 20 that are mounted in a shelf 22. As shown in FIG. 2, the shelf contains a backplane 24 which provides a physical media for allowing the modules 10, 12,14, 16, 18, and 20 to communicate with each other. Each module 10, 12, 14, 16,18, and 20 includes a connector 25 for providing electrical communication pathways between the backplane 24 and components on the processor modules 10,12,14,16,18, and 20.
As shown in FIG. 3, the exemplary multiprocessor system 2 is a multiple services Garner node 26 that can be used in networks carrying frame- , packet-, and cell-based traffic. The processor modules in this node 26 are either traffic carrying modules, i.e., modules that carry IP or ATM traffic to or from the node, or cross-connect modules, i.e., modules that pass IP or ATM traffic from one traffic carrying module to another traffic carrying module.
The present invention provides a highly robust, completely redundant, high-speed communication system for inter-processor communication in a multiprocessor environment. The communication system is implemented using Ethernet medium access control (MAC) over a backplane physical media.
As shown in FIG. 4, processor modules 10, 12, 14, 16, 18, and 20 are interconnected to allow for inter-processor communication. The connnunication scheme is based on an Ethernet protocol that is implemented using a different to physical media, the backplane. Each processor module includes a device that allows the processor module to communicate over the backplane.
The preferred multiprocessor system includes a set of redundant switches 28 and 30 that interconnect processor modules 10, 12, 14, 16, 18, and 20 via the backplane. Switches 28 and 30 could optionally reside on one or more of the processor modules 10,12, 14, 16, 18, and 20 or could optionally reside on a separate module. In the illustrated embodiment, switches 28 and 30 reside on processor modules 10 and 12, respectively, referred to hereinafter as the system processor modules. The switches 28 and 30 are the devices for backplane communication for the system processor modules.
2o The other processor modules 14, 16, 18, and 20, referred to hereinafter as the line processor modules, each include an output communication device 15 for backplane communication. In the preferred system, each communication device 15 is coupled to each switch 28 and 30 via a dedicated communication channel on the backplane. In the illustrated embodiment, the communication device 15 of processor module 14 is coupled to switch A via chaimel B 1 and coupled to switch B via channel B2. The other line processor modules are similarly coupled.
Finally, switch A and switch B are coupled to each other via channel A. Inter-processor communication is accomplished by the switches 28 and 30 passing data traffic from one processor module to another via the dedicated communication channels.

As shown in FIG. 5, the system processor modules 10 and 12 each include a high speed communication link, preferably 100 Mbits/s, between the on-board processor 11 and the on-board switch. Each switch 28 and 30 includes a plurality of ports. One port is coupled to a high speed link A, preferably 100 Mbits/sec, that provides a high speed commuiucation path between the switches. In addition, each switch 28 and 30 has a plurality of ports that are coupled to communication channels to the line processor modules 14-20. Optionally, each switch 28 and 30 could include a debug port.
The function of the Ethernet switches 28 and 30 is to allow a processor on to one of the processor modules to communicate with a processor on another of the processor modules. The protocol used for the corrununication is a modified Ethernet protocol. Because Ethernet is a widely known protocol and many CPUs have built-in media access controllers, the present invention provides a versatile and less complex system for inter-processor communication in a multiprocessor environment.
Communication between processor modules is via data packets that are formatted using an Ethemet media access control (MAC) protocol. Ethernet protocols and Ethernet MAC are well-known.
The physical media for connnunication includes the bacl~plane which 2o provides the communication channels a~ld the processor module connectors 25.
The I/O communication devices 15 and the switches 28 and 30 contain the circuitry to provide for the transmission of data over the communication channels.
As shown in figure 6, each switch 28 includes a switch agent 32, a transmitter block 34, a receiver block 36, and a data multiplexor 38. The switch agent 32 cornlnunicates with the on-board processor 40 to transfer data and instructions between the two. The switch agent 32 also sends data packets to the transmitter block 34 for transmission to another processor module and receives data packets from the receiver block 36 that were sent by another processor module. The switch agent 32 also has access to an address table in which it stores the addresses of the processor modules with which it can communicate.
The transmitter block 34 forwards data paclcets to a multiplexor 38 which routes the data packets to the port 46 assigned to the recipient of the message.
The multiplexor 38 also forwards data packets received from a port 46 to the receiver block. The multiplexor 38 is also capable of forwarding data packets to and from the debug port 48 and the high speed communication port 44 to the other switch.
As shown in figure 7, the communication devices 15 for the line processor l0 modules include a transmitter block 34, a receiver block 36, a data multiplexor 50, and two I/O ports 46. The preferred line processor modules use a PowerQUICC (MPC860) processor, which already has a built-in Ethemet MAC.
The MAC address does not need to come from a configuration memory on the board. The MAC address can be constructed based on a fixed number, and the slot ID in which the module physically resides.
The transmitter block 34 forwards data paclcets from the on-board processor to the data multiplexor 50. The receiver block forwards data packets from the data multiplexor 50 to the on-board processor. The data multiplexor selects which of the two ports 46 data paclcets are to be forwarded to from the 2o transmitter. The on-board processor instructs the data multiplexor 50 to select a particular port via the use AB Line.
The I/O ports 46 are coupled to the communication channel and transfers data thereon. Functionally, the I/O ports 46 are the same on both the switches and 30 and on the communication devices 15. The communication channels have two data paths, an upstream path with a direction of data flow from a communication device to a switch and a downstream path with a direction of data flow from a switch to a communication device. Because the data sent over the paths are differential signals, each communication channel requires four lines, two for each path.

_g_ As shown in figure 8, the I/O ports 46 include a transmit section and a receive section. The transmit section transfers data from the data multiplexor to the upstream path of the communication channel as a differential signal via a differential driver 56. The transmit section also includes a blip generator 52 that generates a blip on the upstream path after each millisecond of inactivity on the data path. A blip is a simple '1', Manchester encoded signal. It is not long enough to activate the detector (it lacks the Ethernet preamble), but it does trigger the activity detector that says the switch at the remote end is there and available. A 1 ms detector 54 monitors the port to determine if a millisecond has passed since to the last blip or transmission of data packets and signals the blip generator 52 to generate a blip when a millisecond has passed. This blip generation mechanism is used by the switch to determine if the communication channel is available.
The receive section of the I/O port 46 includes a differential receiver 58 for receiving data packets from the downstream data path and forwarding it to the receiver block via the data multiplexor. The receive section also includes a 4 millisecond activity detector 60 and a 1 Mhz detector 62. The activity detector 60 detects whether there has been activity, either data packets or a blip, on the data path within the past 4 milliseconds and communicates this information to the on-board processor as the link status. The 1 Mhz detector 62 looks for a special 1 2o Mhz pattern on the receive data , such as a Manchester encoding is invalid pattern and outputs a reset pulse to the processor if one is received.
An exemplary transmitter block 34 is shown in block diagram form in figure 9. The transmitter block 34 receives data packets from the on-board processor, encodes the bits using the Manchester encoder 64 and transmits the Manchester encoded data packets to the data multiplexor 50 (shown in figure 7) for forwarding to the active I/O port 46. The transmitter block 34 also generates a 20 Mhz clock signal from a 80 Mhz source for use by the Manchester encoder 64 and a 10 Mhz clock for use by the processor when transmitting the data packet.

_g_ An exemplary receiver block 36 is shown in block diagram form in figure 10. Received data is received from the I/O port 46 via the data multiplexor.
Using a local high-speed clock, the received data is first oversampled, and stored in a small FIFO 66. Resampling is re-synchronized on each data edges, which will give 3,4 or 5 samples per data bit. The data is then Manchester decoded via a Manchester decoder 68, and passed on to the Ethernet MAC along with the RxEN (data present) status.
The present invention provides many advantages and features. An advantage of the present invention over traditional Ethernet switches is that it l0 requires less physical space and uses fewer heat generating components because of the use of fewer analog circuits. In traditional Ethenzet switches a lot of physical space is devoted to the analog physical components and to heat dissipation resulting from the analog physical components. For example, traditional Ethernet switches use analog signal treatment methods for signal shaping, filtering, etc. Traditional switches also implement analog phase locked loops (PLLs) for clock recovery and magnetics for isolation. The present invention uses Manchester encoding, Iike in traditional switches, but eliminates signal shaping and the need for a PLL and magnetics.
Normally, the Ethernet clock is 10 Mhz. On the communication device 15, since there is no 10 Mhz analog PLL, a simple Digital PLL is implemented using an 8x-oversampling clock, i.e. 80 Mhz. Since data is transmitted via Manchester encoding, the clock and data is combined into a single signal and that signal is transferred as LVDS levels over the backplane. The clock recovery from the combined clock/data signal is via a digital mechanism and not through the use of a PLL. The clock recovery is performed using an x8 clock (80 Mhz).
Essentially, the signal is sampled at 80 Mhz, and converted to 10 Mbit clock by determining the signal transition edges. One advantage is that the clock recovery is entirely digital and can be implemented in a low cost device such as a field programmable gate array (FPGA).

Another advantage of the present invention is that it has a built-in remote processor reset feature. A special pattern (1 Mhz) can be sent from the system processor modules to the line processor modules to cause the line processor modules to reset. The communication devices 15 are configured to activate a reset command when they receive a "special" invalid pattern on the physical link, preferably a 1 Mhz clock. The communication devices 15 will not recognize the special pattern as normal data because it is an invalid signal in Ethernet world, but the communication devices will detect it and reset the processor. In the preferred system, the reset pattern will only be listened for on the link coming l0 from switch A. Any reset patterns received from switch B will be ignored.
Aiso, in the preferred system, the reset "detection" is active at all time. Reset detection is active even when the link status is down. This remote processor reset command allows fox the line processor modules to be reset without having to send a technician out to perform this function.
Another advantage of the present invention is redundancy. Each Ethernet interface on the line processor modules has access to two different Ethernet physical links, for redundancy. The two switches 28 and 30 provide the redundancy. When both system processor modules are operational, the line processor modules can communicate with either switches 28 or 30 without any difference in performance. When the linlc with one of the system processor modules is down, the line processor module will choose to communicate using the other link. Control over which switch the line processor module connects with is implemented using the UseAB control line as shown in figure 7.
Still another advantage of the present invention is its link active mechanism for keeping track of active links between line processor modules and the switches 28 and 30. The communication devices 15 are responsible for providing a minimum level of activity on each of the linlcs, regardless of whether the link is the active link or the standby link.
On the transmit side, a link is kept "active" by the transmit section of the 3o I/O port 46 sending a blip signal after one milliseconds of inactivity. On the standby Iink, the blip will be transmitted every millisecond. On the active link, the blip will start transmitting each 1 millisecond after the last transmission and will continue until traffic resumes. The cornlnunication devices 15 will communicate to the on-board processor, the link status with respect to each switch.
On the receive side, if the receive section of the I/O port 46 does not detect activity on the link after 4 milliseconds (complete silence), the link is declared down. It will be declared up and ready for use when activity is detected again.
1o Having described in detail the preferred embodiment of the present invention, including preferred modes of operation, it is to be understood that the present invention could be carried out with different elements and steps. The preferred embodiment is presented only by way of example and is not meant to Iimit the scope of the present invention, which is defined by the following claims.

Claims

What is claimed is:

1. A multiprocessor system having a plurality of processor modules coupled together via a backplane, comprising:
a first processor module having a first processor and a switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O
ports of said switch, said switch being operable to route data packets formatted according to an Ethernet MAC protocol;
a second processor module having a second processor and a first communication device that is operable to communicate with said switch via a first communication path on the backplane; and a third processor module having a third processor and a second communication device that is operable to communicate with said switch via a second communication path on the backplane; and wherein the switch is operable to route Ethernet MAC protocol data packets from one of said first, second or third processors to another of said first, second or third processors.

2. The system according to claim 1 wherein said first processor module is operable to transmit a reset command to said second processor module via said first communication path and wherein said first processor module is also operable to transmit a reset command to said third processor module via said second communication path.

3. The system according to claim 1 wherein said first and second communication devices each comprise an I/O port, a transmitter block, and a receiver block.

4. The system according to claim 3 wherein said I/O port comprises a transmit section and a receive section, said transmit section including a blip generator and an activity detector for generating a blip after a predetermined period of time worth of inactivity on the communication path, said receive section including a second activity detector for monitoring a communication path and generating a link status indication signal.

5. The system according to claim 4 wherein the blip is a Manchester encoded digital one signal.

6. The system according to claim 3 wherein said I/O port further comprises a differential transmitter for transmitting a low voltage differential signal onto the communication path.

7. The system according to claim 3 wherein said I/O port further comprises a differential receiver for receiving a low voltage differential signal from the communication path.

8. The system according to claim 3 wherein the transmitter block comprises a Manchester encoder for Manchester encoding a data packet to be transmitted on a communication path.

9. The system according to claim 3 wherein the receiver block comprises a Manchester decoder for decoding Manchester encoded data packets received from the communication path.

10. The system according to claim 3 wherein the receiver block comprises a digital phases lock loop circuit for recovering a receiver clock from the data packets received from the communication path.

11. The system according to claim 10 wherein the digital phase locked loop comprises a clock source having a higher frequency than the frequency of the received data and means for oversampling the received data to recover the receiver clock.

12. A multiprocessor system having a plurality of processor modules coupled together via a backplane, comprising:
a first processor module having a first processor and a first switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O
ports of said first switch, said first switch being operable to route data packets;
a second processor module having a second processor and a second switch with a plurality of Il0 ports and a plurality of communication paths coupled to the I/O ports of said second switch, said second switch being operable to route data packets;

a third processor module having a third processor and a first communication device that is operable to communicate with said first switch via a first communication path on the backplane and operable to communicate with said second switch via a second communication path on the backplane; and a fourth processor module having a fourth processor and a second communication device that is operable to communicate with said first switch via a third communication path on the backplane and operable to communicate with said second switch via a fourth communication path on the backplane; and wherein said first switch is operable to route data packets from one of said first, second, third or fourth processors to another of said first, second, third or fourth processors; and wherein said second switch is operable to route data packets from one of said first, second, third or fourth processors to another of said first, second, third or fourth processors.

13. The system according to claim 12 where the data packets are Ethernet MAC
protocol data packets.

14. The system according to claim 12 wherein said first processor module is operable to transmit a reset command to said second processor module via said first communication path and wherein said first processor module is also operable to transmit a reset command to said third processor module via said second communication path.

15. The system according to claim 12 wherein said first and second communication devices each comprise a first I/O port, a second I/O port, a transmitter block, and a receiver block and data multiplexor.

16. The system according to claim 15 wherein said first and second I/O ports each comprises a transmit section and a receive section, said transmit section including a blip generator and an activity detector for generating a blip after a predetermined period of time worth of inactivity on the communication path, said receive section including a second activity detector for monitoring a communication path and generating a link status indication signal.

17. The system according to claim 16 wherein the blip is a Manchester encoded digital one signal.

18. The system according to claim 15 wherein said I/O ports further comprises a differential transmitter for transmitting a low voltage differential signal onto the communication path.

19. The system according to claim 15 wherein said I/O ports further comprises a differential receiver for receiving a low voltage differential signal from the communication path.

20. The system according to claim 15 wherein the transmitter block comprises a Manchester encoder for Manchester encoding a data packet to be transmitted on a communication path.

21. The system according to claim 15 wherein the receiver block comprises a Manchester decoder for decoding Manchester encoded data packets received from the communication path.

22. The system according to claim 15 wherein the receiver block comprises a digital phases lock loop circuit for recovering a receiver clock from the data packets received from the communication path.

23. The system according to claim 22 wherein the digital phase locked loop comprises a clock source having a higher frequency than the frequency of the received data and means for oversampling the received data to recover the receiver clock.

24. The system according to claim 15 wherein said data multiplexor is operable to select one of said first or second I/O ports for transmitting data over a communication path.

25. The system according to claim 15 wherein said data multiplexor is operable to route data received from a communication path via said first or second I/O
ports to said receiver block.

26. The system according to claim 12 wherein said first and said second switches each comprise a data multiplexor and a plurality of I/O ports.

27. The system according to claim 26 wherein said data multiplexor is operable to route data packet received from one of said I/O ports to another of said I/O
ports.

28. The system according to claim 27 wherein each of said I/O ports is coupled via a distinct communication path to distinct I/O port on one of said communication devices.

29. The system according to claim 12 wherein the microprocessor system is a multiple services carrier mode.

30. A system for facilitating communication between a plurality of processor modules in a multi-processor system, comprising:
a backplane that provides a plurality of communication paths for coupling the processor modules, a first processor module having a first processor and a first switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O
ports of said first switch, said first switch being operable to route data packets;
a second processor module having a second processor and a second switch with a plurality of I/O ports and a plurality of communication paths coupled to the I/O ports of said second switch, said second switch being operable to route data packets;
a third processor module having a third processor and a first communication device that is operable to communicate with said first switch via a first communication path on the backplane and operable to communicate with said second switch via a second communication path on the backplane; and a fourth processor module having a fourth processor and a second communication device that is operable to communicate with said first switch via a third communication path on the backplane and operable to communicate with said second switch via a fourth communication path on the backplane; and wherein said first switch is operable to route data packets from one of said first, second, third or fourth processors to another of said first, second, third or fourth processors; and wherein said second switch is operable to route data packets from one of said first, second, third or fourth processors to another of said first, second, third or fourth processors.

31. The system according to claim 30 where the data packets are Ethenlet MAC
protocol data packets.