CA1315373C - Distributed control rapid connection circuit switch - Google Patents

Abstract

DISTRIBUTED CONTROL RAPID CONNECTION CIRCUIT SWITCH
Abstract A high capacity metropolitan area network (MAN) is described. Data traffic from users is connected to data concentrators at the edge of the network, and is transmitted over fiber optic data links to a hub where the data is switched.
The hub includes a plurality of data switching modules, each having a control means, and each connected to a distributed control space division switch.
Advantageously, the data switching modules, whose inputs are connected to the concentrators, perform all checking and routing functions, while the 1024x1024 maximum size space division switch, whose outputs are connected to the concentrators, provides a large fan-out distribution network for reaching many concentrators from each data switching module. Distributed control of the space division switch permits several million connection and disconnection actions to be performed each second, while the pipelined and parallel operation within the control means permits each of the 256 switching modules to process at least 50,000 transactions per second. The data switching modules chain groups of incoming packets destined for a common outlet of the space division switch so that only one connection in that switch is required for transmitting each group of chained packets from a data switching module to a concentrator. MAN provides security features including a port identification supplied by the data concentrators, and a check that each packet is from an authorized source user, transmitting on a port associated with that user, to an authorized destination user that is in the same group (virtual network) as the source user. Distributed control of the space division switch is implemented by dividing that switch into disjoint sets of connection and switching elements and controlling each such set with a differentcontroller. These controllers communicate via a separate control network with the data switching modules which request circuit connections.

Description

`- 1 31 5373 DISTRIBUTED CONTl~OL R~PID CONNECTION CIlRClJIT SWITCH

Technical Field This invention relates to telecommunications circuit switches.
Problem For a large metropolitan area network handling millions of transactions per second, it is advantageous to have a central switching network which can be used for switching all of the data messages of such a system. A network topology for such a switching network has been described in G. W. Richards, U.S.
Patent 4,566,007 ~Richards). Such a central switching network presents complex 10 problems of circuit control since the number of circuit connections which must be set up in such a central switching network if data is to be switched far exceeds the capability of any controller using present technology. It is therefore necessary to resort to some form of distributed control of such a network.
A problem in distributing the control of a large network is that two or 15 more controllers are usually required to cooperate in searching for a path and controlling such a network. Alternatively, single controllers may be used for searching for a path but these controllers must use a common data base and arrangements must be made so that two controllers do not access the same portion of the data base at the same time thus creating conflicts in the use of the switches 20 represented by that portion of the data base. In either arrangement, major inefficiencies are encountered in the cornmunications between network controllers and the lockout arrangements to prevent different c~ntrollers from accessing andchanging a portion of a common data base representing the state of the network. A
problem of the prior aTt, therefore, is that there is no efficient way of distributing the 25 control of a network wherein each of a plurality of controllers can set up complete connections without communicating with other controlle~s.
,,~.

Sol llti~n In accordance with one aspect of the ;nvention there is provided in a circ~lit switching network for establishing connections between ecach ot a plur~llity ot input terminals on one side and each of a pl-lrality of output terminals on anotller side, a method of control]ing said switching networlc using a pl~lrality ot` controllers comprising the steps of: dividing said network into a plurality of disjoint sets ot switching and connection elements, wherein each of said disjointed sets is ~Ised for establishing a dit`ferent set of connections trom ones of saicl plllrality ot terminals on a tirst side to al] of said pl-lrality ot terminals on a second side; ~Issignill~l to dilt`elellt ones ot said plllrality ot controllers, the control ot dit`t`erellt ones ot said diSjOillt sets:
and responsive to a request to set up a connection, controlling the estahlisl1lnent ot a connection using the controller assigned to the di.sjoint set used for that connection.
The above problem is solved and an advance is made over the prior art in accordance with the principles of this invention which advantageollsly partitions a network in such a ~vay as to permit each ot a multiplicity o~

controllers to control a disjoint subset of the network. ~dvantageously, each of the multiplicity of controllers can hunt for a path and set up a complete connection for its associated subset of such a network; the control rate for such a network is therefore the control rate per controller multiplied by the number of controllers which corresponds to the 5 number of subsets of the network.
In one specific embodiment of the invention, a 2-stage network of the type described in Richards has 1,024 inlets, 1,024 outlets, and is controlled by 64 controllers.
Each inlet generates requests for several thousand connections per second so that the total number of connections set up is several million per second. Such a network can readily be 10 broken down into independent segments each controlled by a separate controller. Each controller therefore independently sets up several tens of thousands of connections per second, a number which is well within the capabilities of present technology controllers.
In accordance ~vith this invention, each controller controls an independent segment comprising, in this embodiment, one output stage switch and the links and input 15 stage crasspoints for connecting to that output switch. The data set for maintaining the busy idle status of the output switch, the connected links, and the associated input crosspoints connected to these lirlks is disjoint from a similar data set of each other controller associated with a different output switch. Thus, each controller can perforrn the operation of hunting for a path to an output for its associated output switch consulting only its own disjoint data 20 set, a data set which is not affected by operations of any other controller.
Brief Description of the Drawings FIG. 1 is a graphic representation of the characteristics of the type of communications traffic in a metropolitan area network.
FIG. 2 is a high level block diagram of an exemplary metropolitan area network 25 (referred to herein as MAN) including typical input user stations that communicate via such a net vork.
~ IG. 3 is a more detailed block diagram of the hub of MAN and the units communicating with that hub.
FIGS. 4 and 5 are block diagrams of MAN illustrating how data flows from input 30 user systems to the hub of MAN and back to output user systems.
FIG. 6 is a simpli~led illustrative example of a type of network which can be used as a circuit switch in the hub of MAN.

~IG. 7 is a block diagram of an illustrative embodiment of a M~N circuit switch and its associated control network.
FIGS. 8 and 9 are flowcharts representing the flow of requests from the data distribution stage of the hub to the controllers of the circuit switch of the hub.
FIG. 10 is a block diagram of one data distribution switch of a hub.
FIGS. 11-14 are block diagrams and data layouts of portions of the data distribution switch of the hub.
FIG. 15 is a block diagram of an operation, administration, and maintenance ~OA&M) system for controlling the data distribution stage of the hub.
FIG. 16 is a block diagram of an interface module for interfacing between end user systems and the hub.
FIG. 17 is a block diagram of an arrangement for interfacing between an end - user system and a network interface.
FIG. 18 is a block diagram of a typical end user system.
FIG. 19 is a block diagram of a control arrangement for interfacing between an end user system and the hub of MAN.
FIG. 20 is a layout of a data paclcet arranged for transmission through MAN
illustrating the MAN protocol.
FIG. 21 illustrates an alterrlate arrangement for controlling access from the data distribution switches to the circuit switch control.
FIG. 22 is a block diagram illustrating arrangements for using MAN to switch voice as well as data.
FI(3. 23 illustrates an arrangement for s,vnchronizing data received from the eircuit switeh by one of the data distribution switches.
FIG. 24 illustrates an alternate arrangernent for the hub for switching packetized voice and data.
FIG. 25 is a block diagram of a MAN circuit switch controller.
General DescriPtion The Detailed Description of this specification is a description of an exemplary metropolitan area network (MAN) that incorporates the present invention. Such a network as shown in FIGS. 2 and 3 includes an outer ring of network interface modules (NIMs) 2 connected b~Y ~lber optic links 3 to a hub 1. The hub interconnects data and voice packets from any of the NIMs to any other NIM. The NIMs, in turn, are connected via interface modules to user devices connected to the network.
The large access circuit switch, MAN switch (MA~S) 10, in the hub of the MAN
network is a key element in providing the high throughput of the MAN networlc. Such a 5 switch must be controllable by a plurality of controllers operating in parallel and independently to increase the transaction rate (number connects and disconnects per second) of that switch. The switch is described with respect to FIGS. 6-9. Since the switch is controlled by the MINTs, the description of MINT operation with respect to FIGS. 10-15 is also highly pertinent.
10 1 INTRODUCI`ION
Data networks often are classified by their size and scope of ownership. Local area networks (LANs) are usually owned by a single organization and have a reach of a few kilometers. They interconnect tens to hundreds of terminals, computers, and other end user systems (EUSs). At the other extreme are wide area networks (WANs) spanning continents, 15 owned by common carriers, and interconnecting tens of thousands Oe EUSs. Between these extremes other data networks have been identified whose scope ranges from a campus to a metropolitan area. The high performance metropolitan area network to be described herein will be referred to as MAN. A table of acronyms and abbreviations is found in Appendix A.
Metropolitan area networks serve a variety of EUSs ranging from simple 20 reporting devices and low intelligence terminals throu~h personal computers to large mainframes and supercomputers. The demands that these EUSs place on a network vary widely. Some may issue messages infrequently while others may issue many messages each second. Some messages may be only a few bytes while others may be files of millions of bytes. Some EUSs may require delivery any time within the next few hours while others may 25 require delivery within microseconds.
This invention of a metropolitan area network is a computer and telephone communications network that has been designed for transmitting broadband low latency data which retains and indeed exceeds the performance characteristics of the highest performance local area networks. A metropolitan area network has size characteristics similar to those of 30 a class 5 or end-office telephone central office; consequently, with respect to size, a metropolitan area network càn be thought of as an end-office for data. The exemplary embodiment of the invention, hereinafter called MAN, was designed with this in mind.
However, MAN also fits well either as an adjunct to or as part of a switch module for an end-office, thus supporting broadband Integrated Services Digital Network(ISDN) services. MAN can also be effective as either a local area or campus areanetwork. It is able to grow gracefully from a small LAN through campus sized networks to a full MAN.
S The rapid proli~eration of workstations and their servers, and the growth of distributed computing are major factors that motivated the design of this invention. MAN was designed to provide networking for tens of thousands of diskless workstations and servers and other computers over tens of kilometers, where each user has tens to hundreds of simultaneous and different associations 10 with other computers on the network. Each networl~ed computer can concurrently genera~e tens to hundreds of messages per second, and re~quire I/l~ rates of tens tO
hundreds of millions of bits/second tMbps). Message sizes may range from hundreds of bits to rnillions of bits. With this level of pe~formance, MAN is capable of supporting remote procedure calls, interobject comrnunications, remote 15 demand paging, remote swappingj file transfer, and computer graphics. The goal is to move most messages (or transactions as they will be referred to henceforth3 from an EUS memory to another EUS memory within less than a millisecond for small transactions and within a few m~lliseconds for large transactions. FIG. 1 classifies transac~ion types and show desired EUS response times as a function of 20 both transaction type and size, simple (i.e., low intellîgence) terminals 70, remote procedure calls (RPCs) and interobject communications aOCs) 72, demand paging 74, memory swapping 76; animated computer graphics 78, computer graphics still pictures 80, file ~ransfers 82, and packetized voice 84. Meeting the response time/transaction speeds of FIG. 1 represents part of the goals of the 25 MAN network. As a calibration, lines of constant bit rate are shown where the bit rate is likely to dominate the response time. MAN has an aggregate bit rate of 150 gigabits per second and ean handle 20 million network ~ransactions per second with the exemplary choice of the processor elemen~s shown in FIG. 14.
Furthermore, it has ~een designed ~o handle ~affic overloads gracefully.
MAN is a network which performs switching and routing as many systems do, but also addresses a myriad of other necessa~y functions such as e~or handling, user interfacing, and the like. Significant privacy and security features in MAN are provided by an authentication capability. This capability prevents unauthorized network use, enables usage-sensitive billing, and provides non-35 forgeable source identification for all information. Capability also exists for defining virtual private networks.

MAN is a transaction-oriented (i.e., connectionless) network. It does not need to incur the overhead of establishing or maintaining connections although a connection veneer can be added in a straightforward fashion if desired.
MAN can also be used for switching packetized voice. Because of 5 the short delay in traversing the network, the priority which may be given to the transmission of single packet entities, auld the low variation of delay when thenetwork is not heavily loaded, voice or a mixture of voice and data can be readily supported by MAN. For clarity, the term data as used hereinafter includes digital data representing voice signals, as well as digital data representing commands, 10 numerical data, graphics, programs, data files and other contellts of memory. MAN, though not yet completely built, has been extensively simulated. Many of the capacity estimates presented hereinafter are based on these simulations.

2 ARCHlTECTURE AND OPERATION
15 2.1 Architecture The MAN network is a hierarchical star architecture with two or three levels depending upon how closely one looks at the topology. FIG. 2 shows the network as consisting of a switching center called a hub 1 linked to network interface modules 2 (NIMs) at the edge of the network.
The hub is a very high performance transacdon store-Md-forward system that gracefully grows from a small four link system to something very large that is capable of handling over 2û million network ~ansactions per secondand that has an ag~egate bit rate of 150 gigabits per second.
Radiating out from the hub for distances of up to tens of kilometers 2~ are optical fibers (or alternative data channels) called external links (XLs) (connect NIM to MINT), each capable of handling full duplex bit rates on the order of 150megabits per second. An XL terminates in a NIM.
A NIM, the outer edge of which delineates the edge of the network, acts as a concentrator/demuldplexer and also iden~fies ne~wo~c ports. It 30 concentrates when moving infoImation into the network and demultiplexes when moving information out of the network. Its purpose in concentrating/demultiplçxing is to interface multiple end user systems 26 (EUSs)to ~he network in such a way as to use the link efficiently and cost effectively.
Up to 20 EUSs 26 can be supported by each NIM depending upon the ElJSs 35 networking needs. Examples of such EUSs are the increasingly common advanced function workstations 4 where the burst rates are already in the 10 Mbps range (with the expectation that much faster systems will soon be available~ with average rates orders of magnitude lower. If the EUS needs an average rate that is closer to its burst rate and the average rates are of the same order of magnitude as that of a NIM, then a NI~I can either provide multiple interfaces to a single 5 EUS 26 or can provide a single interface with the entire NIM and XL dedicated to that EUS. Examples of ElJSs of this type include large mainframes 5 and file servers 6 for the above workstations, local area networks such as ETHERNF,T(~) 8and high performance local area networks 7 such as Proteon(g~ 80, an 80 MBit token ring manufactured by Proteon Corp., or a systern using a fiber distributed10 data interface (FDDI), an evolving American National Standards Institute (ANSI) standard protocol ring interface. In the latter two cases, the LAN itself may dothe concentration and the ND~I then degenerates to a single port network interface module. Lower performance local area networks such as ETHERNE~T 8 and IBM
token rings may not need all of the capability that an entire NIM provides. In 15 these cases, the LAN, even though it concentrates~ may connect to a port 8 on a multiport NIM.
Within each EUS there is a user interface module (UIM) 13. This unit serves as a high bit rate direct memory access port for the EUS and as a buffer for transactions received from the network. It also off-loads the EUS firom 20 MAN interface protocol concerns. Closely associated with the UlM is the MAN
EUS resident driver. It works Wit}l the IJIM to format outgoing transactions, receive incoming transac~ions, implement protocols, and interface with the EUSs operating system.
A closer inspection (see FIG. 3) of the hub reveals two different 25 functional units - a MAN switch (MANS) 10 and one or more memory interface modules 11 (MINTs). Each MINT is conn&cted to up to four NlMs via XLs 3 and t us can accommodate up to 80 EUSs. The choice of four NIMs per MINT is based upon a number of factors including transaction handling capacity, buffer memory size within the MINT, growability of the network, failure group size, and30 aggregate bit rate.
Each M~NT is connected to the MANS by four internal links 12 (ILs) (connect MINT and MAN switch), one of which is ihown for each of the MINTs in FIG. 3. The reason for four links in this case is different than it is for the XLs Here multiple links are necessary because the MINT will normally be sending 35 information through the MANS to multiple destinations concurrently; a single IL
would present a bottleneck. The choice of 4 Ls (as well as many other design ~315373 choices of a similar nature) was made on the basis of extensive analydcal and simulation modeling. The ILs run at the same bit rate as the external links but are very short since the entire hub is colocated.
The smallest hwb consists of one MINT with the ILs looped back and 5 no switch. A network based upon this hub includes up to four NIMs and accommodate up to 80 EUSs. The largest hub that is currently envisioned ccnsists of 256 MINTs and a 1024 x 1024 MANS. This hub accommodates 1024 ~Is and up to 20,000 EUSs. By adding MINTs and growing the hIANS, the hub and ultimately the entire network grows very gracefully.
10 2.1.1 UWUs, Packets, SUWUs, and Transactions Before going further several terms need to be discussed. EUS
transactions are transfers of units of EUS info~nadon that are meaningful to theEUS. Such transactions might be a remote procedure call consisting of a few bytes or the transfer of a 10 megabyte database. MAN recognizes two EUS
15 transaction unit sizes that are called long user work unit (LUWUs) and short user work units (SUWUs) for the purposes of this descripdon. While the delimiting size is easily engineerable, usually transaction ~mits of a couple of thousand bits or less are considered SUWUs while larger transaction units are LUWUs. Packets are given priori~ within the network to reduce response time based upon criteria20 shown in FIS~. 1 where it can be seen that the smaller EUS transaction units usually need faster EUS transaction response dmes. Packets are kept intact as a single frame or packet as they move through the network. LUWUs are ~agmented into frames or packets, called packets hereinafter, by the transrnitting UIM. Packets and SUWUs are sometimes collectively referred to as network 25 transaction units.
TIansfers through the MAN switch are referred to as swit-~h transacdons and the units transferred through the MANS are switch transaction units. They are ~omposed of one or more network transaction units destined for the sam~ NII~I.
30 2.2 Functional Unit Overview Prior to discussing the operation of MAN, it is useful to provide a brief overview of each major functional unit within the network. The units described are the UIM 13, ND!~I 2, MINT 11, MANS 10, end user system link:
(connects NIM and UIM) (EUSL) 14, XL 3, and IL 12 respectively. These units 35 are depicted in FIG. 4.

~315373 2.2.1 User Interface Module - UIM 13 This module is located within the E3US and often plugs onto an EUS
backplane such as a VME~3) bus (an IEEE standard bus), an Intel MULTIBUS
II(~)9 mainframe l/O channel. It is designed to fit on one printed circuit board for S most applications. The UIM 13 connects to the NIM 2 over a duplex optical fiber link called the EUS link 14 (EUSL), driven by optical transmitter 97 and 85. This lir.k runs at the same speed as the ex~ernal link (XL) 3. The UIM has a memory queue 15 used to store information on its way to the network. Packets and SUWUs are stored and forwarded to the NIM using out-of-band flow control.
By way of contrast, a receive buffer memory 90 must exist to receive information from the network. In this case entire EUS transactions may sometimes be stored until they can be transferred illtO End User System memory.
The receive buffer must be capable of dynamic buffer chaining. Partial EUS
transactions may arrive concurrently in an interleaved fashion.
Optical Receiver 87 receives signals from optical link 14 for storage in receive buffer memory 90. Control 25 controls UIM 13, and controls exchange of data between transrnit first-in-first-out (FIFO) queue 15 or receive buffer memory 90 and a bus interface for interfacing with bus 92 which connects to end user system 26. The details of the control of UIM 13 arc shown in FIG. 19.
20 2.2.2 Network Interface Module - NIM 2 A NIM 2 is the pare of MAN that is at the edge of the network. A
NIl!~ per.forms SLY func~ons: (1) concentration/demultiplexing including queuingof packets and SUWUs moving towa~d the MINl' and external link arbitration, (2 par~cipation in network security using polt identification, (3) participation in25 congestion control, (4) EUS-to network control message idendfication, (5) participation in error handling, and (6) network interfacing. Small queues 94 inmemory sirnilar to those 15 found in the UIM exist for each End User System.
They receive information from the UIM via link 14 and receiver 88 and store it until XL 3 is available for transrnission to the MINT. The outputs of these queue~
30 drive a data concentrator ~5 which in turn drives an optical transrnitter 96. An external link demand multiplexer exists which services demands for the use of Ihe XL. The NIM prefixes a port identification number 600 (FIG. 20) to each network transaction unit flowing toward the MINT. This is used in various wayc to provide value added services such as reliable and non-fraudulent sender 35 identification and billing. This prefix is particularly desirable for ensuring that members of a virtual network are protected from unauthoIized access by outsider~

A check sequence is processed for error control. The NIM, working with the hub 1, determines congestion status within the network and controls flow from the UIMs under high congestion conditions. The NIM also provides a standard physical and logical interface to the network including flow control mechanisms.InfoImation flowing from the network to the EIJS is passed through the NIM via receiver 89, distributed to the correct UIM by data distributor 86, and sent to desdnation UIM 13 by transmitter 85 via Ihlk 14. No buffering is done atthe NIM.
There are only two types of NIMs. One type (such as shown in 10 FM. 4 and the upper right of FIG. 3) concentrates while the other type (shown at the lower right of FIG. 3) does not.
2.2.3 Memory and Interface Module - MINT 11 MINTs are located in the hub. Each MINT 11 consists of: (a) up to four external link handlers 16 (XLHs) that tern~inate XLs a~d also receive signals 15 from the half of the internal link that moves data from the switch 10 to the MINT;
(b) four internal linls handlers 17 (ILHs) that generate data for the half of the IL
that moves data from a MINT to the switch; (c~ a memory 18 for sto~ing data while awaiting a path from the MINT through the switch to the destina~ion NIM;
(d) a Data Transport Ring 19 that moves data between the link handlers and the 20 mernory and also carries MINT control inforrnation; and (e) a control unit 20.
All fimctional units within the MINT: are designed to accommodate the peak aggregate bit rate for data moving conc~rently into and out of the MINT. Thus the ling, which is synchronous, has a set of reserved slots for moving infolmation from each XLH to memory and another set of reserved slots 25 for moving irlfonnation ~om memory to each ILH. It has a read plus write bit rate of over l.S (3bps. The memory is 512 bits wide so that an adequate memory bit rate can be achieved with components having reasonable access times. The size of the memony (16 Mbytes) can be kept srnall bècause the occupancy time of information in the memory is also small (about 0.57 milliseconds under full 30 network load). However, this is an engineerable number that can be adjusted if necessary.
I'he XLHs are bi-direcdonal but not symmetric. InfoImation moving from NIM to MINT is stored in MINT memory. Header inforrnation is copied by the XLH and sent to the MINT control for processing. ~ contrast, infonnation 35 moving from the switch 10 toward a NIM is not stored in the MINT but simply passes through the MINT, without being processed, on its way from MANS 10 output to a destination NIM 2. l:)ue to variable path lengths in the switch, theinformadon leaving the MANS 10 is out of phase with respect to the XL. A
phase alignment ~nd scrambler circuit (described in section 6.1) must align the data before transmission to the ND!~I can occur. Section 4.6 describes the internal 5 link handler (ILH).
The MINT pe~forms a variety of functions including (1) some of the overall routing within the network, (2) participation in user validation, (3) participation in network security, (4) queue management, (5) buffering of network transacdons, (6) address translation, (7) participation in congestion control, and (8) 10 the generation of operation, administration, and maintenance (OA&M) primidves.
The control ~or the MINT is a data flow processing system tailored to the MINT control algorithms. Each MINT is capable of processing up to 80,000 network transactions per second. A fully provisioned hub with 250 MINTs can therefore process 20 million network transactions per second. This is 15 discussed further in secdon 2.3.
2.2.4 MAN Switch - MANS 10 The MANS consists of two main parts (a) the fabric 21 tluough which information passes and (b) the control 22 for that fabric. The control allows the switch to be set up in about 50 microseconds. Special properties of the fabric 2û allow the control to be decomposed into completely independent su~controllersthat can operate in parallel. Additionally, each sub-controller can be pipelined.
Thus, not only is the setup time very fast but many paths can be set up concurrently and the "setup throughput" can be made high enough to accommodate high request rates from large numbers of ~s. MANs can be 25 made in various sizes ranging from 16x16 (handling four MlNTs) to 1024 x 1024 (handling 256 MI~lTs).
2.2.5 ~nd User System Link - EUSL 14 I'he end user system link 14 connects the NIM 2 to the UIM 13 that resides within the end user's equipment. It is a full duplex opdcal fiber link that 30 runs at the sarne rate and in synchronism with the eternal link on the other side of the NIM. It is dedicated to the EUS to which it is connected. The length of the EUSL is intended to be on the order of meters to 10s of meters. However, there is no reason why it couldn't be longer if econornics allow it.
The basic format and data rate ~or the EUSL f~r the present 35 embodirnent of the invendon was chosen to be the same as that of the MetrobusLightwave System OS-l link. Whatever link layer data transmission standard is eventually adopted would ~e used in later embodiments of MAN.
2~2.6 External Links - XL 3 The external link (XL) 3 connects the NIM to the MINT. It is also a full duplex synchronous optical fiber link. It is used in a demand multiplexed S fashion by the end user systems connected to its NIM. The length of the XL is intended to be on the order of 10s of kilometers. Demand multiplexing is used for econornic reasons. It employs the Metrobus OS-l forrnat and data rate.
2.2.7 Internal Li~lcs - IL 24 The internal link 24 provides connectivity between a MINT and the 10 MAN switch. It is a unidirectional semi-synchronous link that retains frequency but loses the synchronous phase relationship as it passes through the MANS 10.
The length of the IL 24 is on the order of meters but could be much longer if economics allowed. The bit rate of the IL is the same as that of OS-I. The format, howevçr, has only lirnited similarity to OS-l because of the need to 1~ resynchronize the data.
2.3 Software Overview Using a workstation/server paradigm, each end user system connected to MAN is able to generate over 50 EUS transactions per second consisting of LUWUs and SUWUs. This translates into about 400 network transactions per 20 second (packets and SUWUs). With up to 20 EIJS per NIM, each NIM must be capable of handling up to 8000 networl~ transactions per second with each MINT
handling up to ~our times this arnount or 32000 network transactions per second.Ihese are average or sustained rates. BuTst conditions may substantially increase "instantaneous" ra~es for a single EUS 26. Averaging over a number of EUSs 25 will, however, smoo~h out individual EUS bursts. Thus while each NIM port must deal with bursts of considerably more than 50 network ~ansactions per secolld, NIMs (2) and XLs (3) are likely to see only moderate bursts. This is even mo~e bue OI MINTs 11, each o~ which serves 4 NIMs. The MAN switch 10 must pass an average of 8 million network transactions per second, but the switch 30 controller does not need to process this many switch requests since the design of the MINT control allows multiple packets and SUWUs going to the same destination NIM to be switched with a single switch setup.
A second factor to be considered is network transaction interaIrival time. With rates of 150Mbps and the smalles~ network transaction being an 35 SUWU of lQ()0 bits, two SUWI~s could a~ive at a NIM or MINT 6.67 microseconds apart. NIMs and MINTs must be able to handle several back-to-13t5373 back SUWUs on a transient basis.
The control software in the NIMs and especially the MINls must deal with this severe real-time transaction processing. The asymmetry and bursty nature of data traffic requires a design capable of processing peak loads for short 5 periods of time. Thus the transaction control software stmcture must bç capable of executing many hundreds of niillions of CPU instructions per second (100's ofMIPs~. Moreover, in MAN, this control software performs a multiplicity of functions including routing of packets and SUWUs, network port identification, queuing of netwo~k transactions destined for the same NXM over up to 1000 NIMs 10 (this means real time maintenance of up to 10~0 queues), handling of MANS
requests and acknowledgements, flow control of source EUSs based on complex criteria, network traffic data collection, congestion control, and a myriad of other tasks.
The MAN control software is capable of perfom~ng all of the above 15 tasks in real time. The control software is executed in three major components:
NIM control 23, MINT control 20, and MANS control 22. Associated with ~hese three control components is a fourth control structure 25 within the UIM 13 of the End User System 26. FIG. S shows this ~rrangement. Each NIM and MINT has its own control unit~ The control units function independently but CoOpeTate 20 closely. This partitioning of control is one of the architectural mechanisms that makes possible MAN's real-time transaction processing capability. The other mechanism that allows MAN to handle high transaction rates is the technique of decomposing the control into a logical array of subf~mctions and independen~ly applying processing power to each subfunction. This approach has been greatly 25 ~acilitated by the use of Transputer~ very large scale integration (VLSI) processor devices made by INMOS Corp. The technique basically is as follows:
- Decompose the problem into a number of subfunctions.
- Arrange the subfunc~dons to form a dataflow structure.
- Implement each subfunction as one or more processes.
30 - Bind sets of processes to processors, arranging the bound processors in the sarne topology as the dataflow structure so as to form a da~qow system that will execute the funcdon.
- Iterate as necessary to achieve the real-time performance required.
Brief descriptions of the functions performed by the NIM, MINT, and 35 MANS (most of which are done by the software control for those modules) are given in sections 2.2.2 through 2.2.4. Additional information is given in section 2.4. Detailed descriptions are included later in this description within specific sections covering these subsystems.
2.3.1 Control Processors _ _ The processors chosen for the system implementation are Transputers S from INMOS Corp. Ihese 10 million instmctions/second (MIP) reduced instruction set control (RISC) machines are designed to be connected in an arbitrary topology over 20 Mbps serial links. Each machine has four links with an input and output path capable of simultaneous direct memory access (DMA).
2.3.2 MINI Control_e formance Because of the need to process a large number of transactions per second, the processing of each transaction is broken into serlal sections which form a pipeline. Transa tions are fed into this pipeline where they are processed simultaneously with other transactions at more advanced stages within the pipe.
In addition, there are multiple parallel pipelines each handling unique processing 15 streams simultaneously. Thus, the required high transaction processing rate, where each transaction requires routing and other complex servicing, is achieved by breaking the control structure into such a paralleVpipelined fabric of interconnected processors.
A constraint on MINT control is that any serial processing can take no 20 longer than 1/ (number of transactions per second processed in this pipeline).

A further constraint coneerns the burst bandwidth ~or headers entering ~he controt within an XLH 16. If ~e time between successive network units arriving ~t the XLH is less ~han (header size) / (bandwidth into con!rol) then the XLH must buffer headers. The maximum number of transactions per second assuming uniform a~rival is given by:

(bandwidth into control) I ( size of transaction header).

An example based upon the effective bit rate of transputer linlcs and the 40 byle 30 MAN network transaction header is:

1 3 ~ 5373 (8.0Mb/s for control link)/(320 bit header/transaction) - 25,0()0 transactions/sec. per XLH, or one transaction per XLH every 40 microseconds. Because transaction interarrival times can be less than this, header buffering is performed in the XLH.
S The MINT must be capable, within this time, of routing, executingbilling primitives, rnaking switch requests, performing network control, memory management, operation, administration, and maintenance activities, name serving,and also providing other network services such as yellow page primitives. The paralleV pipelined nature of MINT control 20 achieves these goals.
As an example, the allocating and freeing of high-speed memory blocks can be processed completely independently of routing or billing primitives.
Transaction flow within a MINT is controlled in a single pipe by the management of the memory block address used for storing a network transaction unit tie.
packet or SUWIJ). At the first stage of the pipe, memory management allocates 15 free blocks of high-spçed MINT memory. Then, at the next stage, these blocks are paired with the headers and routing translation is done. Then switch units are collected based on memory blocks sent to comrnon NIMs, and to close the loop the memory blocks are freed after the blocks' data is transmitted into the MANS.Billing primitives are simultaneously handled within a dif~erent pipe.
20 2.4 MAN eration The EUS 26 is viewed by the network as a user with capabilities granted by a network administration. This is an~ogous to a terminal user logged into a time-sharing system. The user, such as a wQrkstation or a fron~ end processor acting as a concentrator for stations or even networks, will be required 25 to make a physical connection at a NIM port and then idendf~,r itself via its MAN
name, virtual networlc identification, and password secunty. The netwo~k adjustsroudng tables to map data destdned for this name to a unique NIM p~rt. The capabilides of this user are associa~ed with the physical port. The example justgiven accommodates the paradigm OI a portable wo*station. Ports may also be 30 conIigured to have fixed capabilities and possibly be "owned" by one MAN named end user. This gives users dedicated network ports or provides privileged adminis~ative maintenance ports. The source EUS refer to the destinadon by MAN names or services, so they are not required to know anything about the dynarnic network topology.

The high bit rate and large transaction processing capability internal to the network yield very short response times and provide the EUS with a means to move data in a metropolitan area without undue networl~ considerations; A MAN
end user will see EUS-memory-to-EUS memory response times as low as a S millisecond~ low error rates, and the ability to send a hundred EIJS transactions per second on a sustained basis. This number can expand to several thousand for high performance EUSs. The EUS will send data in whatever size is appropriate to his needs with no maximum upper bound. Most of the limitations on optimizing MAN performance are imposed by the lim~ts of the EUS and 10 applications, not the overhead of the network. The user will supply the following inforrnation on transmitting data to the UIM:
- A MAN name and virtual network name for the destination address that is independent of the physical address.
- The size of the data.
15 - A MAN type field denoting network service required.
- The data.
Network transactions (packets and SUWUs) move along the following logical path (see FIG. 5):

sourceUIM ~=> sourceNIM -=> MINT ==> MANS==> destinationNIM(via 2û MINT) =-> destinationUIM.

Each EUS transaction (i.e., LUWU or SUWU) is submitted to its UIM. Inside the UIM, a LUWU is further fragmente~ into variable size packets. An SUWU is not fragmented but is logically viewed in its entirety as a network transacdon.
However, the determination that a network transaction is an SUWU is not made 25 until the SUWU reaches the MINT where the information is used in dynamically categorizing data into SUWUs and packets for optimal network handling. The NIM checks inc~ming packets from the EUS to verify that they do not violate a maximum packet size. 7he UIM may pick packe~ sizes smaller Ihan the maximum depending on EUS stated service. For op~imum MINT memory 30 utilization, the packet size is the standard maximum. However under some circumstances, the application may request that a smaller packet size be used because of end user consideration such as tirn~ng problems or data availability timing. Additionally, there may be timing limits where the UIM will send what itcurrently has from the EUS. Even where the maximum size packet is used, the :

last packet of a LUWU usually is smaller than the maximum size packet.
At the transmitting UIM each network transaction (packet or SUWU) is prefixed with a fixed length MAN network header. It is the inforrnation within this header which the MAN network software uses to route, bill, offer network 5 services, and provide network control. The destination UIM also uses the inforrnation within this header in its job of deliveIing EUS transactions to the end user. The network transactions are stored in the UIM source transaction queue from which they are transmitted to the source NIM.
Upon receiving network transactions from UIMs, the NIM receil~es 10 them in queues pe~nanently dedicated to the EUSLs on which the transaction arrived, for forwarding to the MINT 11 as soon as the link 3 becomes available.
The control software within the NIM pro esses the UIM to NIM protocol to identify control messages and prepends a source port number to the transaction that will be used by the MINT to authenticate the transaction. End-user data will 15 never be touched by MAN network software unless the data is addressed to the network as contr~l informativn provided by the end user. As the transactions areprocessed, the source NIM concentrates them onto the external linlc between the source NIl!~I and its MINT. The source NIM to MINT links terminate at a hardware inter~ace in the MINT (the external linlc handler or XLH 16).
The external link protocol between the N~I and MINT allows the XLH 16 to detect the beginning and end of network transactions. The transactionsare immediately moved into a memory 18 designed to handle the 150Mb/s bursts of data amving at the XLH. This memory access is via a high-speed time slotted ring 19 which guarantees each l50Mb~s XLEI input and each l50Mb/s output from 25 the MINT (ie. MANS inputs) bandwidth with no contention. For example, a MINT which concen~tes 4 remote NIMs and has 4 inpu~ por~s to the center switch must have a burst access bandwidth of at least 1.2C;b/s. The memory storage is used in fixed leng~ blocks of a size egual to the maximum paclcet size plus the fixed length MAN header. The XLH moves an address of a ~Sxed size 30 memory block followed by the packet or SUWU data to the memory access ring.
The data and network header are stored until the MINT control 20 causes its transrnission into the MANS. The MINT control 2û will continually supply the XLHs with free memory block addresses for storing the incoming packets and SUWUs. The XLH also "knows" the length of the fixed size network header.
3S With this information the XLH passes a copy of the network header to MINT
control 20. MINl` control 20 pairs the header with the block address it had given 1~ -the XLH for storing the packet or SUWU. Since the header is the only internal representation of the data within MINT control it is vital that it be correct. To ensure sanity due to potential link errors the header has a cyclic redundancy check (CRC) of its own. The path this tuple takes within MINT control must be the S same for all packets of any given LUWU (this allows ordering of LUWU data to be preserved). Packet and SUWU headers paired with the MINT memory block address will move through a pipeline of processors. The pipeline allows multipleCPUs to process different network transactions at various stages of MINT
processing. In addition, there are multiple pipelines to provide concu~rent 10 processing.
MINT control ~0 selects an unused internal link 24 and requests a path setup from the IL to the destination NIM (through thç MINT attached to thatNIM). MAN switch control 21 queues the request and when, the path is available and (2) the XL 3 to the destination NIM is also available, it notifies the source 15 MINT while conculTently setting up the path. This, on average and under full load, takes 50 microseconds. Upon notification, the source MINT transmits all network transactions destined for that NIM, thus taking maximum advalltage of the path setup. The internal link handler 17 requests network transactions from the MINT memory and-transmits them owr the path:
ILH ==> sourceIL =--> MANS ==> desdnationIL ==> XLH, this XLH being attached to the desdnation NIM. The XLH recovers bit synchronization on the way to the destination NIM. Note that information, as it leaves the switch, simply passes through a MINT on its way to the destination NIM. The MINT doesn'~ process it in any way other than to recover bit 25 synchronization that has been lost in going through the MANS.
As info~mation (i.e., switch transacdons made up of one or more network transactions) arrives at the destination NIM it is demultiplexed into network transactions (packets and SUWUs) and forwarded to the destination UIMs. This is done "on the fly"; there is no buf~`ering in the NIM on the way out 30 ~ the network.
The receiving UIM 13 will store the network transactions in its receive buffer memory 90 and recreate EUS transactions (LUWUs and SUWUs).
A LUWU may arrive at the UIM in packet sized pieces. As soon as at least part of a LUWU arrives, the UIM will notify the EUS of its existence and will, upon instructions from the EUS, transmit under the control of its DMA, partial EUS orwhole EUS transactions into the EUS memory in DMA trans~er sizes specified by the EUS. Alternate paradigms exist for transfer from UIM to EUS. For instance, an EUS can tell the UIM ahead of time that whenever anything arrives the UIM
should transfer it to a specified buffer in EUS memory. The UIM would then not need to announce the alTival of information but would irnmediately transfer it tO
the EUS.
2.5 Additional Considerations 2.S.l Error Handling In order to achieve latencies in the order of hundreds of microseconds from EUS memory to EUS memory, errors must be handled in a manner that dif~ers from that used by conventional data networks today. In MAN, network transactions have a header check sequence 626 (FIG. 20) (HCS) appended to the header and a data check sequence 646 (FIG. 20) (DCS) appended to the entire 15 network transaction.
Consider the header first. The so~ce UIM generates a HCS before transmission to the source NIM. At the MINT the HCS is checked and, if in error, the transaction is discarded. The destination NIM performs a similar action for a third time before routing the transaction to the destination UIM. This 20 scheme prevents rnisdelivery of information due to corrupted headers. C)nce aheader is found to be flawed, nothing in the header can be considered reliable and the only option that MAN has is to discard the ~ansacdon.
The source UIM is also required to provid~ a DCS at the end of the user data. This field is checked within the MAN networl~ but DO action is taken if 25 errors a~e found. The information is delivered to the destination WIM who cancheck it and take appropriate action. Its use within the network is to identify both EUSL and in~ernal network problems.
Note that there is never any attempt within the network to correct errors using the usual automatic repeat request (ARQ) techniques found in most of 30 today's protocols. The need for low latency prccludes this. Error correctirlgschemes would be too costly except for the headers, and even here the dme penalty may be too great as has sometimes been the case in computer systems.
However, header error correction may be employed later if experience proves th.
it is needed and time-wise possible.

Consequently, MAN checks for errors and discards transactions when there is reason to suspect the validity of the headers. Beyond this, transactions are delivered even if flawed. This is a reasonable approach for three reasons. First, intrinsic error rates over optical fibers are of the same order as error rates over S copper when common ARQ protocols are employed. Both are in the range of 10-1l bits per bit. Secondly, graphics applications (which are increasing dramatically) often can tolerate small error rates where pixel images are transmitted; a bit or two per image would usually be fine. Finally, where error rates need to be better than the intrinsic rates, EUS-to-EUS Al;LQ protocols can be 10 used (as they are today) to achieve these improved error rates.
2.5.2 Authentication .
MAN provides an authentication feature. This feature assures a destination EUS of the identity of the source EUS for each and every transacdon it receives. Malicious users cannot send transactions with forged "signatures".
15 Users are also prevented from using the network free of charge; all users areforced to identify themselves truthfully wi~h each and every transacdon that they send into the network, thus prov~ding for accurate usage-sensidve billing. This feature also provides the primitive capabili~ for other features such as virtualpnvate networks.
When an EUS first attaches to MAN, it "logs in" to a well known and privileged Login Server that is part of the ne~work. The login server is in an administrative terminal 350 (FIG. 15) widl an attached disk memory 351. The administrative terminal 350 is accessed via an OA&M MINT processor 315 (FIG. 14) and a MINT OA&M monitor 317 in the MINT central control 207 and 25 an OA&M central control (FIG. 15). This login is achieved by the EUS tvia itsUIM) sending a login transaction tO the server through the network. This transac~on contains the EUS identification numb~r (its name), its re~ueseed virtual network, and a password. In the NIM a port number is prefixed to the transactionbe~ore it is forwa~ded to the MINT for routing to the server. The Login SeIver 30 notes the idlport pairing and infonns the MINT attached to the source NIM of that pairing. It also acknowledges its receipt of the login to ~he ElJS, telling the EUS
that it may now use the network.
When using the network, each and every network transaction that is sent to the source NIM from the EUS has, within its header, its source id plus 35 other information in the header described below with respect to FIG. 20. The NIM prefixes the port number to the transaction and forwards it to the MIN~

where the pairing is checked. Incorrect pairing results in the MINT discarsiing the transaction. In the MINT, the prefixed source port number is replaced with a destination pOlt number before it is sent to the destination NIM. The destination NIM uses this destination port number to complete the routing to the destination5 EUS.
If an EUS wishes to disconnect from the netwoTk, it "logs off" in a manner similar to its login. The Login Server inforrns the MINl' of this and theMINT removes the id/port infoIrnation, thus rendering that port inactive.
2.5.3 Guaranteed Orderin~
E~rom NIM to NIM the notion of a LUWU does not exist. Even though LUWUs lose their identity within the NIM to NlM envelope, the packets of a given LUWIJ must follow a path through predetermined XLs and MINls.
This allows ordering of packets arriving at UIMs to be preserved for a LUWU.
However, packets may be disca~ed due to flawed headers. The UIM checks for 15 rnissing packets and notifies the EUS in the event that this occurs.
2.5.4 Virtual S~ircuits and Infinite LUVVUs The network does not set up a circuit through to the destination but rather switches groups of packets and SUWUs as resources become available.
This does not prevent the EUS from setting up virtual circuits; for example the 20 EUS could write an infinite size LIJWU with the appropriate UIM timing parameters. Such a data stream would appear to the EUS as a virtual circuit while to the network it would be a never ending LUWU that moves packets at a time.
The implementation o~ this concept must be h~uldled between the UIM and the EUS protocols since there may be many different types of EUS and ~IMs. The 25 end-user can be transmitting multiple data streams to any nurnber of destinations at any one time. These streams are multiplexed on packet and ~UWUs bvundaIies on the transmit link between the source UIM and the source NIh.
A parameter, to ~e adjusted for optimum performance as the system is loaded, limits the time (equivalent to limiting the length of the data stream) ~hat 30 one MINT can send data to a NIM in order to free that NIM to receive data from other MINTs. An initial value of 2 milliseconds appears reasonable based on simulations. The value can be adjusted dynamically in response to traffic patterns in the system, with different values possible for different MINTs or NlMs, and at different times of the day or dif~erent days of the week.

3 S~iVITCH
The MAN switch (MANS) is the fast circuit switch at the center of the MAN hub. It interconnects the MINTs, and all end-user transactions must pass through it. The MANS consists of the switch fabric itself, (called the data5 network or DNet), plus the switch control complex (SCC), a collection of controllers and links that operate the DNet ~abric. The SCC must receive requests from the MINTs to connect or disconnect pa~rs of incoming and outgoing internal links (ILs), execute the requests when possible, and inform the MINTs of the outcome of their requests.
These apparently straightforward operations must be calTied out at a high performance level. The demands of the MAN switching problem are discussed in the next section. Next, Section 3.2 presents the fundamentals of a distributed-cont~ol circuit-switched network that is offered as a basis for a solution to such switch~ng demands. Section 3.3 tailors this approach to the specific needs 15 of MAN and covers some aspects of the control structure that are critical to high performance.
3.1 Characterizin~ the Problem First we estimate some numerical values for the demands on the MAN switch. Nominally, the MANS must establish or remove a transaction's 20 connection in fractions of a millisecond in a network with hundreds of ports, each runr~ing at 150 Mb/s and each caIrying thousands of separately switched ~ansactions per second. Millions of transaction requests per second imply a distributed control structure where numerous pipelined controllers process ~ansaction requests in parallel.
The combination of so many~ ports each running a high speed has several implications. First, the bandwidth of the network must be at least 150 Gb/s, thus requinng multiple data paths (nominally 150 Mb/s) through the network. Second, a 150 Mb/s synchronous network would be difficult to build (although an asynchronous network needs to recover clock or phase). Third, since30 inband signaling creates a more complex (self-routing) network fabric and requires buffering within the network, an out-of-band signaling (separate control) approach is desirable.
In MAN, ~ansaction lengths are expected to vary by several orders of magnitude. These transactions can share a single s vitch, as discussed hereinaf~er 35 with adequate delay performance for small transactions. The advantage of a single fabric is ~hat data strearns do not have to be separated before switching and recombined afterwards.
A problem to be dealt with is the condition where the requested output port is busy. To set up a connection, tbe given input and output ports must be concurrently idle (the so-called concurrency problem). If an idle input (output) 5 port waits for the output (input~ to become idle, the waiting port is inefficiently utilized and other transactions needing that port are delayed. If the idle port is instead given to other transactions, the original busy destination port may havebecome idle and busy again in the meantime, thus adding further delay to the original transaction. The delay problem is worse when the port is busy with a 10 large transacdon.
Any concurrency resolution strategy requires that each port's busy/idle status be supplied to the controllers concemed with it. To maintain a high transaction rate, this status update mechanism must operate with short delays.
If transaction times are short and most delays are caused by busy 15 ports, an absolutely non-blocking network topology is not required, but the blocking probability should be small enough so as not to add much to delays or burden the SCC with excessive unachievable connection requests.
Broadcast (one to many) connections are a desirable network capability. However, even if the network SUppOltS broadcasting, the concurrency 20 problem (here even worse with the many ports involved) must be handled without disrupting other traf~ic~ This seems to rule out the simple strategy of waiting for all destination ports to become idle and broadcasting to all of them at once.
Regardless of the special needs of the MAN network, the MANS
satisfies the general requirements for any practic~ll network. Startup costs are25 reasonable. The network is growable without disrupting existing fabric. The ; topology is inherently efScient in its use of fabrir and circuit boards. Finally, the conc~rns of operational availability - reliability, ~ault tolerance, failure-group sizes, and ease of diagnosis and repair - are met.
3.2 General Appr~ach - A Distributed-Control Circuit-Switching Network In this section we describe the basic approach used in the MANS. It specifically addresses the means by which a large netwolk can be rlm by a group of controllers operating in parallel and independently of one another. The distributed control mechanism is described in terms of two-stage networ~s, but with a scheme to extend the approach to multistage networks. Section 3.3 35 presents details of the specific design for MAN .

A major advantage of our approach is that the plurality of network controllers operate independently of one another using only local in~ormation.
Throughput (measured in transactions) is increased because controllers do not burden each other with queries and responses~ Also the delay in setting up or S tearing down connections is reduced because the number of sequential control steps is minimized. All this is possible because the network fabric is partitioned into disjoint subsets, each of which is controlled solely by its own controller that uses global static inforrnation, such as the internal connection pattern of the data network 120, but only local dynamic (network state) data. Thus, each controller 10 sees and handles only those connection requests that use the portion of the network for which it is responsible, and monitors the staee of only that portion.
3.2.1 Partitionin~ Two-Stage Networks Consider the 9 x 9 two-stage network example in FIG. 6 comprising three input switches ISl (101), IS2 (102), and IS3 (103), and three output switches 15 OSl (104), OS2 (105), and OS3 (106). We can partition its fabric into three disjoint subsets. Each subset includes the fabric in a given second stage switch(OSx) plus the fabric (or crosspoints) in the first stage switches (ISy) that connect to the links going to that second stage switch. For example, in FIG. 6, the partition or subset associated with OSI (104) is shown by a dashed line around 20 the crosspoints in OSl plus dashed lines around three crosspoints in each of the first stage switches (101,102,103) (those crosspoints being those that connect tO
the links to OSl).
Now, consider a controller for this subset of the network. It would be responsible for connec~ions from any inlet to any outlet on OSl. The controller 25 would maintain busy/idle statos for the crosspoints it controlled. This information is slearly enough tO tell whether a ~onnection is possible. For example, supposean inlet on ISl is to be connected to an outlet on OSl. We assurne that the request is from the inlet, which must be idle. The outlet can be detemuned to beidle from outlet busy/idle status memory or else from the status of ~he outlet's30 three crosspoints in OSI (all three must ~e idle)~ Next, the status of the link between ISl and OSl must ~e checked. This link will be idle if the two crosspoints on both ends of the link, which connect the link to the remaining two inlets and outlets, are all idle. If the inlet, outlet, and link are all idle, acrosspoint in each of ISl and OSI can be closed to set up the requested 35 connection.

Note that this activity can proceed independently of activities in the other subsets (disjoint) of the network. The reason is that the network has onlytwo stages, so the inlet switches may be partitioned according to their links tosecond stage switches. In theory this approach applies to any two-stage network,5 but the usefulness of the scheme depends on the network's blocking characteristics. The network in FIG. 6 would block too frequently, because it can connect at most one inlet on a given inlet switch to an outlet on a given seco,nd stage switch.
A two-stage network9 referred to hereinafter as a Richards network, of 10 the type des~ribed in G. W. Richards et al.: "A Two-Stage Rea~rangeable Broadcast Switching Network, IEEE Transactions on Communications, v. COM-_ 33, no. 10, October 1985, avoids this problem by wiring each inlet port tomultiple appearances spread over different inlet switches. The dis~ibuted control scheme operates on a Richards network, even though MAN may not use such 15 Richards network features as broadcast and rearrangement.
3.2.2 Control Network 3.2.2.1 Function .
In MAN, requests for connections come from inlets, actually, the central control 20 of the MINTs. These requests must be distributed to the proper 20 switch controller via a control network (CNet). In FIG. 7, both the DNet 120 for circuit-switched transactions and the control CNet 130 are shown. The DNet is a two-stage rearra~geably non-blocking Richards network. Each switch 121,123 includes a rudiment~ary crosspoint controller (XPC) 122,124 which accep~s commands to connect a syecified inlet on the switch to a specified outlet by 25 closing the proper crosspoint. The first and second stages' XPCs (121,123) are abbreviated lSC (first stage controller) and 2SC (second stage controller3 respectively.
On the right side of the CNet are 64 MANS controllers 140 (MANSCs) corresponding to and controlling 64 disjoint subsets of the l:~Net, 30 partidoned by second stage outlet switches as described earlier. Since the controllers and their netwoIk are oYerlaid on the DNet and not integral to the data fabric, they could be replaced by a single controller in applications where transaction throughput is not critical.

1~15373 3.2.2.2 Structure The CNet shown in FIG. 7 has special properties. It consists of three similar parts 130,134,135, corresponding to flows of messages from a MINT to a MANSC, orders from a MANSC to an XPC, and acknowledgments or negative S acknowledgments ACKs/NAKs from a MANSC to a MINT; acknowledge tACK), negative acknowledge (NAK). Each of the networks 130,134 and 135 is a statistically multiplexed dme-division switch, and cornprises a bus 132, a group of interfaces 133 for buffering control data to a destination or from a source, and a bus arbiter controller (BAC) 131. The bus arbiter controller controls the gating of 10 control data from an input to the bus. The address of the destination selects the output to which the bus is to be gated. The output is connected to a controller (network 130: a MANSC 140) or an interface (networks 131 and 132, interfaces similar to interface 133). The request inputs and ACK/NAK responses are concen~ated by control data concentrators and distributors 136,138, each control15 data concen~ator concentrating data to or from four MINTs. The control data concentrators and distributors simply buffer data from or to the MINTs. The interfaces 133 in ~he CNet handle statistical demultiplexing and multiplexing (steering and mergir~g) of control messages. Note that the interconnections madeby bus 132 for a given request message in the DNet are the sarne as those 20 requested in the CNet.
3.2.3 Connection Request Scenario The connection request scenario begins with a connection request message arriving at the lef~ of CNet 130 in a multiplexed stream on one of the message input links 137 from one of the data concentrators 136. This request 25 includes the DNet 120 inlet and outlet to be connected. In the CNet 130, the message is routed to the appropriate link 139 on the right side of the CNet according to the ou~et to be connected, which is uniquely associated with a particular second stage switch and therefore also with a particular MANS
controller 140.
This MANSC consults a static global directory (such as a ROM) to find which first stage switches carry the requesting inlet. Independently of other 1~4NSCs, it now hecks dynamic local data to see whether the outlet is idle and - any links from the proper first stage switches are idle. If the required resources are idle, the MANSC sends a crosspoint connect order to its own second stage 35 outlet switch plus another order to the proper first stage switch via network 134.
The latter order includes a header to route it to the correct first stage.

This approach can achie7re extremely high transaction throughput for several reasons. All network controllers can operate in parallel, independently of one another, and need not wait for one another's data or go-aheads. Each controller sees only those requests for which it is responsible and does not waste S time with other messages. Each controller's operations are inherently sequential and independent functions and thus may be pipelined with more than one request in progress at a time.
The above scenario is not the only possibility. Variables to be considered include broadcast -vs- point-to-point inlets, oudets -vs- inlet-oriented 10 connection requests, rea~rangement -vs- blocking-allowed operation, and disposition of blocked or busy connect requests. Although these choices are already setded for MAN, all these options can be handled with the control topology presented, simply by changing the logic in the MANSCs.
3.2.4 Multista~Networks This control structure is extendible to multistage Richards networks, where switches in a given stage are recursively implemented as two-stage networks. The resultant CNet is one in which connection requests pass sequentially through S-l controllers in an S-stage networl~, where again controllers are responsible for disjoint subsets of the network and operate independently, thus 20 retaining the high throughput potential.
3.3 Specific Desi~n for MAN
In this section we first examine those system attributes that drive the design of the ~NS. Next, the data and control netw~rks are described. Finally the functions of the MANS controller are discussed in detail, including design 25 ~adeoffs that affect perifiormance.
3.3.1 System Attributes 3.3.1.1 Ex~e~nal and Internal Interfaces .
FIG. 7 illustrates a prototypical fully-grown MANS composed of a DNet 121 with 10~ incoming and 1024 outgoing ILs and CNet 22 comprising 30 three contr~l message networks 130,133,134 each with 64 incoming and 64 outgoing message links. Ihe ILs are partitioned into groups of 4, one group for each of 256 MINTs. The DNet is a two-stage network of 64 first stage switches 121 and 64 second stage switches 123. Each switch includes an XPC 122 that takes comLmands to open and close crosspoints. For each of the 35 DNet's 64 second stages 123~ there is an associated MANSC 140 with a dedicated control link to the XPC 124 in its second stage switch.

Each control link and status link interfaces 4 MINTs to the CNet's left-to-right and right-to-left switch planes via 4:1 control data concentrators and distributors 136,138 which are also part of the CNet 22. These may be regarded either as remote concentrators in each 4-MINT group or as par~s of their S associated 1:64 CNet 130,135 stages; ;n the present embodiment, they are part of the CNet. A third 64x64 plane 134 of the CNet gives each MAMSC 140 a dedicated right-to-left interface 133 with one link to each of the 64 lSCs 122.
Each MINT 11 interfaces with the MANS 10 through its four ILs 12, its request signal to control data concentrator 136, and the acknowledge signal received back 10 from control data distnbutor 138.
Alternately, each CNet could have 256 instead of 64 ports on it~
MINT side, eliminating the concentrators.
3.3.1.2 Size The MANS diagram in F~G. 7 represents a network needed to switch 15 data traffic for up to 20,000 FUSs. Each NIM is expected to handle and concentrate the traffic of 10 to 20 EUSs onto a 150 Mb/s XL, giving about 1000 XLs (rounded off in binary to 1024). Each MINT serves 4 XLs for a total of 256 MINTs. Fach MI~T also handles 4 ILs, each with an inpu~ and an output tennination on the DNet portion of the MANS. The data networlc thus has 1024 20 inputs and 1024 outputs. Internal DNet link sizing will be addressed later.
Failure-group size and other considerations lead to a DNet with 32 input links on each first stage switch 121, each of which links is connected to two such switches. There are 16 outputs on each second stage switch 123 of the DNet. Thus, there are 64 of each type of switch and also 64 MANSCs 14û ;n the 25 CNet, one per second st;age switch.
3.3.1.3 Traffic and Consolidation The "natural" EUS transactions of data to be switched vary in size by several orders of magnitude, from SUWUs of a few hundred bits to LUWUs a megabit or more. As explained in Section 2.1.1, MAN breaks larger EUS
30 transactions into network llansactions or packets of at most a few thousand bits each. But the MANS deals with the switch transac~ion, defined as the burst of data that passes through one MANS connection per one connect (and disconnect) request. Switch h-ansac~ons can vary in size from a single SUWU to several LUWlJs (many packets) for reasons about to be given. Por the rest of Section 3.
35 "transaction" means "sw~tch ~ansaction" except as noted.

For a given total data rate through the MANS, the transaction throughput rate (transactions/second) valies inversely with the transaction size.
Thus, the smaller the transaction size, the greater the transaction throughput must be to maintain the data rate. This throughput is limited by the individual S throughputs of the MANSCs (whose connect/disconnect processing delays reduce the effective IL bandwidth) and also by concurrency resolution (waiting for busyoutlets~. Each MANSC's overhead per transaction is of course independent of transaction size.
Although larger transactions reduce the transaction throughput 10 demands, they will add more delays to other transactions by holding outlets and fabric paths for longer times. A compromise is needed -- small transactions reduce blocking and concurrency delays, but large transactions ease the MANSC
and MINT workloads and improve the DNet duty cycle. The answer is to let MAN dynamically adjust its transaction sizes under varying loads for the best 15 per~ormance.
The DNet is large enough to handle the offered load, so the switching control complex~s (SCC) throughput is the limiting factor. Under light traffic, the switch transactions will be short, mostly single SUWUs and packets. As traffic levels increase so does the transaction rate. As the SCC transaction rate capacity 20 ;s approached, transaction sizes are dynarnically increased to maintain the transaction rate just below the point where the SCC would overload. This is achieved automatically by the consolidation control strategy, whereby eac'h MINTalways transmits in a single switch transaction all available SUWUs and packets targeted for a given destinadon, even though each burst may contain the whole or25 parts of several EUS transactions. Further increases in ~affic will increase the size, but not so mu~ the number, of transactions. Thus fabric and IL utilizationimprove with load, while the SCC's workload increases only slightly. Section 3.3.3.2.1 explains the ~eedback mechanism that controls ~ansaction size.
3.3.1.4 Porfonnanco Goals Nevertheless, MAN's data throughput depends on extremely high performance of individual SCC control elements. For example, each XPC 122,124 in the data switch will be ordered to set and clear at least 67,000 connections per second. Clearly, each request must be handled in at most a few microseconds.

Likewise, the MANSCs' functions must be done quickly. We assume that these steps will be pipelined; then the sum of the step processing times will contribute to connect and disconnect delays, and the maximum of these step timeswill limit transaction throughput. We aim to hold the maximum and sum to a few 5 microseconds and a few tens of microseconds, respecdvely.
The resolution of the concurrency problem must also be quick Md efficient. Busy/idle status of destination terminals will have to be determined in about 6 microseconds, and the control strategy rnust avoid burdening MANSCs with unfulfillable connection requests.
lû One final performance issue relates to the CNet itself. The network and its access links must run at high speeds tprobably at least 10 Mb/s) to keepcontrol message transmit times small and so that links will run at low occupancies to minimize the contendon delays from statisiical multiplexing.
3.3.2 Data Network (DNet2 The DNet is a Richards tw~stage rearrangeably non-blocking broadcast network. This topology was chosen not so much ~or its bruadcast capability, but because its two-stage structure allows the network to be partitioned into disjoint subsets for distr~buted con~ol.
3.3.2.1 Design Parameters The capabilities of the Richards net~work derive from the assignment of inlets to multiple appearances on different first stage switches according to a definite pattern. The particular assignment pattem chosen, the number m of multiple appearances per inlet, the total number vf inlets, and the number of links between first and second stage switches determ~ne the maximum number of outlets 25 per second stage switch permitted for the network to be rea~angeably non-bloc~ing.
The D~et in FIG. 7 has 1024 inlets, each with two appearances on the - first stage switches. There are two links between each first and second stage switch. These parameters along with dle pattern of distributing the inlets ensure 3û that Yvith 16 oudets per second stage switch the network will be realTangeably non-blocking for broadcast.
Since MAN does not use broadcast or reaITangement, those par~neters not justified by failure-group or other considerations may be changed as more experience is obtained. For example, if a failure group size of 32 were deemed 35 tolerable, each second stage switch could have 32 outputs, thus reducing the number of second stage switches by a factor of 2. Making such a change would depend on the ability of the SCC control elements each to handle twice as much traffic. In addition, blocking probabilities would increase and it would have to be determined that such an increase would not significantly detract from the performance of the network.
The network has 64 first stage switches 121 and 64 second stage switches 123. Since each inlet has two appearances and there are two links between first and second stage switches, each first stage switch has 32 inlets and 128 outlets and each second stage has 128 inlets and 16 outlets.
3.3.2.2 Operat on Since each inlet has two appearances and since there are two links between each first and second stage switch, any outlet switch can access any inlet on any one of four links. The associa~ion of inlets to links is algorithmic and thus may be computed or alternatively read from a table. The path hunt involves simply choosing an idle link tif one exists) from among the four link possibilities.
If none of the four links is idle, a re-attempt to make a connection is made later and is requested by the same MINT. Alternatively, existing connections could be re-arranged to remove the blocking condition, a simple procedure in a Richards network. However, rerouting a connection in midstream could introduce a phase glitch beyond the outlet circuit's ability to recover phase 20 and clock. Thus with present circuitry, it is preferable not to run the MANS as a rea~angeable switch.
Each switch in the UNet has an XP(~ 122,124 on the CNet, which receives messages from the MANSCs telling which crosspoints to operate. No high-level logic is performed by these controllers.
25 3.3.3 Control Ne~work and MANS Controller Functions 3.3.3.1 Control Network (CNet2 The CNet 130,134,135 briefly descrtbed earli~r, interconnects the MINTs, MANSCs, and lSCs. It must carry three types of messages --connect/disconnect orders from MINTs to MANSCs using block 130, crosspoint 30 orders from MANSCs to lSCs using block 134, and ACKs and NAKs from MANSCs back to the MINTs using block 135. The CNet shown in FIG. 7 has three coIresponding planes or sections. The private ~ANS 140--2SC 124 links are shown but are not considered par~ of the CNet as no switching is required.
In this embodirnent, the 256 MIN~s access the CNet in groups of 4 3~ resulting in 64 input paths to and,64 output paths from the network. The bus elements in the control network perforrn merging and routing of message streams.

A request message from a MINT includes the ID of the outlet port to be connected or disconnected. Since the MANSCs are associated one-to-one with second stage switches, this outlet specification identifies the proper MANSC to which the message is routed.
S The MANSCs transmit acknowledgment (ACK), negative acknowledgment (NAK), and lSC comrnand messages via the right-to-left portion of the CNet (blocks 134,135). These messages will also be formatted with header information to rou~e the messages to the specified MINTs and lSCs.
The CNet and its messages raise significant technical challenges.
10 Contention p~oblems in the CNet may mirror those of the entire MANS, requiring their own concurrency solution. These are apparent in the Control Network shown in FI(3. 7. The control data concentrators 136 from four lines into one interface may have contention where more than one message tries to arrive at one time.
The data concen~ators 136 have storage for one request from each of the four 15 connected MINTs, and the MINTs ensure that consecutive requests are sent sufficiently far apart that the previous request from a MINT has already been passed on by ~he concen~ator before the next arnves. The MINTs time out if no acknowledgement of a request is received within a prespecified time.
Alternatively, the control data concentrators 136 could simply "OR" any requests20 received on any input to the output; garbled requests would be ignored and not acknowledged, leading to a time out.
Functionally what is needed inside the blocks 130,134,135 is a micro LAN specialized for dny fixed-length paclcets and low conten~ion and minimal delay. Ring nets are easy to interconnect, grow ~acefillly, and permit 25 simple tokenless add/drop pr~tocols, but they are, ill-suited for so many closely packed nodes and have intolerable end-~o-end delays.
Since ~e longest message (a MIN7's connect order) has under 32 bits, a parallel bus 132 sen~es as a CNet fabric that can send a complete message in one cycle. Its arbitration controller 131, in handling contention for the bus, 30 would automatically solve contention for the receivers. Bus components are duplicated for reliability (not shown).
3.3.3.2 MAN Switch Controller (MAN3C~ OPerations FIGS. 8 and 9 show a flowcha~ of the MANSC's high level functions. Messages to each MANSC 140 include a connect/disconnect bit, 35 SUWU/packet bit, and the Il:)s of the MANS input and output po~s involved.

3.3.3.2.1 Reg~t Queues; Consolidation (Intake Section, FIG. 8) Since the rate of message arrivals at each MANSC 140 can exceed its message processing rate, a MANSC provides entrance queues for its messages~
Connect and disconnect requests are handled separately. Connects are not 5 enqueued unless their requested outlets are idle.
Priority and regular packet connect messages are provided separate queues 150,152 so that priority packets can be given higher priority. An entry from the regular packet queue 152 is processed only if the priority queue lS0 isempty. This minimizes the priority packets' processing delays at the expense of 10 the regular packets', but it is estimated that priority traffic will not usually be heavy enough to add much to packet delays. Even so, delays are likely to be more user-tolerable with d~e lower priority large data transactions than with priority transactions. Also, if a paclcet is one of many pieces of a LUWIJ, any given packet delay may have no final effect since end-to-end LUWIJ delay 15 depends only on the last packet.
Both the priority and regular packet queues are short, intended only to cover short-term random fluctuations in message a~ivals. If the short-te~ rate of arrivals exceeds the MANSC's processing rate, the regular packet queue and perhaps the priority queue will overflow. In such cases a control negative 20 acknowledge (CNAK) is returned to the requesting MINT, indicating a MANSC
overload~ This is no catastrophe, but rather the feedback mechanism in the consolidation strategy that increases switch trans,action sizes as traffic gets heavier.
Each MINT combines into one transaction all available packets targeted for a given DNet oudet. Thus, if a connection reques~ by the MINT results in a 25 CNAK, the ne~ct reques~ for the same destination may represent more data to be shipped dunng the connection, provided more packets of the LWVUs have arrived at the MINT in the meantime. Consolidation need not always add to LUWU
transmission delay, since a LUWU's last packet n~ight not ~e affected. This scheme dynamically increases effective paclcet (transactiorl) sizes to aecommodate 30 the processing capability of the MANSCs.
The priority queue is longer than the regular packet queue to reduce the odds of sending a priority CNAK due to random bursts of requests. Priority packets are less likely to benefit from consolidation than packets recombining into their original LUWUs; this supports the separate, high-priority queue. To force 35 the MINTs to consolidate more packets, we may build the reglllar packet queueshorter than it "ought" to be. Simulations have indicated that a priority queue of ~ 3 1 5373 requests capacity and a regular queue of 8 requests capacity is appropriate. Thesizes of both queues affect system performance and can be fine-tuned with real experience with a system.
Priority is determined by a priority indicator in the type of service 5 indication 623 (FIG. 20). Voice packets are given priority because of their required low delay. In alternative arrangements, all single packet transactions (SUWUs) may be given priority. Because charges are likely to be higher for high priority service, users will be discouraged from demanding high pr~onty service ~or ~he many packets of a long LUWU.
10 3.3.3.2.2 Bu~/Idle Check When a connect request first arrives at a MANSC, it is detected in test 153 which differentiates it from a disconnect request. The busy/idle status of the destinadon outlet is checked (test 154). If the destination is busy, a busy negative acknowledge (BNAK) is returned (action 156) to the requesting MINT, 15 which will try again later. Test 158 selects the proper queue (pr;ority or regular packet). The queue is tested (160,162) to see if it is full. If the specified queue is full, a CNAK (control negative acknowledge) is returned (action 164). Otherwise the request is enqueued in queue 150 or 152 and simultaneously the destination is seized (marked busy~ (action 166 or 167~. Note that an overworked (full queues) 20 MANSC can still return BNAKs, and that both E~NAKs and CNAKs tend to increase transaction sizes through consolidation.
The busy/idle check and BNAK handle the concurrency problem. The penalty paid for this approach is that a MINrr-to-MANS IL is umlsable during theinterval between a MINT's issuin" a connect request for that IL and its receipt of 25 an ACK or BNAK. Also the CNe~ jams up with BNAKs and failing requests under heavy MANS loads. Busy/idle checks must be done quic~ so as not to degrade the connection request throughput and IL utilization; this explains the performance of a busy test before enqueuing. It may be desirable further to use separate hardware to pre-lest outlets for concuIrency. Such a procedure would 30 relieve the MANSCs and CNets from repeated BNAK requests, increase the successful request throughput, and perrn~t the l~NS to saturate at a higher percentage of its theoretical aggregate bandwidth.
3.3.3.2.3 Path Hunt~ MANSC Service Section (FIG. 9) Priority block 168 gives highest priority to requests from disconnect 35 queue 170, lower priority to requests from the priority queue 150, and lowestpriority to requests from the packet queue 152. When a connect request is unloaded from the prioIity or the regular packet queue, its requested outlet port has already been seized earlier (action 166 or 167), and the MANSC hunts for a path through the DNet. This merely involves looking up first the two inlets to which the incoming IL is connected (action 172) to find the four links with access S ~o that incoming IL and checking their busy status (test 174)~ If all four are busy, a blocked-fabric NAK (fabric NAK or FNAK) fabri~ blocking negative acknowledge (FNAK) is returned to the requesting MINT, which will try the request again later (action 178). Also the seized destination outlet is released(marked idle) (action 176). We expect FNAKs to be rare.
If the four links are not all busy, an idle one is chosen and seized, first a first stage inlet, then a link (action 180); both are marked busy (action 182).
The lnlet and link choices are stored (action 184). Now the MANSC uses its dedicated control path to send a crosspoint connect order to the XPC in its associated second stage switch (action 188); this connects the chosen link to the 15 outlet. At the same time another crosspoint order is sent (via the right-to-left CNet plane 134) to the lSC (action 186) required to connect the link to the inlet port. Once this order arrives at the lSC (test 190), an ACK is returned to the originating MINT (action 192).
3.3 3.2.4 Disconnects To release network resources as quickly as possible, disconnect requests are handled separately from connect requests and at top priority. They have a separate queue 170, built 16 words long (same as the number of outlets) so it can never ~verflow. A digconnect is detected ;n test 153 which receives requests from the MINT and separates connect fi~m disconnect requests. The outlet is released and the request placed in disconnect queue 170 (action 193).
Now a new connect request ~or this same outlet can be accepted even though the outlet is not yet physi~ally disconnected. Due to its higher priority, the disconnect will tear dow3l the switch connections before the new request tries to reconnect the outlet. Once enqueued, a disconnect can always be executed. Only the outlet ID
30 is needed to idendfy the spent connection; the MANSC recalls this connection's choice of link and crosspoints from local memory (acdon 195)) marks these links idle (action 196) and sends the two XPC orders to release them (acdons 186 and 188). Thereafter, test 190 controls the wait for an acknowledgrnent from the first stage controller and the ACK is sent to the MINT (action 192). If there is no 35 record of this connectdon, the MANSC returns a "Sanity NAK." The MANSC
senses status from the outlet's phase alignment and scrarnble circuit (PASC) 29() to verify that some data transfer took place.
3.3.3.2.5 Parallel Pipe n ng Except for seizure and release of resources, the above steps for one request are independent of other requests' steps in the same MANSC and thus are S pipelined to increase MANSC throughput. Still more power is achieved through parallel operations; the path hunt begins at the same time as the busy/idle check.
Note that the transaction rate depends on the longest step in a pipelined process, but the response time for one given transaction (from request to ACK or NAK) is the sum of the step times involved. The latter is improved by parallelism but not 10 by pipelining.
3.3.4 Error Detection and Dia~nosis Costly hardware, message bits, and time-wasting protocols to the CNet and its nodes to verify every little message are avoided. For exarnple, each crosspoint order from a MANSC to an XPC does not require an echo of the 15 cornmand or even an ACK in return. Instead, MANSCs does assume that messages alTive uncorrupted and are acted on correctly, until evidence to the contrary arrives from outside. Audits and cross-checks are enabled only when there is cause for suspicion. The end users, NIMs and MINTs soon discover a defect in the MANS or its control complex and identify the subset of MANS ports 20 involved. Then the diagnostic task is to isolate the problem for repair and intenm work-around.
Once a portion of the MANS is suslpect, temporary auditing modes could be turned on to catch the guilty parties. For suspected lSCs and MANSC.
these modes requ~re use of the comrnand ACKS and e~choing. Special messages 25 such as cxosspoint audits may also be passed through the CNet. This should be done while s~ll carrying a li~ht load of user tra~fic.
Before engaging these internal self-tests (or perhaps to`eliminate ~hem entirely~, MAN can run experiments on the MANS to pinpoin,~ the failed circuil.
using the M3NTs, Ls, and NIMs. For example, if 75% of the test SUWUs sent 30 from a given IL make it to a given outlet, we would conclude ~hat one of the two lhlks from one of that IL's two first stages is defective. (Note this test must be run under load, lest the deterrninistic MANSC always select the sarne link.) Further experiments can isolate that link. But if several MINTs are tested and none can send to a particular outlet, then tnat outlet is marked "out of service lo 35 all MlNTs and suspicion is now focussed on that second stage and its MANSC
If other outlets on that stage work, the fault is in the second stage's fabric. Th~

tests use the status lead from each of a MANSC's 16 PASC.
Coordinating the independent MINTs and NIMs to run these tests requires a central intelligence with low-bandwidth message links to all MINTs and NIMs. Given inter-MINT connectivity (see FIG. 15), any MINT with the needed S firmware can take on a diagnostic task. NIMs must be involved anyway to tell whether test SUWUs reach their destinations. Of course any NIM on a working MINT can exchange messages with any other such NIM.
3.4 MAN Switch Controller FM. 25 is a diagram of MANSC 140. This is the unit which sends 10 control instructions to data network 120 to set up or tear down circuit connections.
It receives orders from control network 130 via link 139 and sends acknowledgments both positive and negative back to the requesting MINTs 11 via control network 135. It also sends instructions to first stage switch controllers via control network 134 to first stage swi~ch controller 122 and directly to the second 15 stage controller 124 that is associated wit~ the specific MANSC 140.
Inputs are received from inlet 139 at a request intake port 1402. Ihey are processed by intake control 1404 to see if the requested outlet is busy. Theoutlet memory 1406 contains busy/idle indications of the outlets for which an MANSC 140 is responsible~ If the outlet is idle a connect request is placed into20 one of two queues 150 and 152 previously described with respect to FIG. 8. Ifthe request is for a disconnect, tbe request is placed in disconnect queue 170. The outlet map 1406 is updated to mark a disconnected outlet idle. The acknowledge response unit 1408 sends negative acknowledgments if a request is received with an error or if a connect request is made to a busy outlet or if the appropriate 25 queue 150 or 152 is full. Acknowledgrnent responses are sent via control network 135 back to the requesting MINT 11 via distributor 138. All of these acdons are performed under ehe control of intake control 1404.
Service control 1420 controls the setup of paths in data network 120 and the updating of oullet memory 1406 for those circumstances in which no path 30 is available in the data network between the requesting ;nput link and an available output link. The intake control also updates outlet memory 1406 on connect requests so that a request which is already in the queue will block another request for the same output link.
Service control 1420 examines requests in the three queues 150, 152, 35 and 170. Disconnect requests are always given the highest priority. For disconnect requests, the link memory 1424 and path memory 1426 are examined to see which links should be made idle. The instructions for idling these links are sent to first stage switches from first stage switch order port 1428 and the instructions to second stage switches are sent from second stage switch order port 1430. For connect requests, the static map 1422 is consulted to see which 5 links can be used to set up a path ~rom the requesting input link to the requested output link. Link map 1424 is then consulted to see if appropriate links are available and if so these links are marked busy. Path memory 1426 is updated to show that this path has been set up so that on a subsequent disconnect order theappropriate links can be made idle. All of these actions are performed under the10 control of service control 1420.
Controllers 1420 and 1404 may be a single controller or separate controllers and may be program controlled or controlled by sequential logic.
There is a great need for a very high-speed operations in these controllers because of the high throughput demanded which makes a hard wired controller preferable.
15 3.5 Control Network Control message network 130 (FIG. 7~ takes owtputs 137 from data concentrators 136 and transmits these outputs, representing connect or disconnect requests, to MAN switch controllers 140. Outputs of concentrators 136 are storedtemporarily in source registers 133. Bus access controller 131 polls thesç source 20 registers 133 to see if any have a request to be transmitted. Such requests are then placed on bus 132 whose output is stored temporarily in interrnediate register 141. Bus access controller 131 then sends outputs from register 141 to the appropriate one of the MAN switch controllers 140 via link 139 by placing the output of register 141 on bus 142 connected to link 139. The action is 25 acccmplished in three phases. During the first phase, the output of register 133 is placed on the bus 132, thence gated to reg~ster 141. During the second phase, the output of reg~ster 141 is placed on bus 142 and delivered to a MAN switch controller 140. During the third phase, the MAN switch controller signals the source register 133 as to whether the controller has received the request; if so, 30 source register 133 can accept a new input from control data concentrator 136.
Othenvise, source register 133 retains the sarne request data and the bus accesscontroller 131 will repeat the transrnission later. ~he three phases may occur simultaneously for three separate requests. Control networks 134 and 135 operatein a fashion similar to control network 130.

3.6 Summary A structure to meet the large bandwidth and transaction throughput requirements for the MANS has been described. I~e data switch fabric is a two-stage Richards network, chosen because its low blocking probability permits a 5 parallel, pipelined distributed switch control complex (SCC). The SCC includesXPCs in all first and second stage switches, an intelligent controller MANSC with each second stage, and the CNet that ties the control pieces together and links them to the MINTs.

10The memory and interface module (MINT) provides receive interfaces for the external fiber-optic links, buffer memory, control for routing and link protocols, and transmitters to send collected data over the lir~ks to the MAN
switch. In the present design, each MINT serves four network interface modules (NIMs) and has four links to the switch. The MINT is a data switching module.
15 4.1 Basic Functions The basic functions of the MINT are to provide the following:
1. A fiber-optic receiver and link protocol handler for each NIM.
2. A link handler and transrnitter for each link to the switch.
3. A buffer memory to accumulate packets awaiting transmission across the 20switch.
4. An interface to the controller for the switch to direct the setup and teardown of network paths.

5. Control for address translation, routing, making efficient use of the switch,orderly transmission of accumulated packets, and management of buffer 25memory.

6. An inter~ace for operation, administration, and maintenance of tbe overall sys~ern.

7. A control channel to each NIM for operation, administration, and rnaintenance functions.
30 4.2 Data Flow In order to understand the descriptions of the individual functional units that make up a MINT, it is first necessary to have a basic understanding of the general flow of data and control. FIG. 10 shows an overall view of the MINT~Data enters the MINl' on a high-speed (10~150 Mbit/s) data channel 3 from 35 each NIM. This data is in the foTm of packets, on the order of 8 Kilobits long, each with its own header containing routing information. The hardware allows for ~315373 packet sizes in increments of 512 bits ~o a maximum of 12~ Kilobits. Small packet sizes, however, reduce throughput due to the per-packet processing required. Large maximum packet sizes result in wasted memory for transactions of less than a maximum size paclcet. The link terminates on an external link 5 handler 16 (XLH), which retains a copy of the pertinent header fields as it deposits the entire pacXet into the buffer memory. This header inforrnation, together with the buffer memory address and length, is then passed to the central control 20. The central control determines the destination NIM from the address and adds this block to the list of blocks (if any) awaiting transmission to this10 same destination. The central control also sends a connection request to the switch controller if there is not already a request outstanding. When the central control receives an acknowledgement from the switch controller that a connectionrequest has been satisfied, the central control transni~ts the list of memory blocks to the proper internal link handler 17 (ILH). The ILH reads the stored data from15 memory and tr~msmits it at high speed (probably the same speed as the incoming links) to the MAN switch, which directs it to its destination. As the blocks aretransmitted, the ILH informs the central control so that the blocks can be added to the list of free blocks available for use by the XLHs.
4.3 Memor~r Module The buffer memory 18 (FIG. 4) of the MlNT 11 satisfies three requirements:
1. The quantity of memory provides sufficie;nt buffer space to hold the data accumulated (for all destinations) while awaiting switch setups.
- 2. The memory bandwidth is adequate to support simultaneous activity on all eight links (four receiving and four transmitdng).
3. The memo~y access provides for efficient strearning of data to and from the link handlers.
4.3.1 O~anization Because of the amount of memory required (Megabytes), it is 30 desirable to employ conventional high-density dynamic random access memory (DRAM~ parts~ Thus, high bandwidth can be achieved only by making the memory wide. The memory is therefore organized into 16 modules 201,...,202 which make up a composite 512-bit word. As will be seen below, memory - accesses are organized in a synchronous fashion so that no rnodule ever receives 35 successive requests without sufficient time to per~oIm the required cycles. The range of mernory for one MINT l l in a typical MAN application is 16-64 Mbytes. The number is seasitive to the speed of application of flow control in overload situations.
4.3.2 Time Slot Assigners The time slot assigners 203,...,204 (TSAs) combine the functions of a S conventional DRAM controller and a specialized 8-channel DMA controller. Each receives read/write requests from logic associated with the Data Transport Ring 19 (see 4.4, below). Its setup commands come from dedicated control time slots on this same ring.
43.2.1 Con~rol . _ From a control viewpoint, the TSA appears as a set of registers as shown in FI(3. 11. For each XLH there is an associated address register 210 and count register 211. Each ILEI also has address 213 and count 214 registers, but in addition has registers containing the next add~ess 215 and count 216, thus allowing a series of blocks to be read from memory in a continuous stream with 15 no inter-block gaps. A special set of registers 220-226 allows the MINT's central control section to access any of the internal registers in the TSA or to perforrn a directed read or write of any particular word in memory. These registers includea write data register 22~ and read data register 221, a memory address register 222, channel status register 223, eIror register 224, memory refresh row 20 address register 225, and diagnostic control register 226.
4.3.2.2 Vperation In normal operation, the TSA 203 receives only four order types from the ring interface logic: (1) "write" requests for data received by an XLH, (2) "read" requests for an ILH, (3) "new address" coxnmands issued by either an XLH
25 or an ILH, and (4~ "idle cycle" indications which tell the TSA to perform a refresh cycle or other special operation. Each order is accompanied by the identity of the linlc handler involved andl in the case of "write" and "new address" requests, by 32 bits of data.
For a "write" operation, the TSA 203 simply perforrns a memory 30 write cycle using the address from the register associated with the indicatedXLH 16 and the data provided by the ring interface logic. It then increments th~address register and decrements the count register. The count register is used in this case only as a safety check since the XLH should provide a new address before overflowing the current block.

For a "read" operation, the TSA 203 must first check whether the channel for this ILH is active~ If it is, the TSA performs a memory read cycle using the address from the register for this LH 17 and presents the data to the ring interface logic. It also increments the address register and decrements the5 count register. In any case, the TSA provides the interface logic with two "tag"
bits which indicate (1) no data available, t2) data available, (3) first word ofpacket aYailable, or (4) last word of packet available. For case (4), the TSA will load the LH's address 214 and count 213 registers from its "next address" 21~
and "next count" 215 registers, provided that these registers have been loaded by 10 the ILH. If they have not, the TSA marks the channel "inactive."
From the above descriptions, the function of a "new address"
operation can be inferred. The TSA 203 receives the link identity, a 24-bit address, and an 8-bit count. For an XLH 16~ it simply loads the associated registers. In the case of an ILH 17, the TSA must check whether the channel is 15 active. If it is not, then the normal address 214 and count 213 registers are loaded and the channel is marked active. If the channel is currently active, then the "next address" 216 and "next count" 215 registers must be loaded instead of the normaladdress and count registers.
In an altemative embodiment, the two tag bits are also stored in buffer 20 memory 201,...,202. Advantageously, this permits packet sizçs that are not limited to being a multiple of the overall width of the mlemory (512 bits). In addition, the ILH 17 need not provide dle actual length of the packet when reading it, thus relieving the central control 20 of the need to pass along this information to the ILH.
25 4.4 Data TrallSPOrt Rin~
It is the job of the Data Transport Ring 19 to carry control commands and high speed data between the link handlers 16,17 and the memory modules 201,..,202. The nng provides sufficient bandwidth to allow all the linksto run simultaneously, but carefully apportions this bandwidth so that circuits 30 connec~ng to the ring are never required to transfer data in high-speed bursts.
Instead, a fixed time slot cycle is employed that assigns slots to each circuit at well-spaced intervals. The use of this fixed cycle also means that source and destination addresses need not be carried on the ring i~self since they can be readily determined at any point by a properly synchroni2ed counter.

4.4.1 Electrical escription The ring is 32 data bits wide and is clocked at 24 MHz. This bandwidth is sufficient tv support data rates of up to 150 Mbit/s. In addition to the data bits, the rings contains four parity bits, two tag bits, a sync bit to identify 5 the start of a superframe, and a clock signal. Within the ring, single-ended ECL
circuitry is used for all signals except the clock, which is differential ECL. The ring interface logic provides connecting circuits with TTL-compatible signal levels.
4.4.2 Time Slot Sequencin~R quirements In order to meet the above objectives, the time slot cycle is subject to a number of constraints:
1. During each complete cycle there must be a unique time slot for each combination of source and destination.
2. Each connecting circuit must see its data time slots appearing at reasonably regular intervals. Specifically, each circuit must have a certain minimum interval between its data time slots.
3. Each link handler must see its data ~rne slots in numerical order by memory module number. (This is to avoid making the link handler shuffle a 512-bit word.) 4. Each TSA must have a known interval during which it can perforrn a refresh cycle or othçr miscellaneous memory operation.
5. Since the TSAs in the memory modules must examine every control time slot, there must also be a minimum inte~val between control time slots.
4.4.3 Tirne Slot Cycle Table I shows one data *ame of a tin~ing cycle which meets these requirements. One data fIame consists of a total of 80 time slots, of which 64 are used for data and the remaining 16 for control. The table shows, for each memory module TSA the slot during which it receives data from each XLH to be wr~tten into memory and during which it must supply data that was read from 30 memory for each LH. Every fifth slot is a control time slot during which the indicated link handler broadcasts control orders to all the TSAs. For the purposes of this table, XLHs and ILEIs are numbered 0-3, and TSAs are numbered 0-15.
TSA 0, for example, during time slot 0 receives data from XLH 0 and must supply data for ILH 0. During slot 17, TSA 0 performs similar operations for 35 XLH 2 and ILH 2. Slot 46 is used for XLH 1 and ILH 1, and slot 63 is used forXLH 3 and ILH 3. The re-use of the same time slot for reading and writing is ~5 permissible since XLHs never read from memory and ILHs never write~ thus effectively doubling the data bandwidth of the ring.
The control time slots are assigned, in sequence, to the four XLHs, the four ILHs, and the central control (CC). With these nine entities sharing the S control time slots, the control frame is 45 time slots long~ The 80-slot data frame and the 45-slot control frame come into alignment every 720 time slots. This period is the superframe and is marked by the superframe sync signal.
There is a subde synchronization condition that must also be met for the ILHs. The words of a block rnust be sent in sequence beginning with word 0, 10 regardless of where in the ring timing cycle the order was received. To assist in meeting this requirement, the ring interface circuitry provides a special "word 0"
sync signal for each ILH. For example, in dle timing cycle of Table I a new address might be sent by LH 0 during time slot 24 (its control dme slot). It is necessary to ensure that TSA number 0 is the first TSA to act on this new address 1~ ~requirement 3 in section 4.4.2) even though the data time slots for reads from TSAs numbered S through 15 for ILH 0 imrnediately follow time slot 24.
Since the number of time slots in the superframe, 720, exceeds the number of elements on the ring, 25, it is apparent that the logical time slots do not have a perrnanent existence; each time slot is, in effect, created at a particular 20 physical location on the ring and propagates around the ring until it returns to this location, where it vanishes. The effective creation point is different for data time slots than for control time slots.

TABLE I
RING TIME SL(:)T ASSI{~NMENT
Write to From Read from To Control T~me Slot TSA XLH TSA ILH Slot Source og XLH1 11 g 1 g 13 ~ 3 6 3 3 0~ 3 16 10 1 ~ 10 :: 19 ~ XLH3 4 ~ 4 0 21: 11 ;1 11 1 : 22 1 2 1 2 23 S :3 8 3 3Q ~ 5 0 5 0 : 26 12 1 12 ~7 2 ~ 2 2 3S 7 ~ 7 0 42 5 2 ~ 2 lS 43 12 3 12 3 ~:
44 (~C

9 : 0 9 0 46 - :0 1 0 47 6 2 : 6 2 49 : XLH0 ;:: 50 ~10 0 10~ 0 51 1 1 1~ 1 52 7 2:: 7 2 ~ 53 14: 3 14 3 4 ~ ~ XLHl `~ : 56 : 2 1 2 : : 57 8 2 8 2 : ~ 30: 58 15 3 15 3 59 : ; XLH2 12~ 0 ~ 12 0 61 3 : 1 3 62 ` 9 2 9 2 , --48- 131~373 ~7 10 ~ 10 2 `~ 73 2 3 2 3 74 ILHl ::
lS 0 lS O
:~ 76 6 1 6 1 .

~ 78 3 3 3 3 `; ~ 79 : ILH2 : ~ :

' 1 ~1 5373 - 4~ -4.4.3.1 Data Time Slots Data time slots can be considered to originate at the owning XLH. A
data time slot is used to ca~y incoming data to its assigned memory module, at which point it is re-used to carry outgoing data to the corresponding LH. Since 5 XLHs never receive info~nation from a data time slot, the ring can be considered to be logic.ally broken (for data time slots only) between the ILHs and the XLHs.
The two tag bits identify the contents of the data time slots as follows.

1 1 Empty 10 Data 01 First word of packet 00 Last word of packet The "first word of packet" is sent only by memory module 0 when it sends the first word of a packet to an ILH. The "last word of paclcet" indication is sent` only 15 by memory module 15 when it sends the end of a packet to an ILH.
4.43.2 Control T rne Slots Control time slots originate and tern~inate at the station of central control 20 on the ring. I~e link handlers use their assigned control slots only to broadcast orders to the TSAs. The CC is assigned every ninth control time slot.
20 The T3As receive orders from all control time slots and send responses back to the CC on the CC control time slot.
The two tag bits identify the contents of a control tim~ slot as follows:

1 1 Empty 10 Data (to or from CC) 01 Order 00 Address & count (from a link handler) 4.5 External Link Handler The principal function of the XLH is to terminate the incoming high-30 speed data channel from a NIM, deposit the data in the MINT's buffer memory, and pass the necessary inforrnation to the MINT's central control 20 so that thedata can be ~orwarded to its destination. In addition, the XLH terminates an - so incoming low-speed control channel tha~ is multiplexed on the fiber link. Some of the functions assigned to the low-speed control channel are the transmission of the NIM status and control of flow in the network. It should be noted that the XLH
is only telminating the incoming fiber from the NIM. Transmission to the NIM is 5 handled by the internal link handler and the phase alignment and scrambler circuit that will be described later. The XLH uses an onboard processor 26~ to interfaceto the hardware of the MINT central control 20. The four 20 Mbit/sec links coming from this processor provide the connectivity to the central control section of the MINT. FIC3. 12 shows an overall view of the XLH.
10 4.5.1 Link Interface The XLH contains the fiber optic receiver, clock recovery circuit and descrambler CilCUit needed to recover data from the fiber. After the data clock is recovered (block 250) and the data descrambled (block 252) the data is then converted from serial to parallel and demultiplexed (block 254) into the high-15 speed data channel and the low-speed data channel. Low level protocol processing is then performed on the data on the high-speed data channel (block 2S6) as described in 5. This results in a data stream consisting of only packet data. The stream of packet data then goes through a first-in-first-out (FIiFO) queue 2~8 to a data steering circuit 260 which steers the header into the 20 header FIFO 266 and sends the complete packet to the XLH's ring interface 262.
4.5.2 Ring Interface The rmg interface 262 logic controls transfer of data from the packet E7IFO 258 in the link interface to the MINT's buffer memory. It provides the following functions;
25 I. Establishing and maintaining synchronizadon with ~e ring's timing cycle.
2. T~ansfer of data from the link interface FIFO to the proper ring dme slots.
3. Sending a new address to the memory TSAs when the end of ~ packet is encountered.
It should be noted that resynchronization with the ring's 16-word (per XLH) 30 timing cycle will have to be performed during the processing of a packet whenever the link interface FIFO becomes temporarily empty. This will be a no~nal occurrence since the ring9s bandwidth is higher than the link's transmission rate. The ring and TSA, however, are designed to accornmodate gaps in the data stream. l~us, resynchronization consists simply of waiting for 35 data to become available and for the ring cycle to return to the proper word number, marking the intervening time slots "empty." For example, if the -51- l 31 5373 FIFO 258 becomes empty when a word destined for the fifth memory module is needed, it is necessary to ensure that the next word actually sent goes to that memory module, in order to preserve the overall sequence.
4.5.3 Control The control portion of the XLH is responsible for replenishing the free block PIFO 270 and passing the header information about each packet received to the MINT's central control 20 (FIG. 4).
4.5.3.1 Header Processm~
At the same time a packet is being transmitted on the ring, the header lO of the packet is deposited in the header FIFO 266 that is subsequently read by the XLH processor 268. In this header are the source and destination address fields,which the cenlral control will require for routing. In addition, the header checksum is verified to ensure that these fields have not been corrupted. The header information is then packaged with a memory block descriptor (address and 15 length) and sent in a message to the central ~ ontrol 20 (FIG. 4).
4.5.3.2 Interaction with Central Control There are only two basic interactions with the MINT's central control.
The XLH control attempts to keep its free-block FIFO 270 full with block addresses obtained from the memory manager, and it passes header infonnation ~0 and memory block descriptors to the central control so that the bloc~ can be routed to its destination. The block addresses are subsequently placed on the ring 19 by ring interface 262 upon receipt of the address from control sequencer 272. Bo~h interactions with the central control are carried out over links from XLH processor 268 to the appropriate sections of the central control.25 4.6 Internal Linlc Handler The internal link handler (ILH) (FIG. 13) is the first part of what can '~e considered a distributed link controller. At any instant in time this distributed link con~oller consists of a particular ILH, a pa~h through the swi~ch fabric and a particular ~hase Alignment and Scrambler ci~cuit 290 (PASC). The PASC is 30 described in section 6.1. It is the PASC that is actually responsible for thetransmission of optical signals over the return fi'oer of fiber pair 3 to the NIM
from the MINT. The infonnation that is transmitted over the fiber comes from the MANS 10, which receives inputs at different times from the ILHs sending to that NIM. This kind of dis~ributed link controller is necessary since path lengths 35 through the MAN switch fabric are not all equal. If the PASC did not align all o f the information coming from different ILHs to the same reference clock~

~;

information received by the NIl!/I would be continually changing its phase and bit ali~nment.
The combination of the ILH with the PASC is in many ways a mi~or image of the XLH. The ILH receives lists of block descriptors from the central 5 control, reads these blocks from memory, and transmits the data over the serial link to the switch. As data is received from memory, the associated block descriptor is sent to the central control's memory manager so that the block canbe returned to the free list.
The ILH differs from the XLH in that the ILH performs no special 10 header processing, and the TSAs provide ~he ILH with additional pipelining sothat multiple blocks can be transmitted as a continuous stream if desired.
4.6.1 Link Interface The link interface 289 provides the sçrial transrnitter for the data channel. Data is transmitted in a frame-synchronous format compatible with the 15 link data format described in 5. Since the data is received from the ring interface 280 (see below) asynchronously and at a rate somewhat higher that the link's average da~a rate, the link interface contains a FIFO 282 to provide speed matching and frame synchronization. The data is received from MINT memory via dala ring interface 280, stored in FIFO 282, is processed by level 1 and 2 20 protocol handler 286, and is transmitted to MAN switch 10 through the parallel to serial converter 288 within link interface 289.
4.6.2 Ring Interface The ring inteIface 280 logic controls the ~ansfer of data *om the MINT's buffer memory to the FIFO in the link interface. 1~ provides the 25 following functions:
1. Establishing and maintaining synchronization with the ring's dming cycle.
2. Transfer of data from the ring to the linlc interface FIFO during the proper nng dme slots.
3. Notifying the con~ol section when the last word of a packet (memory block) is received.
4. Sending a new address and count ~if available) to the memory TSAs 203,...,204 (FI~3. 10) when the last word of a packet is received and the condition of the FIFO 282 is such that the new packet will not cause an overtlow.
35 Unlike the XLH, the ILH relies on the TSAs to ensure that da~a words are received in sequence and with no gaps within a block. Thus, maintaining word 53 l 3 1 5373 synchronization in this case consists simply of looking for unexpected empty data time slots.
4.6.3 Control _ The control portion of the ILH, controlled by sequencer 283 is 5 responsible for providing the ring interface with block descriptors received via the processor link inter~ace 284 from the central control and stored therefrom in address FIFO 285, notifying the central control via the processor link interfacewhen blocks have been retrieved from memory, and notifying the central control 20 when transmission of the ffnal block is complete.
10 4.6.3.1 Interaction with Central Control There are only three basic interactions with the MINT's central control:
1. Receiving lists of block descriptors.
2. Informing the memory manager of blocks that have been retrieved from memory.
3. Inforlrung the switch request queue manager when all blocks have been transmitted.
In the present design, all of these interactions are carried out over Transputer links to the appropriate sections of the central control.
20 4.6.3.2 Interaction with TSAs Like the XLH, the ILH uses its control time slots to send block desc~iptors (address and lengths) to the TSAs. When the TSAs receive a descriptor from an ILH, however, they will immediately begin reading the block from memory and placing the data on the ring. The length field from an ILH is 25 significant and deter~es the number of words that will be read by each TSA
befoIe moving on to the next block. The TSAs also provide each ILH with registers to hold the next address and length, so that successive~ blocks can betransrnitted without ~aps. Flow control is the responsibility of the ILH, however, and a new descriptor should not be sent to the TSAs until there is enough room in 30 the packet FIFO 282 to compensate for reframing time and the difference in transrn~ssion rates.
4.7 MINT Central Control FIG. 14 is a block diagram of MINT central control 20. This central control is connected to the four XLH 16s of the MINT, the four ILH 17s of the 35 MINT, to data concentrator 136 and distributor 138 of the switch control (SeeFIG. 7), and to an OA&M central control 352 shown in FIG. 15. The relationship of the central control 20 with other units will first be discussed.
The MINT central control communicates with XLH 16 to provide memory block addresses for use by the XLH in order to store incoming data in the MINT memory. XLH 16 communicates with the MINT central control to S provide the header of a packet to be stored in MINT memory, and the address where that packet is to be stored. Memory manager 302 of MINT central control 20 communicates with LH 17 to receive information that memory has been released by an LH because the message stored in those memory blocks has been delivered, so that the released memory can be reused.
When queue manager 311 recognizes that the first network unit arriving for a paxticular NIM has been queued in switch unit queue 314, which contains FIFO queues 316 for each possible destination NIM, queue manager 311 sends a request to switch setup control 313 to request a connection in MAN
switch 10 to that NIM. The request is stored in one of the queues 318 (priority)lS and 312 (regular) of switch setup control 313. Switch setup control 313 administers these requests according to their priority and sends requesls to MANswitch 10, specifically to switch control data concentrator 136. For normal loads, the queues 318 and 312 should be almost empty since requests can nolmally be made almost immediately and will generally be processed by the appropriate 20 MAN switch controller. For overload conditions, the queues 318 and 312 becomea means for deferring transmissîon of lower priority packets while retaining therelatively fast transmission of priority packets. Xf experience so dictates, it may be desirable to move a request from the regular queue to the priority queue if a priority packet for that destinadon NIM is received. Requests queued in 25 queues 318 and 312 do not tie up an IL, an ILH, and an output link of circuitswitch 10; this is in contrast to requests in the queues 150,152 (EIG. 8) of an MAN switch controller 140 (FIG. 7).
When switch setup control 313 recognizes that a connection has been established in switch 10, it notifies NIM queue manager 311. The ILH 17 30 receives data from a FIFO queue 316 in switch unit queue 314 from NIM queue manager 311 to identify a queue of the memory locations of data packets which may be eransrnitted to the circuit switch, and for each packet, a list OI one or more ports on thç NIM to which that packet is eo be transrnitted. NIM queue manager 311 then causes ILH 17 to prefix the port number(s3 to each packet and 35 to transrnie data for each packet from memory 18 to switch 10. The ILH then proceeds to transmit the packets of the queue and when it has completed this task, notifies the switch se~up control 313 that the connection in the circuit switch may be disconnected and notifies memory manager 302 of the idenhty of the blocks of memory that can now be released because the data has been transmitted.
The MINl central control uses a plurality of high speed processors 5 each of which have one or more input/output ports. The specific processor usedin this implementation is the Transputer manufactured by ~MOS Corporation.
This processor has four input/outout ports. Such a processor can meet the processing demands of the MINI' central control.
Packets come into the four XLHs 16. There are four XLH managers 10 305, source checkers 307, routers 309, and OA&M MINT processors 315, one corresponding to each XLH within the MINT; these processors, operating in parallel to process the data entering each XLH increase the total data processing capacity of the MINT central control.
The hPader for each packet entermg an XLH is transmitted along with 15 the address where that packet is ~eing stored direcdy to an associated XLH
manager 305, if the header has passed the hardware check of the cyclic redundancy code (CRC) of the header performed by the XLH. If that CRC check fails, the packet is discarted by the XLH which recycles the allocated memory block. The XLH manager passes the header ancl the identity of allocated memory 20 for the packet eo the source checker 307. The XLH manager recycles memory blocks if any of the source checker, router, or ~IM queue manager find it impossible to transmit ~he packet to a desdnation. Recycled memory blocks get used before memory blocks allocated by the memo~y manager. Source checker 307 checks whether the source OI the packet is properly logged in and whether 25 that source has access ~o the virtual network of the packe~ Source checker 307 passes information ab~ut the packet, including the packet address in MINT
memory? to router 309 which translates the packet group identification, effectively a virtual network name, and the destination name of the packet in order to find out which output link this packet should be sent on. Router 309 passes the 30 identification of the output link to NIM queue manager 311 which identifies and chains packets received by the four XLHs of this MINT which are headed for a comrnon output link. After the first packet to a NIM queue has been received, the NIM queue manager 311 sends a switch setup request to switch setup cont~l 313 to request a comlection to that NIM. NIM queue manager 311 chains these 35 packets in F~FO queues 316 of switch unit queue 314 so that when a switch connection is made in the circuit switch 10, all of these packets may be sent over that conne tion at one time. Output control signal distributor 138 of the switchcontrol 22 replies with an acknowledgment when it has set up a connection. This acknowledgment is received by switch setup control 313 which informs NIM
queue manager 311. NIM queue manager 311 then informs ILH 17 of the list of S chained packe~s in order that ILE~ 17 may transmit all of these packets. When ILH 17 has completed the transmission of this set of chained packets over the circui~ switch, it informs switch setup con~rol 313 to request a disconnect of the connection in switch 10, and informs memory manager 301 that the memory which was used for storing the data of the message is now available for use for a 10 new message. Memory manager 301 sends this release information to memory distributor 303 which distributes memory to the various XLH managers 305 for allocating memory to the XLHs.
Source checker 307 also passes billing information to operation, administration and maintenance (OA&M) MINT processor 315 in order to perform 15 billing for that packet and to accumulate appropriate statistics for checking on the data flow within the MINT and, after combination witb other statistics, in the MAN ne~work. Router 309 also informs (OA&M~ MINT processor 315 of the destination of the packet so that the OA&M MINT processor can keep track of data concerning packet destinations for subsequent traffic analysis. The output of 20 the four OA&M MINT processors 315 are sent to MINT OA&M monitor 317 which summarizes the data collected by the four OA&M MINT processors for subsequent transmission to OA&M central control 3S2 (FIG. 14j.
MINl' OA~M monitor 317 also receives information from OA&M
central con~rol 352 for rnaking changes ~via OA&M MINT processor 315 in the 25 router 309 data; these changes reflect additional terminals added to the network, the movement of logical terminals (i.e., terminals associated with a particular user~
from one physical port to another, or the removal of physical terminals from thenetwork. Data is also provided from the OA&M central control 352 via the MINT operation, OA&M monitor and the OA&M MINT processor 315 to source 30 checker 307 for such data as a logical user's password and physical port as well as data concerning the p~ivileges of each logical user.
4.8 MINT Operation, Adrninistration, and Maintenance Contr~l System FIG. 15 is a block diagram of the maintenance and control system of the MAN network. Operation, administration, and maintenance ~OA~M) 35 system 350 is connected to a plurality of OA&M central controls 352. These OA~M controls are each connected to a plurality of MINTs~ and within each - 57 ~ l 3 1 5373 MINT, to the MINT OA&M monitor 317 of MINT central control 20. Since many of the messages from OA&M system 350 must be distributed to all the MINTs, the various OA&M~ central con~ols are interconnected by a data ring.
This data ring transmits such data as the identification of the network interface 5 module, hence the identification of the output link, of each physical port that is added to the network so that this infonnation may be stored in the router processors 309 of every MINT in the MAN hub.

5.1 Link Re~ements The links in the MAN system are used to transmit packets between the EUS and the NIM (EUSL) (links 14) and between the NIM and the MAN
hub (XL) (links 3). Although the operation and the characteristics of the the data that is transferred on these links varies slightly with the particular application, the format used on the links is the sarne. Having the fo~rnats be the same makes it 15 possible use comrnon hardware and software.
The link format is designed to provide the following features.
1. It provides a high data rate packet channel.
2. It is compatible with the proposed Metrobus "OS~l" forrnat.
3. Interfacing is easier because of the word oriented synchronous format.
20 4. It defines how "packets" are delimited.
5. It includes a CRC for an entire "packet" (~md another ~or the header.) 6 The format insures transparency of the data within a "packet".
7. The format provides a low bandwidth chaimel for flow control signaling.
8. Additional low bandwidth channels can be added easily.
25 9. Data sc~nbling insures good transition density for clock reeovery.
5.2 MAN Link Descripdon and Reasoning From a performance point of view, the faster the links are the better MAN will perforrn. This desire to operate the links as fast as possible is tempered by the fact that faster links cost more. A reasonable tradeoff between 30 speed and cost is to use L~D transmitters (like the AT~T ODL-200) and multimode fiber. The use of ODL-200 transrnitters and receivers puts an upper limit on the link speed of about 200Mbit/sec. From the MAN architecture poinl of view, the exact data rate of the links is not important since MAN does not dosynchronous switching. The data rate for the MAN links was chosen to be the 35 same as the data rate of the Metrobus Lightwave System "OS-l" link. The Metrobus forrnat is described in M. S. Schaefer: "Synchronous Optical -58- l 31 5373 Transmission Network for the Metrobus Lightwave Network", EEE International Communications Conference, June 1987, Paper 30B.l.l. Another data rate (and format) ~hat could be used in MAN will come from the specification of SONET, a link layer protocol specified by Bell Communica~ions Research Corp. for 150 S Mbit/sec unchannelized links.
5.2.1 Level 1 Link Format The MAN network uses the low level link forrnat of Metrobus.
Information on the link is ca~ied by a sirnple frame that is continuously repeated.
The frame consisis of 88 - 16 bit words. The first word contains a framing 10 sequence and 4 parity bits. In addition to this first word, three other words are overhead words. These overhead words, which are used for internode commlmications in the Metrobus implementation, are not used by MAN ~or the sake of Metrobus compatibility. The word oriented nature of the protocol makes using it much simpler. A simple 16 bit shift register with parallel load can be 15 used to transrnit and a similar shift register with parallel read out can be used to receive. At ~e 146.432Mbit/sec. link data rate, a 16 bit word is transmitted or received every lO9ns. This approach makes it possible to implement much of the link formatting hardware at conventional l-rL clock rates. Tbe word oriented nature of the protocol does put some restrictions on the way the link is used, 20 however. To keep the complexity of the hardware reasonable it is necessary to use tbe bandwidth of the link in units of 16 bit words.
5.2.2 Level 2 Link Forrnat The link is used ~o move "packets", tbe basic unit of ir~ormation transfer in MAN. To identify packets, the ~orrnat includes the specification of 25 "SYNC" words and an "IDLE" word. When no packets are being transmitted the "IDLE" word uill fill all of the words that make up the primary channel bandwidth (words not reserved for other puIposes). Packets are delim~ted by a leading STA~T_SYNC and a trailing END SYNC word. Tbis scheme works well as long as the words with special meanings are never contained in the daea within 30 a packet. Since restricting the data that can be sent in a packet is an unreasonable restriction, a transparent data transfer technique mus~ be used. MAN links employ a very simple word stuffing ~ransparency technique. Within the packet data, any occulTence of a special meaning word, like the START SYNC word, is preceded by another special word the "DLE" word. This word stuffing tIansparency was 35 chosen because of the simplicity of implementation. This protocol requires simpler, lower speed logic than is required for bit stuffing protocols like HDLC

The technique itself is sirnilar to the time proven techniques used in IBM's BISYNC links. In addition to the word stuffing used to ensure transparency, "FILL" words are inserted if the data rate of the source is slightly less than the link data rate.
The last word in any packet is a cyclic redundancy check (CRC) word. This word is used to insure the that any corruption of the data in a packet can be detected. The CRC word is computed on all of the data in the packet, excluding any special words like "DLE" that may need to be inserted in the data stream for transparency or oeher reasons. The polynomial that is used to compute10 the CRC word is ~he CRC-16 standard.
To ensure good transition density for the optical receivers all of the data is scrambled (e.g., block 296, FM. 13) prior to transmission. The scrarnbling makes it less likely that long sequences of ones or zeros will be transmitted on the linlc even though they rnay be quite cornmon in the data actually being 15 transmit~ed. The scrambler and descrambler (e.g., block 2S2y FIG. 12) are well known in the art. The descrarnbler design is self synchronizing, which makes it possible to re over from occasional bit e~rors without having to restart the descrambler.
5.2.3 Low Speed Channels and Flow Control Not all of the payload words in the level 1 format are used for the level 2 format that caIries packets. Addidonal channels are included on the linkby dedicating particular words within the frame. These low rate channels 255,295(FIGS. 12 and 13) are used for MAN network control pu~poses. A packet delimiting scheme similar to that used on the primary data channel is used on 25 ~these low rate chalmels. The dedicated words that make up low rate channels can be fi2rther divided down into individual bits for very low bandwidth channels like the flow control channel. Ihe flow control channel is used on the MAN EUSL
(between the EUS and the NIM) to provide hardware level flow control. I~e flow control channel (bit) from the NIM to the FUs~ indicates to the EUS lin~c 30 transmitter whether or not it is allowed to transmit more information. The design of the NIM is such that sufficient storage is available to absorb any ~ata that is transmitted prior to the EUS transmitter actually stopping after flow control isasserted. Data transmission can be stopped either between packets or in the middle of a packet transmission. If it is between packets, the next packet will not 35 be sent until flow control is turned deasserted. If rdow con~ol is asserted in the middle of a packet, it is necessary to suspend data transmission irnmediately and start sending the "Special FILL" code word. This code word, like all others, is escaped with the "DLE" code word when it appears in the body of a packet.

.. .
The MAN switch, as described in section 3, is an asynchronous space 5 switch fabric with a very fast setup controller. The data fabric of the switch is design to reliably propagate digital signals with data rates from DC to in excess of 200Mbits/second. Since many paths can simultaneously exist through the fabric, the aggregate bandwidth requirements of the MAN hub c;~n be easily meet by the fabric. This simple data fabric is not wi~hout drawbacks however. Because of 10 mechanical and electrical constraints in implementing the fabric, it is not possible for all paths through the switch to incur the same amount of delay. Because the variations in path delay between different paths may be much greater than the bit time of the data going through the switch, it is not possible to do synchronous switching. Any time that a path is setup from a particular ILH in a MINT to an 15 output port of the switch, there is no guarantee that data transmitted over that path will have the same relative phase as the data transmitted over a previous path through the switch. To use this high bandwidth switch it is therefore necessary to very quickly synchronize data coming out of a switch port to the clock being used for the synchronous link to the NIM.
20 6.1 The Phase Ali~nment and Scrambler Circuit (PASC) The unit that must do the synchronization of data corning from the switch and drive the outgoing link to the NIM called the Phase Alignment and Scrambler Circuit (PASC) (block 290, FIG. 13). Since the ILHs and the PASC
circuits are all part of the MAN hub, it is possible to distribute ~he same master 25 clock to all of them. This has several advantages. By using the same clock reference in the PASC as is used to transmit data from the ILH, one can be sure that data can not be coming into the PASC any ~aster than it is being moved out of it over the link. This eliminates the need ~or large FIFOs and elaborate elastic store controll~rs in the PASC. The fact that the bit rate of all data that comes into 30 a PASC is exactly the the same makes the synchronizatlon easier.
The ILH and the PASC can be thought of as a distributed link handler for the format described in the previous section. The ILH creates the basic frarning pattern into which the data is inserted and transmits it through the fabric to a PASC. The PASC aligns this framing pattern with its own framing pattern, 35 merges in the low speed control channel and then scrambles ~he data for transmission.

-Sl 1 31 5373 The PASC synchronizes the incoming data to the reference clock by inserting an appropriate amount of delay into the data path. For this to work the ILH must be transmitting each frame with a reference clock that is slightly advanced from the reference clock used by the PASC. The number of bit times of 5 advance that the ILEI requires is determined by the actual minimum delay that may be incurred in getting from the ILH to the PASC~ The amount of delay that the PASC must be capable of inserting into the data path is dependent on the possible variation in path delays that may occur for different paths through theswitch.
FIG. 23 is a block diagram of an illustrative embodiment of the invention. Unaligned data enters a tapped delay line 1001. The var~ous taps of the delay line are clocked into edge sampling latches 1003,...,1005 by a signal that is 1~0 degrees out of phase with the reference clock (REFCLK) and is designated REFCLK . The outputs of the çdge sampling latches feed selection logic 15 lmit 1007 whose output is used tO control a selector 1013 describPd below.
Selection logic 1007 includes a set of internal latches for repeating the state of latches 1003,...,1005. The selection logic includes a priority circuit connected to these internal latches, for selecting the highest rank order input which carries a logical "one". The output is a coded identification of this selected input. The 20 selection logic 1007 has two gating signals: a clear signal and a signal from all of a group of internal latches of the selection logic. Between data streams, the clear signal goes to a zero state causing the internal latches to accept new inputs. After the fir-st "one" input has been received from the edge sampling latches 1003,...,1005 in response to the ISrst pulse of a data stream, the state of the 25 transparent latches is maintained until the clear signal goes back to the zero state~
The clear signal is set by out of band circuitry which recognizes the presence of a data stream.
The output of the tapped delay line also goes to a series of data latches `lO09,...,1011. The input to the data latches is clocked by the reference 30 clock. The outputs of the data latches 1009,...,1011 are the inputs to selector circuit 1û13 which selects the output of one of these data latches based on the input from selection logic 1007 and connects this output to the output of the selector 1013, which is the bit aligned dala stream as labeled on FIG. 23.
After the bits have been aligned, they are fed into a shift register (not 35 shown) with tapped outputs to feed the driver XL3. This is to allow data streams to be transrnitted syncnronously starting at sixteen bit boundaries. The opera~ion of the shift register and auxiliary circuitry is substantially the same as that of the tapped delay line arrangement.
The selection logic is implemented in cornmercially available priority selection circuits. The selector is simply a one out of eight selector controlled by 5 the output of the selection logic. If it is necessary to have a finer alignment circuit using a one of sixteen selection, this can be readily implemented using the same principles. The arrangement described herein appears to be especially attractive in sihlations where there is a common source clock and where the length of each data stream is limited. The common source clock is required since the 10 clock is not derived from the incorning signal, but is, in fact, used to gate an incoming signal appropliately. Ihe limitation on the length of the block is required since a particular gating selection is maintained for the entire block so that if the block length were too long, any substantial amount of phase wandering would cause synchronism to be lost and bits to be dropped.
While in the present embodiment, the signal is passed through a ~apped delay line and is sampled by the clock and inverse clock, the alternativearrangement of passing the clock through a tapped delay line and using the delayed clocks to sample the signal could also be used in some applications.
6.2 Clock Distribution The MAN hub operation ;s very dependent on the use of a single rnaster reference clock for all of the ILH and PASC units in the system. The master clock must be distributed accurately and reliably to all of the units. Inaddition ~o the basic clock frequency that must ~ distributed, the frame start pulse must be distributed to the PASC and an advanced frame start pulse mus~ be 25 distributed to the ILH. All of these functions are handled by using a single clock distribution link (fiber or twisted pair~ going to each unit.
The info~nadon that is ca~ied on these clock distribution links comcs fr~m a single clock source. This informadon can be split in the electrical and/or optical dom~in and ~ansmitted to as many destinations as necessary. There is no 30 attempt to keep the information on all of the clock distribudon links exactly in phase since the ILH and PASC are capable of correcting for phase differences no matter what the reason for this difference. The information that is transrnitted is simply alternadng ones and zeros with two excepdons. The occurrence of two ones in a row indicates an advanced frame pulse and the occurrence of two zer 35 in a row indicates a normal frame pulse. Each board that terminates one of the~
clock distribution links contains a clock recovery module. The elock recovery module is the same as that used for the links themselves~ The clock recovery module will provide a very stable bit clock while additional logic extracts the appropriate frame or advanced frarne from the data itself. Since the clock recovery modules will continue to oscillate at the correct frequency even without S bit transitions for several bit tirnes, even the unlikely occ~rence of a bit eIror will not affect the clock frequency. The logic that looks for the frarne or advanced frame signal can also be made tolerant of errors since it is known that the frame pulses are periodic and extraneous pulses caused by bit errors caul be ignored.
7 NETWORK INlERFACE MODULE
10 7.1 Overview The network interface module (NIM) connects one or more end user system links (EUSL) to one ~AN external link (XL). In so doing, the NIM
pe~forrns concentration and demultiplexing of network transaction units (i.e.
packets and SUWUs~, as well as insuring source identification integrity by a~fixing 1~ a physical "source port number" to each outgoing packet. The latter funcdon, in combination with the network registration servicç described in 2.4, prevents a user from masquerading as another for the purpose of gaining access to unauthorized network-provided services. The NIM thereby represents the boundary of the MAN network proper, NlMs are owned by the network provider, 20 while UIMs (described in 8) are owned by the users themselves.
This section describes the basic functions of the NIM in more de~ail, and presents the NIM architecture.
7.2 Basic Functions The NIM must perform the following basic functions:
25 EUS Link interfacing. One or more interfaces must be provided to EUS link(s) (see 2.2.5). The downstream link (i.e. from ~M to UIM) consists of a data ch~nnel and an out-of-band channel used by the NlM to flow con~ol the ups~eam link when NIM[ input buffers become full. Because the downstream link is not flow controlled, the flow control channel on the ups~eam link is unused. The 30 Data and Header Check Sequences (DCS, HCS) are generated by the UIM on the upstream link, and checked by the UIM on the downstream link.
External Link interfacing The XL ( 2.2.6) is very similar to the EUSL, but lacksDCS checking and generation on both ends. This is to allow erroneous, but still potentially useful data to be delivered to the UIM. The destination port numbers35 in network transaction units arriving on the downstream XL are checked by the NIM, with illegal values resulting in dropped data.

Concentration nd_demultip exin,~. Network transaction units arriving on the EUSLs contend for and are statistically multiplexed to the outgoing XL. Those arriving on the XL are routed to the appropliate EUSL by mapping the destinationport number to one or more EUS links.
S Source port identification. The port number of the source UIM is prepended to each network transaction unit going upstream by port number generator 403 (FiG. 16). This port number will be checked against the MAN address by the MINT to prevent unauthorized access to services tincluding the most basic data transport service) by "imposters".
10 7.3 M Architecture and Operation The architecture of the NIM is dçpicted in FIG. 16. The following subsections briefly describe the operation of the NIM.
7.3.1 Upstream ~Qeration Incorning network transaction units are received from the UIMs at 15 their EUSL interface 4ûO receivers 402, are converted to words in s~ial to parallel converters 404 and are accumulated in FIFO buffers 94. Each EUSI. interface is connected to the NIM transrnit bus 95, which consists of a parallel data path, and various signals for bus arbitration and clocking. When a network transaction unit has been buffered, the EUSL interface 400 arbitrates for access to the transrnit20 bus 95. Arbitration proceeds in parallel with da~a transmission on the bus. When the current data ~ansmission is complete, the bus arbiter awards bus ownership to one of the competing EUSL interfaces, which begins transmission. For each transaction, the EUSL port number, inserted at the beginning of each packet by port number generator 403, is transrnitted first, followed by the network 25 ~ansaction unit. Within an XL interface 440, the XL transrnitter 96 provides the bus clock, and peIforms parallel to serial conversion 442 and data transmission on the upstream XL 3.
7.3.2_~S~9 Network transaction units arriving from the MINT on the downstream 30 XL 3 are received within XL interface 440 by the XL receiver 446, which is connected via seIial to parallel converter 448 to the NIM receive bus 430. The receive bus is similar to, but independent of the transmit bus. Also connected to the receive bus via a parallel to serial converter 408 are the EUSL interface transmitters 410. The XL receiver performs serial to parallel conversion, provides 35 the receive bus clock, and sources the incoming data onto the bus. Each EUSL
interface decodes the EUSL port number associated with the data, and forwards the data to its EUSL if appropriate. More than one EUSL interface may forwarcl the data if required, as in a broadcast or multicast operation. Each decoder 409checks the receive bus 430 while port number(s) are being transmitted to see if the following packet is destined for the end user of this EUSL interface 400; if so, S the packet is forwarded to transmitter 410 for delivery to an EUSL 14. IllegalEUSL port numbers (e.g. violations of the error coding scheme) result in the data being dropped (i.e. not forwarded by any EUSL interface). Decode block 409 is used to gate informadon destined for a particular EUS link from trans nit bus 95to the paralleVserial converter 408 and transmitter 410.

8.1 Oven~iew A user interface module (UIM) consists of the hardware and software necessary ~o connect one or more end user systems (EUS), local area networks (LAN), or ded;cated point-to-point links to a single MAN end user system link 15 (EUSL) 14. Throughout this secdon, the term EUS will be used to generically refer to any of these network end user systems. Clearly, a portion of the 'iJlM
used to connect a pa..ticular type of EUS to ~AN is dependent on the architecture of that EUS, as well as the desired performance, flexibility, and cost of the implementation. Some of the functions provided by a IJIM, however, must be 20 provided by every UIM i n the system. It is therefore convenient to view the architecture of a UIM as having two distinct halves: the network interface, which provides the EUS-independent functionality, and the EUS intertace, which implements the remainder of the UIM ~unctions for the particular type of EUS
being cormected.
Not all EUSs will require the perfo~mance inherent in a dedicated external link. The concentration provided by a NIM (desclibed in 7) is an appropIiate way to provide access to a number of EUSs which have stringent response ~ime requirements along with the instantaneous VO bandwidth necessary to effectively utilize thë full MAN data rate, but which do no~ generate the 30 volume of traffic necessary to efficiently load the XL. Similarly, several EUSs or LANs could be connected to the same UIM via some interrnediate link (or the LANs themselves). In this scenario, the UIM acts as a multiplexer by providing several EUS (actually LAN or link) interfaces to go with one network interface.
This method is well suited to EUSs which do not allow direct connections to their 35 system busses, and which provide only a link connection that is itself limited in bandwidth. End users can provide their multiplexing or concentration at a UIM

and MAN can provide further multiplexing or concentration at the NIM.
Th;s sect;on examines the arch;tectures of both the network interface and EUS inte~face halves of the UIM. The funct;ons prov;ded by the network ;nterface are described, and the architecture is presented. The heterogene;ty of5 EUSs that may be connected to MAN does not allow such a generic treatment of the EUS ;nterfaces. Instead, the EUS interface des;gn opt;ons are explored, and a specific example of an EUS is used to illustrate one possible EUS interface design.
8.2 UIM - Network Interface The UIM network interface implements the EUS-independent functions of the UIM. Each network interface connects one or more EUS
interfaces to a single MAN EUSL.
8.2.1 Ba c Functions The UIM network interface must perfonn the following functions:
15 EUS Link ;nterfac;ng. The interface to the EUS L;nk ;ncludes an optical transmitter and receiver, along with the hardware necessary to perform the link level functions required by the EUSL (e.g. CRC generadon and checking, data formatting, etc.).
I)ata buffering~ Outgo;ng network transaction units (i.e. packets and SUWUs) 20 must be buffered so that they may be transrnitted on the fast network link without gaps. Incoming network transaction units are buffered for purposes of speed matching and level three (and above) protocol processing.
Buffer memorv mana~ement. The packets of one LUWU m~ay arrive at the receive UIM interleaved with those of another LUWU. In order to support this concurrent 25 reception of several LUWUs, the network interface ~ust manage its receive buffer memory in a dynamic fashion, allowing incorning packets to be chained together into LUWlJs as they arrive.
Proto ol processing. Outgoing LUWUs must be fragmerlted into packets for ~ansmission into the network. Similarly, incoming packets must be recombined 30 into LUWUs for delivery to the receiving process within the EUS.
8.2 2 Arch~e~
Clearly, all of the functions enumerated in the previous subsection must be perforrned in order to interface any EUS to a MAN EUSL. However, some architectural decisions must be made regarding where these functions are 35 performed; i.e., whether they are internal or external to the host i~self.

-67 l 31 5373 The first two functions must be located external to the host, although for different reasons. The first and lowest level function, that of interfacing to the MAN EUS Link, must be implemented externally simply because it consists of special purpose hardware which is not part of a generic EUS. The EUS link S interface simply appears as a bidirectional I/O port to the remainder of the UIl\~
network interface. On the other hand, the second function, data buffering, cannot be implemented ~n existing host memory because the bandwidth requirements are too strin~gent. On reception, the networl~ interface must be able to buffer incoming packets or SUWUs back-to-back at the full network data rate (150 Mb/s). This 10 data rate is such that it is generally impossible to deposit incorning packets directly into EUS memory. Similar bandwidth constraints apply to packet and SUWU transmission as well, since they must be completely buffered and then transmitted at the full 150 Mb/s rate. These constraints make it desirable to provide the necessary buffer memory external to the EUS. It should be noted that15 while FIFO memory will suffice to provide the necessary speed matching for transmission, the lack of flow control on reception along with the interleaving of received packets necessitate that a larger amount of random access memory be provided as receive buffer memory. For MAN, the size of receive buffer memory may range from 256 Kbytes to 1 Mbyte. The particular size depends on the 20 interrupt latency of the host and on the maximum size LUWU allowed by the host software.
The final two functions involve processing, which could conceivably be pe~o~med by the host processor itself. The third function, buffer memory management, involves the timely allocation and deallocation of blocks of receive25 buffer memory. The latency requirement associated with the allocation operation is ;~ stringent, due once more to the high data rates and the possibility of packets a~riving back-to-back. However, this can be alle~iated (for reasonable burst sizes) by pre-allocating several blocks of memory. It is possible, therefore, for the hos~
processor to manage the receive packet buffers. Similarly, the host processor mav 30 or may not assume the burden of the fourth function, that of MAN protocol processing.
The locatioll of these final two functions determines the level at whi~h the EUS connects to the UIM. If the host CPU assumes the burden for packet buffer memory management and MAN protocol processing (the `'local"
35 configuration), then the unit of data transferred across the EUS interface is a packet, and the host is responsible for fragmenting and recombining LUWUs~

~ 31 5373 on the other hand, those functions are off-loaded to another processor in the UIM, the front end processor (FEP) configuration, the unit of data trans~erred across the EUS interface is a LUWU. While in ~heory, subject to interleaving constraints atthe FUS interface, the unit OI data transferred may be any amount less than or S equal to the entire LUWU, and the units delivered by the transmitter need not be the same size as those accepted by the receiver, for a general and uniform solution, useful for a variety of EUSs, the LUWU is to be preferred as the basicunit. The FEP configuration offloads the majority of the processing burden from the host CPU, as well as providing for a higher level EUS interface, thereby 10 hiding the details of network operatlon from the host. With the FEP, the hostknows only about LUVVUs, and can control their transmission and reception at a higher, less CPU intensive level.
Although a lower cost interface is possible utilizing the local configuration, the network interface architecture described in the following section 15 is a FEP configuration more characteristic of that required by some of the high perfonnance EUS that are natural users of a MAN network. An additional reason for choosing the FEP configuration initially is that it is better suited ~or interfacing MAN to a LAN such as ETHERNET, in which case there is no "host CPU" to provide buffer memory management and protocol processing.
20 8.2.3 Network Interface Architecture .
The architecture of the UIM network interface is depicted in FIG. 17.
I~he following subsections brielqy describe the operation of the UIM network interface by presenting scenarios for the transrnission and reception of data. An FEP-type architecture is employed, i.e., receive buffer memory rnanagement and 25 MAN network layer protocol pr~essing are performed external to the host CPU
of the EUS.
8.2.3.1 Transrnission of Data The main responsibilities of the network interface on transrnission are to fragment the arbi~ary sized ~ransrnit user work units (UWUs) into packets ~if30 necessary), encapsulate the user da~a in the MAN header and trailer, and transmit the data to the network. To begin transmission, a message from the EUS
requesting transmission of a LUWU traverses the EUS interface and is handled by network interface processing 450, which also implements memory management and protocol processing func~ions. Por each packet, the protocol processor portion 35 of the interface processing 450 formulates a header and wlites it into the transmit FIFO 15. Data ~or that packet is then transferred across the EUS interface 451 ,59 1315373 into the transmit FIFO 15 within link handler 460. When the packet is completelybuffered, the link handler 460 transmits it onto the MAN EUS link using transmitter 4~4, followed by the trailer, which was computed by the link handler 460. The link is flow controlled by the NlM to ensure that the NIM
S packet buffers do not overflow. This transrnission process is repeated for each packet. The transmit FIFO 15 contains space for two maximum length packets so that packet transmission may occur at the maximum rate. The user is notified viathe EUS interface 451 when the transmission is complete.
8.2.3.2 Reception of Data Incoming data is received by receiver 458 and loaded at the 150 MB/s link rate into elastic buffer 462. Dual-ported video RAM is utili~ed for the receive buffer memory 90, and the data is unloaded from the elastic buffer and loaded into the shift register 464 of receive buffer memory 90 via its serial access port. Each packet is then transferred from the shift register into the main memory lS array 466 of the receive buffer memory under the control of the receiver DMA
sequencer 452. The block addresses used to perform these transfers are provided by the network ;nterface processing a~ran~gement 450 of UIM 13 via the buffer memory controller 456, which buffers a small number of addresses in hardware to relieve the stnct latency requirements which would otherwise by imposed by 20 back-to-back SUWUs. Block 450 is composed of blocks 530, 540, 542, 550, 552, 554, 556, 558, 560, and 562 of FIG. 19. Because the network interface processinghas direct access to the buffer memory via its random access port, headers are not stripped off; rather they are placed into buffer mlemory along with the data. The receive queue manager 558 within 450 handles the headers and, with input from 25 the memory manager 550, keeps track of the various SUWUs and LUWUs as they anive. I'he EUS is notified of the arrival of data by the network inter~ace processing arrallgement 4S0 via the EUS interface. The details of how data is delivered to the EUS are a fonction of the par~cular EUS interface being employed, and are described, for example, in section 8.3.3.2.
30 8.3 UIM - EUS Interface 8.3.1 Philosophy This section describes the "half' of the network interface that is EUS
dependent. The ~asic function of the EUS interface is the delivery of data between the EUS memory and the UIM network interface, in both directions.
35 Each particular EUS interface will define the protocol to effect delivery, the fonnat of data and control messages, and the physical path for control and data.

Each side of the interface has to implement a flow control mechanism to protect itself from being overrlln. The EUS must be able to control its own memory and the flow of data into it ~om the network, and the network has to be able to protect itself as well. Only at this basic functional level is it possible to talk 5 about cornmonality in EUS interfaces. EUS interfaces will be different because of EUS hardware and system software differences. The needs of the applications using the networl~, coupled with the capabilities of the EUS, will also force interface design decisions dealing with performance and flexibility. There will be numerous interface choices even for a single type of EUS.
This set of choices means that the interface hardware can range from simple designs with few components to complex designs including sopbisticated buffering and memo~y managemçnt schemes. Control functions in the interface can range from simple EUS interfaces to handling network level 3 protocols and even higher level protocols for distributed applications. Software in the EUS can 15 also range from straightforward data transmission schemes that fit underneathexisting networking software, to more extensive new EUS software that would allow very flexible uses of the network or allow the highest performance that the netwoTk has to offer. These interfaces must be tailored to the specific existingEUS hardware and software systems, but there must also be an analysis of the 20 cost of interface features in comparison to the benefits they would deliver to the network applications running in these EUSs.
8.3.2 EUS Interface Desi~n Options . .
The tradeo~f between a ront end prc~cessor (FEP) and EUS processing is one example of different interface approaches to accomplish the samç basic 25 function. Consider variations in receive buffering. A specialized EUS
architecture with a high performance system bus could receive network packet messa~ges directly ~om the network links. Howevel; usually the interface will atleast buffer packet messages as they come off the link, before they are delivered into EUS memory. NoImally EUSs, either transrnitting to or receiving from the 30 network, do not know (or want to know) anything about the internal packet message. In that case, the receiving interface might have to buffer multiple packets that come from the LUWU of data that is the natural sized transmission unit between the ~aansmit and receive EUSs. Each one of these three receive buffering situations is possible and each would require a significantly di~ferent 35 EUS interface to ~ansfer data into the EUS memory. If the EUS has a particular need to process network packet messages and has the processing power and system bus perforrnance to devote to that task then the EUS dependent portion ofthe network interface would be simple. However, often it will be desirable to off-load that processing into the EUS interface and improve the EUS performance.Different transmit buffering approaches also illustrate the tradeoff 5 between FEP and EUS processing. For a specialized application, an EUS with high performance processor and bus could send network packet messages directly into the network. But if the application used EUS transaction sizes that were much larger that the packet message size, it might take too much of the EUS
processing to produce pa~ket messages on its own. An FEP could offload that 10 work of doing this level 3 network protocol formatting. This would also be the case where the EUS wishes to be independent of the internal network message size, or where it has a diverse set of network applications with a great variation in transmission size.
Depending on the hardware architecture of the EUS, and the level of 15 performance desired, there is the choice between programmed I/O and-DMA to move data between EUS memory and the network interface~ In the programmed VO approach, probably both control and data will move over the same physical path. In the DMA approach there will be some kind of shared memory interface to move control information in an EUS interfacing protocol, and a DMA
20 controller in the EUS interface to move data between buffer memory and EUS
memory over the EUS system bus without using EUS processor cycles.
There are several alternatives that e3dst for the location of EUS
buffering for networl~ data. The data could be buffered on a front end processornetwork controller circuit board with its own private memory. This memory can 25 be colmected to the EUS by busses using DMA transfer or dual ported memory accessed via a bus or dual ported memory located on the CPU side of a bus using private busses. The application now must access the data. Various techniques areavailable; some involve mapping the end user work space directly to the address space used by the UIM to store the data. Other techniques require the operating 30 system to further buffer the data and recopy into the user's private address space.
Options exist in writing the driver level sof~are in the EUS that is responsible for moving control and data inforrna~ion over the interface. The driver could also implement the ~US interface protocol processing as wçll as just moving bits over the interface. For the driver to still run efficiently the protocol 3~ processing in the driver might not be very flexible. For more flexibility based on a particular application, the EUS interface protocol processing could be moved up to a higher level. Closer to the application, more intelligence could be applied to the interface decisions, at the expense of more EUS processing time. The EUS
could implement various interface protocol approaches for delivery of data to and from the network: prioritization, preemption, etc. Network applications that did5 not require such flexibility could use a more direct interface to the driver and the network.
So, there are a variety of choices to be made at different levels in the system in both the hardware and the software.
8.3.3 Implementation Example: SUN Workstation Interface To illustrate the EUS dependent portion of the interface we describe one specific interface. The interface îs to the Sun-3 VME bus based workstationsmanufactured by Sun Microsystems, Inc. This is an example of a single EUS
connected to a single network interface. The EUS also allows connection directlyto its system bus. The UIM hardware is envisioned as a single circuit board that15 plugs into the VME bus system bus.
First, there follows a description of the Sun VO architecture, and then a description of the choices made in designing the interface hardware, the interfaee protocol, and the connection to new and existing network applications software.
20 8.3.3.1 SUN Workstation I/O Architecture The Sun-3's 1/0 architecture, based on the YME bus structure and its memory management unit (MMU), provides a I)MA approach called direct virtual memory access (DVMA). FIG. 17 shows the Sun DVMA. DVMA allows devices on the system bus to do DMA directly to Sun processor memory, and also 25 allow main bus masters to do DMA directly to mam bus slaves without going through processor memory. It is called "virtual" because the addresses that a device on the system bus uses to communicatc with the Icernel are virtual addresses similar to those the CRU would use. The DVMA approach makes sure tha~ all addresses used by devices on the bus are proeessed by the MMU, just as if 30 they were virtual addresses generated by the CPU. The slave decoder 512 (FIG. 18) responds to the lowest megabyte of VME bus address space (OxOOOO
0000 -> OxOOOf ffff, in the 32 bit VME address space) and maps this megabyte into the most significant megabyte of the system virtual address space ~OxffO ()(3(~
-> Oxfff ffff in the 28 bit virtual address space). (OX means that the subsequenl 35 characters are hexadecimal characters.) When the driver needs to send the buff~r address to the device, it must st~;p off the high 8 bits from the 28 bit address. ~--that the address that the device puts on the bus will be in the low megabyte (20bits) of the VME address space.
In FIG. 18, the CPU 500 drives a mernory management unit 502, which is connec~ed to a VME bus 504 and on board rnemory 506 that includes a S buffer 508. The VME bus communicates with DMA devices 510. Other on board bus masters, such as an l::T~ERNET access chip can also access memory 508 via MMU 502. Thus, devices can only make DVMA transfers in memory buffers that are reserved as DVMA space in these low (physical) memory areas. The kernel does however support redundant mapping of physical memory 10 pages into multiple virtual addresses. In this way, a page of user memory (orkernel memory) can be mapped into DVMA space in such a way that the data appears in (or comes from~ the address space of the process requesting that operation. The driver uses a routine called mbsetup to set up the kernel page maps to support this direct user space DVMA.
~5 8.3.3.2 SUN UIM - EUS Interface Approach As mentioned above there are many options in designing a particular interface. With the Sun-3 interface, a DMA transfer approach was designed, an interface with ~EP capabilities, an interface with high perforrnance matching the system bus, and an EUS software flexibility to allow various new and existing 20 network applications to use the network. FIG. 19 shows an overview of the interface to the Sun-3.
The Sun-3's are systems with potentially many simultaneous processes running in support of the window system, and multiple users. The DMA znd Fl~P
approachs were chosen to offioad the Sun processor while the network transfers 25 are taking place. The UlM har~ware is env~sioned as a single circuit board that plugs into the VME bus system bus. With the chance to connect directly to the system bus it is desirable to attempt the highest performance interface possible.
Sun's DVMA provides a means to move data efficiently to and from processor memory. I~ere is a DMA controller 92 in the UIM (FIG. 4) to move data from 30 the UIM to EUS memory and data from EUS memory to the UIM over the bus, and there will be a shared memory inter~ace to move control information in the host interfacing protocol. The front end processor (FEP) approach means that thedata from the network is presented to the EUS at a higher level. Level 3 protocol processing has been performed and packets have been linked ~ogether into 35 LUWUs, the user's natural sized unit of transmission. With the potential variety of network applications that could be running on the Sun the FEP approach mean~;

74 l 31 5373 that EUS software does not have to be tightly coupled to the internal network packet format.
The Sun-3 DVMA architectllre will lirnit the EUS transaction sizes to a maximum of one megabyte. If user buffers are not locked in, then kernel 5 bu-ffers would be used, as an intermediate step between the device and the user, with the associated performance penalty for the copy operation. If transfers aregoing to be made directly to user space, using the "mbsetup" approach, the user's space will be locked into memory, not available for swapping, during the whole transfer process. This is a tradeoff; it ties up the resources in the machine, but it 10 may be more efficient if it avoids a copy operati.on from some other buffer in the kernel.
The Sun system has existing networlc applications running on ETHEE~NET, for example, their Network File System (NFS). To run these existing applications on MAN but still leave open the possibili~r for new 15 applications that could use the expanded capabilities of MAN, we needed flexible EUS software and a flexible interface protocol to be able to simultaneously handle a variety of network applications.
FIG. 19 is a functional overview of the operation and interfaces among the NIM, UIM, and EUS. The specific EUS shown in this illustrat*e 20 example is a Sun-3 workstation, but the principles apply to other end user systems having greater or lesser sophistication. Consider first the direction from the MINT
via the NIM and UIM to the EI~S. As shown irl FIG. 4, data that is received frorn MINT 11 over link 3 is distnbuted to one of a plurality of UIMs 13 over links 14 and is stored in receive buf~er mernory 90 of such a UIlM, &om which 25 data is transmit~ed in a pipelined fashion over an EUS bus 92 having a DMA
interface to the appropriate EUS. The control structure for accomplishing this transfer of data is shown in FIG. 19, which shows that the input from the MINT
is controlled by a MINl to NIM link handler 520, which transmits itS outpu~
under the con~ol of router 522 ~o one of a plurality of N~ to UD!I~I link handlers 30 ~N/U LH) 524. MINT/NIM lmk handler (M/N LH) 520 supports a variant on the Met~obus physical layer protocol. Th~ NIM to UIM link handler 524 also supports the Metrobus physical layer protocol in this implementation, but other protocols could be supported as well. It is possible that different protocols could coexist on the same NIM. The output of the N/U LH 524 is sent over a link 14 35 to a UIM 13, where it is buffered in receive buffer memory 90 by NIM/UIM linkhalldler 552. The buffer address is supplied by memory manager 55û, which manages free and allocated packet buffer lists. The status of the packet reception is obtained by N/U LH 552, which computes and verifies the checksum over header an data, and outputs the status information to receive packet handler 556, which pairs the status with the buffer address received frorn memory mana~ger 550 S and queues the information on a received packet list. Information about received packets is then transferred to receive queue manager 558, which assembles packetinformation into queues per LUWU and SUWU, and which also keeps a queue of LUWUs and SUWUs about which the EUS has not yet been notified. Receive queue manager 558 is polled for inforrnation about LUWUs and SUWUs by the 10 EUS via the EUS/UIM link handler (E/U LH) 540, and responds with notificationmessages via UIM/EUS link handler (U/E LH) 562. Messages which notify the EUS of the reception of a SUWU also contain the data for the SUWU, thus comple~ing the reception process. In the case of a LUWU, however, the EUS
allocates its memory for reception, and issues a receive request via E/U LH 540 to 15 recehte request handler SoO, which formulates a receive worklist and sends it to resource manager S54, which controls the hardware and effects the data transfer over EUS bus 92 (FIG. 4) via a DMA arrangement. Note that the receive request from the EUS need not be for the entire amount of data in the LUWU; indeed, all of the data may not have even arrived at the UIM when the EUS makes its first 20 receive request. When subsequent data for this LUWU arrives, the EUS will again be notified and will have an oppo~tunity to rnake additional receive requests.
In this fashion, the recepdon of the data is pipelined as much as possible in order to reduce latency. Following data transfer, receive request handler 560 informs the EUS via U/E LH 552, and directs memory manager 550 to de-allocate the 25 memory for that p~ion of tKe LUWU that was delivered, thus making that memory available for new incorning data.
In the reverse direction, i.e., from EUS 26 to MINT 11, the operation is controlled as follows: driver 570 of EUS 26 sends a transmit request to transmit request handler 542 via U/E LH 562. In the case of a SUWU, the 30 transmit request itself contains the data to be transmitted, and transmit request handler 542 sends this data in a transmit worklist to resource manager ~54, which computes the packet header and writes both header and data into buffer 15 (FIG. 4), from which it is transmitted to NIM 2 by UI~M link handler 546 when authorized to do so via the ~ow control protocol in force on link 14. The 35 packet is received at NIM 2 by UIM/NIM link handler 530 and stored in buffer 94. Arbiter 532 then selects among a plurality of buffers 94 in NIM 2 to select the next packet or SUWU to be transmitted under the control of NIM/MINT
link handler 534 on MINT link 3 to MINT 11. In the case of a LUWU, transmit request handler 542 decomposes the request into packets and sends a transmit worklist to resource manager S54, which, for each packet, forrnulates the header, 5 writes the header into buffer 15, controls the hardware to effect the transfer of the packet data oveT EUS bus 92 via DMA, and directs U/N LH 546 to transmit the packet when authorized to do so. The transmission process is then as described for the SUWU case. In either case, transmit request handler 542 is notified by resource manager 554 when transmission of the SUWU or LUWU ls complete, 10 whereupon driver 570 is notified via U/E LH 562 and may release its transrnit buffers if desired.
FIG. 19 also shows details of the internal software structure of EUS 26. Two types of arrangements are shown, in one of which blocks 572, 574, 576, 578, 580 the user system performs level 3 and higher funcdons. Shown in 15 F~G. 19 is an implementation based on Network of the Advanced Research Projects Adminis~ration of the U.S. Department of Defense (ARPAnet,) protocols including an internet protocol 580 (level 3), transmission control protocol (TCP) and user datagram protocol (UDP) block 578 (TCP being used for connection oriented service and UDP being arranged for connectionless service). At higher 20 levels are the remote procedure call (block 576), the network file server (block 574) and ~he user programs 572. Alternatively, the services of the MAN
network can be directly invoked by user (block 582) prograrns which directly interface with driver 570 as indicated by the null block 584 between the user and the driver.
25 8.3.3.3 EUS Inte~face Functions The main funceional parts of the transrnit EUS interface are a control inter~ace ~vith the EUS~ and a DMA intefface to transfer data between the EUS
and the UIM over the system bus. When transrnitting into the network~ control infolmation is received that describes a LUWU or SUWUs to be transrnitted and 30 info~nadon about the EUS buffers where the data resides. The control information fiom the EUS includes destination MAN address, desdnation group (virtual network), LUWU length~ and type fields for type of service and higher level protocol type. The DMA interiFace moves the user data over from the EUS
buffers into the UIM. I'he network interface portion is responsible for formatling 35 the LUWUs and SUWUs into packets and transmitting the paclcets on the link tothe network. The control interface could have several variations for flow control.

~315373 multiple outstanding requests, priority, and preemption. The UIM is in control of the amount of data that it takes from the EIJS memory and sends into the network.
On the receive side, the EUS polls for information about packets that S have been received and the control interface responds with LUVYU inforrnation from the packets header and current infonnation about how much of the EUS
transaction has arrived. Over the control interface, the EUS requests to receivedata from these messages, and the DMA interface will send the data from memory on the UIM into the EUS memory buffers. The poll and response mechanism in 10 the interface protocol on the receive side allows a lot of EUS flexibility for receiving data from the network. The E~US can receive either partial or entire transac~ons that have come from the source EUS. It also provides the flow control mechanism for the EUS on receive. I'he EUS is in control of what it receives, when it receives it, and in what order.
15 8.3 3.4 SUN Software This section describes how a typical end user system, a SUN-3 workstation, is connectable to MAN. Other end user systems would use different software. The interface to MAN is reladvely s~aightforward a;nd efficient for a number of systems which have been studied.
20 8.3.3.4.1 Existin~ Network Software The Sun UNIX(9 operating system is derived from ~he 4.2BSD UNIX
system from the University of California at Berkeley. Like 4.2BSD it contains aspart of the kernel, an implementation of the AR~P~net protocols: internet protocol ~lP), transrnission control protocol (TCP) for connection-oriented seNice on top of 25 IP, and user datagram protocol (UDP) for connectionless serv~ce on top of IP.Current Sun systems use IP as an internet sublayer in the top half of the network layer. The bottom half of the network layer is a network specific sublayer. It currently consists of driver level software that interfaces to a specific network hardware connection, namely an ETHERNET controller, where the link layer 30 ~AC protocol is implemented. ETHERNET is the network cunently used to connect Sun workstations. To connect Sun workstations with a MAN network. it is necessary to fit into the framework of ~his existing networking software. Thesoftware for the MAN netwo~k interface in the Sun will be driver level software The MAN network is naturally a connectionless or datagrarn type ~-f 35 network. LIJWU data with con~ol information forrns the EUS transaction crossing the interface into the network. Existing networlc services can be provi~e~

using the MAN network datagram LUWUs as a basis. Software in the Sun will build up both connectionless and connection-or~ented transport and application services on top of a MAN datagram network layer. Since the Sun already has a variety of network application software, the MAN driver will provide a basic 5 service with the flexibility to multiplex multiple upper layers. This multiplexing capability will be necessary not just for existing applications but for additional new applications that will use MAN's power more directly.
There needs to be an address transladon service function ;n the EUS
at the driver level in the host software. It would allow for IP addresses to be 10 translated into MAN addresses. The address translation service is sirnilar infunc~ion to the current Sun address resolution protocol (ARP), but different in implementation. If a particular EUS needs to update its address translation tables, it sends a network message with an IP address to a well known address translation server. The corresponding MAN address will be returned. With a set of such 15 address translation services, MAN can then act as the underlying network for many different, new and existing, network software services in the Sun environment.
8.3.3 4.2 Device Driver On the top side, the driver rnultiplexes s~.veral different queues of 20 LUWlJs from the higher protocols and applications for transmission and queuesup received LUWUs in several different queues for the higher layers. On the hardware side, the driver sets up DMA transfers to and from user memory buffers.The driver must communicate with the system to map user buffers into memory that can be accessed by the DMA controller over the main systern bus.
On transm~t, the driver must do address translation on the outgoing LUWUs for those protocol layers that are not using MAN addresses, i.e., the ARPAnet protocols. The MAN destination address and destination group is included in MAN datagram control information that is sent when a LUWU is to be transmitted. Other transmit control information wil~ be LUWU length, fields 30 indicating type of service and higher level protocol, along with the data location for DMA. The UIM uses this control info~nation to forrn packet headers and to move the LUWU da~a out of EUS memory.
On receive, the driver will implement a poll/response protocol with the UIM notifying the EUS of incorning data. The poll response will contain 35 control infolmation that gives source address, total LUWU length, arnount of data that has arrived up to this point, the type fields indicating higher protocol layers7 and some agreed on arnount of the data from the message. (For small messages, the whole user message could arrive in this poll response.) The driver itself has the flexibility based on the type field to decide how to receive this message and which higher level entity to pass it on up to. It may be, that based on a celtain 5 type field, it may just deliver the announcement, and pass the reception decision on up to a higher layer. Which ever approach is used, eventually a control request for the delhery of the data from the UIM to the EUS memory is made, which results in a DMA operation by the UIM. EUS buffers to receive the data may preallocated for the protocol types where the driver handles the reception in a 10 fixed fashion, or the driver may have to get buf~`er information from a higher layer in the case where it has just passed the announcement on up. This is the type offlexibility we need in the driver to handle both existing and new applications in the Sun environment.
8.3.3 4.3 Raw MAN Inter~ace Software Later, as applica~ions are written that wish to directly use the capabilides of the MAN network, the address transladon function will not be necessary~ The MAN datagram control in~orrnadon will be specified directly by special MAN network layer software.
9 MAN Protocols 20 9.1 Overview The MAN protocol provides for the delivery of user data from source UIM across the network to destination UIM. The protocol is connectionless, asyI}~netric for ~eceive and send, implements error detection without correction, and discards layer purity for high perormance.
25 9.2 Me sage Scenalrio The EUS sends datagram transactions ealled LUWUs into the network. Ihe data that comes from the EUS resides in EUS memory. A control message from the EUS specifies to the UIM the data length, the destination address for this LUWU, the~destination group and a type field which could contain 30 înfonnadon like the user protocol and the ne~wor~c class of service required.Together, the data and the control information form the LUWU. Depending on the type of EUS inter~ace, this data and control can be passed to the UIM in different ways, but it is likely that the data is passed in a DMA transfer.
The UIM will transmit this LIJWU into the network. To reduce 35 potential delay, larger LUWUs are not sent into the network as one contiguousstream. The UIM breaks up the LUWU into fragments called packets that can be up to a certain maximum size. An UWU smaller than the maximum size is called a SUWU and will be contained in a single packet. Several EUSs are concentrated at the NIM and packets are transmitted over the link from the UIM to the NIM
(the EUSL). Packets from one UIM can be demand multiplexed on the link frors S the NIM to the MINT tthe XL) with packets from other EUSs. Delays are reduced because no EUS has to wait for the completion of a long LUWU from another EUS sharing the link to the MINT. The UIM generates a header for every packet that contains information from the original LUWU transaction, so that each packet can pass through the network from source UIM to destination UIM and be 10 recombined into the same LUVirU that was passed into the network by the source EUS. The packet header contains the inforrnation for the network layer protocol in the MAN network. `
Before the NIM sends the packet to the MINT on the XL, it adds a NIM~MINT header to the packet message. The header contains ~he source port 15 number identifying the physical port on the NIhI where a particula~ EUS/UIM is connected. This header is used by the MINT to verify that the source E~US is located at the port where he is authorized to be. This type of additional checl~ is especially important for a data network that serves one or more virtual ne~works, to ensure privacy for such virtual networks. The MINT uses the packet header tO
20 deterrnine the route ~or the packet, as well as other po~ential services. The MINT
does not change the contents of the packet header. When the ILH in the MINT
passes the packet out through the switch to be sent out on the XL to the destination NIM, it places a different port number in the NIM/MINT header. I'hisport number is the physical port on the NIM where the destination EUS/UIM is 25 coMected. The destination NIM uses this port number to route the packet on the fly to the proper EUSL.
The var~ous sections of a packet are identified by delimiters zccording to the link foImat. Such delimiters occur between the NlhVMlNT header 600 and the MAN header 610~ and bet veen the MAN header and the rest of the packet.
30 The delimiter at the MAN header/rest of packet border is required to signal the header check sequence circuit to insert or check the header check. The NIM
broadcasts a received packet to all ports in the NIM/MINT header field.
When the packet arrives at ~he destination UIM, the packet header contains the original inforrnation from the source UIM necessary to reassemble the 35 source EUS transaction. There is also enough information to allow a variety of EUS receive interface approaches including pipelining or other variations of EUS

transaction size, prioritization, and preemption.
9.3 MAN Protocol Descripdon 9.3~1 Link Layer Functions The link functions are described in Section 5. The functions of 5 message beginning and end demarcation, data transparency, and message check sequences on the EUSL and XL links are discussed there.
A check sequence for the whole packet message is perforrned at the link level, but instead of corrective action being taken there, an indication of the error is passed on up to the network layer for handling there. A message check 10 sequence error results only in incrementing an error count for administrativepurposes, but the message transrnission continues. A separate header check sequence is calculated in hardware in the UIM. A header check sequence er or detected by the MINT control results in the message being thrown away and an erroF count being incremented for administrative purposes. At the destination 15 UIM a header check sequence error also results in the message being thrown away. The data check sequence result can be conveyed to the EUS as part of the LUWU arlival notification, and the EUS can determine whether of not to receive the message. These violations of layer purity have been made to simplify the processing at the link layer to increase speed and overall network perforrnance.Other ''standard" link layer functions like error correction and flow con~ol are not performed in ~he conventional manner. There are no acknowledgement messages retumed at the link level for error correction (retransmission requests) or for flow control. Flow control is signaled using special bits in the framing pattern. The complexity of X.25-like protocols at the 25 link level can be tolerated for low speed links where the processing overhead will not reduce pe~formance and does increase the reliability of links that h~ve higherror rates. However, it is felt that an acceptable level of error-free throughput will be achieved by the low bit error rates in the fiber optic links in this network ~Bit Error Rate less than 10 errors per trillion bits.) Also, because of the large 30 amounts of buffer memory in the MINT and the UIM necessary to handle data from the high-speed links, it was felt that flow control messages would not be necessary or effective.
9.3.2 Network Layer 9.3.2.1 Funcoons The message unit that leaves the source UIM and travels all the way to the destination UIM is the packet. The packet is not altered once it leaves the source UIM.
The information in the UIM tO UIM message header will allow the following functions to be perforrned:
- fragmentation of LUWUs at the source UIM, - recombination of LlJWUs at the destination UIM[, - routing to the proper NIM at the MINT, - routing to the proper UIM/EUS port at the destination NIM, - MINT transmission of variable length messages (e.g., SUWU, packet, n packets), - destination UIM congestion control and anival announcement, - detection and handling of message header errors, - addressing of network entities for internal network messages, - EUS authentication for delivery of network services only to authorized users.
9.3.2.2 Format FIG. 20 shows the UIM to MINT Message format. The MAN
header 610 consists of the Destination Address 612, the Source Address 614, the group (virtual network) identifier 616, group name 618, the type of service 620,the Packet Length (the header plus data in bytes) 62~, a type of service indicator 623, a protocol identifier 624 ~or use by end user systems for identifying ~he contents of EUS to EUS header 630, and the Header Check Sequence 626.
The header is of fixed length, seven 32-bit words or 224 bits long. The MAN
25 header is followed by an EUS to EUS header 630 to process message fragmentation. This header includes a LUWU identifier 632, a LUWU length indicator 634, the packet sequence number 636, the protocol identifier 638 for i~entifying the contents of the internal EUS protocol which is the header of user data 640, and the number 639 of the initial byte of data of this packet within the 30 total LUWU of info~nation. Finally, user data 640 may be preceded for appropriate user protocols by the identity of the destination port 642 and sou~:e port 644. The fields are 32 bits because that is the most efficient length (integerc ~
for present network control processors. Error checking is perfo~med on the he;lder in control software; this is the EIeader Check Sequence. At the link level, error 3S checking done over the whole message; this is the Message Check Sequence The NIM/M~r header 600 (explained below) is also shown in the figure for completeness.
The destination adclress, group identification, type of seIvice, and the source address are placed as the first five fields in the message for efficiency in MINT processing. The destination and group identification are used for routing, 5 the size for memory management, the type fields for special processing, and the source is used for service authentication.
9 3`2.2.1 Destina~ion Address _ _ _ . _ The Destination Address 612 is a MAN address that specifies to which EUS the packet is being sent. A MAN address is 32 bits long and is a flat 10 address ~hat specifies an EUS connected to the network. (In internal network messages, if the high order bit in the MAN address is set, the address specifies an internal network entity like a MINT or NIM, instead of an EUS.) A MAN
address will be permanently assigned to an EUS and will identify an EUS even if it moves to different physical location on the network. If an EUS moves, it must15 sign in with a well-known routing authentication server to update the correspondence between its MAN address and the physical port on which it is located. Of course, the port number is supplied by the NIM so the EUS cannot cheat about where it is located.
In the MI~T the destination address will be used to determine a 20 destination NIM for routing the message. In the destination NIM the destination address will be used to determine a destinadon UIM for routing the message.
9.3.2.2.2 Packet Length The Packet Length 622 is 16 bits long and represents the length in bytes of this message fi~agment including the fixed leng~h header and the data.
25 This leng~h is used by the MlNT for transrn~tting the message. It is also used by the destination IJIM to determine the amount of data av~ulable for delivery to the E~US~
9.3.2.2.3 T~e Fields The type of seNice fi~ld 623 is 16 bits long and contains the type of 30 service specified in the original EUS request. The MINT may look at the type of service and handle the message differently. The destinadon UIM may also look at the type of service to de~errnine how to deliver the message to the destination EUS, i.e., deliver even if in error. The user protocol 624 assists the EUS driver in multiplexing various strearns of data from the network.

9.3.2.2.4 Packet Sequence Number This is a Packet Sequence Number 636 for this particular LUWU
transmission. It helps the receiving UIM recombine the incoming LUWU, so that it can determine if any fragments of the transm~ssion have been lost because of S error. The sequence number is incremented for each fragment of the LUWU. The last sequence number is negative to indicate the last packet of a LUWU. (An SUWU would have -1 as the sequence number.) If an infinite length LUVVU is being sent, the Packet Sequence Number should wrap around. (See UWU Length, Section g~3.2.2.7, for an explanation of an infinite length LIJWU.
10 9.3.2._5 Source Address The Source Address 614 is 32 bits long and is a MAN address that specifies the EUS that sent the message. (See Desdnation Address for an explanation of MAN address.) The Source Address uill be needed in the MINT
for network accounting. Coupled with the Port Number 60Q from the NIM/3!~NT
15 header, it is used by the MINT to authenticate ~he source EUS for network services. The Source Address will be delivered to the destinadon EUS so that it knows the network address of the EUS that sent the message.
9.3.2.2.6 UWU ID
The UWU ID 63~ is a 32 bit number that is used by the destination 20 UIM to recombine a UWU. Note that the recombination job is made easier because fragments cannot get out of order in the network. The UWIJ ID, along with the Source and Destination Addresses, identifies packets of the same LUWU, or in other words, fragments of the original datagram transaction. The ID must be unique for the source and destination pair for the time that any fragment is in the 25 network.
9.3.2 2.7 UWU Length The UWU Length 634 is 32 bits long and represents the total length of UWU data in bytes. In the first packet of a LUWU this will allow the destination UIM to do congestion control, and if the LU~U is pipelined into the 3û EUS, it will allow the UIM to beg2n a LUWU announcement and delively before the complete LUWU arrives at the UIM.
A Length that is negative indicates an infinite length LUWU, which is like an open channel between two EUSs. Closing down an infinite length LUWU
is done by sending a negative Packet Sequence Number. An infinite length 35 LUWU only makes sense where the UIM controls the DMA into EUS memory.

9~3.2.2.8 Header Check Sequence There is a header check sequence 626, calculated by the transmitting UIM for header inforrnation so that the MINT and the destination UIM can determine if the header information was received correctly. The MIN'r or the 5 destination UIM will not attempt delivery of a packet with a header check sequence error.
9.3.2.2.9 User Data The user data 640 is the portion of the user UWU data that is transmitted in this fragment of the transmission. Following the data is the overall 10 message check sequence 646 calculated at the link level.
9.3.3 NIM/MINT Layer 9.3.3.1 Functions This protocol layer consists of a header containing a NIM port nurnber 600. The port number has a one to one correspondence to an EUS
15 connection on the NIM and is prepended by the NIM in block 403 (FIG. 16) so that the user cannot enter false data therein. This header is positioned at the front of a packet message and is not covered by the overall packet message check sequence. It is checked by a group of parity bits in the sarne word to ¢nhance its error reliability. The incorning message to the MINT contains the source NIM
20 port number to assist in user authentication for network se~vices that might be requested in the type fields. The outgoing message from the MINT contains the destination NIM port number in place of the source port ~00 in order to speed the demultiplexing/routing by the NIM to the proper destination EUS. If the packct has a plurality of destination ports in one NIM, a list of these ports is placed at 25 the beginning of the packet so that section 600 of the header becomes several words long.
10 LOGIN PROCEDURES AND VIRTUAL NEI~WORKS
10.1 General A systern such as MAN is naturally most cost effective when it can 30 serve a large number of customers. Such a large number of cus~omers is likely to include a number of sets of users who require protection from outsiders. Such users can conveniently be grouped into virtual networks. In order to provide still further flexibility and protection, individual users may be given access to a number of virtual networks. For example, all the users of one company may be 35 on one virtual network and the payroll department of that company may be on asepa~ate virtual network. The payroll department users should belong to both of these virtual networks since they may need access to general data about the corporation but the users outside the payroll departrnent should not be members of the virtual network of the payroll department virtual network since they should not have access to payroll records.
The login procedure method of source ch~ckirlg and the method of routing are the arrangements which perrnit the MAN system to support a large number of virtual networks while providing an optimum level of protection against unauthorized data access. Further, the a~rangement whereby the NIM
prepends the user port to every packet, gives additional protection against access 10 of a virtual network by an unauthorized user by preventing aliasing.
10.2 Buildin~ Up the Authorization Data Base FIG. lS illustrates the administrative control of the MAN network. A
data base is stored in disk 351 accessed via operation, administration, and maintenance ~OA&M) system 350 for authorizing users in response to a login 15 request. For a large MAN network, OA&M system 350 may be a distributed multiprocessor arrangement ~or handling a large volume of login requests. This data base is arranged so that users cannot access restricted virtual networks ofwhich they are not members. The data base is under the control of three types ofsuper users. A first super user who would in general be an employee of the 20 cornmon carrier that is supplying MAN service. This super user, referred to for convenience herein as a level l super user, assigns a block of MAN names which would in general consist of a block of numbers tc~ each user group and assigns type 2 and type 3 super users to particular ones of these names. The level 1 super user also assigns virtual networks to particular MAN groups. Finally, a level 1 25 super user has the authonty to create or destroy a MAN supplied service such as electronic "yellow page" service. A type 2 super user assigns ~/alid MAN names from the block assigned to the particular user communi~y, and assigns physical port access restrlctions where appropriate. In addition, a type 2 super user has the authority to restrict access to certain virtual networks by sets of members of his 30 customer community.
Type 3 super users who are broadly equal in authority to type 2 super users, have the authority to grant MAN names access to their virtual networks.
Note that such access can only be granted by a type 3 super user if the MAN
narne's type 2 super user has allowed this MAN name user the capability of 35 joining this group by an appropriate entry in table 370.

The data base includes table 360 which provides for each user identification 362, the password 361, the group 363 accessible using that password, a list of ports and, for special cases, directory numbers 364 from which that user may transmit and/or receive, and the type of service 365, i.e., receive 5 only, transmit only, or receive and transmit.
The data base also includes user-capability tables 370,375 for relating users (table 370) to groups (table 375) potentially authorizable for each user.
When a user is to be authorized by a super user to access a group, this table ischecked to see if that group is in the list of table 370; if not the request to 10 authorize that user for that group will be rejected. Super users have authority to enter data for their group and their groups in tables 370,375. Super users also have the authority for their user to move a group from table 375 into the list of groups 363 of the user/group authorization table 360. Thus, for a user to accessan outside group, super users from both groups would have to authorize this 15 access.
10.3 Login Procedure At login tirne, a user who has previously been appropriately authorized according to the arrangements described above, sends an initial loginrequest message to the MAN network. This message is destined not for any other 20 user, but for the MAN network itself. Effectively, this message is a header only message which is analyzed by the MINT central conhol. The password, type of login service being requested, MAN group, MAN name and port number are all in the MAN header of a login request, replacing other fields. This is done because only the header is passed by the XLH to the MI:NT central control, for further 25 processing by ~e OA8cM central control. The login data which includes the MAN name, the requested MAN group name (virtual network name), and the password are compared against the login authorization data base 351 to check whether the particular user is auth~ized to access that virtual network from thephysical po~ to which that user is connected (the physical port was prepended bv30 the NIM prior to reception of the lo~in packet by the MINI'). If the user is in fact properly authorized, then the tables in source checker 307 and in router 30(FIG. 14) are updated. Only the SOUrGe checker table of the checker that processes the login user's port is updated from a login for terminal opera~ions lf a login request is ~or receive functions, then the routing tables of all MINTs mu~
35 'oe updated to allow that source to receive data from any authorized connecta~l~
user of the same group who may be connected to other MINTs to respond to requests. The source checker table 308 includes a lis~ of authorized name/group pairs for each port connected to the NIM that sends the data stream to the XLH
for that source checker. The router tables 310, all include entries for all users authorized to receive UWUs. Each entry includes a name/group pair, and the S corresponding NIM and port number. The entries in the source checker list are grouped by group identification numbers. The group identification number 616 is part of the header of subsequent packets from the logged in user, and is derivedby the OA&M system 350 at login time and sent back by the OA&M system via the MAN switch 10 to the login user. The OA&M system 350 uses the MINl`
10 central control's 20 access 19 to the M~T memory 18 to enter the login acknowledge to the login user. On subsequent packets, as they are received in the MINT, the source checker checks the port number, MAN name and MAN group against the authorization table in the source checker with the result that the packet is allowed to proceed or not. The router then checks to see if the destination is an 15 allowable destination for that input by checking thç virtual network group name and the destination name. As a result, once a user is logged in, the user can reach any destination that is in the routing tables, i.e., that has previously logged in for access in the read only mode or the read,~write mode, and that has the same virtual network group name as requested in the login; in contrast unauthorized users are20 blocked in every packet.
While in the present embodiment, the checking is done for each packet, it could also be done for each user work unit (LUWU or SUWU), with a recorded indication that~ all subsequerlt packets of a LUWU whose or~ginal packet was rejected are also to be rejected, or by rejecting all LUWUs whose initial 25 packet is missing at the user system.
Those super user logins which are associated with making changes in the login data base are checked in the same way as conventional logins except that it is recognized in OA~M system 350 as a login request for a user who has au~hority for changing the data base stored on disk 351.
Super users types 2 and 3 get access to the OA&M system 350 from a computer connected to a user port of MAN. OA&M system 350 der~ves statistics on billing, usage, autho~izations and performance which the super users can access f~om their computers.
The MAN network can also serve special types of users such as 35 transmit only users and receive only users. An example of a transmit only user is a broadcast stock quotation system or a video transrnitter. Outputs of transmit ~9 only users are vnly checked in source checker tables. Receive only units such asprinters or monitoring devices are authol~zed by entries in the routing tables.

11 APPLICAl~ON OF MAN TO VOICE SWITCHING
FIG. 22 shows an arrangement for using the MAN architecture to 5 switch voice as well as data. In order to simplify the application of this architecture to such services, an existing switch in this case, the SESS@) switch manufactured by AT~T Networ~ Systems, is used. The advantage of using an existing switch is that it avoids the necessity for developing a program to control a local switch, a very large development effort. By using an existing switch as 10 the interface between the MAN and voice users, this effort can be almost completely eliminated. Shown on FIG. 22 is a conventional customer telephone connected to a switching module 1207 of SESS switch 1200. This customer telephone could also be a combined integrated services digital network (ISDN) voice and data customer station which can also be connected to a SESS switch.
lS Other customer stations 1202 are connected through a subscriber loop carrier system 1203 which is connected to a switching module 1207. The switching modules 1207 are connected to a time multiplex switch 1209 which sets up connections between switching modules. Two of these switching modules are shown connected to an interface 1210 comprising Common Channel Signaling 7 20 (CCS 7) signaling channels 1211, pulse code moduladon (PCM) channels 1213, an special signaling channels 1215. These are connected to a packet assembler and disassembler 1217 for interfacing with an MAN NIM 2. The function of the PAD is to interface between the PCM signals which a~e generated in tbe switch and the packet signals which are switched in the MAN network. The function of 25 the special signaling channel 1215 is to inform PAD 1217 of the source and destinadon associated with each PCM channel. The CCS 7 channels transmit packets which require furtber processing by PAD 1217 to get them into the form necessary for sYvitching by the MAN network. To make the system less vulnerable against the failure of equipment or transrnission facilities, the switch is 30 shown as being connected to two dif~erent NlMs of the MAN ne~work. A digital PBX 1219 also interfaces with packet assembler disassembler 1217 directly. In a subsequent upgrade of the PAD, it would be possible to interface directly with SLC 1203 or with telephoDes such as integrated services digital network (ISDN) telepbones that generate a digital voice bit stream directly.

The NIMs are connected to a MAN hub 1230. The NIMs are connected to MINTs 11 of that hub. The MINTs 11 are interconnected by MAN
switch 22.
For this type of configuration, it is desirable to switch substantial 5 quantities of data as well as voice in order to utilize the capabilities of the MAN
hub most effectively. Voice packets, in particular, have very short delay requirements in order to minimize the total delay encountere~ in transmitting speech from a source to a destination and in order to ensure that there is no substantial interpacket gap which would result in the loss of a portion of the 10 speech signal.
The basic design parameters for MAN have been selected to optimize data switching, and have been adapted in a most straightforward manner as shown in FIG. 22. If a large amount of voice packet switching is required, one or moreof the following additional steps can be taken:
1. A form of coding such as adaptive differential PCM (ADPCM) which offers excellent performance at 32 Kbit/second could be used instead of 64 Kbit PCM. Excellent coding schemes are also available which require fewer than 32 Kbit/sec. for good performance.
2. Packets need only b,e sent when a custcrner is actually speaking. This reduces the number of packets that must be sent by at least 2:1.
3. The size of the buffer ~or buffering voice samples could be illcreased above the storage for 256 voice sarnples (a two packet buffer) per channel.
However, longer voice packets introduce more delay which may or may not be tolerable depending on the characteAstics ~ the rest of the voice network.
4. Voice traffic migh~ be concentrated in specialist MINTs to reduce the number of switch setup operations for voice packets. Such an arrangement may enlarge the number of customers affected by a failure of a NIM or MINT and might require arrangements for providing alternate paths to another NM andlor MINT.
5. Alternate hub configurations can be used.
The alternate hub configuration of FIG. 24 is an exarnple of a step 5 solution. A basic problem of switching voice packets is that in order to minimize delay in transmitting voice, the voice packets must represent only a short segment 35 of speech, as low as 20 milliseconds according to some estimates. This corresponds to as many as 50 packets per second for each direction of speech. If a substantial fraction of the input to a MINT represented such voice packets, the circuit switch setup time might be too great to handle such traffic. If only voice traffic were being switched, a packet switch which would not require circuit setup operations rnight be needed for high traffic situations.
S One embodiment of such a packet switch 1300 comprises a group of MINTs 1313 interconnected like a conventional array of space division switches wherein each MINT 1313 is connected to four others, and enough stages are added to reach all output MINTs 1312 that carry heavy voice traffic. For added protection against equipment failure, the MINTs 1313 of the packet switch 1300 10 could be interconnected through MANS 10 in order $o route traffic around a defective MINT 1313 and to use a spare MINT 1313 instead.
Ihe output bit stream of NIM 2 is connected to one of the inputs (XL) of an input MINT 1311. The packet data traf~ic leaving input MINT 1311 can continue to be switched through MANS 10. In this embodiment, the data 15 packet output of MANS 10 is merged with the voice packet output of data switch 1300 in an output MINT 1312 which receives the outputs of MANS 10 and data switch 1300 on the XL 16 (input) side and whose IL 17 output is the input bit stream of NIM 2, prodoced by a PASC circuit 290 (FIG. 13). Lnput MINT 1311 does not contain the PASC circuit 290 (FfG. 13) for generating the 20 output bit stream to NIM 2. For output M~T 1312 the inputs to the XLs from MANS 10 pass through a phase alignment circuit 292 (FIG. 13) such as that shown in FIG. 23, since such inputs come from many different sources through circuit paths that i~llsert different delay.
This a~Tangement can also be used ~or switching high priority data 25 packets through the packet switch 1300 while retaining the circuit switch 10 for switching low prionty data packets. With this arrangement, it is not necessary to connect the packet switch 1300 to output MINTs 1312 car~ying no voice traffic; in that case, high priori~ packets to MINTs calTying no voice traffic would have tobe routed through circuit switch MANS 10.
30 12 MINT ACCESS CONTROL. TO MAN SWITCH CONTROL
FIG. 21 illustrates one arrangement for controlling access by MINTs 11 to the MAN switeh control 22. Eaeh MINT has an associated aeeess controller 1120. A data ring 1102,104,1106 distributes data indicating the availability of output links to eaeh logie and count cireuit 1110 of each access35 eontroller. Eaeh aceess eontroller 1120 maintains a list 1110 of output links such as 1112 to which it wants to send data, each link having an associated priority indicator 1114. A MINT can seize an output link of that list by marking the linkunavailable in ring 1102 and transmitting an order to the MAN switch control 22 to set up a path from an LH of that MINT to the requested output link. When the full data block to be transrnitted to that output link has been so transmitted, 5 the MINT marks the output link available in the data transmitted by data ring 1102 which thereby makes that output link available for access by other MINTs.
A problem with using only availability data ls that during periods of congesdon the time before a pardcular MINT may get access to an output link can 10 be excessive. In order to even the accessibility of any output link to any MINT, the following arrangement is used. Associated with each link availability indication, called a ready bit transmitted in ring 1102, is a window bit transmitted in ring 1104. The ready bit is controlled by any MINT that seizes or releases anoutput link. The window bit is controlled by the access controller 1120 of only a 15 single MINT called, for the purposes of this descr~ption, the controlling MINT. In this particular embodimen~, the controlling MINT for a given output link is the MINT to which the corresponding output link is routed.
The ef~ect of an open window (window bit = 1) is to let ~he first access controller on the ring that wants to seize an output lir~ and recognizes ilS
20 availability as the ready bit passes the controller, seize such a link, and to let any controller which tries to seize an unavailable lir~ set the priority indicator 1114 for that unavailable link. The effect of a closed window (window bit = û) is tO
permit only controllers which ha~re a priority indicator set for a colTespondingavailable link to seize that available link. The window is closed by the access 25 con~oller 1120 of the controlling MI~T whenever the logic and count circuit 1100 of that controller detects ~hat the output link is not available (ready bit = 0) and is opened whenever that controller detects that that output link isavailable (ready bit = 1).
The operation of an access controller seizing a link is as follows. lf 30 the link is unavailable (ready bit = 0~ and the window bit is one, the accesscontroller sets the priority indicator 1114 for that output link. If the link isunavailable and the window bit is zero, the controller does nothing. If the link i~
available and the window bit is one, the controller seizes the link and marks the ready bit zero to ensure that no other controller seizes the same link. If the link :~
35 available and the window bit is zero, then only a controller whose priority indicator 1114 is set for that link can seize that link and will do so by markin~ IhC

ready bit zero. The action of the access controller of the controlling MINT on the window bit is simpler: that controller simply copies the value of the ready bit into the window bit.
In addition to the ready and window bits, a frame bit is circulated in S ring 1106 to define the beginning of a frame of resource availability data, hence, to define the count for identifying the link associated with each clear and window bit. Data on the three rings 1102, 1104 and 1106 circulates serially ~nd in synchronism through the logic and count circuit 1100 of each MINT.
The result of this type of operation is that those access controllers 10 which ars trying to seize an output link and which are located between the unit that first successfully seized that output link and the access controller that controls the window bit have priority and will be served in turn ~efore any other controllers that subsequently may make a request to seize the specific output link.
As a result, an approximately fair distribution of access by all MXNTs to all output 15 links is achieved.
If this alternative approach to controlling MIN~ 11 access control to the MANSC 22 is used, priority is controlled from the MINT. Each MINT
maintains a priority and a regular queue for queuing requests, and makes requests for MANSC services first from the M~T priority queue.

It is to be understood that the above description is only of one pre~erred embodiment o~ the invention. Numerous other arrangements m~ay be devised by one skilled in the art without departing from the spirit and scope of the invention. The invention is thus limited only as defined in the accompanying 25 claims.

APPENDIX A
ACRONYMS AND ABBREVIATIONS

lSC First Stage Controller 2SC Second Stage Controller ACK Acknowledge ARP Address Resolution Protocol ARQ Automatic Repeat Request BNAK Busy Negative Acknowledge CC Central Control CNAK Control Nega~e Acknowledge CNet Con~ol Network CRC Cyclic Redundancy Check or Code DNet Data Network DRAM Dynamic Random Access Memory D~MA Direct Virtual Memory Access EUS End User System :
EUSL End Us~ Link (Connects NIM and UIM) FEP Front End Processor ~IFO First In First Out FNAK Fabr~c Blochng Nega~ve Acknowledge IL Internal Link (Connects MINT and MANS) ILH Internal Link Handler IP : Internet Protocol : LAN Local Ar:ea Network LUWU Long User:Work Unit MAN Exemplary~Me~opolitan Area Network : MANS MAN Switch MANSC : MAN/Switch Controller MINT ~emory and Interface Module MMU: : Memo~y Management Unit NAK : Negative Acknowledge NIM Network Interface Module OA~M Operation, Administration and Maintenance PASC Phase Alignment and Scramble Circuit SCC Switch Control Complex SUWU Short User Work Unit S TCP . I`ransmission Control Protocol TSA Time Slot Assigner UDP User Datagram Protocol UIM User Interface Module IJWU User Work Unit VLSI Very Large Scale Integration VME(~) bus An EEE Standard Bus ~; WAN Wide Area Network XL Extemal Link (Connects NIM to MINlr) XLH Extesnal Link Handler XPC Crosspoint Controller :

:~ ~ : : : :: : :

;
:
:
: ~ :

.

Claims (9)

1. In a circuit switching network for establishing connections between each of a plurality of input terminals on one side and each of a plurality of output terminals on another side, a method of controlling said switching network using a plurality of controllers comprising the steps of:
dividing said network into a plurality of disjoint sets of switching and connection elements, wherein each of said disjointed sets is used for establishing a different set of connections from ones of said plurality of terminals on a first side to all of said plurality of terminals on a second side;
assigning to different ones of said plurality of controllers, the control of different ones of said disjoint sets; and responsive to a request to set up a connection, controlling the establishment of a connection using the controller assigned to the disjoint set used for that connection.

2. The method of claim 1 wherein said switching network is a two-stage network and wherein said dividing step comprises the step of dividing said network into disjoint sets of elements wherein each such disjoint set comprises at least one switch connected to terminals on said first side.

3. The method of claim 2 wherein each of said controllers comprises a local data base for storing data for status information for each set of disjointelements controlled by said controller, and wherein said controlling step comprises the step of controlling the establishment of a connection using the local data base of said controller used for controlling the establishment of said connection.

4. The method of claim 3 wherein said circuit switching network is a space division network.

5. A circuit switching network comprising:
a plurality of first stage switches;
a plurality of second stage switches; and a plurality of controllers;
wherein each of said first stage switches is connected to a plurality of second stage switches;

wherein each of said plurality of controllers controls at least one of said second stage switches and sends control signals to any first stage switch that can access one of the second stage switches controlled by that controller;
wherein each of said controllers comprises a data base local to that controller for identifying and for determining the availability of a path from any first stage input to an output on one of the second stage switches controlled by that controller;
wherein each of said controllers comprises another data base local to that controller for determining availability of each output on each second switch controlled by that controller; and wherein said controllers are divided into groups of at least one, and wherein each of said groups of controllers controls a disjoint set of said second stage switches.

6. The circuit switching network of claim 5 wherein said circuit switching network is a space division network.

7. The circuit switching network of claim 6 wherein each of said controllers controls one and only one of said second stage switches.

8. A circuit switching network for establishing connections between each of a plurality of input terminals on one side and each of a plurality of output terminals on another side, comprising:
a plurality of switches, each comprising a plurality of switching elements, interconnected by a plurality of connection elements; and a plurality of controllers;
each of said plurality of controllers for controlling a disjoint set of switching and connection elements, each of said disjoint sets used for establishing a different set of connections from ones of said plurality of terminals on a first side to ones of said plurality of terminals on a second side, and responsive to requests to set up a connection using a disjoint set controlled by said each controller, wherein at least some of said switches are controlled by a plurality of controllers for controlling different disjoint sets.

9. The network of claim 8 wherein each of said controllers controls connections from ones of said terminals on said first side to all of said terminals on said second side.