CA2068048A1 - Fault tolerant processing section with dynamically reconfigurable voting - Google Patents

Abstract

Abstract of the Disclosure A fault-tolerant digital data processor includes three identical logic CPU boards connected to a voting bus and a system bus. The three boards are initially designated as a master board, a slave board 0 and a slave board 1. The master board drives the system bus and the two slave boards serve as backups in case the master breaks. The master board issues signals, at different instances with the aid of multiplexing, to the slave boards. On the slave boards, corresponding signals are compared and the result is broadcast to all three boards. When all three boards are compared equal, the master board remains as master. If there is a miscompare between one slave board and the master but not between the other slave board and the master, the master board remains master, and the slave board with which the miscompare occurred will be disabled after another miscompare. If a miscompare occurs between the master board and both slave boards, a re-execution of the previous cycle occurs. After a master board failure is confirmed, a slave board becomes a master board, and if there is another comparison failure the former master board is disabled.

Description

Ch A PT ER 2 Sur7Jey on ~t ~ferant Computer~lrch~tectu~

As comput~s take inc~cas~ngly cridcal roles in the sDciety, si~nply aug1nenting the qual-ily of components and impro~ing design ~cchniques un not provide adequa~e eomputer reliability ~ meet eslablished ne~ds. Compu ets must be able to funcd~n s ~iluJes are encountered. In today's computer parlance. this t~ansla~ into computer faull tolerance.
Fault ~olerance is essential for envi~nmCms tht ~equire suppo~t lo cn-line continuous processing sueh s the banlcing industry, 6nance, telc communic~ion, ~-traffic control, military application, retailing, etc. Since the ~art of the bst dullde, oDmmetcial com-puter manufacturcrs have devised a variety of tchitea~es lo SuppOTt f~uh tDlerance.
In thiC ehapter, the basic coneepts of eomputet fauJ~ toleJ nee ~I be e~Lamincd and a swey of sevetal ~ull ~oleran~ canpu~et rmplemensations win be presentcd.

2.1 Fundamenta1 Concepts of Fault Tolerant Computing :
There re Ihroe ma,iot eomponents in tbis seaion. F~. the common ~ources of eompul-et failutes ~l be ~rnineJ. ~en a more rigorous concep~ of computcr fsilurc wil1 bc defined. ~t Isst, thc necd and the dvanugc of building ~ult tolet nt computcr ~D be bncfly ~esa~
2.1.1 Ar~s o~ computer bllures With thc advcn~ of microprocessors, compusct ~ysums hav~ moved f om the clean cn-i-ronmcnts of computcr rooms to industrial cmu~nmcnts~ .~ harshcr cnv~onrncnt makcs env~onmental vanables ota c~mputcr sys~m fluctuatc dramatically, e.~. tcmpcrature IIOP R~port on R.un Tobr-nl CornpuUno 6 F ~lm~nt~ >nwpt3 ~ F,~n ~lsr~n~ ~mpu ~d hlmidiry kvtl, ~lrimuy power ~upply Ruc~ ion Imd Iccrromagnctic inu:rrer~ncc.~hcse Lac~s ~iU li~tly ~duce corn;pu~r failur~
A ~d rcRson ~or ct~mpuLcr ~ilun~ is commc~nly ci~d by compulcr industry el-~cr~s i5 thc i~dvcncnl user abusc due to ~c prolire~ation or computers arld Lhe ]DwerC~
kvel or c4tnpu~cr lita~cy rnong av~ut ~ompuLtr uscr~
ThirtSly, BS compu~cr ~ys~uns grow c1.~cr largcr, IhC~ arc morc componcnts in Ihe system ~md ~ubs~uen~ly, the ~robability of ~ysLcm failurt due lo the rnalflmclion ot ~ny com-ponet~l iDc~ases.
In ~ cornpa~cr sys~em, failures a~n ~. 8eneraLed frDm scveral levels that are ~arallel to Ihc kvels in ~c di6i~1 CompuLer r,yr~m. n~e first is Ihc circuit leveL Tbis level consis~s of such componcn~s ~s resislors, capachors, inductors, and power souJces. Thc r,scond 1evcl consisLs of lo~ic gates al~d dats operalors built oul of gates. Logic gaLes can be fw-thcr dividçd into Iwo cJlegories of cornbinuional and sequcntial logic pLCS. The ne~u IcveJ is ~he pro~ rarn IeveL The pro~am Ievel is unique to digital compuLers. ~1 Ihis kv-el a scqucncc otinslrucLions in the device is interp~ed and causes ~aion upon a dala structure. This is thc Ins~ucdon Su Processor sublcvd. the lSP descripion is used in lurn lo c~aLe rioftwarc components tlull ue essily rnanipula~d by prograrnmers the higb-level-bnguage sublcvd. The resul~ is sofiwa;re, such s opcntin~ systems, run-dme syslerns, ~pplica~ion p~g~ams, and application systuns. The lasl levd in a compul-u sysiem is thc pro~amming memory riwilch a'MS) Icvel. This kvd includes the in-put/outpul devices, memories, mass su~sge, communications, u~d processors. They are all interconneclcd lo form a compulcr systcm.
2.1.2 Definltlon~ In the roalm of computer fault tolerance There are Iwo sets of def~nitions ID k inlroduccd in ~his sectiorL One is rcla~d to com-puter failwes and the other is relaleJ to the measurement of a compuw's depcndabilil) .
Firs~, the sou~es of computer failu~s can k alegorizcd in Ihe folJowing way:
5~PhYS;CaIChanBCinhadVarC~ IPU~ I r I
S~ Erroncous slalc of hardv.~are or of rware ~~
rcsulting from failwes of components, physieal ~-interference from the unvua~rneltt~ q~r er- ~wn~
or, inconea desigm .~. \
r. Manifcstadon of a fault ~hin a ~ro~ram ~_ Enot or da~a s~ucture. The ror n~y ocAIr some dis- U~c ~ ~~
lance fsom tbe f~ll dle. ~-~i~ , a 27~nn~L Dcscribes f~ilure, faul~, or crror lha is ca~tinuous and slabk~ Ln hsrdwarc, pma- ~
nenl failure re~ects n irrevasiblc physical i~-- ~ _ change. Thc v ord h~rd is used interclungeably withpcrmanenL ~ ;~_ I mi~ult~ I
Fig. 1 S~urces of err~rs ~oP F-port on F.un Tobnnt Compullng 7 ~P'~4U~ np~n~ Dmpu~sr 13~L Describes a fault S~ t is only o~iiorL~ly prescnt due LO w~ita bk bAr~w~re or varying hardws~ hws~ ~Les.
nbeti ~ fault ~r ~rrDr t~ultin~ 6~m t~7lporary ~nvilJrqmcr.~al csn~i-~ons. th~ ~sd 6~ft i3 11ge4 int~r~:hRngc~bly ~h tr~nsJ~L
;Oll*~p of th ese clnSSe5 Or l~ompUW un~r iri Dlus~a~d in Fi~ a.
n)eJ~ ue ~wo unpo1ant m~ur~m~ s oS f~h ~41e~n~ c~pabilitics in ~ computcr sys-em. T~e finit is ~vail~bility. /~vaila~ility i~ ~ function Or ~ne. Il is thc probabiii~y lhat Ihe systun is o~ic nal u ~ instant Or t~une. 17~e Se~Dnd m~cmcnt i~s t~eliability. It ~s Ihe condi~ion~t p~abi~ity ahu ~e sys~n has ~urvivcti the intemi [O, IJ, ~ivcn uhat it was op~tional ~t ~ne ~0. Reliability is t3s~d ~o dc~ibe systcms ~ which tcpair canno~ ts~e pla~ (~s in rltcDiie computct~) or in which Ihe cornput~ iis ~ing ~ criti-ca1 ~uncti~n &nd c~t be 10st cven foT tbt dwuion Or D r~ir (tlS In fiight comput~as on ~r~af~) or t;n which Ibe t~a~r t~i prohibidvdy ~pensivc.
2.1.3 The need ~Ind Incetltlve t~ bulId f~uH lobt~nt comput~r As;Je h~m thc spparent b~ of having a csmputcr systcm that clm fimcdon w.ithoutintcrruption tn thc cvtm of c~mputcr faiiurc, il ti econDmically ~ound to o~vn ~ ~ull tol-cJant computes cystan. Ihe cosl of ~ computcr tynem is DOl limi~d ID initial p~hase;
~ignific~nt costs r~cur during Ihe life of a ~ystcm. ~ f~ull lol~anl computcr systcm not only reduces the ~st Or rn~untcnanK, il rcduccs ~e cosl of downtimc significantly.

~2 Various Types of Impîement~tlon In Fault ~oler~nt Computer . . _ . . .
In ~his section, two comparisons wili bc made on Ihc difrescnl app~ches jD designing a fault toluam computes syslcm. The sewnd compasison is made on a narrowcr scope.
2.2.1 Sottwar~lault vs. I~ardwat~hult tolet~nce So~twue fault tolcrancc u~d hudwarc-faul~ tole~ancc rharc many common featutc such as t~dundancy, self~checking, e~c. ~VitD difrcrcnt focuscs Sofn4~ faul~ tolcrancc aims at diminating computcr eJror duc to f~llsc s,ofn~arc dcsign nd prograrntniD~. WDCr~S
hrdWarC~fJUIt tOIC~CC airDs u cJimina~ng compu~cr ~or due lo h~rdwJre f~ilurcs.
2.2.1.1 ~ot~a~c~aun toIcranc~
Both c~pcrimcnal Ant r~ lirc f~ult tok~n~ ~ystcms hvc begun to use design diversity to lole~te sohware fa~ts. Such sysLcms focus ~ngly on design r." ~s~ wDeJe the tcrm "dcsign~ a~compasses cvuylhing bom ~cm requ~emcn~s to rali~tion during both i~itial prodvcdon Jnd fulurc modifica~icns. Dcsgn liwlts ~c J ~ourcc of eomrnon-mode f~ilures, which dcfeal fault tolcrance ~ategies ~d on ~ria ~cplicatia~ and gcncsally havc c t~rophic ewlsogucnccs.
In a diversif cd dcsign, thc diftc~nl systuns pr~ced Srom s comrnon ~crvicc spccifi cstion re cslled vuiantS, ~ diversified design has at least two var~ants plus a decider.
which monilors the rcsults ot vanant c~ecution, given consistent initial condi~ions and .
~oP R-pon on F~u~ Td nnt Computino B

V~ ~4 ~ Impbrr~n~lbn ~n F~un Tobra-~:ompulor . _~ .

inpu~ ll~e comrnon spe~ific~Lion mu~ t~1icil1y ~d~ress Lhc dt~i5ion poinLs, Lh t IS~ il mus~ nAIc whcn U) m~c a~ci5iO Ls ~nld wh~l ddU~ ~o base Lhun on.
~c bcs)-d~cwnemed Icchniques for l~olen~ing sohware dcsign I~UILS re ~he rccovery b10c~ (RB) sppr~h nd N-vc~sion pr~grarnn~ng (NVP). In Ihe fir~ ~pproach, Lhc vui-~nLs ~rc calkd cltemales u~ Lhe kci ~r is n ~ L~nCC L~SL, which is ~Ued scquen-lo Lhe ~l~rnUcs' ~ LS. U Ihe ~ LS of Ihe prirna~ trnaLe do no~ ~Lisfy LhC
ance les~, the ~ondary ~l~Le c~lecules. Ln ~he socond pproach, lhe vananLs re called ve~s~orLs, and ~e dceider is a voLe bascd on uU versions' leSUlL
M~s~ of Ihc ral lire ~yslems donol implcment chhcr ~ r~covay blosl~ ~pproach or N-vcrsion prog~nming, bul rsLhcr are bssed on ~cU-choc~ing Sofiwuc ~vhich ConsisLs Or CiLh a Varial~l and an ~cccpLancc LcSI or two v runLs and a comparison algori~m.
In implcmcnlin~ dcsi~n divasity, ~re ~ two msin eonsidcrsLions. F~L is thc numbcr of varian~s. ~sidc from cconomic considcruions. Ihe numbcr Or vuiams tor a givcnaohware fJul~ ~Icrancc metbod is dixaly rcl-Lcd o thc numbcr orf ulLs Lo bc Loleral-cd. lbt: soh or olid n~turc of the Saflw~C f~UILS significJnlly ~ffecls thc uehiLeclurc only when it musl toluuc morc Ihan onc f~UIL ~Iso nOLC Ihl an rchilecLurc LoJcr~Ling a solid f~ull can Iso tolcraLc ~ infinile ~cqucnct of soh faulLs, lxrovidcd Lhcrc arc no ~UIL coi~cidalucs.
n c sccond considaaLion is Ihe Icvd of f~ull ~erance application. n c Icvel of ppli-alLion involvcs two qucsLions: How much shoult Lhc sys*cm bc docomposed into com-p OncnLs Lo bc divcrsi9cd? Ant which l~ycrs (applicsLion sonwart~ a~cemivt, hsrdwarc) musl bc divcrsiIicd? l-lc nswcr lo Lhe firsl qucsLion involvcs a tntc off bcLwccn LWo cpposing consideraLions: srn~er componenLs ~ow a bc~r masLering of Lhe decision algorithms, bul l~u COmponenLS id diversily. In ddiion, the decision pOinLS re "non-diversity" poin~s (and ~cb~i~ poinLs f N s~elf eheeking pro~amming.
NSCP nd NVP); s sueh, Lhcy must be bmited. Doeision points ar;e necessary only for inLersctions wiLh the environment ( ensor dala cql~isiion. delivering orders to aelua-lors, opentor inLuaCtion, eLe.). However, perfomunce consideraLions could prompl sd-diional eomp~miscs.
2.2.12 Hatdw rot-uKtobr~
Softwart faull toleranee primui1y deals wiLh fsults ~sl re ~en~ucd from rors insoftwsre design nd prognnuning. H rdw re f ull Lolu~ce dsds wiLn er,ms thal sredue to hrdware design or ei~dt emss. It un be achieved Lhrough r~dundaney in hard-wsre, sofiware, infonnation, uld/or canputaions. A ~ult-Lole~nee malcgy ineludesone or more of Lhe following demen~s:
n~r~u~ ~, ~arnc~orL By ct~mparing dsLa from ~edundam hardware ComponCnLs, erross ue de~eLed, Fullty dsta will bc dyruunically corrccLed. thc sys-tem must Iso be ~ble Lo prevenl Lbe propaguion of fauhy dala aeross Lhe defined boundarics.
s ~a~. rnis is s proeess in ~hieh the eompuLer SySLan idenLifies the fauhy hard-w re CornponenLs, (e,g. module. dala paLh, scaion an log~c board, eLC.).
sys~em is repaired eiLher by replaeing Lhe ~ailcd modulc wiLh a sparc or by reconfiguring Lhc sysLCm SlruCLurC or worlc load disLribuLiOn to CirCum-. . . _ AIOP R port on Feult Tobt-nt C~nputlno 9 7 ~

~rk~uY l~a cl ~V~ 7~n ~ F~u~ rO~pr~ rnplttsr he mod~lle. Thcre Are ~hol" r~ls~crr len~ ~nd cold rtpbcemcnL A hol spare a~curr-nlly putormi t~ ~une o~eruions as the moJult it is lo r~pl~e, n~in~ no ~iri~ Rtion ~h~n h is ~wiLch~d into ~e ~yslcm~ sparc is cilhcr nol powcrt~
or useJ for o~r tasl~s, requirin~ i~li ializalion wl~n swilched inlo Ihe ~rn.
~C~ W~ ~crc is ~ faull occurre~ in thc compu~r ~slrm, corrx ion nccds lo bc rrade lo r~s~ the syslcm to ~he ini~ e of continucd op~ion.
22.2 En~r Corr~îlon Co~ v8. Sp~lrg ~nd Pllllr ~hi5 scaion dcs~riba r~o diff~rcnl ap~7rwu:hcs in hardwsre fau11 Iclcrancc irnplcmcn-~tiolL
2.22.1 En~r c~rr~tbn K>d~
SophisLi~sd a l ~r~Cling codcs ue cornrnonplacc in bolh comrnucisl nd noncom-~ercial compul~3, prirr~1y cmplo~od in data ~orage and b~ckp2ane d~ transmission~or dc~ction ar,d conection. Dtpcnding on thc levd of L~uh proleaion, a numbcr Or chcc~ng bils u~ ~ddcd lo s dala wofd, or by~c. Onc commonly used ECC is the Ham-ming SEC (~ingle^crmr~roaion) code which ad&s ~t parity bits Lo an N ~ characler.
forming s ncw cha~cur of 16ngth N+K bi~ Thc bits ~e numbe~cd su~ u 1, r~ 0, wi~h bil I the Ichmasl biL ~11 bits ~vhose bil numbcr is a power of 2 ~e parity bits, and ~11 thc remaining bits ~ used for da~ Each pari~y bit checlcs r,~ccific bit positixs, nd the p~ity bi~ is ~ to Ihat the lotal number of 1 `s in Ihe checlced positions is even. Thc bil positions chccl ed by the psrity bils ~rc bi~ l~checksbits l,3,S,7,9,11 bh 2-checks bits 2.3~,7,10,11 bh 4~checks bits 4,5,6,7 bil 8- chccks bhs 8,9,10,11 BusA BusB
~ imple method fr~r finding thc inconcct IZ!~
bil is fi~st to co(nyuu 11 the panity bits in the rxeived cllaractcr. U all are conecl /P~or ~ Processor B
Iherew~snoerror,then ddupalJthem- /
COrreCI parity bits, counting I f bit 1,2 for bh 2, 4 for bil 4, and so on, Ihe resull- ~ ~a ing sum is the position of Ihe inu~ct biL ~ ~
\ CPV2 2.222 Sp r~nd-PJlrlmpbment tbn /Processor~P~cessorB\
lbe ~are~nd-pair implemenu~ion recorLs hard- /
w~e redundancy to achieve bull tolance. By cornparing critiul signals from two identical hard-~vare comyonenLs, th~ sysum de~cts e~ror rcmove faulty canponents fmm the system.
Fig 2 Str~tus Spare-and-Pair Figure 2 iDustlales a typicat implementadon of this schcme. In a StJams On-Line Con-tinuous Processing Sysum, all hardware comyoncnLs are mirrored meaning uhat two componenLs are identicat and caecuting the same function. In Figure 2, tbeTe arc two CPU boards, CPU 1 and CPU2.

uoP R port on FOun Tobnnt Compu~ln~l 19 c~lu~ --On a~ ard sherc cre Iwo id~nu~l indepcndan~ hs ~ch has proccessor ~vhich are ulkd A Sidc and B Si~ ch sidc rectivcs inpu~s ~r~m i~s own bus and ~rives hs ov.n bus. Each bus is win~d-OR otone half of c2ch board. On each boar4 Lhe h~o sidcs c~s~ntly comparc with u~ch olhcr. In Ihc uuc or a disagrec nenl Lhe rcd iighl i5 nlrned on cnd Ihe boar~ is removcd from Lhc ~s~crn. In such cvenl ~`he o~her board b~comcs Ihe sdc CPU board in Ihe syslan.

2.3 3::onoluslon Compulcr ~uh lolcrancc am b~ hcbievcd by hardw~rc w sofrwarc cdur~dancy. Hy comparing Lhc redundam execu~ions rors ~or f9UI15 ue ~eu:cLul. ln loday s indus-~y various rypes o~impltrrlcntalions c~ tach represcnuing ~ balancc bawecn pc~or-mancc l~ull bvcl pro~eaion cos~ uld Lhe c~mplcxily o~ dcsign irl cach syslcm.

.
LIOP Fl-porl on F~u~l Tobr~nl Computin~ 11 CH A PTER 3 'Ihree-~ay~ 5Fau~t lo~erant ~llS ~rcf~tecture Introduction Thc objective of this MQP is lo build a fauh-tolennt computer experimenl platrorm bascd on a three-wsy-~roing mechanism with future ~plicabiliry. By th~, I mcan the algorithm devised in this MQP shou~d be applicable ~o the more dvalwd processorsand computer syslcms ~rith biggeJ memory space. ~ s J resull~ a numbu of rc4uiJements are ~mposcd on the design of the three-~y-voing algo~itbm.
- One~ there shall be no performance pa~alty due to hull tole~ance.
a Two, the bus musl ha~e a minimum number of bits of signals and a minimal bus bandwidth.1 lbe microF~occer used in this pro,ect is a MC68000 wiLh 16 bil da~a,23-bil dd~ss, a ~nall number o c~ol signa~ d a Seqr~cncy of 8MHz If thc samc ~Iprithm uscd in this projocl is ~pplie~ on a D ad~nccd nicroproccssor wiLh ~ wider data and ~ddress bus u~t a higher froqua~cy, Ibe number of biLs on thc bos nd thc bus hndwidth ~ill be mulifold.
~hree, the syslcm must bc blc to tokr lc up to l~bits of dala eno~, 23-bils of ad-d~css error.
Tbis chapLcr is de~ted to the desi~n Or the th~ee-way-voting fu~lt tolerance bus archi-teuure for this projecL It consists of the following th~ee major secdons:
~ dcscripion of the Lhree-way-voting JugoriLhm;
u An overvicw of thc bus ~rchiLcctctrc n lbe opaations of the bus and the rcspcctiYc timing L~agram.

. .
1. n~e n~nbn of bi~ second wcd by ~ memory. ~tpUttoy~yl device. oT prDCCSsOn _ LIOP R-,oon on F~ull Tobr-nt CornpYtln~ 12 T:~ Dsio~opm~nt o~ A Thr~W~y--'D~ln j Al,~ornhm i~nd ~1~ 7hb~r~ltlc~
...., . . _ ~he implemen~ n o( fault toleran~ bus in~tace and Crror ch~clcing methanism on ~-in~ividu~l CPU boards wilh M68000 sp~cific~ ue n~ conLained in this chap~ efer lo Ch~er4 ~ r~4aivc ms~rials.

3.1 The Development of A Thre~W~y-Votin~i Al~iorlthm and tls Theoretical JusSlflcaSlon lbe compuleT ~s~em will havt nvo s~Le buses, u SysLcm Bus ~nd ~ Bus. The ~y~em bus which comains ~11 the da~ dd~ss. and control signal lines, connccts the CPU BKud wilh Ihe mcmory bo~rd. In a ~Iy Saull toleran~ cornputtr synem, sD signals ~re pso~cled by llardwue OT sofrwase rcdundancy. To pro~ct the compuleT system ngains~. f~ults on the sy~em ;us, this bus should be irnplement,ed with Tedun~cy. Bul.
d in Chap~ 1 and the introduction of Ihis ehapt~, the ob,~ctive of this MQP is lo design ~ three-way-vuung algorithm baw~n CPU's, und f the simplieity of the de-sign, Ihe systl:m bus w~l nol conuin fealures of r~dundaJucy and th~etere il d~s no provide bus faull tok~anee.
The ~ ;ng Bus is a bus th t eonneas thr~ idcntieal b~ie CPU boards. C ri~ signsls will be sem onto Ihis bus for eomparis4n and vo~ing. The dailod dcscription of the bus will be deall with in subscquem s~aior~ lllis soction c~plains the three-way-voing al-godth n which is implememeJ in the Vodn~ Bus.
3.1.1 Premlse ol tl~ de~l~n Baull loluance ean be achiwed by h~dware o~ sofiware rcdundancy. By eompar-ing the reduDdam c~oeutions, mrs ~ f~ultsl are deteeted. Inc~itsbly, Ihere ~e simple~ signals and/oq ~mple~ logic eireuisries, i~e. the COn~p~isDn device uld the resuhs of Ihe comparison whieh the device geners~ ~Ibeit the degroe of replica-tion varies unong differem comme~ial computer dcsigns. Ihe comp~ra~r is ~Iways u the single point of f~ure or unprolected dau pa~ ll is ~ablished that the reli-~bilily of the comparison circuitry is sufScicm p ovided th~ il has the minimal comple~dty. The three-way-vo~ing algorithm employed in this computer system is based on the prunise tha~ thc colT~rats re reli ble deviccs nd w~l nol be repli-c~ted 2 SpecificaDy, if signals from Board ~ nd B re compared on Board ~, they ~I not bc comparcd on Board B
3.12 The Thr~wdy~votlno Aloorlthm rnere re three bo rds, Bo~rd ~ B, ~nd C in ~ compu~ rys~em. ~U thrce boards sre mirrors of each ouher. Namally~, ~11 dgnals coming out of Ihe th~e boards re identi-cd. ~nèrefore, only one board is rcqurr6d to ~ive ~hc sysliem bus. The three boards in 1. Reter lo aupler One ~nd Chptcr X for Clp]~ia~ of Iheir dif~eren~ del~uuor~.
2. ~he r~uon for m Icin~ thi~ usumpwn ~ ~ premise of ~he design w~l be e~ph~ed rn fuD deLail rn ~e ne~u ~ecuon 3. Norm~y rnuns e ~i~ion in ~hich ~ rc r~ nDr. feull or f ilure in dle amlpum UOP R-porl on F.un Tohr n~ Computino 13 ~ ~loprn3nt n7 ~ Thr~Ws~y-Volln~ Al~rl~ n~ nG 7h~or-tlc~l ~ .. __ _ the ~cm wilJ be assiEncd ns mast~ tlave bo~rd O ar~l slavc board ~ rcspectivce ma~cr bwd drive5 ~C ~yS~ bus u~d ~e t~vo slavc boa~ds ~VC5 5 backups 'r~ble 3.1 L~ ~ ubi~ary assi~nrnent ormaster board ar~d slavc t~ s.

~bblt:3.1 Mssler and Sl~-e Bosrd AsslEnmenl In Figure 3~2, A is IhC nus~ a Boatd B nd Bo~ud C arc the sJ~vc boards.
Board ~ issues dgnals, u dittcren~ a~s with Ihe ud ot multiplc~dingl, ~o thc slavc( ~
boards, Board B nd Boa;rd C On Board B nnd ~ `r ~ `r C, cTe~onting signals arc comparcd u~d ~e I \ /1 1 \ /0 rcsult is brDadcasl ~o ~D Ihroe bo~\
In Ihe casc ota), whcn Boards ~, B, nnd C
eompa~d cqual, logic Icvcl "1-' is gcn~led s ~) b) he rcsu]l indicsting com~rison c~u~. Board A
w~l rtmain lo bc ~e rnaslu bo~
In rit~ion b), Ihcrc is ~ is bctwccn Board AC~) ~) (~) ~nd Bo~rd C E~ecause Boand A vld B ne tqual, \ T
thc voting m~ t idcntit~cs Bo~d C s ~t mi- \ /
nority bwd. Bou~d A rtmains ~o be Ihc mas~
board with no effecl on thc t~-eculions on bolh Board A and Bærd B, i.e., no re~ecution of the psevious cycle is requir~ Bo rd C will be C) d) disabled ~ftc7 ~e addi~ional miscompare.
F~S~ 3.2 lhree-v,a~ oting In rhu don c), miscompcr.e occurs bctwcen both Bd A ~d Bo rd B, nd Board A
nd Board C. This indic~cs thl Boud A is either a minity or ~ ~ee boards re un-equal. Unlil~e r,itualion b) where th~e brolten board is ~ot a m~er ba~, n re~ecuion of Ihe p~vious cycle is requircd n this case. ~fter Baud A ~ilu~ it duccted, Board B
becomes the masLer board Boud A loscs con~rol ov thc syslan bus. AD threc boards rc~eculc thc previous cycle. ~s indica~ed in sia~ation d), comparison wiD occur be-twecn Boa~ B nd Board C, Boud B nd Board A. I~ard A will be dlsabled af~cr a second comparison failure.

-1. See Chpl-s 3 ~ ~nplemcnu~ion ~pCCiSIcs.

. .
~IOP a.pO I on F~un Tobr~n~ Cornputln~ ~ 4 ~.s- ~ 3 ~
Arch'~uro o7 th~ ~y~tom ---~s ~e hve ~ocn, this nlg~ithm wll~ DCCdS r~)t ~t ot sigllals trom onc board lo bc triv-en ~n lo ~e bus nd ~e hVo individua1 slsve boards w~l comparc llut sct ot sign~ls fr~m Ihe nus er board v,~ilh hs own corresponding ti~nals lo~lly. The ~sum~lion m~dc hcre is Ih~ Ihe cornpantor on Ihe s~avc b~rds uc r,u~ic cnlly reL;ablc th l il d~ct na equirc rcdundancy. ln Ihis c~sc, Ihe si~ s ccnera~od on thc dsve bo~rds re nol com-~rcd by thc n~cr board conve~scly. F~ hu monc, by mulli;e~ding ~ddress, dals ~ndc~nlrol dgnals onlo onc bus, ~he number of bi~.s on ~c bus nc t~uly reduccd ~vilh the tr~de off or ir~se i~ b~ndwidth of tJ~e bus.
By u~ladin~ f~uhs in the sysl~m, Ihis n1gorithn ~110v.~s one eycle of tnnsiem em~.
Uhen n miscompare occ~r, the tailing board nee4 nol be disabld imrnedi~1y ~s bng es the masltr bo~rd is ~ pcrly locetcd.
Again, it is spparenl th~tl Ihe design nelies on the ~ccu~cy Or Ihe compantor lo~ie.
Jhcrefore, ComparalOr bgie eircoitry must have the Ieasl comple~tity.

3.2 Bus Arc~lltecture of ~he Systam .
As mentioned betore, ~herc ue two disc~eLe buscs in the ~em. One is the ~cm bus which eonne~ts dl logicDI boards2 in the ~yslcm. Ihe olhcr irS thc Voting Bos s de sc~ibcd in the previous stion. l lis section provides an ovcrview of lhe bLts uehitee-~re.
32.1 Vo~lng f~us Archtlecture lbe micropr~cssor used in Ihc design it ~ Mo~ln 68000 mic~w~5. Il hs 16-bit dala bus (D0-DlS), 23-bil ~ddress bus ~Al-A23 nd A0 inten~ o the micJoproccs-sor), ~hrec bil funcdon code, nd other con~ol signals~. A bus eycle consists of eighl states. The v~rioos signals sre sscncd durin~ s~ccific sl tes of ~ sead cycle or write cy-cle. Utilizing Ihis fealure, differenl signals can be multiple~ed on the same bus and cap olred at a differenl tirnc. Therefone, thc maximum ~uunber of bits r~quired in thc d-csign will be 23 bits.
Following is a description o eompare signals on the Voting BUS.
SLV COMP DA.TA[0..22] lhis is Ihe d~ drivcn onto the Voting BLIS by the currem mas~crboard. Aficr the da~a ue driven onto the Vodng BOS, they ~ be eompared with Ihe ~sresponding k~l signals by IWO sl ve bosrds Tbe ~ t~f Ihe eomparwn w~l bc b~oadusted onto the Voting Bus.

-L l~ nwnber ot Ir~iau ror ~llowod i~ decidcd ~rb~ily.
S-v~rl rod~uhnl bD~ds ~hich p~rorm the ~ne fimclion, e, eoL~~ive~ c~Ded atc loBic~
~rd. ~ e~nple. the thsoc idauic l CPU bo~ rn thir ry~n re collectivdy c~Ded ODC logi-c~ CPU bo~r~
3. For da ilt of the rnicropn~ or. rc~ to MC68000 Mic~o~ocesr~T UKr'~ Mu~
_ _ .
~oP Ropor~ on F-un ~robr~n~ Computlno 15 q ~ ~ ,t, ~nJs ~ Dr~n~ .t:~ In~rrr~t~oT C~isTw~
- , llrchlt4~r- o~ S~t~m SLV MISCOI U L~O:I] ~ w~ s re drivcn by Ihe two slave boards. ~ecordin~
Ic the VDJue ot Ihcse two bi~s eDch indivjdual bo~rd d x:ide; the ~ction of neu eyele.l A_I~DEX10_11 B_l~DEXIO..lJ lC IhDEX[O_~]Thiistwobi~indieJtoridenti~esthe board in a ~nicular S]ol in ~e sysu:m. Esch bh is ~ulled low or ~ulled high ~eording ollowing schemc PhysicaDy ~ùs indie~tor iis loc~Led u the Syslem Bus.
~DEX A
O ___ __ ~
,. O .._ C
_ _ Nol I llid Tsble:3.2 Index 10:1]
3.2,2 System Bus Archltecture ~he sys~em bus will have 24-bit ~ddress bus. 16-bil dau bus. and a eon~ol logie bus Ihat eonsisLs ~ number of signals BUS ~DR10.23] rhis is a ~4-bi) sddress bus line that earries ~he address issued by ~he CPU Board.
BUS D~TAlO_lSl l-fis is 9 bidireetiDnsl l~bit data ~us whieh eonnens the CPV
board snd the memory board. When there is ~ 8-bil wri~.e or r~ad dau wiU be sligned lo bilO.
FCIO..I] This 2-bit function eode i5 issued by the ~U board which spccifies a 16-bit or B-bil ~peralion.

Opaation O O 16 bil Op 01 Lowtr ~bil Op_ 10 Upper 8-bh Op . . Not Vscd Tsble:33 FC[1:0]

1. Scc nc~ ~ccaon ~or deuil Or ~uion.

-~QP ~,oort on F.un ~bnnt CompuUno 16 ~r~tbn ~n~ ss~l~rTlmlno o~ram FC121 This iis s~ifics ~ r~d or wniLc ~per~iion.

FC12~ OP
_~_ I _ Re;~
O 'Wri~e Table:3.4 FC12]
hat i~ is ~or sim~licity of ~bul~ ~ F~10..1] and FC`12] are r~cd. ln bus opcra~ion, tilty arc assal~ al l~he same ~nc. By us~ F~10..2], no bus ope~tion, 8-bh rçad/wri~, and l~bi~ ~d,/wri~ can be dcfined 3.3 Bus Operatlon snd Staterrimln~ Dla0ram ~ . . .
'Jhis ~CIion describes Ihe opcra~ion of ~he bus in Ibo~ rading nrd ~Tidn~ wi~ or wilh-oul errors~.
3.3.1 Sta~e dla~ram o~ ul~ T~rance ~p~r~lon ln Figure 23, UhC over-all ~a~ disgram of ~e faull toleranl compuler syslem is presenl-c~ Following is a panial flow of s~ale ~nsi~ix.
8- glr, Bo~sdA 1- t~- t5ST, BocrdB lc SI,0, BocrdC i~ Sll;
P-sSorm Bis~ Comp~r-c-~e 11,11 no cbcnge~
~1, 0~ Bocrd El l-t m--eomp-r- 1- r-gict-r-d Begin p-rform S-cond ComF~re c~ 1,l Bo~rd B i~ di~ d 1, lj Coto ln~tl-l etot-~0,1~ Bo~rd B lr 0 1~ Bo-rd C l-t micconlp I û, l Bo~rd C i c t~ST, Bocrd A l-t mireomp endc ce B-gln perfor~ Sl~rd eomp~r t~, ll Bo-rd C l-t ~-conp-re i- r-gl-tor-d ~-si~l-r to /' c~
[0,0l Bo~rd B iJ }IST /- no ~-eomp r~ r-oi-t-s-d B oin S-cond Cong\~r-. . .

1. ~or mo~t da~iled u~fosm don on Ihc rc d ~d wr,l~ o~ion of ~,c M680uO. p]~ rer lo ~e MC68000 Vscs'r m nu~l.

_ UQP R pon on F~un Tobrcn~ CompuUn~ 1?

. .

bn Dna s~At~ mlng 5~Dr~m ~oa~on Orslalcs ue ~din~ lo the rollowing I;chemc:
Tx: X is Ibe l7as~r board Al thc p~ aLe, no miscompate ~ regis~te4.
Tx~ X is ~he rnasL~t b~ Isl miscomp~re is r~gis~cd ~or Bo~d Y.
Txy X is ~h~ rn~sLct k~rd. 2nd mis~ompare rot Boar~ Y ~nd is rcmovcd ~tom the sys~m.

=;~TB~) SLV-~SCOMPIO]
~ ~ ---................... ~LV ~SCOMP~ I ]
Fia 2. 3 Fstult Tolerant System St9te Diagram U~P R~pon on F~uH Tol~nnt Co npusln~ 18 nd ~ c ~r~m .2 nmln~ Di~ram 10r Yotln~ Bu~ Op~r~bn maslu k~ ~ri~ i~s ~d~ess nnd ~au~ ,b~ng Bus COMP_D~A
lir~s a~ two diflcrenl liJnt. ~ uu CPU b~ ut res~nslblc lo gen~ale 2 sel of sig-nals Ihal sr~nldinS tl~ e counl on each lo~l bo~r~
A~ Ihc nsin~ edge Or S'rA~EI, Addrtss is ~did on tht U~lin~ ~. Tl e ~avc bs~ds ~h~uld r~gisu r Ihe ~ss r t ~ing Bus u uhis ~sne.
Al lht rising edge of STA~E2, Da~ ~s ~;d wl ~ Bus. ~he sbvc bo~rds should register d~e da~a si6r~als from d~e VoLing Bus ~ ~his ~mt.
Al the ~isin~ ed8c of S?A~E3, all dsu ar~ v~L;d fo7 comparison on slsvc boards. SLV
MISC5~ O] ~ pro~uced as Ihe 7~sull of ~bt co~npanson.
At Ihc rising edBe ot STA~, all Ihree boards resolve Ihe slale (being ~he MST, or SLO
or SLI~ of nc~l cycle.

8M}~z ==
COMP_DAT~22:0] --STAT~I
STAlE2 SWE3 ' STATE2 is suppresscd in Read ycle. l~je reison is ~ here is no necd ~D
compare dala t Read cycles. ' ', ~io ~ . 4 rlmin~ diag~m for Votine Bus Oper~tion Al ~his poinl, none of thc specifics of ~e CPU boud dcsign rclalin~ to M68000 is dis-cusscd. Bw in mind ~ Ihc SlaLes rcfc~d in F~cr 2A rc nol thc s~mc s ~e Slatcs for Ihc M68000 Read/Wrilc op~dons. Umil now, thc fcbon li~ of thc M68000 mi croproccssor is nol discusscd For tnc s ke of simplici~, Ihc ~es fQr thc Vodng Bus opcradon rc nol rumed n rcfc~enccd lo Ihc microproc~s~.
In a~apw 3, ~t implemcnlation Or Jn~crfacc lo thc Voling Bus and the emlr dcaionmechanism is discussed.

~IOP Ruporl ~r~ F-ult Tr~r nl CompulJn~l 19 CHAPTER 4 ~ s Inte~ace ~
rror ~ete~ ec~nism Introduction The Vonng Bus protocol and iming are iDusuatcd in Cha~er 2. In Ihis chap~, ~he )ogic ci~uh Dn she CPV board which inlerfaces wilh the Voting Bus is desi,6ned in accw-dance to thal pr,olocol.
The following three rnajw seclions sre cont~ined in this chapter A dis~ription on the read and wri~e ~cral~on o~ Ihe Mo8000 mic1eprocessw. The rcason il is included in this chapw and p~cccds the s~ction des~nbin~ the votingbus intsr&lce is that the dming of the rnic~ccs~r dectates the design the ~ollow-ing ~wo sections of the logic design.
The Voint Bus In~ce. The throe-way voting Igorithm descnbed in Ibe previous chapw is implemented by dis~ributing the voing decision maldng on all tbree boards. The tbroe CPU boards monitor ~he two dave misoompare signals on the Vot ing Bus nd deride wbal funclional rolel it should ssume for the ne~u eyele.
The error dc~ction mechan~n. This ~ocLion Or Ihe bo~trd conuins bgie CiTCUU tha eompare two scts Or dau nd gener,alc a miscompare sigNl Depending on the func tional ole of Ibe board, the rniscomp~e signal drives one of the two ~ave miscom-psre bits.

. Flmcaonel role ~nd function~l bo rd hv~ inu~rch~ngcable mewng in this TepOTt. A boaTd's ion~ a ~.~vey.voùngfullltoleT ntsys~ is ~;th~T M slts,Sl~veO OJSbvel.

~IQP R pon on Faull Toi-nn~ CompuUn~ 20 n4 Wrh- O~r-~bn o~ 1~6~0 4.- The ~ead and YUrl1e Oper~ttDn o1 M68000 . . _ .
Du~g ~ ~d cyck, thc a~J bc~ud ~civcs onc or t~o by~cs ot dala trom the rncmc-y board via Ihc syslem. If the ir~r~c~ion ~ccifics ~ word long-word Operalin, thc MC68000 p~ssor re~ bo~h the upper ~nd lowcr bylcs simultancously by csscning both uppcr ~nd lower dsu str~bes (IDS, UDS). U'hen thc ins~ruetjon ~ccifies byl~ o~
eration, the proccssar uscs ~hc inlc~al J~O bil lo dc~nninc which byte to read ~nd issucs thc a~propnatc da d ~bc. ~en ~0 aquals zu~, the upper d~a strobc is issued; when/~0 equals one, the low dau s~c~be is issuc~ Whu Ihc ~au is reccivcd, the processor in~nally positions Ihc bytc 9ppr~ t~ly. BUS_~DRS[O] is de~mined by the value lo LDS nd UD5. F~lO. ?] re de~ni~tI by Ihc v~lue of LDS, ~JDS, cnd R/W_.
In a wrilc cyclc, thc processor scn~s by Les of da~ lO lI c rDcrnory. It' thc insrruaion spec-ilies ~ word ope~, Ihc p~r issucs boLh UDS_ and LDS_ nd writes bo~h bytes.
Whcn Ihe instruClion r.~ if ics ~ byte opcsation, Ihc proccssor uscs thc rnterrul AO bil lo d~rminc which bylc lo wri~e ~nd issues thc ~propriatc dala urobc. When thc AO cqual 2ero, UDS_ is assened; whcn ~0 bil oquals one, LDS_ is ssscr~.
SO Sl S2 S3 S4 SS S6 S7 SO Sl S2 S3 S~ SS S6 S7 8 MHz QK

AS_L
UDS_,LDS_ ~ ~=~ _ RrW~
DTACr_L ' D[O~
~ . X
Read, Writc Figure:4.7 Timin~ di~ m for Re~d ~nd followed b~ ~ Write 4.1.1 Re~d Cyck ~0 Ihc rad cyck stans in 5~t~C 0 (S0). ~ne yrocesso~ drives R~W_ high lo identify a tead cycle.
STAlE 1 Emering ~te I tS 1), the ~oce~or drivcs a valid ddress on thc addrcss bus on CPU b~rd.
- STATE 2 On ~he nsing edge ot ~ue 2 ~S2), the ~occssor ~s AS_L and 1.1DS_, LDS_. During S2, FC[0_2J are la~ ed lo the system bus.
~ Dlrring ~tlC 3 (S3), Da1a are r~ed fmm the memory board. Fhm ~he rising edge of S3 liu tne faUing edge of S6.
~ During state 4 (S4), the processor wsi~s for a cycle terminalion signal (DTACK_ LERR_) or VPA_(lhis is nol used in this system). If neithe~ termina-~OP R~pnn on F~ull Tobr nt Compulln~ 21 otlng Bus hl~na~ _ ~ { ~

ti~n si~Lal is as~er~ berorc Ihc f~lling ~dge ~ ~e end ot S4, Lhc ,L7rOCessot inscrL~ail s~aLes ttUO cbc~ cyclçs) un~ll eilher DTACK_ or BERR_ is assened.
~ During t,U~ 5 (S5), no bus signa3s ue nlLeted STA~ During s~ale 6 (S6), dala t-~n Lhe devicc is driverl c~n~o Ihc dala bus.
n LhC taIling cdge ot ~- clock 6nu~rinS ~e 7 (S7), Ihe p~cessot la~hes da~ tJom Lhe ~ddrcsscd device ~md nagales f~S_ nnd UDS_. LDS_. FC10..2] ~re driven ~D high impe~ance Lale. All Ihe rising c~ge of S7, ~he ptOCCS50t pbccs LhC ~Id dress bus in ~he high-impedance s~a(e. rne p~cessc>r places Lhc address bus in Lhe high-impctlance 1a~e u Lhe tisillg edge of S7. nne device nega~s DT~CIC_ u Lhis~im~.
4.1.2 Write Cyck ~Q Tne wrile eyck su~,s in S0. The processor drivcs RJW_ high (it a preced-ing writc cyele has kfl R/W_ low).
~1 En~ing Sl Ihe pro~cssor ~rives a ~lid ddrcss on Lhe o~dress bus.
~IQ~ On Ihe nsing cdge of S2 Ihe psocessor Dssa ls AS_ and drivcs R/W_ lou .
a ~ During S3 Lhc daLa bus on Lhe CPU board is driven out of thc high-irnped-~nce siale as Lhe data to be writLtn is placcd on Lhe bus.
STATE4 Al Lhe tising edge of S4 Ihc pt~ccssor sscns UDS_ LDS_. The processot waiLs for ~ eyclc tcnninaLion sign~l (I)T~ or BERR~ or VPA_ which is nol uscd in Lhis design. If nciLher LCtminaLion signal is asser~ befotc Lhc falling edge at UhC cnd of S4 Lhe process~ inscrLs wah s~aLcs (full elock eyclcs) until ciLhcr DT~CK_ or BERR_ is ass~cd.
STATE S During SS no bus signals re altcr~d.
STAlE6 Duting S6 no bus signa~s uc nlLcrcd.
ST~TE7 On Lhe falling edge of Lhe eloclt enLCtinE S7 Lhe pt~essor negaLes AS_.UDS_ LDS_. FC10.2] is put in high-irnpedancc Sultc by tbe CPV boart. As the elock nscs at ~hc end of S î the ptocessot placu tbc addrcss and d~ta buscs in the high-impe~ancc s~e snd drivcs RlW_ hi~b. Thc dcviec negates DTACK_ at tbis ~me.

4.2 The Votln~ Bus Interhce Inslead of baving on~ eentral deviec wbich pvarns thc ~latcs of liuee CPU bMtds. the statc machine des~n~ in Fgltrc 2.3 is b~Jtcn down lo tbtec idcntical sub-slatc ma-chines prog~ned in tbrx PALsl which re dt~ ed on each CPU board. Tbere are fout signals that tbc dcvice monilDrs BRD_INDEXll:0] and SLV_MISCOMPl[l:0~. The outpul of Ibe device uc only used in Ihat individual board. Tbe ou~put siEnals are IM_MST D~.S_SL0.

1. P,~. ii ~ cl~nsnon ~b~vi~ion ~ bl- Logic ~sr-y.
~IOP R p~rt on F-ull 'robrnnt Cornpu~in~ 22 ~ oUn~ BL~ hl~
_ _ . .. .

42.1 Sl~ ~lno ~o Vo~lna St~t~ 14acllln~
ID~ Ol lbcsc two signals uc fr~rn t~ btm~ ~lus. ~n Slot ~., B, und C, Ihesc ~gnals uc ticJ lo Ihc ~_BRD_II~ l 0~, B_BRD_INDEXll:O], Ind C_BRD_~DEX~I:O] ~spcc~ive~y.
QV MlSCOMP Lll:Ol n~esc two ~ugnals u~ ob~ ed ~om Ihc ~ting Bus. Thc val-llC of ~hest two 6ignals at~ idcmical an all th~c boards u ny timc.
~M_MSJ Whal this signal ~ logic bigh, it i~ndic~tes thnt the board is the Masta Board ot ~hc ~ys~m.
IM_SLO Whcn this cignsl i~ bgic high, i~ icl Cs t~ut the bo~rd is the Sl-vc O E~oard of the ~an.
IM_SLI Wb~n this tignal is logic high, it in~leS tb~t the bo ud is the Slave 1 Board o~ the syslcm. Fw the simplicily of the PAL ~rognunmint, this signcl is not ~ OUtpUl of the slalç machine. Bcc~use the outputs of the PAL used hn Ihe design re regislcrcd oul-puls, h is valid one cloclt dcl~ ~f~ the IM_SL0 nd IM_MST re v~i~
BUS ADRS ~L~D At the rising edge of ~is si~l, the nddress on the voting bus is valid nd should be regis~cd on~o !he bal CPV board The ~ddress is Yslid dll the falling edge of Ihis signal.

s~ 3 w~ w2 s~ s5 s6 s7 8 MHz J J
COMP_DATA122:03 i , ADRS1~2:0] ~ . . i -DATA[lS:0~ __ AS_L
BUS_ADRS_VALID
BVSpATA_VALlD ~--COMP_CYC ! t. _ SLV_MISCOMP_LII o] ~ ~
IM_SL0 ~ ~ ~--=
- ~_SLl BVS_D~_V~LlD r~m bw ~ d~s.
Figure:4.8 rlmin~ di9~r9m ror Votin~ Bus Intfsce . . . _ IIOP R-p~ on F.un Tobr nt CornpuUno 23 Yotln~ Bu~ ht-tbc~

BUS DATA VALID Al the n~in~ edge of ~his ~i~al, uhc da~ on hc v~jng bus is va~-id and should k registc~ onlo thc k~al CPU ~1 Ihc data is v~lid till thc r~lling~gc o~ this signal.
COMP CYC ~t Ihc rising edge of this si~nal, SLV_MISCO~[I:O~ is valid on thc Voting Bus u~d s~ould bc rcgis~l on the local board. Al Ihis dme, a signal indit~ling a bus~r requiring a prcvious tyt:lc r~clccution 9 dala-acblo~vlcdge should bc is-~ut.
42.2 St8to dla~r~m A
~ ~0 L,~

S~alc namc: ~~ x~OI
ll~SI: MaslCr, SLalel \
lM2: Maslcr, S~atc2 \
l'SOl:SlavcO Sla~el \
TSO': SlavcO Sta~c2 \
TSll:Slavel S~alcl ~12: Slavcl Sta~e2 \ OOOO /
~000\~~ ~/ 010~ 0 ~01 ( Reset SLV_~SCOMPI0]
SLv_MSCOM
~ MSr~Dtl~]

Figure:3.9 Stste t~agrsm ror Distribuled Votin~ Me~hsnism ~IOP R-por~ on F.un Tobrnn~ Compullno 24 T~- ~tlng ~u~ ht~

A numbc~ o( rea~3re uc emph~sizc be~o~e n ~o~a~ional te~e. An~ board c~n be eilh M~er~ Slave O ~ Sl~ve I depending on ~e pr~sen~ SUIC and d~e value ot Ihe SLV_MISCOMP_L[l:~. Tbe pallern is ilJusual- s ~ollo~vcd ~ogln I ~ Sl-v-O
I un o ~ no ch~ng--- 11,0] I m b d ~or th- Sir~t tlme ~gln g-t n-cond -t oS ml~con4nr~
C-D- l~t,O~ I'm ~d for tb- 2nd tlme, r-mov-l~,l] I-t b-d s-cord r--- I'~ tlll Sl-veO
rdc---P-gln got n~xt mi-comp r-. . .
ID,O~ I m b d, o 1~ tbe otb-r bo-rd I ~com- ll~nter Ihgln S-cond Co~r-c--- 10,01 I'm b-d for tb- 2nd tlm, r-mov-5 lO,l~ I gr-- Ylth on- or botb l-v-, I m n~nt-r B gln g-t n-xt 0~comp r-lS 10, O] th-n Both lv di~gr--I ~com 51-~no cbang-Live-lnscnion Avulsbility. When a board is rcmovcd tTorn ~he system, the system is ino longer pracc~cd against ~aulL To recover tault lo1crance o~ ~he syslun. third board musl be ~ought on-line synchrwous with thc ouher ullrce boards. Following is a scheme by which thc Dismbwed Voting Machine dccidcs which funetion board i should bc.
Ebgin g-t ~c~re c-~e 10, 01 b gin lnltl~l lnJtlmnt /~lt i~ lmpo--ible to /-b-v- 1,l ~b~n tb-re /~ar- 1-~ tb n tbr--/-bonrd~ ln tb- ~y~t-m c--- Slot~ t-r SlotB Sl-v-O
SlotC Sl-v-l ndc-~e l,OI g~to Sl-v-O
0,1~ goto Sl-v-l jl,l~ ln~oscibl- ca~ h~n SLV ~ISCOMP L is not /-drlv-n by n cl-v-, lt is /-pull lo~
ndca s~
. . .

~oP Raport on F-un Tobr-nt Computlno 25 ~ 'vot~ ull ht~
____ _ _ _ _~ _ __ _ _ __ Sta~ m~:hinc ~aul~ Lec~ion. In ~e cvenl ~tl PAL has error, Ihe syst~rl is panially p~clcd.
o Un-rasovcrable ~ophy. Sincr Lhe vo~ing mechsnism is dismbuLed onlo ~e boards, the system ts protec~d from PAL e~ror wiLh one e~c~Lion. Thal is when Iwo boards assume Ihe role Or ~he Mt~er and one asstlmcs thc rolc or Sbvel, ~erc is only onc sbvc tmd ir ~cre is a miscomp~trc on ~hal d~vc, SLV_MISCOMP_L[ l :O~ i510,0]- Conscqt~ntly, boLh m~stcr bccomes Slavcl and Ihe (onn Slavel becomes SlaveO. rnc s~ond compa~ will undoublly Cul bccause Iherc is no mas~r board to drivc LhC d~tLa onto Lht ~btin~ B~. All thre,e boards vviU be t~oved from Lhe sys-tem. This is what is commonly ~RIIcd doublc error.

~IOP Ft port on Fault Toh nt Computino 26 C~IA~T t~s 5 System ~chitec~ ~e an~oa~
.~un~ cn~n Intr~ducti~n In the hst two chaptcrs, the primary ocus has bc~n placed on thc &wlt iolcrant aspcct otthe computcr system. Chaptcs Two c~cplains thc devclo,sJmcnl u~.d dcsign of the three-way-votin6 faull ~olc~ mcchanism. Ils implemenution is descnb6d in a more detailed ~ashion in ChapLes lhrce in which thc ~ling Btts, Voling Bus inLesfacc ~nd ~heu opcsa-tions were eovcred. In this ehap~es, Ihc comput s sys~cm is pscscnLcd m its toLality.
The chaptcs is organi~ed in thrcl main scctions caeh focuses on one major componcm of thc compuLes system.
Systcm Architccwsc. In Lhis secaon, thc ovcrall compu~u sysLem architccturc i5 pre-scnts~ The Systcm Bus by which boards in the CompuLCs sys~m inLeJact WiLh one ano~hcr is illus~
CPU Architcaure. Thc CPU board cos lains tbree maJos ~ccdorts, Ihe micsoprocessor and its in~erfacc, bcal mcmory, Voting BLIS inLer~cc ~nt Syslcm Bus in~uface.
Mcmory Boasd ~hi~c. HhhesLo, thc mt~nosy board hs r ot been discusscd. In Ihis chapLer, a complcLe dcsign of Ihe mcmosy boaJd ~ill be presen~ed, irlcluding the bloclt dia~ram, the iming d~m, thc systcm bus intesface u~d a spa~al sccLion on the boasd which links the compu~er systcm ~o a vcdio tesmhal.

IIIOP R~port on Fault Tobr~nl computin~ 27 ~nv~nclbh Cornput~r Sy~l~n C~>nli~Jr3tbn ~ ,3 5.1 Invlnclble Compuler Sys~em Con71~ur~tlon ....
In the Invinciblc Comr1u~r Sys~m, lhere 3~ two logical boardsl, 1hC CPU ~nd the mem-ory. As il is ~ scn~ m Fi~5 . lQ th~ src 11uee CIU boards c~3ch dcsi~na~ed as Bo~ Board B ~d Board C w~lich luc iden~ica3 ~o c~lch other. Each one ot Ihese ~ree b~ds pcr~orms idx~ t~ctions dlrLn~ ~ad cyck. On thc olhcr hand, dunng wri~e cyck, only ~e rnaslcr board h~ s io Ihe SyslCrn u~d ~bung Bus.
The munory b~ard is a simpl~ lo~ic b~ It int~racss wilh 3~hc CPU ~ he Sys~cm Bus. Includc4 in Ihc munory board is a ~cct,ion which utilizcs a Univ~l Asynchronous R3~ceivcrtr~ansDeiYer (UART~ lo {n3-~aoc ~vith ~ Vl02 ~L
rc~ ~

Syst-m_Bus Figure:S.10 In-~lincible System Configur~tion 5.1.~ Sys1em Bus ~lonal descrlptlon ~ddress Bus (SYS A.DRS d~ ) lhis 23 bit~ unidircclional, threc-stalc bus is capa-blc of addrcssing up lo l6 Mby~cs of data lt ~o idcs thc ~ddrcss for bus opa7 tion dur-ing r~ad and wmc cydcs of ~hc mcmory ba~rt.
Dsts Bus (SYS_D~ 0..lS~) 7his bidirectional, ~uec-~stc bus is ~hc gcncrdl-pur-posc dala pah ll is 16 bils widc ~nd c~n ~nsfu dala of ~vidlh 8 bhs and 16 bils.
Syncbronous Control (SYS CODE~, SYS CODE2, SYS CODE3, SYS CODE4) Thcsc thrcc signals ~ alid or~y d~ing sys~m bus cyclcs Thcy ~ ridc thc tuning conrrol foT mcrnory bollrd during ~at nd wme cycbs. ~cn SYS_CODEl is low, ~ialidddrcss is plucd on Ihe Sys cm Bus by thc CPU boa~s~ whsn SYS_CODE2 is hw, the mcmo~y board miay rc~islCr thal ddrcss onto thc memory board. Whcn SYS_CODE3 is 1. Se~enl red~d~n~ bo rdsi whjch p(orrn ~e ume funciorl are coDeaively c~llcd one lo~icd boiu~

UOP ~-port on F~ull Tobnnt Co npu~ino 28 ~ r~bb ;~mput~r SYQt~m COnr1~Ur~ n ~, bw, during bus rcad cyck, ~c memt~ b~ l~uld pl~t~ tlata on Syslem Bus, ~rise ~isa, dunng bu5 wnle cyclc, Ihc CPI J board sht>uld plaec da~ on Ihe sys~n bus. When SYS_CODE4 is bw, daul on Ihc Sys~cm BUS shO~d be rcgis~crc4 on Ihc memory board or ~c CPU board during syst~m ~ril~ arld rcad cyck respbctivcly. SYS_CODE1 goinghigh in~ic~tes ~he ~d or ~ bus c~clc.
Srstem Funct;on Code (FC .c0~) n~ bi~ func~on code issued by Ihc CPV board ~vhich spccifies ~ bi~ ~ 8-bi~ read or ~vri~c opc~ation.
__ _ eration _ _ 16 bi~ vmLc 1 IJpp ~bil wri~c ¦
010 _ Lowu8 ~b wnlt m 1 Nol Uscd 100 10-bit re~d 10~ Uypa 8-bilread ¦
110 _ Lowcr 8-bit read 111 Not U~
Tsble:~ Functlon Code S~stem Memor~ Intcrfscc CODtnOI (SYS MEM SEL L) Durin~ systcm bus cycles, Ihc mcmory bclard scnds oul a dgnal Lo ~he CPU board indicating thal thc mcmory has becn selecLcd by assating Ihis sigrlal low.
Cbck (SYS CLJi) lbis is a 8 ~z cla:k which i5 gcncraled from a 8 MHz clock card moun~ed on ~hc sys~em bus bac!c p~ncl.
Bosrd Indi~tor (SYS MSTR INDEX c0_1>) lbis 2-bh sigrlal delermincs thc loca-tion of thc CPU board. By pulling thesc bils high or low ir ~ panicular slol on thc back pand, CPU boards irlserLcd in these slas arc ~ssigncd as Board ~, Board B, nd Board dat ~0..1> BRD

O~ ~

I I Nol Used Table:~.7 BoardIndex UOP R-,ODrt on F-ul~ Tol~rrnt C omputing 29 h C~rnp~n~ S~ m C4n~1~aur~sn ~ ar,~
. .

i.l.2 Sys1em ~u~ Tlmln~
In the roDowin~ ~gurcs, rcad u~d ~riu opc~ ons on ~hc Sys~crn Busi uc illus~.
SO S~ S2 S3 __ S4 SS S6 S7 SO Sl S2 a.oc~ IJ ~1=
BUF~ L ~__ SYS_CODI ~-- _ SYS_CODE2 ~ _ SYS_CODE3 SYS_CODE4 ~ r--_~
SYS~RS ~--SYS_DATA _C ~ ~ ~
Ad~ess is rcgislaed ~n~ Ihc mcmor~ board.
on~othcmcmorybo~rd.
Figure~.ll System Bus W nt~Cycle SO Sl S2 S3 ~I W2 S4 SS 56 S~ SO Sl S2 CLOCK ~ I ~ ~IJ=
BUF_AS_L- ~~~~~~~~~C ~___ SYS_CODEI ~-~~----~~~~~__ _ SYS_CODE2 . ~
SYS_CODE3 r~~
SYS_OODE4 , r-~ ~ __ _~ _~---- _~ __ SYS_~DRS ~
C2> ,Cc _ SYS_DATA f { ~
~dd~ rcg~u~td rht~r~o~d 7~ gn~o~ye~sui~ ~cCPVboad.
F~gure~.12 Sydem Bus R~td Cycle IIQP R~por~ on F.un Tobnnt Computln~ 30 08-d L~ bn 5.2 CPU 8~ard Leve1 Desl~n ~s i~dic~cd in Figurc 5.13, ~hc CPU bosrd inclL~cs ~cur m9jot scclions, Ihc micropro-c~ssor an~ ssocia~d logic Ci~CNi~, ~e local mcmory. and Syslcm Bus in~ ~e hnd ~in6 Bus in~cc ~hich in runn includes ~ u~ oJ' comparuvri nd a voUn~ ma-c.
... "' ~

68000 ~ _ tur d~ = ~ SysricmBus t~u __ ~ _ ~ _ l _ A ~ 1 1~ ~ r ~ = ~
~;~1 -- ~1 2~ 1 4~ ~ungB ~s Fgure:S.13 CPU Block Diagrsm S2.1 Local memory Thc îocal mcmory on thc CPU board is a 321C~8 Xicor l~ctrially E~sablc Prom ~~om). Tbe mcmory is di~tly mapped to Ihe CPU mcmo y space in word s~c and i~ is addresscd by 14 addrcss lincs. Thc LOW_BY~E_DTSl kwmincs Ihc higb by~e or low bylc be ~ad or wriùcn lO. Only 8-bil opcra~ior~s arc allowed whcn ~cessing Ihe EEprom. rnc addrcss sp~tce of ~c local mcmoJy is 000000,~, ~ 007FFF,0~.
EEprom wri c cycles arc comrolled by cbip selecl (CS_L) bits on the EFprom.

UOP R porl on Rwn Tobnn~ Compv~)no 31 ~ = . . ~ .

6.2.2 ~y~4m Bu~ Ir~rtæc~
ut Ihr~ sels Or signals bcin~ sent ~ ~e Sysscm Bus, nddress lines, dau lines.
;Dn~ol si~7lals. Tt~ Sys~m Blls inlufacc gn ~hc CPU board is rcsponsiblc ror con-~rOI]iJlg the "g81C-' bCIW~I lhC SigTIals C~\ ~hC board ~nd ~c bus lines. All SigT als ~rc l~nchcd on~ dx bus on~y ir, one, ~e board is ~e m~r b~rd in Ihc ~ysum, two, il is ~ssing Ihc ysltm n~em~ spacel. Oshcrwise, ~e guc ~AlI bc hcld u ~i-s~u.
S.2.3 Error de1ectlon mechanl~;m 3nd Yotlng BU~ QrbC
~11 Ih~c C~U boards in IhC compLTItr tys~cm lalchcs thc COM_DA~A signals fJom ~hc bting Bus. Tl~e d~nals are cornpar~d whh thc addrcss nd da~a sigr~ls on uch in-dividual board nt appr~ tc imc. The cquality of thc 5ignals is rcso1ved bcfore ~he COM_CYC during Ivhich eycie onc Ihe n~t r~uc o( cach board is tecidcd r~nd Iwo Ihe da~ r~om ~he masur board is writun onto the Sys~m Bus. rhe aTor de~oction mechanism ~ibLing Bus in~crr~cc u)d ~he fi~uh lolerant ryslcm su~e machine is chbora~-cd in dt~aiJ in Chap~er 4.

5.3 Memory Bo~rd Level Desl~n _ Hi~h~o the dcsign of ~he CPV bwd is dealt wsth in grca~ dcuiL Thc following is aminulc dcscripdon Or the Mcmory Board uscd in the sys~em. The scction is sub-divided in~o ~our pans. Thc first part describcs tbe opa ion of thc natic rarn. The sccond pan includcs a blocl~ d agram of tbc memory boanL In hc tird pan tbc tirrung of the rcad/
wri~e operuion or the mcmory k)ard is iLus~ Thc last pan dcscribcs thc S10 oper-~don.
5.3.1 Cypress 2048x8 StatlC RNV RAM
Thc CY7C128 is a high pcrforTnance CMOS ~cic R~M organizcd as 2~48 words by 8 bils. An ~ivc LOW wriLe enable signal ~WE_L) conoDls thc ~ritin~freading operaLion or the memory. Whcn Ihe chip enable (CE_L) nd wrile enable ~WE_L) inpu~s are bo~LOW da1a on the dgh~ data inpul,loutpul pirls a~O through 1~) is writ-~n imo tbememory bcadon sd4rcssed by the ~ddrcss prescr~ on the sddness pin (Ao through A~Rcadin6 thc dc~icc is ccompUsbed by selccung the device snd cnabling the ou~pu~s.
CE_L nd OE_L cdvc LOW whilc (WE_L) ~msins in ctivc or }~GH Und thcsc condidons Ihe comcms of thc loc~lion aMr~cd by the ilnformstion on sddress pins is prcscn~ on thc eigh dau inpu~/output pins. Soe ~ppcndi~ X ~or specihca~ion publishcd by thc rnanu~mcr.
S3.2 Block dlagram ot the mem~ry board Tbe memory boJud cccss thc Sys~m Bus only. During systun operation cyclcs thc mcmory board ~egis~crcd thc ddrcss ~nd control signals on tbe Systc;n Bus. A addJsss decodcr on the memory board divi4cs tbc board into two addrcss spaccs. the SRAM

1. Al~ou~h Ihis is not ~ in ~e cun~l ~esigrL 1~ m~kes more a~ineenn~ sesue th~t Ihe CPU bo~d shol~ld rele se Ihe bus if h is not u~ ing ~e iu _ ~IOP ~op~ on F.un Tob~nl Cornputino 32 ~p~ and ahc I )A~ ~ c. 7~ SRAM C~ rfonn hi~h by~ DpcraLiorl, IOW by-e opcr-~L~on ~d word op~icn, ~her~s ~hc VA~ m~y only p~rrDrm low by~ opcruion.

mcrn ~nG!3..1~
~____ Hi;b 1 Low ~11 =L ~L
_ ~ Decod~ ~ L l I . LZ
Sys~m ~ adr ~ _ ~ 1_ Bus ~23.. 1> _ _ ~ _ 1~ '-11 t- ~1 RS232 .
_ _ Y
Figure:5.14 s~er~ Board Bloek Diagr~m Tht ~dd~ss spacc of thc memory u~d Ihe VART ~s listed as ~ollowcdl:
Memory: 010000 - 07BFJ:FtEX
U~RT: 800~ - FFFFFEHE3~
S.3.3 Memory read~wrlte oper~lon Refer to Figure S.14, S.IS for bming diagTam of rcad u~d wri~e operation.
Due to thc timing constn~int, the buff that dsives thc memory d~a is cnablcd at the ume time the data from the SRAM is valid. Thc conscqucnce is that therc is a small smounl of ume d~ing which thc data on Ihe system bus is not valid. This docs not ham-p~ the opcT~tion of the syssem be~ ause onc, Ihe SYS_DATA is valid whcn CPU board is ready to latch it in, nd Iwo, no companson cyclc is pfonncd dusing T~ad cycle.
S.3.4 S10 1unc~bn The saial 1/0 port on thc mem~y is to pro~de u rminal 1/0 c~pabili~es f ~e use of stand-alonc debugging. The design is bascd on the Mo~ola 68681 Dual AsynchronousRecciverl~Jansmitter (Du~Rn. which provides two xrial UO pons, 8 parallel 1/0 lines~ a bi4irestional dala bus to the CPU.

lhe d~tss p ce ~lloc~lion it fOT the u~nv~nience of ~he design. 1~ is nol opum~l in ~he sense ~u ~he memory c~s~ be 1~T~ V/hile Ihe U~RT Tqui7es onJy two byle memory sp-ce.

biOP ~ por~ on F-ul~ Toh~n~ Compu~in~ 33 ~' ~ry ~s~rl~ L4v~ ~gn Tne MC6~K81 DU~kT is a MC68000 f~rnily pcriphal which int~f~ wi1h the ~r;ro-ccssc~r vi~ cn asyn~nous bus nruurt. ~he m~jor func~ir.~s of the DV~RT u~ilizcd in ~ S10 ~yplic~tio~l ue describ~d be}G~.
5.3.4.~ I~ernal Con~
ll)e in~Lal control bgic reLeivcs o~al;on corrLmands from the CPU and generates ~ppr~riale ~ignals ~ ~e intcrnal seaions ~o control dcvicc opcralion. Il allows lhe reg-islos wilhin the DU~r ~o bc ~ccsscd u~ various commands lo be performed by de-coding thc four Iregis~r xkct lincs (RS I ~lurough RS4). Elesidcs ~hc four regist4r-select lir~s, thcre are ~e o~h i~nputs lo Ihe iin~l control logic ~m the CPU: r~d/
ite_l, chip selecl (CS_L), ~d r~(RESFr_L).

SO Sl S2 S3 ~1 U'2 S4 SS S6 S7 SO Sl S2 CLOCK ~--IJ--LJ~_ SYS_CODEI ~_ _ SYS_CODE2 ~_ _ _ SYS_CODE3 ~
SYS_CODE4 , SYS_~DRS ~= _ ~--MEM_ADRS c FC c2> ~
_ _ _, FC c0.. 1> ~C =~--SYS_DATA J { ~--'~! ~
omo ~nc mcrnory coard. Da~a is regis~cd onto the mernory board Figure S.lS Memory Bo~rd write timing di~gr~m .3.4~ runir~ Logic The tirning lo~ic conD#s of ~ crystal oscill~or, J bau~rate genc~or (BRG), 8 pro-grammab1e counlcrttimcr, ~nd four cloclc selec~ors. The crystal oscillator operales di-realy frn a 3.68684 MHz ~. Tnc cloclt scrves s the basic timing refe~ence for the BRG, thc counw~limer, u~d other ir~tcrnal circuits.

UOP R-p~ on F~ull ~obr nl Cwnpulin~ 34 . ................................................................. ~

~3~ IJ~VOI ~ n ~3`~ ' ? ~

Th; BRG op~s from Ihc oscills~r and i5 capablc of genuaing 18 cornmonly uscd data cornmunication b~ud ra~ts ranEinB ~rom SO ID 38.4K by prc~ucing inLerns] clock ou~ns ~t 16 tim^s Ihc sclual baud ra~e. nlc baud rale for Ihe S10 spplication i,s set a 9600.

SO Sl S2 S3 l W2 S4 SS S6 S7 SO Sl S2 CLOCK ~J _ J
SYS_CODEl =__ _ SYS_CODE2 ~ _ _--SYS_CODE3 ~ _ __ ~--SYS_CODE4 ~ .~ _ SYS_ADRS _= = _ MEM_ADRS C =
FC ~> ~ _ _ FC ~l.. O> ~ ~ =
MEMpATA f _ C
SYS_DATA -- ~ _ ~
CPU_DATA _ I~ ~ ~~

Ad~ i i ' Data strobed onlo ~c mc~rd. Dau iis regis~ercd onto the memay board Figurc:~.16 Memory Board read ti~ung disgram 5.3.4.3 ComnwnicaUon Channels A and B
Each conun~ulieaion ebannd comp~ses ~ full duple~ ~ recciv/~ansmi~cr (U~. Tne opaadng frequency for each reccivcr #Id s~nsmit~cr can be sclocted in-dependently from th~ baud-rue gCr~Lar.
lbe sransmit~a accepts psn~lel da~a fsorn the CPU, eonvcrts it to r saial bil stream, in-se~ts she appr~riue stan, SlOp and optia~al pari~y bits, nd w~puts a e~nposite serial str~un of data on the T~D output pin. Thc receiv coep~s xriaî dasa on she RxD pin, eonvens Ihis serial input so parallel f~7nal, chccl~s for a stan bit, ~top bit, parity bi~, w break condi~ion, nd transfer~ an ssembled characw to ~he CPU dLtring a read opcra-tion. Tl~ para~neters used for tbe S10 defining ssop bits, woad length, pari~y, and baud rate are:

~IOP R-p~ on F~ult ~ol rnnl Cornputin~ 35 BAUD:9600 WORD: 7 bit P~Y: ODD
S~.l CO~: F~X
}~ND: X-ON
The eha~act~ outpu~ utine beins by checkin the UART slatus regis~er ~or the T~RDY and T~EMT bits being sc- n~Sc bits irldicate thl the IransmiUer butrer is tlnpty l~nd the Iransrnit~r is nol eun~ly r~nding a ehaacuer. By eheel~ng Ihese bils bero7c ~nding cach ch~raeter, i~nsmii~cr ~vUT~ is prevalt~ ~AIhoul usin~ the RTSI
Cl S praocoL
Sinc~ thuact~r output is inillaled by Ihe CPU, ~re is no danger Or losing data during t~ansmission.

_ _ _ _ ~t~P R port on P~uil ~r~bnnt Computinp 36 6.4 Evaluation ~nd Concluslon ot thls MQP
nte objcc~i~e of the MQP Wl1S to de~ f~tull tolennl compulc~ synem using thrcc-~ay-vo ing algoritbm. It v as fulfilled ~vit~t t~vo major acltievcments. One is ~ thc sys-em tOlualcs morc tlYtn one faul~ single-poin~-fulure by comparin~ sll critical si~nals 1. Componcnt mc~ bo rd kvd carn~nenu in thc ~lon hcrc.
_.
~oP R-pDn cn F-uh ~GI-r~nl Cornputln~ 40 r~ ? i~
7u~nbn ~d ~onclu~bn ~ hl~ ~op r~3 f~m mirrored p~s ot d e Iyslan. This is c mJ~or irnpr~vcmuu on somt c~ ing com-m~cial ~u~ un~oy hadwal~ t~ul~ ~tnmcc.
Secondly, by ~dopnne ~ masta ~alvt alg~rilhm, csscn~ially Ihcre is only ont su of tignal on thc comn~on bus. lllis al8ori~m eluninalcs Lht pr~blem or having Ihrce sc~s ot ~gruls on 7be con~T~n bus ~d ~heJcrorc c7r~uing c ~mlmageablc ~idt bus b;~d-W ;~ LS thc IyS~UT becomes Ir~t ~nd mDsc cornpScs~1, Iht issuc ot bus band-wid~h bccornes incr~sing cTinc8L Al!;O, due u~ thc rngs~r-slave ~Igori~sm, Ihe broken boa~ can be isola~ crn the syslem ~qlhoul any C45! of per~orrnance degradaoon Finally, trom n tc07~0mic pcrspocti~ Yi~U Ihroc minDred pans D IhC rJSlem is mo~e coSI effe~ive Ihan hving Jou~
On ~c otllcr h~nd, ~ is c~l mueh irnp~rovemenl ~ cr~n be m dt Tnt voLing bus rnusl be mo~e refint~ if il is ~nak ~ conune~ial produa, mong ~ e prOlCaion or lhe clocJ~, VCC nd Iht ~ardwale guann~eing 7~he synchronism ot Ihe elock on lhree r~ale bo&rds Tnese ques~ons arc AS un-addrcs~ ~ ~is projecl A n~jor improvanenl ot the InYincible Compuw S~em whicb is outside Or Ihe sc~pe of the go~l of Ihis MQP is 10 rnalce Ihe ~cm bus rnore efficienl by employing some ad-nnced en~ineaing tecnniques such as pipelining ~s ~ fir~l wort, ~e resc~rch donc m this M~2P n prescnts a new implemenulion of faulI
tolerance bascd on ~ proven simple the~sy The nlgorilhm is a improvemenI ov the e~-ising ~ur-and-spre clgo~ithm because h is morc reli~ble, and e~fiaenL

F~r ~unple ~he 7~e 0~' ~ rnore ~dv~ced n~i~ro~ocesr~r, ~d~ ddresr ~md du~ lines a~.

UoP P-oon on F-un Tobr~nl Computlnp 41 A p p e n d i ~; A Three Way Votillg S tate Machine PAL ~quations ~.~
module ~tate n~c~ine:
fl~g `-r3';
title ` State Machine or ~hree-Wny-Voting Douglaa Cheunq 11/16t90 Stratus Computer Inc';
ftO101 device `P22V10';
REV 01 ~ato~y Rev 01-01: Diatrlbute the ~tnte n~chine to ~11 three cpu boards.
11/2~/90 ~ NP~TS ~
gnd, vccpin 12, 24;~ po~er and ground slv_mi~comp_l 0 pin 10;
slv_mi~comp 1 1 pin 11;
clk pin l;
clr pin 2;
~str_index_O pin 3;
mstr index 1 pin 4;
~ t OUTPVT5 ~
im_mst pin 15;
im_~10 pin 16;
lm_a pin 17;
im_b pin lB;
im_c pin 19;
code_O pin 20;
code_l pin 21;
code_2 pin 22;
break pin 23;

I,IOP R-port on F-un Tobr-nl Compullng 42 ~ CONSTUNT5 ~
x, ck - .X. , .C.;

~ STATES ~
FTOO - Ic3de 2, code 1, code_O~:
~inlt ~ ^bOOO;
TMl - ^bOOl;
TM2 - AbO10;
TSOl - ^bDll;
~S02 - ^blOO;
TSll - ^blOl;
~S12 - ^bllO;
~ a ~ NP~ M~DE ~
Mode - lclr, mstr ~ndex 1, m~tr lndex 0, slv niJComp ~ lv m~com-p 1 0~;
brd a ~ I 0, , , , 01;
brd_b - 1 O, O, 1, O, O~;
brd c - 1 O, 1, O, O, O~;
both slv ni~comp - 1 O, x, x, O, O~;
slv_l mi~comp only - 1 O, x, x, O, 1~;
slv O mi~comp only - I O, x, x, 1, 0~;
bot~ ok - 1 O, x, x, 1, 1~;
reset - I 1, x, x, x, x~;
~ state dia~ram equations state_diagram rToo State Tin~t: case (Mode -- brd n) :TMl;
(Mode brd b) :TSOl;
(Mode brd c) :TSll;
~Mode -- nlv 0 ni~comp only) :TSOl;
- (Mode -- slv 1 ni~co~p only) :TSll;
~Mode -- re~et) :~init;
endca~e;
State TMl: ca~e (Mode -- slv 1 ni5co~p only) :~Ml;

UOPRqx~tonFnullTobnntComputln~ 43 IMode ~- ~lv 0 mu~comp only) :~Ml;
lMode -- both ok) :5Ml:
(Mode -~ bol~h_slv mlscomp) :5S12;
endc~se:
Stste 5M2: C~3e ~Mode ~n 31v 1 m~8Comp only) :5M1;
~Mode ~ slv 0 ml~comp only) :5Ml;
~Mode n~ bo~h o~ :SM1;
;Mode ~ both slv mi9Comp) :5init;
endcr~e:
St~te 5501: C~D~ ~od~ ~ ~lv 0 ~lseomp only) :5S02:
~Mode -- ~lv 1 miscomp only) :~S01:
~Mode ~- both ok) :5S01;
~Mode -- both 51v mi~comp) :SM2:
endcs~e;
Stste 5502: csse (Mode -- slv 0 miscomp_only) :5init;
(Mode -- slv_l miscomp_Dnly) :5S01;
(Mode -- both_ok) :TSDl:
(Mode -~ bDth slv miscomp) :5init:
endc~se:
St~te T511: c~se (Mode -- slv_l miscomp only) :S512:
(MDde - ' 51v 0 miscomp only) :5Sll:
(Mode -- both ok) :SSll:
(Mode -- both slv miscomp) :S502:
endcsse;
Stste 5S12: csse ~Mode -- slv 1 miscomp only) :Sinit:
(Mode - slv 0 miscomp only) :5Sll:
(Mode -- both ok) :5S11:
~ Mode ~- bDth_slv_misccmp) :~init:
endcsse;

equ~tions ![code 2 code 1 code_0~ :- (Mode -- reset) ~ Il 1 1~;
im mst :- !code_2 ~ ~code_l S code 0):

~IOP R port on F-ull Tohr~nt Computlng 44 :~ ~code 2 1; !code_l ~ !code_O) ¢~ ( !code_2 ~ code 1 code O );
br2-~k :- ~code 2 ~ code 1 ~ !code O ~ ! tlv ra~ucomp 1 1) ~ ~ !code 2 code 1 ~ !code O ~ !Dlv m~atcomp 1 1 ~ !alv_~nisconç 1 O) (code 2 ~ !code 1 ~ !code 0 ~ !clv m$sco~p_1 0);
:- !m~ttr index 1 ~ !mstr $ndex_0;
~.t ~ :- !n~ttr lndex 1 ~ m~tr index_0:
$m_c :~ nstr $ndex 1 ~ !n~ttr lr.tdex 0:
end ~IOP P~porl on F-ull Tohnnt Compu~lno 45

Claims (14)

1. In a fault-tolerant digital data processor of the type having plural functional units interconnected along system bus means for transferring signals therebetween, wherein a first such functional unit responds to an input signal received on said system bus means from one or more other functional units for performing a first processing function on that input signal to generate an output signal, and for applying that output signal to said system bus means for transfer to one or more other functional units, the improvement wherein said first functional unit comprises (n) processing sections interconnected along voter bus means for transferring signals therebetween, and being coupled to said system bus means for receiving corresponding ones of said input signal therefrom, each said processing section comprising A. processing means responsive to said input signal for performing said first processing function thereon to generate said output signal, B. master means actuable in response to a master state signal for applying to at least one of said bus means the output signal generated in response to at least a selected input signal, C. slave means actuable in response to a slave state signal for generating and applying to said voter bus means a signal representative of a comparison of i) the output signal generated by the processing means of that processing section in response to the input signal corresponding to said selected input signal, and ii) the output signal generated by the master means of another of said processing sections, D. state means, coupled to said voter bus means and to each of said master and slave means, for maintaining a state of the respective processing section and for generating a signal representative of that state, said state means including means for selecting said state from a group of states including a master state and one or more slave states, E. said state means including voter means responsive to said state signal and to a combination of one or more match signals received on said voter bus means for selectively changing the state of the respective processing section.

2. In a fault-tolerant digital data processor according to claim 1, the further improvement comprising synchronizing means, coupled to the state means of each of said processing section, for placing one of those processing sections in said master state and the others of those processing sections a slave state at a selected time.

3. In a fault-tolerant digital data processor according to claim 2, the further improvement wherein said state means include means for maintaining only one of said processing sections in said master state at any given time.

4. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein (n) is equal to three.

5. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said state means includes means for selecting said state from a group of statesincluding a master state and (n-1) slave states.

6. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said voter means includes means for changing the state of the respective processing section as a function of the current state of that processing section and said combination of one or more match signals received on said voter bus means.

7. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said slave means include means for wire OR'ing their respective match signals onto said voter bus means.

8. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said first functional unit is a central processing unit and wherein said master means include means for generating said output signal to include at least one of data, address and control information.

9. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said voter means includes means responsive to said combination of one or more match signals received on said voter bus means indicating that all the output signals generated by the respective processing sections match for not changing the state of cthe respective processing section.

10. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said voter means associated with a processing section in said master state includes means responsive to said master state signal and said combination of one or more match signals received on said voter bus means indicating that the output signal generated by the respective processing section matched the output signal generated by at least one other processing section for not changing the state of the respective processing section.

11. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said voter means associated with a processing section in said slave state includes means responsive to said slave state signal and to i) said combination of one or more match signals received on said voter bus means indicating that the output signal generated by the respective processing section did not match the corresponding output signal generated by a processing section associated with said master state, and ii) said combination of one or more match signals received on said voter bus means indicating that the output signal generated by another processing section in said slave state did match the corresponding output signal generated by a processing section associated with said master state, for changing the state of the respective processing section from one slave state to another slave state.

12. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said voter means associated with a processing section in said slave state includes means responsive to said slave state signal and to i) said combination of one or more match signals received on said voter bus means indicating that the output signal generated by the respective processing section did match the corresponding output signal generated by a processing section associated with said master state, and ii) said combination of one or more match signals received on said voter bus means indicating that the output signal generated by another processing section in said slave state did not match the corresponding output signal generated by a processing section associated with said master state, for changing the state of the respective processing section from one slave state to another slave state.

13. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein said voter means of each of said processing sections includes means responsive to said combination of one or more match signals received on said voter bus means indicating that the output signal generated by a processing section associated with said master state did not match the corresponding output signals generated by a selected number of other processing sections in said slave state for i) changing the state of the respective processing section, ii) re-performing said first processing function on said selected input signal to generate said output signal.

14. In a fault-tolerant digital data processor according to claim 1, the further improvement wherein A. said master means includes means for applying said output signal to said voter bus means, B. said slave means includes means for comparing the output signal generated by the processing means of that processing section with the output signal received on said system bus means.