CA2078315A1 - Parallel processing apparatus and method for utilizing tiling - Google Patents

Abstract

Abstract An improved parallel processing apparatus and method executes an iterative sequence of instructions by arranging the sequence into substasks and allocating those substasks to processors. This division and allocation is conducted in such a manner as to minimize data contention among the processors and to maximize the locality of data to them. The improved apparatus and method have application to variety of multiprocessor systems, including those which are massively parallel.

Description

~07~

Ba~aroun~ 5~ I~v~a~ion The ~nvention r~la~0s ~o digital ~ata processing and, more particularly, ~o metho~s and apparatus for ~xecuting programs on parallel processing computers.
Early computers typically relied on a ~ingle processi~g unit, or CPU, to per~orm processing functions. Source code programs written for those computers were translate~ into a ~equence of machine instructions which were ~hen e2ecute~ one-by-one by the CPU. Where repetitive ~equences of s~eps~ or loops, existed in the original program, the ~ingle processor would take up each instruction in the loop, one at a ~ime, and repeat those same instructions for each iteration of the loop.
A later advance made it possible to execute some sets of instructions ~in parallel~ w;th one another. This advance, referred to ~s co-prscessinq, provided alongside the CPU a special purpose processor. Execution o programs on ~uch a machi~e was thus divided between ~he ~oproce~sor and the CPU.
With the advent of computer~ with mul~iple processors, i~ became possible to allocate entire tasks to separate concurrently op~rating CPU'~. A
special class of these multiprocessors, referred to as parallel processors, are equipped with speeial synchronizing mechanisms and thus ~re particularly ;.
suited for concurrently e~ecuting portions of the same program.
~ Parallelizing~ e~ecution of computer programs o that they can be run efficiently on parallel processors ~s a daunting task. First, the data-flow and control-flow of the program mu~t be understood and, then, rearranged to define a set .

-2- 2 ~ 7 ~ 3 1 ~3 clearly divi~ible tas~ he most ~lgn~ficant g3in~
attained to aate have been in the rearrangernent of loop e~ecution, i.e., ~loop interchange,~
~ynchronization an~ ~o a limited degree) tilin~.
Although much paxalleli2~0n work i8 ~one by hand, recent inroads have been made ~nto automat;ng that task. Thi~ i~ typically performe~ ~n connection with the compil2tion proces~, which converts the 80urce co~e program into mach~ne co~e.
One commercially available product, the KAP/KAI
Preprocessor available from Kuck ~nd Associates, Inc., of Illinoi~, perform~ some of these functions.
Particularly, that preprocessor provide~ capabilities for loop interchange and synchronization.
In view o~ th~ foregoing, an object of this invent;on is to provide improved digital data processing apparatus and methods.
More particularly, an object is to provide an improved mechanism for executing programs on parallel processing ~omputers, including those which are massively parallel.
Still another object is to provide an improved compiler for facilitatiny parallelization of computer programs.
Yet another obJect is to provide a computer run-time mechanism for parallelizing and e~ecuting a computer program on a multiprocessor system.

3 2 O ~ 3 ~ J '~

Sumna ry.Q~he InYe2lt-i~
These objects are attained by the invenkion which provides an ~mprove~ parallel processor for executing an iter~tive sequence of instructioAs by arranging the eguencs into s~ta~k~ an~ ~llocat~ng those to ~he proces~ors. This divi~ion an~
allocation i~ conductea in such a manner as to minimize data contention among the proce~sors ana to maximize locali~y of data to the proces~or~ which access that ~at~.
In one aspect, the invention provide~ an improved parallel processor of the type hav~ng a plurality of processing units, each for e~ecuting instructions; a memory for storing aata and instructions; and a mechani~m for facilitating communication be~ween the processors. The memory can itself include a plurality of memory elements, each capable of storing data and instruetions. The communication mechanism can be, for ~ample, a signalling protocol using common areas of memory.
The improvement i5 characterized, in part, by storing a tile~ ~equence of instructions, representing ~he i~erative se~uence, in memory.
Further, each processor signals its av~ilability to execute a portion of the tiled ~equence. Still further, a ne~t-tile element responds to that signalling by generatin~ a signal representing th~
boundaries o a Utile~ -- that i~, a portion of an iteration space a~sociated with the it~rative ~equence. The sa~nalling processor then execute~ the tiled equence over that tile.
The tiles generated by the ne~t-tile element ~o not overlap one another; however, all the tiles together cover the iteration ~pace. As noted above, .

~f3 J183lr~

these tiles are gener~te~ 60 ~S to minim~ze contention for ~ata be~ween the processors, while maximizing the locallty of data to them.
A parallel proces~or as descri~e abovs can inclu~e a tile-builder ~hat qenerates D til~-shape signal de~ining the dimensions wi~hin the ~teration space of the tile~. For e~ample, an iterat~on ~pa~e may be defined by the indices (i) and ~); where ti) ranges from ~ to 100, an~ where (j~ ranses from 1 to 25. ~ tile for ~uch a ~pace may be defined to cover 16 increments slong ~i) and 25 increments ~long (~)~
Thus, dividing the iteration space into 6 tiles of equal size (i.e., 16 increment~3, an~ one of ~maller tile (i.e., of 4 increments) at the edye.
The tile-builder ~enerates the tile-shape signal in viPw of the dependencies which will e~i~t between the resultant tiles. Dependency in this regard is an inter-tile characteristic referring to the relationship ~etween them with respect to the data they access so that the serial e~ecution order is preserved whenever it matter~. For e~ample, if first til~ must write a datum befor~ ~t can ~e read by a second tile, then that s~cond tile is ~aid to depend on ~he first.
More particularly, ~ata dependency e~ist~
between two tiles where i) an instruction in the irst tile writes a selected datum, which an instruction in the second tile subsequently reads, ii) an instruction in the first tile reads a selected datum, which an instruction in the ~econd tile that subseguently writes, or .
~ ,:
~ . . , . ~

iii) an instruction ~n the fir~t tile writes a selected datum, which ~n instruction ~n the ~ec~na t;le also subsequently write~.
; A more complete under~ta~ding of ~epen~ency itself may be obtaine~ ~y reference to Wolfa, Optimizin~ ~u~er~p~LL~or Superc~mpu~er~ (The MI~
Pres~, l9B9).
Th~ tile-buil~er optimizes, ~mong other things, memory utilizatis~. For esample, ~t c~n choose a tile ~hape that minimizes the number o~
individual datum subject tc wri~-type acce~ ~y different ones o the tlles to minimize data movement~ Further, to minimize contention, it can choose the tile shape that minimi~es the number of individual datum ~ubject to write-type access by plural concurrently-e~ecuting tiles.
In another 2spec~, the tile-builder of 3 parallel processor as described above qenerate~ an ~affinity~ signal ~or ~sb~) repreoeenting a ~eguence for tile execution that minimlze~ the transfer of a data between proces~ors.
The ~ile shape i8 generated a~ a function of at least a dependency direction of the tiles; an affinity signal (s~b), ~n estimate o~ the cost (e.g., the number o~ machine ~ycles) o~ eYecuting the t~les;
~he size of the iteration ~pace; the number of processors ~vailable fsr execution of the tiled ~e~uence; asld whether the tiles lie within an ~affinity region~ -- that ~8~ ~ re~ion of the program where the iteration ~pace defined by a plurality of tile sequences, rather than by a ~ingle one.
In yet another aspect, a parallel proce~sor according to the invent;on includes a tile-strategy element that selects a manner and ~eguence for , , .

~ ~ 7 ~

generat~ng tiles from ~mon~ a ~et of ~trateg~e~. The tile-strategy element gener3tes a correspo~ng signal to which the next-tile element respona~ in pro~ucing til~.
"slice" strategy ~ivi~es the iter~tion ~pace by the number o~ ~vallable proces~Drs~ One eaeh of these ~iles are ass~gne~ to a respective one of ~he proc~ssors. Thu~ there ~re 10 proce~sor~, these will b~ 10 tiles: the ~irst ~ile w~ll be assi~ned to the ~r~t processor, the second t~le to the second processor, an~ ~o ~orth~
This strateqy i8 chosen when there i~ no data depe~dency between the resultant tiles. As well as when there i~ little affinity between them -- that is, when little data accessed (whether ~or reads or writes) by any one of them is also accessed by another.
A Umodulo~ strategy divides the iteration space into a number of tiles which can be greater ~han the number of available processors, and assigns the resulting tiles based on the modulus of the t~le ~umber. Thus, for exampls, if there ar¢ 3 available processorS and 9 tiles, regardless of their timing of availability, the first processor w;ll be assigned tiles 1, 4 and 7; the second processor, tiles 2, 5 and 8; and the third ~rocessor, til~s 3, 6 and 9 This ~trategy is also Relected when there is no ~ependence and little affinity between the t~les.
Additionally, the strategy iæ chosen where the xesultant t;les and tile assignments will ma~imize the re-use of data by each of the processor~, even if the ~ize o~ the iteration ~pace ~hanges. ~, A "wavefro~t~ ~trategy also Rivides the iteration space suCh that there can be more tiles , - ~

. . .
~, ~

~ n ~ ~ t4~ ~ ~3 than available processors. ~t ~ ~hosen wher0 the resultant t~les exhibit data ~ependence ~nd, accordingly, the tile~ must be generated in ~
sequence determined by the dependency ~irection of the tiles.
~ y way of e~ample, a ~ir~t tlle may bs required to be execute~ before ~ secona tile and a ~hird tile. Under the wavefront ~trat~gy, ~ven three processors were simultaneously ~vailabl~ to take those tiles, only one of them woula be ~iven a tile. Particularly, the fir~t tile woula be assigned for execution by one processor. Only when that completed could the second and third tiles be executed, relying on the results o execution o~ the fir~t tile.
A modulo-wavefront ~trategy divides the iteration space and assign~ the tiles in accord with both the modulo and wavefront strategie~. ~hiæ
strategy is ~hosen where there i6 data dependence between the tiles an~ where data reuse by the processor can be ma~imized, again, even in the presence of a change of a ~i~e of the iterativs space.
A grab trategy too divides the iteration space such that there are more tiles than available processor~. The re~ultant tiles are sssigned ts re~uesting processors on a irst-come-~irst-~erve basis. Unlike the modulo and wavefront strategies, this strategy is employed where therP is no dependence and little affinity between th~ tiles. ~t facilitates load-balancing between the proeessorsO
In addition to the ~on~itions discussea above, the tile strategy element ~an ~hoose any of the oregoin~ ~trategie~ upon demand of the user.

.

, ,__., ! . i ~ 2~37 33~i A parallel proceGsor as described above can include an affinity region-build element for defining an iteratlon space that includes more than o~e tile~
~equence. In addition to generating a sign~1 representing that reg~on, this elem~nt can to gener~t~ a signal defining tile ~imen~ion: a ~ign~l definin~ a ~equence and manner ~or genersting tiles;
and a ~ignal aef~ninq which proce~or~ are to esecute the tiles.
In yet ~nother aspect, the invention provides an improved compiler of the type for translating a computer program into object ~ode .uitable for loading ~or execution by ~ plusality of parallel processors.
The improvement i~ charac~erized by a til~ng element that generates a tiled ~equence o~
instructions representing an iterative ~equence ~n the source program. The tiling element also generates signal~ providing a framework for use in parallel processor of the type describea above for defining and generating tiles over which to execute the tiled se~uence at run time. In this regar~, the tiling element is responsive to the iterative seguence, as well as to user-defined parameter~.
In accord w;th thi~ asp~ct of the invention, the tilin~ element can include a parallelizer responsive to the ~ependency dir~ction of the iteratiYe sequence, as well as to the equence itsel~, for choo ing inde~es in the iteration ~pace over which to execute the tiled sequence. The parallelizer can automatically choose the indices for tiling based on that. While the parallelizer can accept user-defined preferences for those indices, it - , ~
~, : :
.
.
. ~ .
' _g~ .l .C) compares them with ~he ~u~oma~ically identified one~
to insure their viability.
A compiler ~s defined ~bove can al~o includ~
an optimizer for ~enerating ~n ~ffinity ~iqn~ sb) represen~ing ~ tile execution ~eguenc~ that minimizes a transfer o4 data ~ub~ect to ~ny of re~fl-type or write-type access during execution thereo~ ~n plural tiles.
The optimi~er can ~160 generate an ~f~nity region sign31 identify~ng one or more ~erat~on sequences in the source program tha~ ~cces~ the same data. While the optimizer too can accept user-defined affinity regions, it compare~ them with the automatically identified ones to check whether the user-defined region~ are reasonable.
in another aspect, the optimizer can determine a cost associated with e~ecution of a tiled sequence ~nd generate a signal representative thereof.
~ call-qenerating element within the compiler can replace the iterative equence in the source code with an instruction represent~ng a call to a code-dispatohing ~ubroutine. A run-time ~lement can execute that call to initiate e2ecution of the tiled ~eguence by the processor~.
This and other aspects of the invention are evident in the drawings and in the description wAich follows.

-1 o- 2 ~ 7, 3 1 ~

~i e~Q&
A more complete understanding o the invention may be attained by referenGe to the drawings, ~n wh~ch:
Figure~ 1 an~ 2 depict the ~tructure of multiproces~ing sy~tem for use ~n a preferred practice of the inv~ntion;
~ igure 3 ~ep~ct~ ~he modulss utilized or parallelization an~ ¢xecution of software program~
includin~ iterative ~eguences;
~ igures 4~ - 4D depict loop tables generated by the preprocesscr 60a of Figure 3;
Figur~ 5 present~ preferred directives and assertions for preprocessor 60a of Figure 3, Figure 6 depicts runtime environment parameters for use in execution of the system o~
Figure 3;
Fi~ure 7 illustrates the tra~sformation of a tile into a task-subroutine;
Figure 8 depicts a partial ~ile ordering by the system of Figure 3; ;
Figures 9A - 9C depict wor~ plans carried out by the system of Figure 3 ~o e~ecute a ~iled sequence; and Figure 10 is a high level block-aiagram of Runtime ~ibrary 66 of Figure 3.

' ''' ~ i ' 2~7.,~3~ ~

Det~ D~ iP~ ~
Figure 1 depict~ a preferre~ multlprocessing system usea to practice the invention. The : illustrated ~y~tem 10 include~ three information trans~er level~: level:0, level:l, and level:2.
Each information trans~er level ~ncluaes one or more level segments, characterized by a bu~ element ana a plurality o~ inter~ace element~. Part~cularly, l~vel:0 of the illustrated syst~m 10 includes ~is ~eqments, designate~ ~A, 12B, 12C, 12D, l~g and 12F, respectively. Similarly, ~evel:l includ~s egment~
14A and 14B, while level:2 includes segment 16.
Each segment of level:0, ~.e., segment 12A, 12B, ..0 12F, comprise a plurality o processi~g cells. F~r example, ~egment 12A ~ncludes cell~ 18A, 18B and 18C; se~ment 12B includes cell~ 18D, 18E and 18F; and so forth. Each of those cells include a central processing unit and a memory elemen~, interconnected along an intracellular processor bus tnot shown). In accord with the preferred prac~ice of the invention, the memory element contained ~n each cells stores ~11 control and data ~ignal~ used by its associated central processing unit.
Certain cells o~ the processing system 10 are connected to secondary ~tora~e device~. In the illustrated sys~em, ~or e~ample, cell lRC i~ coupled with disk drive l9A, cell 18D iæ ~oupled with flisk driY~ l9B, ~nd cell 180 is coupled with ~i~k ~rive l9C. The disk arives l9A - 19~ are o~ conventional design and can be seleoted from ~ny of several commercially ~vailable device~. It will be appreciated that secondary ~tora~e devices other than disk drives, e.~., tape ~rives, can al~ be used to store informat;on.

' ~
' ' -12- ~ ~ r~ ~ 7 ,~ ,, ~ igure ~ illustzates in greater deta$1 processing cells and their interconnection within the processing system of ~igure 1. In the drawing, plural central processing units 40A, 40B ana 40C are couple~, respec~ively, to assoc~te~ memory element~
42A~ 428 an~ 42C. Communications between ~he processing and memory units of each pair ~re carrie~
alon~ buses 4~A, 4~B and ~4C~ a~ ~hown. Networ~ 46, representing the aforsmentione~ level 6egment~ an~
souting cells, transfers ~nformation packets ~passed to the network 46 over bu~s 48A, 48B and 48C) between the illu~trated processing cell~ 42A - 42C.
In the illu~trated embodiment, the central processing units 40A, 40~ and 40C sach incluae ~n access request element, labelled 50A, 50B an~ 50C, respectively. These access reguest elements qenerate requests for acces~ ~o data ~tored in the memory elements 42A, 428 and 42C. Among access reque~t signals ~enera~ed by elemen~s 50A, 50B ana 50C is the ownership-request, representing a reguest or e~clusive, mo~ificat~on access to a ~atum ~tored in the memory element~. In a preferred embodiment, access request elements 50A, 50B and 50C comprise a subset of an instruction ~et implemsnted on CPU's 40A, 40B and 40C. Thi~ in truetion subset i~
descri~ed below.
The central processing units 40~, 40B, 40C
operate under ~ontrol of an operatang ~stem 51~
portions 51A, 51B and 51C of which ~re resident on respective ones of the central processing unit~. The operating s~stem 51 provides ~n interface between applications programs e~ecuting on the central processing units ana the ~y~tem 10 facilities, and , ;
.: ., ~' , -13- 7~31 ~

include~ a virtual memory management ~y~tem for managing data acce~e~ ~nd alloc3tions.
A preferred oper3ting ~ystem for controll~ng central proce~sing unlt~ 40A, 40B ~n~ ~0C $8 a UNIX-like operating sy~tem an~, more p~efera~ly, OSF/l, modifie~ ~n accord w~th the teachings herein.
The memory elements ~0A, 40B ~n~ 40C inclu~e cache control uni~ 52A, 52B an~ 52C, re~pectively.
Each of these cache control unit~ interface~ a ~ata storaye area 54A, 548 an~ 59C vi~ ~ correspondin~
directory element 56A, 56B an~ 56C, as shown. ~tores 54A, 54B and 54C are u~ilize~ by the illustrate~
system to provide physical ~torage space for data ~d instruction ~ignals neede~ by their respective central proce~sing unit~.
~ further apprecia~ion of the ~tructure and operation of the illustrated digital data processing ~ystem 10 may be attained by reference to the following copending, commonly as~igned applications, the teachings of which are incorporated herein by reference:
l) United ~tates Patent Application Serial No. 136,930, filed December 22, 1987 (Attorney Docket: XSP-001), ~or ~MULTIPROCESSOR DI~I~AL DATA
PROCESSIN~ SYSTEM; n 2) United States Pa~ent Application Serial No~ 696,291, filed Nay 20, 1991 (Attorney Docket: KSD-002C2);
3) United States Patent Application No. 370,325 (Attorney Docket No. KSP-006), filed June 22, 19~9, for "MULTIPROCESSOR ~YSTEM WIT~
MULTIPLE INSTRVCTION SOURCES~;
~ ) United States Patent Application No. 370,~41 (Attorney Docket No. KSP-007), Piled ~
. .
,.
. .

.

-14- 2~37~331~

June 22, 19~9, for "IMPROVED MEMOR~ SYSTEM F0R A
MULTIPROCESSOR~;
5) Unite~ ~tates Patent Appllcatlon No. 370,2R7 (Attorney Docket No. XXP-007CP~, f~le~
June 22, 1989, for ~IMPROVED MULTIPROCESSOR SYSTEM~;
6) United 5tate~ Patent Applicat;on No. 499,182 ~Attorney Doc~et No. ~P-014~, ~ile~
March 26, 1990, ~or "HIGH-SPEED PAC~ET SWI~CHING
APPARATUS AND METHODU;
7) United ~tates Patent Application ~o. 521,7~8 (Attorney Docket No. KSP-011~, fil~a MaylO, 1990, for ~DY~AMIC PACKET ROU~ING NErwoRK~;
8) United States Patent Applica~ion No. 526,396 (Attorney Docket No. KSP-015), filea Mayl8, 1990, for ~PAC~ET ROUTING SWITCHn;
9) United State~ Patent Application No. 531,506 (Attorney Docket No. KSP-016), filed May 31, 1990, for ~DYNAMIC HIERARCHICAL ASSOCIATIVE
MEMORY";
10) United ~tates Patent Application No. (Attorney Docket No. XSD-043~, filed ~his day h~rewith, for ~DIGITAL DATA PROCESSOR WITH
IMPROVED PAGING~;
11) United ~tates Patent Applie~tion No. . (Attorney DocXet No. XSD-044), filed this day herewith, ~or ~DIGI~AL DATA PROCESSOR WITH
IMPROVED CHECXPOINTING & FORKING~; a~d 12) United States Patent ~ppli~ation No. ~Attorney Docket ~o. KSD-045), filed this day herewith, for ~IMPROVED DIGITA~ DATA
PROCESSOR WITH DISTRIBU~ED MEMORY ~YSTEMS.~

-15~ ,233~J

Co~e par~,eliz~ltiQn ~ CU~Q~I
~ iqure 3 depict~ a preferred arrangement of so~tware modules utilized ~n digital data processor 10 or paralleliz3tion an~ execution o~ softwar2 proqrams including ~ter~tive sequence~. A
compilation system 60 translates gource code input into object coae. The ~ourc0 code can be of convent;onal format, e.g., Fortr~n 77 or Fortran 90, or C programming language ~ource files, and typically includes iterat~ve ~equences or aloop~.~ In ~ddition to conventional programming ~tatement~, the gourc~
code can include user-~pec~ie~ directives for parallel;zation, Those aire~tives are preferably provided as comments, a i.e., non-e~ecutable ~tatement~. To distingui~h them from conventional comments (which are ~ypi~ally used as explanatory text) the directives preferably take a ~pecial format, as discusse~ further below.
The compilation system 60 includes preprocessor 60a and a compiler 60b. The preprocessor SOa preforms preliminary analy~i~ of iterative ~equences in the ~ource code to determine the dependency dire~tions thereof, and performs certain loop interchanges. The preprocessor 60a also qenerates diractives, o the type referred to above, for use by the compiler 60b. Technigues ~or dependency direct;on determination and loop interchaning are known. Modifications on those known techniques for improved parallelization o~ iterative sequences are described below.
The compiler 60b o~ the compilation ~ystem 60 translates program ~tatements from the preprocessed source code format to object eode format. In addition to translating conventional i -16~
207 ?~1~

codet the compiler 60b co~vert~ iterative sequence~
in the preprocessed source co~e to ~tiled" ~eguences for use in par~llel execution. This proce~ure ~s referred to as ~tiling~ and ~ controlle~, ~n p~rt, by the ~rect~ves generate~ by the pr~processor 60 as well as those includea in the ~ource code 62 itr~elf.
~ he object code output by the comp~lation system 69 i~ linked wi ~h a Runtime Library 66 by link editor ~4 to proauc~ co~e ~uitable Por e~ecution on the di~ital data processor 10.
Describe~ below ~s ~ pre~erre~ preproces~or 60a. Although the teehni~ues are appl~cable to ~
variety of ~oftware languages, such as the Fortran 77 or 90 and C programming 13nguages, the di~cu~sion below concerns translation of Fortran 77 ~ource code.
The techniques below can be adapte~ to operate in connect;on with previously known preprocessing systems -- particularly, as adapted to determine the dependency direction of the ~ource cod~
iterative sequences in the manner described below.
Such prior systems include the commercially aYailable KAP/K~I preprocessor of Ruck and Associates, of ~llinois, as well as preprocessors available from Pacific Si~rra (e.~., the ~Vast~ preprocessor).

The~E~e~cessor 1. Overview The main role of preprocessor 60a is to put annotations into a Fortran pro~ram that enable parall~l execution; that i~, to insert tiling directives. Preprocessor 60a also optionally ;
!
. . ' .
;

' ' ~ ' 1 , -1~- 2~7~3 per~orms some ~ran~formations o the code, ~uch a~
loop interchanges, either to enable tillng or to optimize.
In the illustratea embodiment, the ~nput ts preproces~or 60a ~ a Fortran program ~ource, an~ ~t~
primary output i~ o a Fortran progrsm -- albeif ~preprocessea~ one -- which include~ the tiling directives. That ~econd pro~ram is a vali~ ~nd correct Fortran program, which when executea serially computes the same result~ as the oriqin~l program.
Tha~ second program i~ the input to th~ compiler 60 2. The Tiling Directive~ - Preprocessor Output The tiling directives which preprocessor 60a puts into the code have the general form of:

C*KSR~ TILE(<tiling-args>) C*XSR* END TILE

2.1 The tiling-args The tiling-args which preprocessor 60a can put into the tiling directives have the general ~orm o~ :
tiling-args>:=~ctile-inde~-list>
l,<tiling-p~ram> ...
<tilins-param>:--<param-~eYWOrd~ ~ ~param-value~

The tiling-params which are ~pecified in the tiling directive ~ive information about attributes o~
the tiled-looP. For esample, a 'localGt' tiling-arg in the tilin~ directive means ~this tiled-loop ;

2 ~ 3 3.3 eontains a variable ~t~ which ~houl~ be local to each process~.
The tili~g params which preproces~or 60 generates are 6hown in ~he following table. Thes~
are referred to hereinafter a~ the primary set o~
parameter~.

Primary Set of Pa~ameter~

Synt~ E~ampl~
order ~ cdep-list> orDer~k las~value ~ cvar~ t~ smalle~t local ~ cvar~list> ~mp or ~tl, t2) reduction ~ <var-list~ ~um The primary set of til;ng-param~ is a subset of the full list of tiling-params which the compiler 60b can accept, as discussed below. ~owever, these tiling parameters contain information which ~an affect the correctness of the program, and it will be seen belo~ how this affect~ the tili~q.

2.2 Tilin~ Directiv~-Summary The syntas o the preprocessor 60a output tiling directive is:
C~KSR~ TILE( cinde~> ... l,order cdep-list>]
[,last~alue~cvariable-list~
[,local-cvariable-li5t~]
[,reduction=cvariable-list>~
[,ce~tra-params3~) ~ . .
C~KSR~ END TI~E

- ~. . ..

-19- 2~7J~3~ ~

The TIL~ directive mu~t occur befor~ the loops ~i,e., the Do ~tatement~) who~ indice~ are included ~n the ~tile-inde~ t~.
~ h~ loop-~n~iceg in th~ ~tile-~n~ex~ t>
ars ln the ~ame order they appeAr ~n the loopne~t;
left-to-ri~ht corre~pona to outer~to-~nner in the loopnest, (Thi~ is a con~ent~on for rea~ab~ y~
The tilin~-params which prepro~e~sor 60a can create ~s a subset (some~$me8 c~lled the primary ~et) of the tiling-param~ w~ich the compiler 60b can under~tand. The primary se~ of tiling-params i~:
order, lastvalue, loc~l, re~uction. Preproc~ssor 60a does not ~now the full lîst of the tiliny-param8; it relies on the ~ynta~ ~efinitions ~or i~ paræing.
Tiling parameter~ whi~h are ~pecifi8d in the TILE input dlrective will be passed ~a8 i ~ to the TILE output directive. (That i8, if tho~e tiling directives do not belonq to the primary ~et. I~ they do, it will be flagged as an error an~ the loop will not be tiled.) Formal ~yntax definition:

<tlle~-loop> 1~ <tila-begi~ <~ortrzD-~o-loop> <tile-end>
<tile-begin> ~ C*XSR~ ~ILE ~ <tll~Dg-arg8>) <tile-en~ C~SR~ TIEE END
<tiling-~rgs> ls ~tlla-~ndex-list~ 1, <t~llDg-p~ram-llst>~
<tile-~n~e~> is <loop-inaex-var~able-D~me>
<tilln~-param> 1~ ORDER e ~oraar-llseiag~
or EASTVALVE ~ <v~r-or-lnvelt-ll~tlng>
or REDVCTION ~ <var-or~lnv0~t-list~Dg>
or PRIVA~E ~ <var-l~tln~>
or <0~tra-~yword> 6~asy-to-par~ tring~

~' ,, ~ ;
"
~, ,: .
' ~

. -20-~7 ~3~

cor~er-l~stlng> ~ ~ord~r~ , or ~ <or~er~ t>~ -~ord~r~ ~8 1-~ ~1oop-l~de~-v~r~ab1o-~ams~
<var-or-lnv~lt-llxt~n~ svar-or-~nv~lt~
or ( ~v~r-or l~v~lt-l~t>~
<var-or-~n~olt> 1~ ~a varlablo ~am~ or ~rr~y oloms~t aams~
ln tho c~ o~ ~D arr~y eleme~t ~m~, all tho flUb5C~pt ~xp~s~lon~ mu~t b~
Invarla~t wlth re~pac~ to ~ho tllo~ loop ~e~t>
<var-listlng~ ~ <var~
or ( ~var-l~t>) ~var> ls <a variable ~amo>
~eYtra-k~ywor~> 1~ <~ Fortra~ ldent~ r>

3. Au~omatic Tiling Thi~ ~ction concentrates on the preprocessor 60a output ~iling direc~ives ~rom the functional point of view. ~he discuss;on (unless otherwise explicitly stated~ ~ssume three ~hings:
that preprocessor 60a is doin~ ully-automatic tilin~, that preprocessor 60a optimization directives (AUTOTILE. ROUNDO~F) are ~t to allow the m~imum parallelization, and that there are no As~ertions.
The pri~ciples or the tiling are t~ til~
from the outermost-inde~ inbound . And to tile as much as possible ~i.e, as many indices as possible).

3.1 Overview ~ given loopnest cannot always be tiled in all the dimension~. However, there can be more than one possibility for correct tiling. This section ! ~ -.

~- ,: ' , ~8~ ~

discusses wh;ch tiling possibil~ty preproce&sor 60 will choose. (How the user intervenes in choos~ng the indice~ to tile will be discussed in the next ~ection.) The preprocessor ~0~ perform~ dependence analysi~ on the i~erative ~equence. The principles of the aata dependence analysis may be understooa by reference to ~Data Dependence and It~ Applic~tion to Parallel Processing,~ Michael Wolfe ~nd Utpal Banerjee, International Journal of Parallel Pro~ramming, Vol. 16, ~o. 20 April 1988, aOptimi~ing Supercompilers ~or Supercomputers,~ Michae1 Wolfe, Ph.D. Thesis~ Dept. of Comp. Sci., Report No.
82-1009, Univ. of Illinoi~, Urbana, IL, October 1932;
and ~Advanced Compiler Optimization or Supercomputers,~ Davi~ Padua ana Michael Wolf~, Communications of the ACM, Vol. 29, No. 12, Decemb~r 1986.
It will be notefl ~hat ~at~ depen~enc~ i6 carried by loops. And that in the contest of preprocessor 60a, ~ependence i~ between tiles.
Several tiling obstacles ean prevent tiling, $or example, a cycl~ in the dependence, Since a dependenc~ is carried by a loopO it prevents tiling of that loop, while other loop(s) in the same loopnest can ~till be tiled. Moreover, some statements are not tilable. These can include a goto out of the loopnest and 8 subroutine call. ~his tiling obstacle affeets the whole loopnest which encloses the non-tilable statement(s).
An~ther obstacle i~ a loop that i~
imperfectly nested~ This occur~ where there are statements (other than DO's) between the loops in the loopnest. Imperfect nesting introduces a ' - 2 2 ~ r~

restriction, as lf there ~ a "wall~ wher~ the imperfect nesting occur~. ~n thi~ case the ~iling can take place either ~above~ or ~under~ that wall, but not on both ~iae~.
- Further is wher~ the bound(s) of a loop depend on the inde~ of an outer loop. This creates a nonrectangular iteratio~ ~pace, ~nd implle~ a restriction that those two loop~ ~re mutually e~clusive for tiling. It will be noted that thi~
restriction can be eased Zor ~pecial cases, ~uch a~
triangular loop~.
Based on it~ analy~is o~ the loopne~t (which - ~mong other things - finds out ~11 the tiling obstacles), preprocessor 60a tiles the loop whil2 avoiding the til~ny ob~tacle In ~o doiny, it produces a loop table which ~hows the tiling-obstacles an~ the tiling decision~ which are based on them.
The final ~ecision whe~her o~ no~ it i~
worthwhile to aotu~lly execute ~ tiled-loop in parallel 1~ take~ by the compiler (or ~t runtime).
Preprocessor 60a can t~le one-dimensional t~lD~) loops with dependence, as well ~ loop~ with a æmall amount o~ work, etc. The main rea on i~ that whil~
preprosessor 60a look at one loop at a time, more glo~al considerations ~u~h as memory distribution may influence the tilin~ strategy. The compiler can ~remove" the T~LE directive to eliminate any runtime overhea~.
Reordering (loop interchange), if any, takes place after the tiling, and only in~ide the tile.

, .
:
, .

-23- 2~7i~3 3.2 Choosinq the inflices for tiling The main role of preprocessor 60a is to insert t~ling ~irec~ives into the co~e:

c~KSR* TILE ( ~tile-in~e~ st~ t,stiling-param~

Choosing the ~iling-inde~ t iB the ma~n deci~ion. The other ~llng-param~ are determined accordingly.
For the ~ake of thi~ discu~sion, as~ume ~hat preproces~or 60a creates the loop tabl~ in two steps. ~irst, it collect~ the information about the loopnest, tiling obstacles, etc. Then, it take~ the decision about which indices to tile ~note that the loops in the loop table are ordered in the ~ame order as the loops in the loopnest, outermost fir~t). So, after preprocessor 60a~s analysi~, and be~or~ any decisions ~re taken about tiliny, the loop table is as shown in Fi~ure 4~.
~ here, the Utilable~u ~ield indicate~
whether there i~ a tiling obsta~le whi~h prevents this particular loop from being tiled, regar~les~ of whether other loops are tiled or not. This occurs when the loop carries a cycl~ in dependence, or the loop body contains a non-tila~le statement, etc.
The ~restriction~ field notes which other loops in the loopnest might be affected ~y tiling this loop. This occurs, e.g., when the loop is imperfectly nested, or non-rectangular. As previously mentioned, the point at which imperfectly nestiny occurs may ~e thought o ~s a ~wall.~ The wall can be ~attached~ either to the previous loop or to the following loop. It can be arbitrarily assumed that it is a~tached to the previous loop.

- ~ ' ~" ( ?

207~2~1~
The obsta~le field conta~n~ desc~iptive information about the tiling obstacle if any.
Now, ~11 th~re i~ lef~ to be done i~ to fill ln the tiling-deci~on field, ~ase~ upon th~
informat~o~ in the tilable? an~ restriction~ ~ield~.
Preprocessor 60fl t~les the loop from the outs4as ~nboun~s, o ~t can be viewe~ a~ i it tart~ from the f~rst row ~n the loop table and move~ down, til~ng ~ much a~ po6~ible, whil~ takin~ care to respect any restriction~.
The loop ~able can ~e use~ to Bescribe the concept~ of restriction handling~ Whenev2r it i8 decided to tile a loop, it is marked as tiled in the tiling-decision ~ield. Then a look i6 taken at its restrictio~ field: If there is an ~IMPERFECT~
indication, the preprocessor 60a goes ahead ana marks the tiling-deci~ion fields of all the rows below as not-tiled; if there i8 an <i~x> ~or more than one)~
the preprocessor 60a mark6 ~h~ tiling-~ecision ~ield of the correspondent loop~s) as not-tiled.
Note that the preprocessor 60a always needs to go Wdownwards~ only.
After preprocessor 60a tiles the ~irst lsop in a loopnest, rows further down the loo~ ta~le may already have a tiling decision entry. This result~
from a restriction imposed ~y a previously tiled loop. In this case, preproces~or 60a ~kips that loop row when it comes to it, and moves on the ne~t.
Conceptually, thi~ i~ th2 way in which preprocessor 60a chooses the indices which will comprise the <tile-inde~-list> ~or the ~I~E
directive. Follow;ng that, the other <tiling-param~
are aetermined, ~nd the process of tiling is complete.

, .

-2 5- 2 ~ 1 ~ 3 ~ j The examples in ~he rest of this ~ect~on demonstrate this behaviox. For each example, the tiled program is provided, with the loop table being ~hown in the accompanying dr~wing.

3,3 E~ample~

Example 1. Inspired by the ~inp~ck benchmark:
~o k ~ l,n-l do ~ ~ k~l, n ~o 1 ~ 1, n-k ~(k~ (k+i,j) ~ t * a(k~i,k3 enado enddo enddo As shown in Figure 4B, the k-loop cannot be tiled, ~înce it carries a cycle in depenaence. Thus, the restriction entries for k did not apply to the tiling decision for i and i.
Preprocessor 60a tiles this loop ~s follows:

do k ~ 1, n C*KSR* TILE( J , I) DO 2 J-k~l,n-l DO ~ I=l,n-k A(X~I,J) ~ A(K~I,J) ~ T ~ A(K+I,R~

C*XSR* END TIhE
enddo ~: , -, . ~. ` ~ .

. ~ .

~ ~ r~

Esample 2. Matrl~ mult~ply do i x l,n do ~ ~ l,m c(i,j) .. ~
do k . 1, 1 c(i,~) .. c(i,~) + a(i,k) ~ b(~
enddo enddo enddo ~ s ref lec~ed in ~igure ~C, the restriction on the j-loop caused the tiling to ~Stop" ~t that point .
E'reprocessor 60a will tile thi~ loop as follows:
, C*KSR~ TILE ( I, J) do i . l,n do ~ .. l,m 9 ~ û
do k ~ 1, 1 c(i, j~ = c(i,j) + a(i,k) * b(k,j) enddo enddo enddo C*XSR* END TILE

Example 3. Inspired by the Legendre Transform:

do 400 1 ~ 1, nlev do 300 ~s ~ 1, nwaves ip ~ nmp(k) ., , . . , ~ . ,.

., ~ : ~ ', . , :;
' ~ , . , . ,:

-27~ 3 ~ ~

~lo 200 ~ ~ 1, nl~t8 do 100 ~ ~ 1, nnp(k) ~d(l,ip~
,ip+~ 5al(1,k, j~*pnm~p~1) ~q(l,~p~
~ q(l,ip+i)+fsgl(l~k,~)*pnma~ip~i) loo continue 200 continue 300 continue 400 continue In order ~o make th;s e~ample work it i~
necessary to put in an assertion ~not ~hown here) to remove assumea dependence. ~lso, ~n this case preprocessor 60a uses forwara substitution technique, so that the k-loop and the ~-loop can be made perfectly nested. Prepro~essor 60a therefore til*s th~ program as ~ the loop was the following.

do 400 1 8 1~ nlev do 300 k ~ 1, nwaves do 200 ~ ~ 1, nlats do 100 i ~ 1, nnp(k) ~d(l,nmp~k)~i)=
sd(l,nmp(k)~ fsdl(l,k,j)*p~md(nmp(k)~i) sqtl,nmp(k)~i)=
sq(l,nmp(k)~ fsql(lOk, j)*pnmd(nmp(k)~i~
100 s:ontinue 200 continue 300 continue 400 continue .

-: ~ ; . . . . .
- . .. , , , .. ~ , , ,.. . . .

, ~07~3~.~

As shown in Figure 4D, whs~ preprocessor 60a deci~es to tile the k-loop, the restriction on ~he i-loop enforce~ ~ not-tiled decision for i.

C*KSR* TILE( L,K,J 3 do 400 1 ~ 1, nlev do 300 k ~ 1, nwave~
do 200 ~ - 1, nl~ts ~o 100 1 ~ 1~ nnp(k~
~d(l,nmp(k)~i)s~d(l,nmp(k)~
~sdl(1,k,~)*pnmd~nmp~k)~i) sq(~ mp(k)~ q~l~nmpska+~ )~
~sqlt2,k,j)*pnmd~nmp(k)~i) 100 continue 200 continue 300 continue 400 continue C*XSR* END TILE

4. Semi-Automatic ~iling This ~ection ~escribe~ the semi-automa~ic method for tiling~ which ~llow the user to partially override the tiling ~ecisions as done by preprocessor 6Ca.

.1 Over~i~w In the general case there is more then one possibility to choose the indices or tiling.
Preprocessor 60a chooses one o~ those possibilaties, ~hrough the automatic tilin~ mechanism.
~emi-automatic tiling allows the user to intex~ene in the process of choosing the indices ~or tiling, by specifyinq explicitly which in~ices he wants to be tiled. Usin~ semi-automatic tiling, the user gains ; ~
, ,, ~
. . . .

.

.

. . ~ ! I
-29- 2~

additional control over the tiling, while keeping the same guarantee of correctness as with automatic g, Thi6 i8 done by u~ng the followinq preprocessor 60a ~nput dlrect~ve:

C~KSR~ TILE ~ctile-in~e~-list> t, 6t~1ing-param>

The ~tiling-param~ ean be a~y parame~er which is not one of the primary set of tiling-parameters. The reason for that is that, as mentioned before, the tiling parameters in the primary set (order, lastvalue, local~ reduction) can affect the correc~ness of the program.
Preprocessor 60a transforms the input directive into the following ~tatement:

c~XSR~ TILE (~tile-i~de~ t>
l, ~tiling-param~.,.]~
c~KSR~ END TILE

Where ~tiling-param> contains a~l the tiling parameter~ which were ~pec~ied in the C*K~R*TIL~
directi~e, and probably additional tiling parameters from the primary set. And where ctile inde~-list> is tbe same as the ~ne whieh the user 6pecified. If the user ~pecified a com~inat~on which ~s incorrect ~according to preprocessor 60a criteria~ preprocessor 60a will issue as error.

4.2 Example ~ eferring again to the ~Inspired ~y the Legendre Transform~ e~ample abo~e, by using forward , . . , . .: ~
: . ' . . . : ~, . .

~30- 2~7 ~

~ubstitution technique, the loop i8 tiled ~n 3D.
However, the u~er coul~ tile ~t in 2D by putt~ng the following line before 'she loopnest:

C~KSR* TILE ( 1, k) Thi~ instructs preproceæ~or 60a to t$1e irl those ~ndice~ only. ~ince it ~8 ~ legal ~0~6ibility (as can be ~een from the loop t~bl~3, preproces~or 60a will aO ~o without ~enerating an e~ror mesE3qe.

C*KSR~ TILE (1, k) do 400 1 . 1, nl~v ~o 300 k ~ 1, nwavé~
ip ~ nimp(k) do 200 j, 1, nlat~
~o 100 i ~ 1, nnp~k) sa (l ~ i p+i ) ~d ~l , ip~
f dl(l,k, j3*pnm~(ip~i) ~q(1,ip~ g(1,ip~
f s~l ~1, k, j ~ ~pnm~ ~ ip+i ) 100 con'cinue 2 0 0 cont i slue 3 oa con~ inue 4 0 0 cont i nue Preprocessor 60a tiles it as ~ollows: :

C*KSR* TILE( L,X ) do 400 1 ~ 1, nlev do 300 k ~ 1, nwaves :
aO 200 j ~ 1, nlat~
do 100 i ~ 1, nnp(k) .

, " ' ', , ., , ~ 2 ~ 7 3 3 ~ ~

,n~p~ ,nmp(k)~
fsal~,k,~)~pnmd(nmp(k)+~) sq(~,nmp(k)~ q(~,nmp(k3~i3 ~gl~,k,~*pnmd(nmp(~
100 continue 200 continua 300 continue 400 ~on inue C*KSR~ END T~LE

5. Related Is ueR
The ~bove section~ focus on the tll~nq aspect of preproce~sor 60a operation. Below, i~ a brief discu s~on of other aspects o~ operat~on o preproces~or 60a which ar~ not ~irectly relatea to ~iling but which may effect the results of the tiled program.
Distribution i~ per~ormed when ~t can help the til~ng. For esample, to tile part o~ the loop when there are 1~0 stateme~ts in ~t.
In ~ome cases coae transformation needs to take place ~n ord0r to t~le ~ pro~r~m (for e~ampl~, in the presence of r~duction, or when the last-value is needed). 8Ome of those transformation require to know the bounds of a tile - ~ runt~me value, which is available when ~ tile i~ being executed.
In most ~ases the transformation i~ done by preprocessor 60a. However, if ~or some reason users (or preprocessor 60a~ do this kind of tran~formation~, they mi~ht need to know the runtime value of the bounds of the tile~ Th;~ can be obtained by the use o intrinsic ~unction.

.

:~ , , , . ~ . ; ... .
,, ,., ~. . , .,; , ,- :
~ 1 ,,. . . , - . . , ~ .
. . . , , . i:

-32- ~7,~31 ~

Inner loop in~ic~ o a s~ri~l loop inside an outer tile~ loop are treate~ ~y the compiler locals. For e~ample .

C*KSR* TILE( ~,X ~
~o 400 1 ~ 1, nlev do 300 k ~ 1, nwave~
~o 200 ~ 8 1~ nlat6 do 100 ~ ~ 1, nnp(k) s~l,nmp~k)~i3~a(l,nmp~k)~ ~+
f~ pnmd(nmp~k)~
~q(l,nmp(k)~ sq(l,nmp(k)+i)~
~sql(l,k,j)~pnmd(nmp(~
100 continue 200 continue 300 continue 400 continue C~KSR* END TILE

. The compiler treats thi~ loop ~s if there iæ
an implicit local.(i,;).

T~e R~ntime Library 1. Overview The following ~ections describe the operation o~ the ~untime Library 66.

2. The programming model Runtime Library 66 iF language independent.
It can be called from a variety of languages, e.g.
Fortran 77 or 90, C or the like. However, the ~ollowing ~ections di~cuss it w;t~ r~spect to it~ use from Fortran programs~

- ' .~ . '' ` ' , 2 ~ 31~

The three parallel construct~ which Runtime Libra~y 66 ha~les are ~lling, parallel regions, and parallel sections. The tiling construct allows the user to execute ~ortr~n ~o-loop~ ln parallel. The parallel ~ections cons~ruct enable~ the u~ex to execute different co~e ~egmentg o ~ program in parallel. The parallel regions construct allow~ the user to have a ~ngle code ~egment o ~ program run mult~ple times imultaneously.
All parallel construct~ may b~ nested.
However, the Runtime ~lhrary 66 may run ~ny par~llel construct ~erially if sufficient resources are not available.
The rest of thi~ section ~ontains a short description o the parallel construct~ which are supported by Runtime Library 66. The following sections will discuss each one in deta;l.

2.1 Tiling ~ iling of a Fortran loopnest i~ 2 partitioning o the iteration pace into rectangular parallelipiped chunks called tiles. Hence, a tile is a collection of iterations. The group of til~s which construct a loopnest is called tile-fam;ly. The tiles are the b~ic entities which can be executed in parallel. Numerous processors c~n e~ecute the same loopnest, each one of them working on a ~eparate tile simultaneously.

E~ample:

C*XSR* TILE ( i, j) do 10 i ~ l,n do 10 j = l,m , ` '' :

`. J

~-3~ 7~3~

o . o 10 continu~
C*KSR* END TI~E

In thi~ case, the loopnest ~8 til~ ~n the two indi~es 1 ana ~. It i~ possible to tile only part of ~he loop ~nflice~, ~.g.- in the ~bove ~xampl~
the following tiling i~ o possibla:

C*XSR~ TI~E ( 1 ~
do 10 ~ ~ l,n do 10 ~ ~ l,m o . o 10 continue C~KSR~ END TILE

The tiling mo~el has two qualities which are important to Runtime ~ibrary 66. First, 1esibility in terms of work~overhead ratio. The Runtime ~ibrary 66 it provides a general way ~o hand~e granularity of parallelism ranging from one iterat;on to any number of iterations. Second, convenience in handling dependency: The Runtim~ ~ibrary 66 pro~ides a ~imple way to define a par~i~l order ~iles dep~ndency~, and a way to exploit paralleli~m in the presence of dependency.

2.2 Affinity Reg~on~
~ he afinity re~ion mechanism applies to the ~iling parallel construct. It provides a me~hod ~or the user to convey optimization information to Runtime Library 66. An afinity region i~ a collection o~ tile ~amilie~ which Runtime Libr~ry 66 attempts to e~ecute in 3 fashion ~o as to avoid data ;;

.
.
' ' .

2~7 ~3 .1'-'3 contention an~ movement. ~untime Library 66 ke~ps ~ome information about the entire set of t~le families, and u~es that to di~tribute tile~ to processors ~o that proces~or~ w$11 execute on the æame ~at~ from tile family to tile family.
To declare ~n affinity re~ion, the user must enclose the desired code with~n the AFFIN~TY REGION
and END AFEINITY REGION d~rectives. The dir~ct~ves must not interrupt ~ t~le family. I~ ~ecl~r~d w~hin a parallel section, the affinity reqion mu~t be within one æect~on blo~. The declaration must be within a single subroutine or main program.
These are param2ters which ~f~ect efficiency of execu~ion rather than correctness. A~finity region requires ~lo~al ~ecision ma~ing, an~ this i8 the way for the user ~o ~pecify them. If th2 u~r specified the same parameters in a ~ILE direct~ve embedded within an AFFINI~Y REGION, the parameters in the AFFINITY REGION override the one~ in the TILE
directive.
Affinity regions can be n~stea.

E~ample:

C~KSR* AFFINITY RE~ION ( i,j, ~TRATEGY . MOD, NUMTHREADS ~ 8) do k C*R~R~ ~ILE ~ i,j) do i do j . . .
enddo en~do C*KSR* END TILE ( 1, j, 3 ~ ' . ' . :
' . ' ' ' ' ' . ' ' .

! ; , -36- 2 ~ 7 ~ 3 ~1 ~

endBo C~KSR* END AFFINITY ~E~ION

~.3 Team Operator~
- Parallel con6tructs are execute~ in Runtime L;brary 66 by ~roups of pthreadQ. In the default mode, these pthread groups ar~ invi~i~le to ~he user. ~owever, Runtime ~brary C6 aoe~ ~mplement an interface to th~se pthread ~roup~ for th2 user who want~ a greater degree o control o his pro~ra~.
The functions that manage pthrea~ group& are calle~
"teama operator~. The ~ntsrface a~ d~scrib~d in detail at the end of this ~ection.

2.3.1 Definit;on of a team Each pthread group, or team, consist~ of o~e or more pthreads, where one pthread is designat~d a ~leader~. Each team member has a member ~d uniqu¢
within the team, ~tarting at 0 anfl a~cending, with ~o ga ps in the ~guence. The team leader~ fi member i~
will be 0.

2.3.2 Default Team Usage Runtime ~ibrary 66 will creat~, manage, and d;sband teams automatically without direction from the user. However, the Runtime ~ibrary 66 interface does allow the user to explic~tl~ ~pecify team creation, dispersion, and usage.
If the user does not specify team u~age, Runtime Library 66 follows the general practice of creatinq a new thread team ~or every ~ew parallel construct. The thread team i~ ~isban~ed at the end of the construct. An e~ception i~ made for TILE
constructs that are le~ic~lly enclosed within an :- .
~' . .

2~ 3~
AFFINITY REGION ~irective; all ~uch ~ILE construct~
are execute~ by the 6ame thread team.

2.3.3 T~am 1~8 Team ID~ ~re uniqus throughout the program.

2.3.4 T~am Cre~t~on The pthre3a that runs acro~ ~n ~pr_create~team c~ll execute~ the call an~ becom~#
the team lea~er. Pthread~ may be member~ of ~everal teams.

2.3.5 Restriction~ in U~e o Teams A team may not be use~ in parallel -- it can only execute one construct at a ~ime. Howev~r, if ~onstructs are neste~, ~ pthread may be a member of several teams, and may execu~e multiple constructs.
A parallel construct may snly be e~ecuted by a team where the pthread that encounter~ the construct i. a mem~er an~ i~ the leader of the team.
The motivation ~or thi~ restrict~on ~s a fundamental implementation i6~ue. ~he pthr~ad that encounter.
the construct i~ the only pthread that has the conte~t to e~ecute the serial code before and after the parallel ~onstructO It could be possible for Runtime Library 66 to allow a pthread to ~all a team that it i~ not a member of to e~ecute th~ construct~
hut the original pthread will be ~orced to idle durin~ the p~rallel e~eeut;on.

: ' ~ . .
.: ' . :

-38 2 ~7~31~J

3. ~nt0rface~

3.1 Runt~me ~brary 6~/User ~nter~ace Users pa~ input to Runtime ~ibr~ry 66 through run ~ime par~meter~, program ~irective~ or ~ubroutine c~ h~ run time parameters enabl~ the user to control the res~urce~ ~n~ c~lculation~ ~one by Runt~me Library 660 allowing her to tun~ for performance. ~he prs~ram d~rectiY0s ~llow the u~r to indicat~ oppor~uni~ e8 for par~lleli~m. Progr~m directives may be ~ddressed to ~he k~r compiler or preproGessor COa or both. Subroutine call~ are u~ed to esplicitly control Runtime Library 66 thread group (team) management.

3.1.1 Program directive~
As noted above, program dir~ctives are in the form of ~ortran comments. When a progrDm d;rective is present, the compller 60b gener~tes call~ to the Runtime ~brary 66 runtime library to cause the parallel execution of the loopn~st.
The parallel ~ection, parallel region, ~nd set dirsctives are yenerated only by the user ~nd understood only by the compiler J Af f ini ty regisn directives are generated by the user or the ~ompiler, and understood only by the ~ompiler. Tilin~
directives ar~ generated ~y the us~r a~d~or the compiler 60b~
To tile ~ Fortr~n program, the u~e~ can either put in the iling directives by han~, or rely on the prepro~essor 60a to ~o ~o. The preproces or 60a takes a Fortran program as an lnput, an~ create the trans~ormed Fortran program wh~ch has the tiling directiYes in it. Ik is possible to use a mi~ture of .

, , -39- ~ '31~

manual and automat~c tillng; hence, the preproce~sor 60a can take a parti~lly tile~ prograrn, ret~in the loopnest~ which are already tiled ~lone, and til~ the other loop The ou~put o~ the preproces~or 60 legal Fortr~n pro~ram;
With the ~ully automatic mo~e, the u~er invokes ~he compila~ion system 60, which in turn invokes ~he preproce~sor 60a. It w~ll bB appreCiatea that the Runtime ~ibrary 66 it~el~ ~ not aware o~
the difference between automat~c ana ~eml automatic tiling. These ~ifferent metho~s produce identicAl input from Runtime Library 66'~ point o vlew.

3.1.2 Run Time Parameter The runtime environment parameter~, which are set forth in ~igures5 are ~e~ined using Uni~
environment Yariables. These parameters can also be set ~sing the SET directive.
In order to achieve parallel e2ecution, the code within a parallel construct i~ tran~forme~ into a special kind of subr~ut;ne. The task-subroutine(æ) resemble~ a ne~ted ~ubroutine in the manner o Pascal. It will be appreciated that thi~ i~ not an estension to programming langua~e itself, the task-subroutines are created in the internal data structures of the the compilation ~ystem 60 only.
Figure 7 illustrate~ the transformation of a tile into a t~sk-~ubroutine. In the drawing the original program (ll.f) is denoted as bloc~ 70a. The tiled program ~ll.cmp) is denoted as block 70b. ~hat tile program ;5 internally transformed into (~us if~) the ~ode 6howD in block 70c.
By doin~ thi~ transformation, the tile~
loopnest turnsd into ~ ~ubroutine. The arguments to , ~ : ,',, , ., ~. ~ , .

2 ~ 7 ~ 3 1 r~
this ~uhroutine zre the boun~s of the ~le. In the e~ample shown ~n Figur~ 7, ~o 10 i~
~o 10 ~l,n WBS transforme~ to 1~0 10 i-~l,i2 ~o ~ 2 and the boun~ become the argument~ to the tas~-subroutine. For ~xample, ~f a tile with a 16~16 tile-size i~ used, one thread will is~ue ~ call to task foo_$1 (32, 47, lS, 31). Th~s will cause the execution of do 10 i.32, 47 do 10 j.l6, 31 a~i,;) . O.o continue ~ ence, this threa~ e~ecutes 16 iterations in the i dimension, and 16 iterations in the dimension. Runtime ~ibrary 66 will invoke the calls to task foo_$1 rom the aiferent threads with the appropriate ~rquments such that all the iterat~ons will be e2ecuted. The parallelism is e~rci~ed by having many threads calling the t~sk-subroutine di~ferent arguments ~i.e., bounæs).
The e~istence o~ the parallel constructs tri~gers a call to "execute~ routin~ in the Runtime Library 66, and the compi1er passeæ the name of the task-subroutine ~s an argument. Th0 arguments of . . . . . . .
;, ' ( ) ~13 r~ r~

these ~xecute~ rout~n~ conta$n ~11 th¢ information about th~ construct whlch i~ ~eeaea ~n order to execute lt in par~llel. ~ome of thi~ ~nformation come~ from th0 program directive ltself (i.~., which indices to tile, ~ependency inform3t~0n e~c.); some in~orma~on comes from the ~ource program ~i.e.
bounds a~d tri~e o the loopne~t); some of ~he information ls generated by the compilation ~ystem 60 ~i.e., ~sb, coaesi~ as ~iscussed ~elow). ~her~ i also ~ome ~ata which i~ neede~ to ~nterace between the ~ode ~nsi~e th~ ta~k-subroutine an~ outsi~e ~t ~pointer to the task-subroutine, frame pointer, fl~g to support la~t value).
Thi~ embo~iment provi~es a ~imple way for th~ tile to be ezecuted ~8 an independent ent~ty ~by being a subroutine) and at the ~ame time recognize th~ ~ariables o it8 parent routine using an e~isting compiler mechanism ~by being a nested subrout;ne).
In this particular example, it recognizes the array a.

3.2 Runtime ~ibrary/Operating æystem interÇace ~ untime Library 66 parallelism ~8 implemented with the OSF implementation of pthreads.
The interf ace between the OS and Runtime Library 66 and pthread~ ~nd Runtime Sy~tem 66 i~ not ~iscussed in detail here, but there are some basic assumptions about the world in which Runtim~ Library 66 live wh;~h are needed in order to establish the ramework.
Runtime Library 66 uses variable number of threads durin~ the life o~ the program. ~ single thread is initiatea at startup, and becomes the program leader thread. This thread $~ responsible for e~ecutin5 all B~rial portions of the pro~ram.

, ~ , , ~;
~, ( ) ) 2~7~3~

Each parallel construct is e~ecute~ by ~
team of threads called ~ ~thread group~. Each thread group has one threa~ that is desiynated a group leader, whil~ all other mem~r~ ar~ group gldYe~.
Runtime Llbrary 66 aelegate~ much of the load balancing between threads to the operating system ~cheduler. In some cases Runtlme ~ibrary 66 assumes that a ~hread i~ associate~ with a processor, and that this bindlng remain~. ~his ~ an ~mport~nt assumption use~ by Runtime Ll~rary 66 ~n the mo~ulo tiling strategy where wor~ i~ partitioned so that a cell will referencs data it already own6.
Runtime Library 66 may pass inf~rmation to the OS scheduler to help it make more informe~
decisions about load balancing.

4. Tiling Tiling, as state~ above, is a metho~ to execute a loopnest in parallel. This section will explain the ~emantics of the Tiling ~rective a~d illustrate the way a tiled loop i~ executed.

4.1 Tiling Directive - 8emantics ~ ot every loop can be til~d in a ~imple way. So~e loops can't be tiled at all, an~ ~ome loops can be tiled only with special care to ensure correct execution. ~Correct e~ecutiona ~n this context means the same result as by running the ame proyram serially.
The syntax of the tiling directive enables specifications of tiling parameters which will provide the additional information re~uired to ensur2 correct e~ecution. Tbe~e are order, lastvalue, local and reduction.
i - , , -4~-7; 0 ~ ~ 3 1 1 In ad~ition, thera ~re other tlling parameter which ~o not a~fect the correctne~ ~f the program, but do ~ffect the psrformance.

4.2 Tiling with Or~er Tiling Par~meter ~ he order tiling parameter ~peci~es 2 partial oraer ~or exe~ut~o~ o t~le~, which i~
derived Prom the ~ata ~epen~ency within the tile~
loopnest. The orfler tiling parameSer ~e~er~es ~ome ~pec~al attention ~n this ~e~tion because ~t can be confusing, and of~en not intuitive, to aetermine the dependency and thu~ the eorrect order. In a~dit~on, it is one of the tiling-parameter~ which can influence the correctness of ~he program e~ecution.
The fa~t that dependency - and thus ~he execution order tiling directive - i~ not easy to determine is not worrisome, ~ince it is typically detected automatically by the preproce~sor 60~. If the user choo~es not to u~e the preproce~sor ~Oa he or ~he can ~peci~y ~t, an~ than it becomes his or her responsibility.
When a loopnest i~ tile~ with srder tiling parameter, Ru~time Library 66 wlll try to a~hie~e parallel e~ecution of that loop whil~ ensuring the correct order of e~ecution. I~ some ~as~s obeying the order will cause ~erial e~ecution. However, in some cas~s a loop which i~ tiled with oraer can run in parallel.
When a loopnest is e~ecuted in parallel, the iterations will not necessarily be e~ecuted in the ~ame order as by serial ~ecution of the ~ame loopnest. In some cases it doesn't matterO while in other ~ases a data-aependency implies a partial order between the iterations~ This, in turn, implies a , ,.

( ) _ 2~7 Q~1 ~

partial or~er between ~iles ts guaran~ee correct e~ecution.
The way to handle it i~ to ~peci~y a order tiling directive. Thi~ will cause ~untime L~brary 66 to ~o the necessary ~ynchronization between the esecut$on of tile~ to ensure ~he correct execu~ion.

E~ample:
aO 10 ~ ~ 2,n-1 do 10 ~ . 2,m-1 a(i,j) ~ at i~ l ) + a ~ i+l , 10 continue Thi loopne~t can be tiled in both dimensions in the following way:

C~KSR* TILE ( i, ~, ORDER ~ -J, I
do 10 i ~ 2,n-1 do 10 j - 2,m-1 ) 8 E~ j+l) ' 6(i~
10 continue C~R* END TILE

This defines the ~ollowing partial or~er between the tiles: e~ecut~on of ~ tile can start when the tile before in the I-th direction completed, and ~he tile after in the J-th direction completed. In the diaqram presented Figure 8 the til~ marked by ~s~
can be e~ecuted when the tiles markea by ~d~
completed. This will typically cause Runtime ~$brary 66 to choose the wave-front tiling strategy, which enables parallel e~ecution in the presence of order in two dimensions.

; :' ~. .

2~7~3~.~
4.3 The oraer tiling parameter ~n~ Dat~
Depen~ency - by e~arnple A~ noted above, the ORDER ~ -30 I i~
typically insertefl by the p~eproce~oz 60a. ~his section e~plains by way of example the rel~tions between ~ata dependeney and til~ng with or~er.
Referrin~ to the original proDr~m:
do 10 i ~ lOn aO 10 ~

10 co~tinue Fir~t, to deine the order ln which iterations must be e~ecuted, one must look ~t the loop body. Assuming the position of a given iteration is marked by ~', and the poRition of the iteration(s) upon which it depends is marked by the resulting diagram ~:
.
__ ~ *

*
V

In the original loop j is the inner in~s~, hence the one which moves faster; so when '~' is e~ecuted, it must be the case that the iterations on its upper-right ~5 already done (this is indi~ated by a '~'), and the iteration on its lower-le~t is not-done (this is inaicated b~ a '-'). The re~ult is as ~ollows:

: .

, ~ i ~ ~ r~ 3 ~

->
~ s (I) Y

In other word~, it i8 ~afe to execut~ an iteration in position s, if ~n~ only if ~he i~et~ion on its upper-right ~s done. ~his can b~ reduce~ to dependency between two iterat~ons, as follows~

j __> d i Y ~II) I
V

The ne~t ~tep ~s to figure out the partial-order between t;les. ~nside the t~le the original order ~s pre~erved. In ox~er to examine what happens betw~en the ~iles, the ~te~cil~ in (I~, above, can be moved ~roun~ the borders of ~n imaginary 'cile, as follows:

x l -k----The equivalent o~ (IX) will be ~ ~ollows, with capital letter~ used to denote tiles:

;, :
.
- - ~ .-.

., - -q7- 2 __ X D

. V
The D ~n t~e upper-righ~ i5 re~undant: the bottom line implie~ that ~ tile ~ mu~t w3it or the til~ to it~ right before it can ~tar~. æO, if the tile in the lower-le~t wait3 for the tile in itæ
uRper-le~t, it i~ ~nough to en~ur~ that ~he til~ on the upper r~ght w~ll be ~one alrea~y. Hence, the lower-left tile ~which i~ mar~ed by ~ ) mu~t wait for the tiles ~abovP~ an~ ~to its rightW to b~ ~one, and by recursion the cor~ect order is defined. Henc~, the resul~ is as follows:

~ D
i X D
I
V

Which is espEesse~ in the tiling ~irective in the following way:

C*KSR* TILE ( ~,J,ORDER=( -J , I )~

4.4 Tiling with local, lastvalue, redu~tion In ~ddition to ths indices ~nd dependency parameters, there ~re three other tiling parameters which ~an affect the ~orrectness of the program.
~ypic~lly, those tiling parameter~ will be created by the preproeessor 60a. They ~re: :

'~' ' '' ; `

7 ~3 3 ~ ~3 LOCA~ - declorat~on o~ local variabls~ needed by the new lexical subroutine. Thi~ ~ B handled by the compiler 60b, and ~6 not pa~ed on to Runtime Libr~ry 66.

LASTY~UE - indioates whethsr the l~st value of the loop indice~ must be pre~erved after the loop e~ecution. Runtime ~ibrary 66 mu~t hanale th~, because the parallelizat~on o ~he loop af~ect~ the e~ecution order of the ~terat~on~. Runtlme Libr3ry 66 ~alculates ~he la3t value by checking the ~oun~s of each tile e~0cu$e~. When the tile containing the highest bound~ of the iteratio~ ~pace i~ e~ecute~, the las~ value i~ passed by Runtime Library 66 to the compilation ~y~tem 60.

REDUCTION- declares that a reduct~on must be handle~
on a variable within the tile. Reduct~on i8 handle~
by the oompilation ~ystem 60, a~d need ~ome ~upport ~rom Runtime ~ibrary 66.

4.5 Other til~nq parameters There are tilin~ parameter~ which enables the user to intervene with Runtime Library 6S'~
decisions, and in~luence efficiency deci~ions. They are supplied only by the user and never by the compilation system 60 and 3re referred to ~s We~tra~
par~meters in the preceding ~eotions. Thos~
parameters are liste~ below.

~ILESI2E - this i~ ~ user suppliea v~cto~ for the ~ile ~ize of the following tile Pamily. This vector is only vali~ for that tile famil~, anD aoes not apply to any ~ubse~uent loopnest~. Possible ~alues .

:
i ` ;

-~9- 2r~s~l~

are n ~where n ~ a numerical value greater than 0), ~ ~where ~ is a vari~ble), or ~ (a ~ymbol indicating that the ~le should take the entir~
iteration 6pace in that dimension. The vector mu~t supply values for ~11 tiled ~ndices. ~ynt~ follow~
the general compilatio~ sy~tem 60 tlling paramster ~ynta~.

~TRATEoY - a u~er directive on what tiling ~trategy to use on the following tile family. This value i~ only v~lid for that tile family. Possible v~lues ~re GRAB or MOD. ~ynt~x follows the generzl the compilation sy~tem 60 tiling parameter synta~.

4.6 E~ecution Of a Tiled Loopnest When a new tile-family i~ about to start execution, Runtime ~ibr~ry 66 decides on ~
wor~-plan. This decision i~ basea upon the factors including affinîty, dependencyO ~ata-locality.
The work-plan is ~ ~ollection of de~isions which will determine the parallel e~ecution of the tile-family: allocating threads, partitioning o the iteration space, and choosing ~ tiling ~trategy.
Choosing the right work-plan, ~nd - ~n particular -choosing the right strategy, has a major effect on performance. In principle, the wor~-plan i~ chosen when a new tile~family starts. If this tile-family belongs to an affinity region, the work-plan is based upon a ~template~ o the work-plan o~ the affinity region; this will ~e discusseB later.

5~~ 2~3~i 4.7 Allocation of Threa~
On tiling, ~untiMe Library 66 consiaer~ the amount of resources available, an~ or each tile family, uses n threads, where n i~ les~ o~ equal to the number of processor6 available to thi~ proqz~m.
Th~ default will be ~o use the maximum number of processors available; i4 the tile family ~
~tructured so th~t ~t i8 not worth using ~he maYimum number, a ~maller number of th~eads will ~e chosen.
This algorithm i8 u6e~ regardless o nesting.
There are ~ome difference ~n the allocation of threads according to the tiling ~trategy which is used ~tiling strategy is described below):
When using the GRAB strategy, Runtime Library 66 let~ the ~cheduler handle all thread-processor bindings, load balancing, affinity, etc.
When usinq the MODULO ~nd WAVEFRONT
strategy, Runtime Library 66 would li~e to assume that the threa~->processor binding i~ con.tant.
Runtime Library 66 constructs and assigns tiles accordingly. Thi~ binding assumption woul~ make it useful to let the scheduler ~now that ~untime Library 66 would like higher thread->processor ~tability on these kinds of threads.
These rules are also ~ollowed for nested parallel structures.

4.8 Tile size and ~hape A tile i8 defined ~n number of iter~tion ~pace, hence, the tile-size is define~ in terms of the iteration ~pace. The tile-~ize vector speeif;es the number of iterations in each ~imen~on of the tile-family~ For e3ample, a loop -51- ~78~l 3 e 1~ 100 do ~ ~ 1, 200 which i~ d$vided into tiles may have ~ tile-si~
vector of ~ 32~, henc~ - the tile-~ize ig 163~2 ~
512 iteration. Note that the tile-s~ze need not f~t directly into the it~ration ~pace. Runtime ~ibrary 66 will ~trim~ the edges of tiles which over~low outside the iteration ~pac~.
The tile-~ize is detsrmined once at th~
begi~ning of the e~ecut;on of ~ tile-family, ~n~
remains constant during the execution o~ that ~ile-family. However, the same tile-family ran be executed more than once in the program, and the tile s;ze can be different each time - due So, for example, different bounds of the loops.
Til~ shapes mu~t be chosen with two objectives in mind~ maximizing parelleli m snd m~king good use of the allc~che memory ~ystem. The following discuesion wPaves together considerationæ
of ~ependency and ~ubpage access to achieve the two goals.
A tile is a rectangular n-dimension~l parallelipiped with a dimension corresponding to each of the dimensions of the iteration ~pa~e. The til@
shape guestion is this -- how long should the tile bP
in each dimension~ The ~asic idea ~ to ~str~tch~
~he tiles in the di~e~tion of array reference~ and to astretch~ the tiles in the directio~ of depe~dency~
The fir~t point will avoi~ contention between two or more ~ threads for the ~ame ubpage.
The ~econd point will minimize ~ynchronization.
Tiles will be multiplies of subpages, or two subpages.

; ~ . , .
-, ~

;
.

( ) ~
-52~ 2 ~ 7 ~ t~ ~ r3 ~ he final deci~ion ~bout tile-siz~ 18 ~
compromise between contraDicting con~iderations: on the one hana, we want to have big enough til~, to ju~ify the unavo~dable overhea~ o~ ~t~rting each tile. On the othet hanfl, we want to h~ve many tile~
in order ~o optimize the load bal~nca. A~ter the ~hape has been declde~O we ~etermine the actu~l ~ize by loo~ing at the amount of work to b~ done, the number of availa~le processor O etc. If th~ ~ileæ
are too ~mall, we ~htretch~ them.

4.9 Tiling ~trategy ~ he tiling ~trategy ~8 the method usea to divide the work among the pthreads so that all the tiles which comprise the tile-family will be e~ecutea correctly and efficiently.
L;ke tile-size, the tiling strategy i5 determined at ~he be~inning o~ ~xecution of a new tile-family. Runtime Li~rary 66 use~ ~
self-scheduling mechani~m, which means that after an initial setup done by the lea~r, ~ach threa~ can ~ind its ehu~k of work by itself. Hence, the strategy is expressed in terms of what a thread mu~t do to find out what it needs to ~o ~e~t.
There are two funda~ental princ~ples regarding the Runtime Li~rary 66 strategies motivated by a desire ~or design elegance and low runtime overhead. The first ;~ e~actness - the strategy defines e~actly how thP ne~t tile is selected i~ each ~ituation. The idea is to leave as little calculation as possible to the ~oint when a thread nee~s to get the ne~t tile, Dnd therefore minimize o~erhead in the ~untime processing to get the ne~t tile. Once a ;

' ` . ( ) 'I

~trategy is chosen, choosing a new tile should be very fast operation.
The secon~ prlnciple is pr~gress~on - the strateg~es are ~ructure~ ~o execuS~on starts rom a known point in the ~terat~on space an~ proceeds ~n known direction. The mot~vatisn i~ to avoi~ the nee~
for complicated ~ata-~tructure~ that rec~ra a~
remember which par. o~ the iteration space 18 has been covered.
Runtime ~ibrary 66 consider~ the following fac~ors when deciding upon a tilinq ~trategy:

1~ Esistence of data dependencies. Data dependencies create or~ering requirements between the tiles, whieh necessitates synchronization between tiles. In the extreme case, data dependency may cause a tile family to esecute serially, ~ecaus~
the tiles of the tile family are in the ~ame ord~rlng relationship. In other cases, some degree of paralleli~ation is ~v~ ble in the tile amily because e~ch tile ~8 indepen~en~ o ~ome number of other tiles.
2) Specification of strate~y by the user.
The user can specify a ~trate~y by æetting the PLSTRATEGY environment variable, u ing the SET
directive, or passing a strategy value as a parameter to the TILE or AFFINITY ~EGION directi~es . 3) ~peci~ic~tion of tile size by user. If the user ~pecifies that a ~imension with ordQring requirements ~hould be tiled, the til~ng ~trategy may be reguire~ to handle orderinq, Runtime ~ibrary 66'~ tiling strategy decision table is as follows:

, . . . .

.

, ~ . .

don ' t care Usor Nwnb~r o~ Us0r U~0r ~pec 'o~l Cho~x~
~pec'e~l ~nd~co~l Sp~c'0~ t~ lz~ cut3 l~tr~togy StrAtogy w/Ord~r t~ o n orfl~ra~ -- ln~l¢o~ -F~S~ O 8 :c MODULO/~L~CE
FALS~ 1 FALSL ~ MODULO/SL~C~
~A~SIS 1 TRU~ O MODULO/SLIC15 FAr.SE 1 SRU~ 1 WAVEFRONS
F)~SE >-~2 FALSl: a- W~VBFRO~
FALSE ~i.2 TRU~ 9 MOD~O~LIC~ ~q FALS~ ~ ~2 TRU~ 1-2 WAVEFR0~7 r FALS~ )-c2 TRU~ ~2 Error rep~rte~
TRUE O B ~ IJIser ~pecl~
~trategy TRUE 1 FALS~ s U~er speclf 3tr3~0gy TRUE 1 TRU~ O U~r spaclfl~
~tr~teqy TRUE 1 TRUE 1 Err~r lf user ctrat3gy! n WAVEFRONT
TRUE >-2 FAI.SE ~ ~rror l user Atr~tegy!
WAVEFRONS
TRUE ~2 TRUE O Us~r cpec~f~
~tra~ogy TRUE ~2 SRU~ 1-2 ~rror ~ u~r ~tr~tegy!
~AvE~Ro~aT
TRUE )~2 ~RUE >2 ~rr~r reported Note that the fact that a tile fam;ly is tiled and that tiles-are distributed to multiple threads ~oes not mandate paralleli~m. A tile family that has ordering requirements may be tiled but ~till esecute seriall~, ~ue to the synchronization required by the ordering information. This ~ituation may be optimal a~ the lack o paralleli~ation is overcome by the advantage of maintaini~g data ~ffinity.

. , -.
-55~ 3~

The follow~ng i ~ description o Runtime ~ibrary 66 strategie~.

~ hi~ ~tr~te~y ~imply ~ivi~es the t~le family iteration ~pa~e into D tile where n ig equal to the number o pthread par~ic~pating in the construct and asslgn~ a t~le to each ~hrea~. Thl~ i8 the ~efault strategy or tile familie~ no~ enclo~e~ within a~
affinity region and ~ ~esigned to m~n~m~ze til~ng overhead an~ the possibility of ~ata csnt~ntlon ~t tile boundaries.

~ODULO s~te~y ~ his strategy distributes tiles evenly throughout thP ;teration space to th~ thread ~roup.
Assume that thread~ an~ t~les are numbere~ starting from 0. The tile~ whic~ will be egecute~ by a given thread are those such that tile-number MODULO number-of-allocated-threads ~ thread~

When eæpressed in terms of a sel~-~cheduling strategy, it means that a thread whose ~hread-id is P
will esecute the following tiles:

- ~irst ~ile is tile whose number i~ æame as the thread-id - ne~t tile is previou~ tile ~ (nu~ber of participatins threaas) This strategy i~ ~ semi-stati~ one. It is dynamic in terms of the number of processors which .
~ .

-. .

--56- 2~783~ ~

are aY~;lable when the execu~on of the tile-f~mily ~tarted, but cannot adjust to a change of the availability of proce~or~ during the e~ecution of the tlle-family an~ to an unbalance~ loa~.
The major advant~ges o~ thi~ ~trateqy ~re that no ~ynchroniæation between tiles ~ regu~re~.
Al~o, the ~et iteration space->threa~ mapp~ng ~
designed to handle ~ta aff~nity, especially when used with affin~ty region~. Further, ~he ~mo~ulo~
distribution ~y~tem of the iterat~on->thread mapping, is designed to optimize loa~ bal~nc~ng wlth~n affinity region , where each ~le family within the affinity region may cover a different part of the it¢ration ~pace. Because of this ~istribution scheme, the percen~age of w~r~ allocated to each thread is not dependent on the iteration ~pace uæed by a single tile family.
Thi~ is the default ~rategy ~o~ tile families enclosed within affinity regions.

WAVEFRONT s~rat~qy This strategy is ~esigned ~ e~ecute tile families with data depende~cies correctly 3nd with masimum parallelization.
With this strategy, tiles are executed in a ~avefront~ pattern on a two dimensional plane.
Runtime Library 66 c~ooses two indices out of the list of tiled i~dices that have ordering requirement~
to form the 2~ wave~ront pl~ne of the iteration space. One inde~ i~ de~ignated the ~column~ ~nde~
while the other ~ design~t~d the ~ubtilen lnde~.
~he columns of the 2D plan~ are allocate~ ln a mo~ulo fashion to the members of the executing threa~
team. Each column is made up of a number of tiles, .; . .-:
~ .

2 ~ 3. ~
Each ~le ha~ jacent, dominant neighboring tile that ~t i~ depen~ent upon, an~
cannot begin execution until that neighbor ~
~inished, ~ e dominant ne~ghbor will be in the column to the right or left, ~epen~ing on the orderinq requirements o~ the column index. Th~
dominan~ tlle will be on the ~ame sub~ile inde~
value. E~ecution o~ the tile ~amily begi~s from one of the four cornerg of the 2D plane, ~ependin~ on the orderin~ of the ~olumn an~ ~ubtil~ indez.
Thi~ ~trategy ~l~o ~ttempt~ to ~istr~bute tiles evenly throughout the iterat~on space to the executing thread team, and is compatible with the use of the af f inity region ~irective.
Runtime ~brary 66 hanales an iteration space with ordering requirements that haæ more or less ~han 2 inae~ dimensions in a num~er o~ ways. If the iteration ~pace has only one indes, and it has an ordering requirement, the work must be done ~erially, and Runtime ~ibrary ~6 attempts to create one tile.
If the user forces tiling by ~pecifying t~le ~ize, Runtime Library 66 e~ecutes the tile ~amily in the 2D
wavefront pattern, where the ~ubtile inde~ is 1, If the iteration space has more than two dimensions, but only two indices have ordering requirements, Runtime Library 66 processes the tile family as 8 series o 2D planes, where the wavefront ~trategy can be conducted on ~ach plane independently. If there are more than two dimenæionæ, and more than two indices that have ordering requirements, Runtime Library 66 will not tile the aaditional ordered indices~ and will create ~chunkier~ tiles t~ esecuts in the 2D wavefront strategy.

, . ' 2~3~3 User ~peci~led t~le ~ize~ that require t~ling ~n more ~han ~wo or~ere~ indice~ are refu~ea.

Thi~ strategy i~sues tile~ on ~ ~ir~t-come, first-~erve basi~ to the threa~ group. Each threa~
must obtain a common lock ~o ~ccess a tile for execution. This strategy is ~esigne~ to loaa balance between the ~hr~aas of ~he threa~ group.
~ ote that thi~ strategy does not consi~er data af~inity consideration~ when a~s~gning tile~ to processor~, an~ th~ it would not be 2 good ~trategy to choose in conjunction with affinity region~.

4.10 Relative Advantages of the Strategies The semi-~tatic MODULO tiling- trategy has t~o main advantages: ~t is eaRy to maintain data-affinity; and it min mizes synchronization overhead. On the other hand, t~le to threafl assignments are tatic a~d will not 3djust to unbalanced loa~ ~uring e~ecut~on time.
The SLICE tiling ~trate~y is ~lso ~emi static and has the most minimal tili~ overhea~, but makes no attempt to maintain Bata af~inity a~ro~
tile families.
The GRAB tiling strat~gy, is dynamic and will keep each thread busy as much as possible, but will probably cause data migration to occur more fxequently. In addition to ~ffinity loss problems is the additional ~ynchronization overhead ~rom the lock~nq reguired to access a new tile.
The WAVEFRONT tiling strategy is ~uir~d when tile famil;es have ordering re~uirements caused by data dependencies. Data affinity ~s maintained, , . j , , ., . ~ .

( _59_ ~7,~3 a~ wlth ~he modulo strategy, but th~ tile-t~le ~ynchronization require~ creat~ additional overheaa.

4.11 Work-plan - E~amples The followin~ are few e~ample~ of t~e work-plan chosen by Run~ime Li~rary 66 to e~ecut~
tile~-loop~. Det~ of why ~ parti~ular work-plan was chosen are not~ given; rather, thl~ a~tempt~ to give the flavor o~ the behavior of various wor~-plan~.

~ample 1 A 2D iteration pace i~ tiled ~n 2D. ~he number of available processors is ~maller than the number of columns. There i no dependency.
The work-plan is depicted in Figure 9A.
There, num~er of processors: N
~ile-~ize : a whole column strategy : modulo FxampL~ 2 The ~ame a~ above, only this time there is a dependency in bvth dire~tion~: the ~trategy used i~
the modulo ~tr2tegy with dependency (wave front).
Note that this strategy is data-affinit~ ~fficient when used in the same affinity ~roup with the normal modulo strategy whi~h i~ shown ~ th~ previou~
example.
The work-plan is depicted in Fi~ure 9B.
There, "
. -, , , .~' :

- ( ` i -6~- 2~33 number of processors: I~
tile-slze : chunlc~ o~ column strategy : wavef ronl;

E~ample 3 Data ~ff;nity ~ o'c an ~ssue, load bal~nce is important. There ~ no dependlency. The ~b poirlt in both ~ an~ ~ ~n~es.
The worls-~plan ~8 depi~ted ln ~igure 9C.
There, number of processor: N
tile-~ze : columns ~trategy : GRAB

Note that ~ n t}liS e~ampl~ the ~rrows ~hows how the tiles are num~ered. ~his i~ no'c to be confused with order of tiles: the numberin~ Ufit a way to keep ~rack o wha'c bas been eæecuted. The way to number ~he tiles ~B arbitrary, and doe~ nor reflect any p~rticular order in ~h~ch the tiles nee~
to be executed.

5. Affinity Region Runtime Library 66 trie~ to alloca'ce work to ~hreads in a way which will minimize contelltion and data movement.
The creat~on of af f inity re~ion ~ ~ a way to coordinate th~ tiling work-plan decision~ made ~or group of tile familie~. The following a~sumptions are made to support thi~ goal:

.
,.. ~ .~ ~ , :

~, . I
-61- ~ ~ 7 ~ t3 .L ~

1. Data mo~ement i expensive.
2. The ti~e families within the af~inity region reference the ~ame data.
3. The ~ile f3milie~ within the affinity region tend ~o hav~ the ~ame ~at~ ~pace eiteratiOn ~pace mapp~ng.
4. The ~ats ~pace is ~ ne~ on subpa~e boundaries, an~ tiles can be constructea that w~ll avoid aata contention.

In earlier ~ectSons~ the work-plan ~eci~ion was described as a two-part decision, wi~h ~
tile-thread mapping component an~ a tile size component. When affinity region are declared~ these de~isions are made across the whole set of tile families in ~he affinity region.
The tile thread mapping is maintained across tile families ~Q that the ~ame thread (and hopefully, ~ame processor) works on ~ach tile, maintaining ~ata locality. The affinity region directive i~ only effective with ~ome tiling ~trategies. For e~ample, the grab ~iling strategy does not allow the maintenance o~ a tile-thread mapping.
The tile size ~eci~ion i~ alæo ~ade ~y considering the iteration spa~e of all th~ tile ~amilies in the affinity re~ion, r~ther than the iteratio~ space of a single tile family. As a result, ~11 tile families ~hare the ~ame tile ~ize, making it possible to maintain a tile~thread mapping across tile families.
The tile wor~-plan decision may reflect f~ctors that affect only ~ome of the tile f a~ s, but that are extended to ~11 o~ the tile families.
For e~ample, a data dependency may e~ist on a ;

' ' , .
. . .;
.
.. ~ . , .

f, t 2~7~31 particular index ~n ~ome of the tile famil~es. ~he effect the ~ependency ha~ on ~ tilin~ 8trate~y applie~ to all tile amilics in khe affin~ty region.
Note that the cr~atlon o aff~nity region i~
purely an ef~ciency i6~ue.

5.1 Recognizing ~n Affinity Region Affinity reg~ons may be ~eclar~ by the u~er, or ~dentif;e~ by the compilation syst~m 60.

5.2 Common Inde~ ~et It mus~ be possible for all the loopnest8 in an affinity region to be tiled on the same ~et of indices. This i~ the ~common index set~ rule. This is necessary because the affinity orientea strategies need an iteration->processor mapping function which determines which processor executes the tile. ~f the number of tiled indices varies loop~est to loopnest, this mapping will fail. This does not mean that every tile family hav~ identical tiled indices, ~u~t that there is an intersection of indices among the different loop nests.

6. Performance issues 6.1 Overhead There is overhead in the following aspects of operation: ~1) a~in~ty re~ion (~reate the template); (2) ~tart of tile family ~create the MCB);
~3) choose the ne2t tile (by each thre~

6.2 Runtime Libr~xy 66 Deci~ion ~aking ~ untime ~ibrary 66 is ~esigned wi~h efficiency 3s the major consi~eration. One important .
..

:: ' . , - , ' :' ~' ,, ~ :
' :', . ' , ' . ' .:' . :

. . .
' , ~ I

concept i~ the propagation of ~ec~sions to the eaxliest possible time. Obviously, it i~ more eff;cient to take 3 ~eci~ion at compile time r~ther than ~t runtime. Consequently, it i8 more efficlent to ta~e a decision when a tile-~amily ~tart~
execution, rather than at the beginning of e~ecution of each tile.
Runtlme L~br~ry 66'~ method i5 to t~e a decision as ~oon as poxsible, based upon all the i~formation ~vail~bl~. Once the ~ec~&ion has been taken, Runtime Library 66 forgets the reasons. The goal i~ that by the t~me a particular thread ~eeas to ~ind the next tile to e~ecu~e, it will be a very simple operat~on, typically involvin~ a couple of simple comparisons ana ad~itions.
At runtime, when ~untime Library 66 ~tart~
to execute a tile ~amil~, all the information about this loopnest is already known: whether or not there ~s a dependency between the tiles, how many processors are ava~l~ble, the ~i~e of the loop (bounds) etc. Based upon that, Runtime Library 66 can decide upon a tiling strategy.
~ here are many ~actoræ which ;nfluence the choice of tilin~ ~trategi~s. ~ome of them are known at compile time, e4~., dependency, or the amount of work in the loop. Other~ can be known at ~ompile time - e.g., the number o~ a~ailable processor~.
Some of the information i~ available at runtime in the general case, but ~n practi~e it i~ very often either known at compile time tor known to the user who can provide it via directives~.
The system has ~ome decisiun~points, where decisions can be taken. These are at compile time, by compilation 8ystem 60. At runtime, by the Runtime .' .
,~
. , .

~ i ~
-64- ~7~31~3 ~ibrary 66, upon starting a new t~le-fam~ly (_pr_execute_tile~ nd when looking ~or ne~t tile to ~o (_pr_tile_ne~t).
The following i~ a li8t of factor~ which may ~nfluence Run~ime ~lbr~ry 66'~ i8~0n~. ~he or~r o these factor~, a8 ~resente~0 i8 es~ent~ally ~n~om.

- s~ze of loop ~boun~) - ~atic amount of work ~ one lteration ~ dynam~c amount o~ wor~ ln one iteration - dependency bet~een tiles - mapping of iteration~ to ~at~
- number of proc~
- data ~ffinity (~sb) - history (affinity region) - resource (processors) availability ~load balance) - [important array~

7, Runtime Library 66 Architecture 7.1 ~lock diagram Figure 10 is a high level bloc~-diagram of Runtime ~ibrary 66. It describes the main routines and the flow o~ control. The function o these routines is describe~ below.

_pr_program_master init is called once at the Yery beginnin~ of e~ecution a program. It reads Runtime L;brary 66'~ environment ~ariable to ~et u5er's configuration ~uch as the ~esire~ number of processors, or e~ample~. Does ~eneral initialization of Ru~t~me ~ibrary 66.

.

-~5~ 2 ~7 ~3 _pr_slave i~ th~ ~ma~n~ sf the ~l~ves. There are many of them running ~t the ~ame ~lme - thi~ i~
ba~ically an idle loop. When the Lea~er "~lo~es the ~witch" ~t will call the tile-~ener~tor ~pr tile_gen) and when $t fini~he~ ~t~ work ~t returns to pr ~l~ve and will stay in the ldle loop until the ne~t t~me the Leader lets lt go.

_pr_e~ecute_tiles~ r~ to e~ecute a tlle ~mlly:
allocate threads ~either create them or use e~i~ting ones3. Also, initi~lize the e~ecut~on ~ata for a par~icular t~led loop. Thi~ involves de~iding upon tiling-strat0gy, an~ o~her initialization ~tuffo Its output is put in a central datastructure called the MCB (Master Control Block) which ~ visibl~ to ~11 the threads. After this is done, the Leaaer is ~ot needed any more as ~uch; the leader puts the ~laves to work, and joins in, i.e., calls _pr_tile_gen.

Upon returning rom pr~tile_gen, i. resumes its responsibilities as the Leader: it wait~ until all the slaYes ~;nish th~ir wor~, ~o the necessary cleanup and returns to its ~aller.

The MCB holds the information regarding the tiles. The first section holds information about the tile-family. The second æection holds information about each ~imension of the tile ~amil~ ~order) -hence, it contains arrays(d) of fields, wh~re ~ , l.,order. A typical ~alue of ~or~er~ is small (4 is enou~h, 8 is more than enough).
In the followin~ tablei each ~ield is pre~i~ed with a l~tter which indicate~ its characteristics xe~arding to when it gets its value:

. .

. , : .

-66- 2~ 3 1~J

ac" stan~s for Compile time; "I~ ~tand~ for Initialization o~ the MCB, i.e., once for e3ich execution of ~ tile family.

MCB - Mas~r Control Bloek ~iel~ ~-c~criL~iQ~

Family ~ection:
I tile_efp the ~R (~rame Poin~er~ of the parent routine C tile_~tart point~ to the text of the tile ~in fac~, P of the ~ile~
C order as ~pecified by tile-family-parameters C affinity as spe~iied by tile-amily-parameters C dependency as ~peci~ied by tile-family-parameter~
C ~ode_~ize as ~omputed by the ~ompiler (back-end) I tile_strate~ ~ode describin~ ~tr~tegy to cover the tile family.

Order Se~tion:
1 ¦tile_size (d) I num~er o iterat~ons in this dim~nxion C~I loop-low-bound(~) as spe~if ied by tile-family-parameters C~I loop-high-bound~d) ~s ~peci~ied by tile-f~mily-parameter~
C~I loop-~tri~e(~) a~ Bpecifie~ by tile-family-parameterfi : . ' ' : , ,, , - , ,; ~

-~ !' ) -67- 2 0 7 8 ~ 1 ,s pr_tile_gen the routine wh~ch does the actual call to the task-subrout~nes with the appropriate arguments, _pr_tile_next i8 ~he routine wh~ch æelect the nezt ~ile to be e~ecute~ ~y th~ thread. ~t returns ~
li~t o ~nte~er8 which are the li8t o~ low boun~ of a tile, i.e., ~he corner of the tile. ~inc0 the tile-~ize is fi~efl; th~s define~ the tile.
_pr_tile_next consult~ the MCB, ~n~ thi~ $ where the tiling-~trstegy i8 a~tua~ly take~ an ~ffect: ~h~
basic control ~ruc~ure o~ _pr_tile_ne~ is a witch ~tatement ((ac referred to in the C programming language, or a ca~e statement as referre~ to in Pascal) according to the workplan.

_pr_execute_parallel sections allocates sla~es to execute the sections, and gives each of them a ~econd block.

_pr end is executed once when the program ~inished.
In addition to general cleanup, it produces statistic and report~.

RuntLme ~i~rary - Internal Ar~hit~ct~F~

1. Overview ~ he following sections describe in grea~er ~etail the Runt~me ~ibrary 66 and, more particularly, the internal structure of a preferred embodiment thereof.

2. The Programming Model The Runtime Library 66 runtime environment allows ~rograms to run in parallel on the digital - .
.: . . ~ . , : ' ~7~}~3 ~ata proc~sor 10. Par~llel construct~ hanaled ~re tiles, parallel section~, or parallel region~.
Paralleli~m is implemen~e~ in ~11 constructs with the u~e of threaas. ~11 co~e outsi~ the parallel con~truct is executed ~erially ~y one of the program threa~s. All ~erial co~e i~ e~ecute~ by the ~ame, ~master~ program thread.

2.1 Implementation of Parallelism Programs ou~put ~y compilation ~y~tem Z0 ~re linked with the Runtime ~ibr~ry 66 to e~ecute these constructs in parallel. ~uch a program will be called aPresto proqram.X Program~ that contain parallel ~onstruct~ and are compiled with ~ ~no parallel runtime swit h~ will treat the parallel construct directives as comment~ and will e~ecute serially, with no ~untime ~ibrary 66 sv2rhead.
Each parallel construct is e~ecute~ hy a group of threads, calle~ a team, which ha~ one or more member~. ~ach ~eam has one member who i~
designated a team leader, for purpo.es of synchronization. Whçn the pro~ram start~, there is one thread that i~ designatea the program leader.
Runtime Library 66 will manage thread teams transparently to the u~er, but the u~er ca~ control teams explicitly.
A ~hread may have one or more code ~egments to e~ecute. Runtime ~ibrary 66 implements a number of strategie~ the parallel construct i~ a tile family, each thread has at least o~e, and more ~ypically, many tiles to e~ecute. In the parallel section and parallel re~ion constructs, each thxead has only one code ~egment ~o execute.

- ( l ~9- 2~7~3~.~

2.2 Transit~on Between Serlal ~n~
Parallel Par~ of the Progr~m All serial portion~ of the pro~ram are e~ecute~ by a singl~ threa~, th~ program magtez.
When a par~llel construc~ ~ encounter~ group of thread~ i~ assi~nefl to the con~truct. The beqinni~
o the ~onstruct ~ a synchron~zatlon point ~or the group o~ threads, while ~he end o~ the constru~t i~ a synchronizat~on point ~or the ~roup ma~ter. Each group member beglns the parallel por~ion as ~oon ~
the thread Qroup i~ assigneB and the group master has finished the preceding serial co~e the group members must wait on the group master bu~ not on the member~.
At the end, the group master ~oes not execute the code followinq ~he parallel por~ion until all group members hav~ inished; the group master must wait on all members~ During the seri~l portions of the program, ~11 threads eYcept the program master are idle.
The ~ynchronization point at the end of the construct is plac~ at what ~s the ~nown ~oint of code dependency.
Parallel constructi may b~ neste~. The general implementation ~pproach i~ the ~ame at each level; ~or each new construct, there is e local group of threads, with a local master.

.3 Resource Allocation Amon~ Parallel Constructs Runtime Librar~ 66 delegates much oP the duty of l~ad balancin~ ~moDg processor~ to the operating system ~OS~) ~chedul~r. ~untime Library 66 will inPluence resource allocation by ~hoosin~ the number of threads to use for a parallel construct.

,....... ~ , .

.

( ~ ) -70- 2 ~ 7 ~C~

The O~ wlll manage re~ource ~lloc~tion among the threads. There may ~e more cr le~ thread~ than available processor~ at ~ny moment.

2.3.1 ~ n~
On t~ling, ~untime Library 66 cons~ders the amount of resource~ available, ~na for ~ach t~l~
family, uses n threads, where n ~ less or egu~l to the num~er of proces~ors avail~bl~ to thi~ progr~m.
The default i~ to u~e the ma~imum number o~
processor~ available; i~ the ~ile family i~
structure~ 80 that it i~ not worth us~ng ~hs ma~imum number, a smaller number of threads will be chosen.
This algorithm is u~ed regar~less of ne~ting.
When using the GR~B strategy, Runtime Library 66 let~ the sche~uler handle all thread-processor bindings ~nd afanity consideration~.
When using the ~ODULO, ~YEF~ONT, or 8LICE
strategy, Runtime ~ibrary 66 attempt~ to ~ssume that the thread->processor bin~iny i8 ~on~tant. ~untime Library 66 ~onstruc~ ~na as~ign~ t~l~s ~ccordi~gly.
This binding assumption makes it useful to let the scheduler know that Runt;me ~ibrary 6S requires higher thread->processor ~tability on the~e kinds of threads.

3~ INTERFACES

3.1 Runtime Library 66~Compilation ~ystem 60 The preprocessor 60~ tr~nsforms the co~e enclo~ed by a parallel construct into a le~ical ~ubroutine. Creating the nsw subr~utine allowæ
Runtime ~brDr~ 66 th~ead~ to run the ~ame coae in parallel. The compilation ~ystem 60 al~o generates .

.
~ ', !

7~ 3~3 call~ to runtime xoutines which wiil ~e~ up an~ cau~e the parallel execut~on.
In the tile con~truct, the le~ical subrout~ne i8 compo~e~ o~ the ~o~e ~ncloge~ by the TILE ~irective~, One subroutin~ ~8 create~, an~ 18 calle~ by each o the threa~ 9roup member~, with different loop boun~s.
All variable~ vi~ible to the co~e ~hat becomes the ~caller~ of the le~ical ~ubrout~ne mu~t be available ~o ~he ~ubroutine i~elf. ~owever, because th~ subroutine i8 calle~ by a ~untime L~hrary 66 routine, ~ mechani~m is neede~ to patch the scoping. Among the in~ormatio~ pas~e~ from the compilation system 60 to Runt~me ~ibrary 66 i~ a pointer to the le~ical su~zoutine an~ the efp ~enclosinq ~rame pointer3 of the calling co~e. Thi enables Runtime Library 66 threaas to call the subroutine with the appropriate ~coping.

3.1.1 General Interf~ce Runtime Li~rary 66 ~nd the compilation system 60 use the following value~ or certain interface arguments:

- boolean Yalues: 0-FALSE, l=TR~E

- strate~y values~ RAB, 2-MODULO, 3=WAVEFRONT, 4-SLICE, -l~not ~pecifi~d - in general, -1 ~ not ~pecified - sizes: 0~entire iteration space, n~value of n .

3.1.2 Start of Affin~ty Regions When the compilation ~y~tem 60 encounters ~n AFFINITY REGION user directive or when lt reco9nizeB
a potentlal affinity reglon, it calls_pr_Rtsrt affinity wlth ~nformation to init~lize the 8fPin~ty region wor~plan (~rom the compilation ~stem 60 ~o region workplan3. Th;s only transfer~ 3ffinity relate~ inormation ~rom the compil~tion sy~tem 60 to Runtime Library 56'~ ~ntern~l data ~tructure~, ~n~
does nQt trigger ~ny par~llel esecution. Note that this interface applies to both le~ically separ~te tile ~amilie~ and ~ile families th~t ar~ neste~
within an enclosing loop and that Runtime ~ibrary 66 ~oe~ not differentiate between the two.
The user directive is:

c*ksr~ AFFINITY REGIO~ ( <index and bvunds~
ttilesize~<s~ze_l~ t~
tstrat~gy~G~AB, MOD,WAVE,~L~CE~3, tnumthreads~tvalue> ¦ teamia~team id value>], aintercall~tO,11 torder~<list~) This results in the compiler ma~ing a call to:

void pr_~tart a f f inity( num_indices, code_size, numthreads, ~trategy, tile size_~pec, teamid, int~rcall, order_num, ~ependency ~ector, --a~ many #'~ order_num :

: : j . . .

low bound values, --as snany #'~ 8S nu~Ç~n~
high boun~ value~ as many #'e as num_indices afflnity values, --as many ~'8 ~5 nUm~illaiCeS
tile_ ~z~_vals~ many #'~ a~ num_indices long num_in~ice~;
long code_giz~;
long numthread~;
long trakegy;
long tile_~ize_spec;
long teamid, long intercall, long order_num:
va_dcl;
{
}

Arguments are:

: Num_ln~ice~ (lon~): Thi~ i8 the number of inaices liste~ by the user. The compilation system 60 check~ that th~ u~r sp~cifiea ind~ces ar~ a subset of th~ inter~ection o~ all indices used by included t~le fam~lies.
I~ there 1~ no common set of indices The compilation system 60 will produce a warning message to the user that the a~finity region was not possible, and will not issue the call to _pr_start affinity.
Code_~ize (lony3: Average code size across all the tile families covered by the afinity region.
~ umthreads (long)~ not ~pecified, O..n - value pa~ed by user. Number of threads to be used by the tile families within this affinity reQion.

, .

. . .

. .
. , :
.

~ ) 2 ~ ~ ~ t) S ,~

~ trategy (long): 5trategy to be used by all tile families within th~ affinity regisn. Scope only extends to affi~ity re~ion. ~ ot specifie~
GRAB, 2 ~ MODULO.
Tile_~lze_6pec (lsng): O i t~le ~iz~ not passed by user, 1 ~f passed by user.
Team~ (long)~ f not ~pecifie~, otherwise the team ia passea by the user.
Interc~ 1 lf not speciie~, other O or 1.
Order_num ~long): Number of ~ep~ndency vector values providea. No~e that the u~ar may speci~y some dependency values.
Dependency_vector lorder_~um] ~longt]):
Depen~ency values ~cross ~11 tile families covered by the affinity region. This is the union o~ ~11 tile ~amily specific dependency v~lues. For ~xample:

Compiler Generates ~ile Or~er_Num, Order ~irectiv9 Y~.~tor c*ksr TIL~ jok~ orders~j,k~ order_num~2, vector (2,3) c~ksr TI~E((i, jo order~ti,j~) order_num~2, vec or-(1,2 c~ksr TILE(j,k3 order_num_O
, because the num_indices~l, order_num=l dependency vector~tl}

Another example.
c*ksr TILE(i,j,k, order ti,k?) order_num~2, vector-{2,3}
c*ksr TILE~i,j,k, order~{i~j~) ord~r_num-2, ~¢ctor~1,2?
c~ksr TILE(i,~,k) o~der_num=0 .
.

;

-75~ 3 ~ ~

num_indices~3 order_num~3 aepen~en~y ~e~tor~1,2,3 .
Dependency value~ are all positive number~, unlike the dependency ~ector pa~ed in the pr~execute ~11. Direc~or ~ lgnor~ u~er specified any dependency value~, ~8~8 tha~ v~lue ~or that inde~, o~herwi~e, compil~t~o~ ~ystem 60 c~lculates the ~alue.
Low_boun~rnum_~n~ices3~1ony[J): the lowe~t low boun~ v~lue or ~ach index, ~cros~ ~he afin~ty region. User ~pecifie~.
. High~bound tnum-ln~ice~ ongl3~: Analogous to low bound, alway~ provided by the u~er.
~ ffinity tnum_indices]~SSB_T~: ~sb valu~
for each indes ~overed by the a~finity.group. ~al~d ssb values for the compilation ~ystem 60 are NON~ and 6P. Sect~on 6.4.1 on ~sb C~lcul~tion ~escribes how ssbs are generate~.
~ ile_6ize_vals[nu~_indice~]~1On~]): Usex specified tile siæe, passea directly from fortr~n directiv2. Compilation ~stem 60 will chec~ that number o~ tile ~ize values match value o num_indices. Tile æize will onl~ ~tay iD effect for this one affinity region. ~ee action on TILE
directive for format o values.

3.1.3 End of Affinity Regions When the compilation system 60 encoun~ers an END AFFI~ITY REGION ~i~ective, or come~ to the end of a compilation system 60-detected a~f~nity region, it calls:

,, '~ ' ' ' , ':
,, ( l i -76- ~37~3~

Voia_pr_~t_affin~ty_off(~
t This ~oe~ not trigger sny parallel execut~on.

3.1.4 ~et Directi~e When the ~omp~lation system 60 encounter~ a SE~ ~irectiv~, ~he appropr~ate Runt~m~ Li~r~ry 66 runt~me paxam~ter6 are chanse~. The compil~tion system 60 checks that thig ~irectlve i~ not u~e~
within a parallel construc~. Thi~ will not tri~ger any parallel e~ecution.
The user directive is:
*ksr*SET([PL_STRA~EGY ~ ~stra~egy ~alue)~, tPL_ INFO 8 ~0~l}~ ~
tPL_NlJM.,.THREADs -- (vzllu~
~PL_STATISTICS . {0,1~, ~PL~LOG n { 0 ~
lPL_VISUAL ~ t0,1~], lPL_~YNC_DEL~Y ~ (value)~, ~PL_MIN_INST_~N_TILE . (value3], ~PL_ONE_~P_LONG ~ (Yalu~
~ TWO_SP~ONG . ~value)~) Compilatio~ ~ystem 60 make~ call to:

voi~_pr_~et (num_variable~, ~ariable~user value pairs,) long ~al;
va_~cl_- (each value iæ a long) .
"

~, ( -77- ~ J~3 ~_ V

Each ~ariable~user value palr i8 A pair of two lony words, The variablE3 V31Ue 'CO pas~ efined in an inclua~ flle '^Runtime Library 66.h~ and ls the ~eyword ~e~ine~ with two underscore~ before th~
nameJ The compilat~on ~ys~em 60 mus'c par~e the variables lto qener~te the vari~ble value~. For e~ample:

#def~ne_P~T~A~EG~r 0 ~*u~e thi~ ~or varlable value if ke~ord was PR_STRATEGY~

The compilation system 60 will pass ~0,1,2,3,43 for the PL_ STRATEGY values, where the ~trategy value~ map the foll~win~ way. ~NONE-O, GRAB~l, MODUL0~2, WAVEFRONT=2, ~LICE=4). These are defined in Runtime Library 66.h. For all other parameter8, the u~er value ~s that v~lue passe~ by the user in the direc~ive.

3.1.5 ~tart o a ~ile Family When the compilat~on ~ystem 60 encounters ~ -TILE dire~tiv~, it calls a Runtime ~ibrary 66 routine that ~et up tiling in~ormation an~ starts parallel execution. The user ~irective is~

C*ksr TILE~<inde~>, (primary set), ttilesize~ ize li~tJ, [~trategy~{GRAB,MOD,WAVE,SLICE~J, [numthreads~cval> ¦ teamid~team_i~>], taff_member~{O,l~J) ~ hi~ result~ in th~ compilation system 60 making a call to:

' , 2~8~1~

v~i~
_pr_execute_tiles (family_nameO
code_size, fr~me_polnter, num_~n~ice~, ~trategy, t~le_size_~pec, teami~, -or~er, thi~_level_ar, xeduction_ptr, psc_ptr, numthrea~, af~_member, dep_vector, --as many #'~ ~ order, low_bound, -as many ~'~ as num_indices, high_bouna, --as many #'~ as num_indices, loop_stride, --as many #'~ as num_indices, affinity, --as many #'s as num_indices, : areg_map, --as many #~ numLindic~s, tile_æixe_vals) --~s many ~'~ as num_indices, int (*family_name)~ *reduction_ptr)() (*psc Dtr) ~);
long code_~ize;
char ~rame_pointer;
lony num_indic~ trate~y, tile_size~pec, teamid, numthread , aff_member;
long order, thi~_level_ar;
~a_dcl - (each value is a long) " ~ :

- ~9 -~7~3~ ~

Argument6 are:

Family_name(int~: pointer to ~ubroutine hol~ing tll~ body.
Cs~e_s~ze(long): numb~r of instruction~ ~n tile ~ody.
Frame_poin~er(char~3: Enclosing ~r~me poin~er of caller for tile body ~ubroutine.
Num_In~ioe~(long): num~er of 1OGP ln~i~es.
~ trategytlo~g3~ ot usor ~pecifiea, l~GRA~, 2~MODULO. Runtime Libr~ry 66 error chec~s that str~tegy i~ a valid value. 8trat2gy ~tays ~n efect for this one til~ family.
Tile_size_spec(long): O ~f tile ~ize not passed by user, 1 if passed by user.
Teamid(long~: -1 if not specifiea, otherwise the team_id passed by the user.
Order (long): number of order ~ector values.
This_level_ar(long): O if this tile family is not le~ically enclosea within an affinity regio~, 1 if thi~ tile family i~ le~ically ~ithi~ an affinity region.
Reduction Ptr~int~3: pointer to ~ubroutine holding code to ~ini~h any reductions within the tile body. I~ this pointer is not null, ea~h thread participating in this tile family e~ecutes this ~all when it has finishe~ its ~hare of w~rk. If thiæ
point i~ null, there ~s no effect.
Psc_ptr~int~): pointer to ~ubroutine or runtime support of privately ~hared common. If this pointer i~ not NULL, the ~ollowin9 four ~teps are taken:
1) the team leader will call pr_psc init_mwu, i :.

~ ` `.

- - ~o -2 0 7 ~
2) each threa~ will call_pr_psc_use_mwu w~th _sc_ptr ~s an argument, 3) then each thre~d call~_pt~psc_helper le~ve, an~
4~ the master thr~a~ c~ _pr p~c ma~ter_leave.

Numthreads~long3~ not 6peci~ie~, O..n~value passe~ by u~er. ~umber of threads to be used by the tile ~m~ly.
A~f member(lon~ not ~pecif~e~, O~u~er doesn't want this ~amily to be includea ~n the enclosin~ affinity regisn, l~t~le family should be e~ecuted within the af~inity region.
Dep_vector(longt~3: Value~ of indices th~t have depe~dencies. Yector values are negative if the dependency is backwar~, positive i~ the dependency ~
forwar~. For e~ample, ~ Yector value o~ -3 means the thir~ loop ind~, counting from 1, has a backwar~
dependency.
Low-bounfl(long[~): Low boun~ values ~or all loop indices.
Hi~h_bound(longt~: High bound values for all loop indices.
Loop_stride(lon~t~): Loop striae values for all Ioop indices.
Af~;nitytnum_~ndice~(SSB_T): ~sb Yalue~
for each inde~ covered by the tile family. Val~ s6b values ~or the compilation ~ystem 60 ~re NO~E and 8P. Section 6.4.1 on ~sb calcul~tion de~cribeæ how ssbs are generated.
Areg_map~num_indices~(lOn9~3): Only valid if this tile family i~ in the middle of an af~inity region declared by the user or identified by the ;
.
., , '" ,' , :' ' . . :
' ' ' ' ~ , -81- 2~97831 compllati~n ~ystem 60. For ~ach loop index o the tile family~ th l~op inde~ not usea for the affinity region, n: i~ this l~op in~e~ correspond~
to the nth loop in~e~ o~ the af f ~ nity region.

For e:~ample:

C*k8r ~TART AFFINITY REGION
C*~r* TI~E [~,~ 7~ - mapp~ng ~8 ~O~lo~l~
C~ksr* END TI~E
C~ksr~ TILE(;) - mappin~ i8 ~1}
C~ksr~ E~D TILE
C*ksr* TILE (j,i) - mapping i~ (1,0 C*ksr~ END TIL~
. .
C*ksr~ END AFFINITY REGION

Tile_size_vals~num_indices](long[~: User specified tile size, passed directly from fortr~n directive. Compilation system 60 will check that number of tile si~e values match ~alue o num_indices. Tile ~ize will only ~tay ~n ef~ect ~or this one tile ~amily.

C*ksr~ TILE~i,;, tilesize~ 16) -- results in error C*ksr* TI~E(i,j, tilesize-(j:10,i:16~3 --numindices~2, tilesize~l6,10}

Valid values for tilesize are constants, variables and ~*~.

.J

' !

~2 2~7~3~

For exampl~:

n: where ~ ome con~tant, ~: wher~ x i~ ~om~ ~ariable - * : t~le ~ize ~houl~ be the entire iter~tlon ~pace ln thi~ ~men~on, For e~ample, the ~ollowin~ ~ile d~rec~ves ~r~ val~:

TILE(~,~, tll~lz~e~ls15, ~slO))--t~l~sizo~{15,10 TILE(l,~, tll~5~ze~ l9))--t~t~ 4~t~ 8iz~{4,10}
TIL}!~ , t~ 2e~ Z 10) )--(boun~ fo~ fl 2-~lO,tll~s~2e~8,~0) areg_shiftLnum_in~ices~longt3~: Only valid i this ~ile family ~ in the middle of ~n af~nity r~gion declared by the user or identifi~d by the compilation system 60. Used to keep afinity ~ nmen~ when the tile famil;es use indices ~hi~ted over by a constant value.
For each loop inde3 o~ the tile family:
n=amount added to use of this inde~

For esample~

C*ksr~ AF~INISY REGION

C~k~r~ TI~E(i,~) -areg_~hi~t i8 {0~0 do i~l,l0 ~o a~
en~do enddo -2~7~

C*k~r~ END TXLÆ

C*k~r~ I'IIJE~ reg_~hi~t i~ ~1,2 1, 10 ~o ~1, 10 a ( 1~
~n~o snd~o C * ks r ~ END T I l.ES

C~k~r~ TILE~
do i-l~10 -can' reconcile th;~, areg_~hift i~ {0,0}
~o ~ Dl ~ 10 a ~ s a ( a ~ y enddo enddo C*k~r* ~ND TILIS

C*ksr~ TILE~
~o ~-1,10 -ca3l't recorlcile l:his, areg_~hift i~ {O, O}
do j=l,10 enddo enddo C~ksr* Et~D TILE:

C~ksr~ END AFFI~I~Y REGION

.. . .. .
-.. . ~.
.. . ... . . .
.. ~ ; ~ . ; . , .. ,, . . . , ~ .

- ~ , , -89- ~37~3~

3.1.6 E~cecution oP ~ T~le F3mily Runtime Library 6fi executes the tile ~mily by call~ng a le~cal ~ubroutlne containing the tile bo~y code with the bounds of the ~teration spac~
covered by th~ t~leO ana ~ boolean ~alue ~ndicatin~
whether the last value of the loop in~ces i~
nee~ea. The order of argum0nt~ is low boun~ ~n~ high boun~ of each ~n~e~, ln or~er, ollowe~ by last-value-needea. This i8 ~ri9gerea by the comp~lation system 60~ G~ll to pr~e~ecute_t~l~s.

3.2 Inter~ac~ng Runtime L~brary 66 with the Operating System Runtime Library 66 delegat~s much o th~
responsibil~ty for loa~ balancing threads to the Operating System (~OS~) ~cheduler, as describe~
above. Runtime ~ibrary 66 doec have soms communication with the 6cheduler on the following topics.

3.2.1 ~vailable Processors Runtime Library 65 u e~ in~ormation on the number of available processors for a program when generating tiles. At startup, Runtime Library 66 either asks for a ~et number of processors, if the user has specified a number, or will ask ~or the ~MAX~ number of availa~le processors, The OS
scheduler replies with the actual number that are available.
There may be further quali~iers, ~uch ~s the specification of a minimum number of processors, the request that processors be on a ~in~le RingO, or the request not to use io cells~

j : - i . ' . .

.

-B~- 2a ~3 ~

If the ~cheduler ha~ the abil~ty to expand or contract proce~or~ set~, ~un~ime Library 66 w~ll ~hec~ the ~che~uler ~t the beyinning of eYery n parallel construc~ (n i~ configurable~ to upDate it'~ count of a~labl~ pro~es~or~.

3.2.2 ~cheduler Hint~Priorlty ~ he scheduler u~es appli~ation-~upplie~
information to affect resource ~llocation ~mon~
thread. The scheduler implements a slee~wakeup mec~anism that enables appl~cations to aff~c~ the descheduling of thread~.

3.2.2.1 ~ynchronization Runtime Libr~ry 66 synchron~z~tion i~
acc3mplished with the mutex and barrier construct~
supplied by the pthreads library.

3.2.2.2 Iale A threa~ i~ arunning~ t i~ executing the construct body or ~er~ ode, an~ "syn~'inga if it has entered a ~ynchron~zation routine. ~ all o~her ~imes, a thread is Ui~le~. A thread can become idle if it is a member of a thread group that will e~ecute several parallel constructs, an~ it is not the group master, and it has finished entering ~he ~onstruct barrier. An idle thread will ~pin in a loop for ~
~onfigurable amount of time ~ then issue a ~leep call. The thread will be woken up by the group master when it is time to leave the barrier.

~. , ' ' ' .

{ ~ ~
-86- ~7331~3 4. ~ILING

4.1 Generating tile size~
Tile ~ize and sh~p0s ~re fle~erminea by th~
combination o~ lnput~:

1, Memory alloc~on of the iter~t~on ~pace and the t~le family ~ccess of the memory ta8 describe~ by the ~sb of the t~le family~. The ~sb i~
generated by the comp~la~ion system 60.
2. Consider~tion o the m;nimum desirable tile size (as d~scribe~ by the PL_MININST_PER_TI~E
environment Yariable).
3. Dependencies on the tile indiceæ.

Runtime Library 66 makes a ~ir~t p~s ~t the tile size using the æsb an~ dependency information to makin~ the minimum ~ize tile. The two pos~ible compil~tion ~ystem 60 ~alculated ssb Yalues are NONE
and ~P. Thi~ fir~t pas~ amends the ~sb valu~s with dependency information; if thers i~ a Bependency on thi~ indes and there are no other dependenc;es in the tile, the ssb i5 convested to MAX, in order to flvoid dependency ~ynchronization. If there ;~ a ~ependency on this ;ndes but there ~re more dependencie~ on the tile, an intermediate size o ~P i~ ~ho~en. ~ -The following truth table deæcribes the algorithm for generatin~ the fir t pass tile ~ize template.
s)~ don't car~

, ~ :, . . .

.. . ..

:

, `~ ( ?
-a7-~7~3~ ~
Compiler More th~n 1 Pr~to ~b Dependency Dpdncy ~n S~b Val~ Qn thl~ L~E~mdl~

NON~ FAL~E ~ NONE
NO~E TRUE FALS~ MAX
NO~E TRUE ~RUE ~P
~p ~ALSE ~ ~P
æP TRUE ~ALSE MAX
E;P TRUE TRUE ~;J?

Runtime ~ibrary 66 takes thl~ Runtime Library 66 ~sb value to generate minimum tile ~izes.
If the ssb value or ~n inde~ i~ MAX, the tils size in ~hat dimension w;ll be the entire iteration spsce in that dimension. If the ssb ~alue i~ NONE, the size of that dimension will be the minimum o~ one iteration. ~ the ~sb is ~P, the tile ~ize of ~ha~
inde~ will be 32. After this first pas~, there should be a tile shape that obeys dependency and ~ubpaqe boundary con~ideration~ ana i~ the minimum ~ize.
Run~ime ~ibrary 66 then makes a secon~ pss that compares the tile ~ize to the minimum required size for the tile. I~ the tile family i~ e~ecuted under the slice ~trategy, this minimum ~ize is equal to the number of instructions in the tile family divide~ by the number of threads in th~ e~ecutinq team. Otherwise, the minimum size ;s ~pecifie~ by the environm~nt ~ariable specifying the minimum size of he tile in terms o ~nstructions ~PL_MIN_INST_P~R
TILE).
The tile ~B ~tretched until it ~ati~ies this minimum ~i2e, keeping within the constraint~ of the SS8 value. If one side of the tile ~s alre~dy I

f`~ ( ~

-a8- 2~7~31~3 the entlre itera~ion space, there ~ nothin~ ur~her to do with ~hat dimension. I~ thi~ ~ize has an ~SB
value o SP, tha tile will ~e stretche~ in multi~les of 32. I th;~ e ha~ an 8SB v~lue of NONE, the tile slze ln that ~imension will be ~tretch~ ~n increment~ of 1.
Runt;me Library 66 choo~ indices to s~retch the tile in the following or~er:
If ~he t~le ~amily ~trategy is WAVE~RONT, the column index i~ ~tretchsd f~rst, &ollowed by the subtile inde~. The o~ective is to increase tile sizes in the dimen~ions where ther~ ~re dat~
dependencies, to minimize synchronization between tiles.
Indices with ~SB values o ~P ~re chosen next, to optimize afinity. These indices are stretched in turn. For e~ample, if a tile family has indices ij, and k, with indices ~ ~nd j with ~b values of SP, ~untime Library 66 will stretch i~ then j, then ~, etc., until ei~her i ana/or ; i~
exhausted, or the tile has reached its minimum ~ize.
Indices with SSB ~alues of NONE are chosen last.

4.2 Tile/thread assignment There are several algor;thms, ~alled strategies, to assign tiles to threaas. O~ th~ four described above, the grab ~trategy use~ loafl balancing consi~eratlons, while the mo~ulo, wavefront an~ slice strategies uses data affinity ~onsiderationsO The wavefront strategy is ~l~o designed to execute tile families w;th ordering requirements with a~ much parallelization ~s possible.

~ , , .
~, .
, ..
: ' ' ; ' ' ' `

-89- 207~31 ~

4.2.1 Grab Strategy In the grab ~trategy, tile~ ~re assigned to threads with a pure ~ir~t eom0, fir~t ~erve algorithm.

9.2.2 Modulo ~tr~te~y In the modulo ~trategy, t~le~ are evenly di~tributed amon~ thread~. The map~ing ~8 ((tilenumber ~ total_tiles)~threa~ i~). This can modified with respect ~o parallel sect~ons, so that thread i~ will be the local group threaa ia.
~ n addition, ~ the mo~ulo strategy i~ use~
in conjunction with affin~ty region~, Runtime ~ibrary 66 attempt~ to have each thread execute tile~ which al l within the iteration ~pace that thread has used in previous tile ~amilies, in order to maintain data affinity. Runtime Library 66 looks only at the iteration space, and not at the data space, assuming that the iteration ~pace-~data space mapping i~
constant across ~11 tile families within the affinity region.
Runtime ~ibrary 66 ~s able to remember the iteration ~pace assigned ~o each thre3a and uses that to normalize the ((tile_number ~ total_til~s thread ~d) mapping across tile ~amilies.

4.2.3 Data Dependent Strategies 4.2.3.1 Correctness Data dependencies create ~n ordering requirement when e~ecuting tiled code. ~ile~ ~re executed in a ~e~uence that w;ll not defy the ~irection of dependency in order to guarantee correct resul'c#.

:. : , ~ , ~,.
. .
.
~:
. ~

~0 ~ a ~ .3 4.2.3.2 Perform~nc~
~ ince ~t~ ~epen~en~les ~mpose oraering restrictions nn tiles, tiling in the ~irec~io~ o ~epen~ency i~ ofte~ not worthwhil~ when cons~ere~
only in tho conte~t o~ ~hat tile f~mily. For e~ample, ~ one ~imensional lter~tion 8pace with ~
~ependency must b~ e~ecu~e~ in ~trlct eri21 order, an~ the overhead reguire~ to ~erialize the tiles l waste. However, performance considerat$ons of the entire program may lead Runtime Library 66 to consider tiling i~ the direction of a dependen~y.
Affinity i~ an important factor in choosing tiling strategies thaS may lead to the ~ecision to tile in a dependency. The overhead of ~erializing tiles may be worthwhile if it maintain~ af~inity of data for other tile families in the program that don't have restrictions ana can be completely parallelize~. This is true indepen~ent of dat~
dependency; the sge~a rout~ne of linpack is a case where the firstO smaller lD ~terat;on ~pac~ ~ tiled to be consistent with the second, 2D ~teration ~pace.
In addition~ tiling ~ dependent ~imension may also present the opportunity to use a partially serialized tiling ordering. In ~uch ~ases, tiles are or~anized in ~roups, which must be e~Guted serially. ~owever, with;n the groups,the tile can bP
e~ecuted in parallel. A partially ~erialize~ tiling ordering can onlr ~e used when the i~eration space is more than one dimension.

4.2.3.3 Wavefront ~trategy Runtime Library 66 will tile data dependencies in order to pre~erve affinity and to use partially seriali~ed tiling ordering. The tiling .
`` '' ~

f I
--91- 2 ~

ordering to be ~mplemente~ w~ th~ wavefron~
ora~r~ng .

5. Afinity An affin~ty re~ion i~ ~ collectlon ~ t~le families which Ru~tims ~ibrary 66 ~ttempt~ to e~ecute in a fash~on ~o a~ to avoi~ ~ata conteDtio~ an~
movement. When ~untime ~ibrary 66 encounters an aff-nity region, it receives ~nformatlon which summar~zes he iter~lon ~pace coverea hy ths tile famil~es within the region. Runt~me ~ibr~ry 66 doe~
not need to distinguish between lesically separate tile familie~ an~ tile families within ~n outer loop. Runtime Libr~ry 66 uses this informat~on to generate a tile ~ize template and tiling strategy which is used as for all tile families within the re~ion, allowing a thread to access the same portion of the iteration ~pace for each t;le family.
No~e that affinity regions ~re useless if the tilin~ strategy chosen is the pure grab strategy.
~ f~inity regions may be nested. Each new affinity region ~enerates a new thread group, and uses its own tiling workplan.

5.1 Common Indes Set A tile size template is createa by ~ooking at a ~et of loop indic~s, their depende~cies, ~nd their ~sb's. The ~et of loop indices of interest are those used ~or all of the tile families in the af~ini y re~ion. ~t must be possible or all the loopnests in an affinity region to be tiled on the ~ame ~et o~ ~n~ic0~. Thi~ is the ~common ~n~e~ ~et~
rule. ~he commo~ inde~ set i~ a ~u~set of the intersection of the indice~ use~ by the tile families - ( ) -9~- 2 ~ 7 ~

within an ~ffinity region. This i~ nec~ssary because the ainity oriente~ strate~ies nee~ ~n iteration->proce~or mapp~ng function which determines which proce~or e~cute~ the ti le. ~f the number of tile~ in~l~e~ varie~ loopne~t to loopnest, thi~ mapp~ng will f~ll.

For e~ample:

The common ~naes set i~ for the following e~ample:
~ tart afrinity region---1~10 i81 ~ 10 do j~l,10 a~i,j,5) ~ ~;
enddo enddo aO ~ 0 ~o k=l,10 ~ (~,4,~
enddo enddo ~ end affinity region---In the following e~ample, it is necessary tochange the code ~o that a ~ommon index ~et can be identified. Here is the original:

~ tart affinit~ region-~-do n-~,9 do m~2,199 do 1~2, 1999 ..
~','~ ~' ', , , . ' .

.
'` ` ' . ~

-93~ 8 3 ~. ~

u(l,m,n) ~ a~:c;
enddo end~o ena~o " do n~2, 99 ~o m~2, 199 u (l ,m,n) ~
u (200û,m,n) ~ :cs;
enddo en~o do n-2, 99 ~lo lg2, 19~
u(l, l,n3 ~ x~;
enaao enado do m.2, 199 do 1~2~ 1999 u(l,m, 1~
u (l,m, 10û3 8 3~;
enddo ~ndao ----end affinity region---llere is the changed ~o~e. The most optimal version, to create a comrnon inde~ s~t o~ ~n,m) i~:
tart a~inity region~

do n~2, g 9 ~o m=2, 199 do 1=2, 1999 .:, ' ' .
... ~ . ~ . . .. . . .
~ , : .~ .

~ ' ` ) 2~7~
~94-u~l~m~n) n ~z;
enddo enddo en~do ~o r~2, 99 ~o m~2, 199 u ( l, m, n) ~
ut2000,m,n~ %;
enado enddo C Add loop ~or m do n~2, 99 do 1~2, 1999 do m~l,l u(l, l,n) ~e ac2~;
enddo enddo enddo C ~dd loop ~or n, al~o ad~l con~itional 60 that C ~tatement i~ only done once, on one C processor. Runtime Libr~ry 66 will make sure C that the iterat;on goes to the right C processor, but cannot assess memory accesses C hard ~oded inside the loop.

C An alternati~ve is to split to loop into two C loops that look lilse the one above, with C only one statement in each loop. The C tradeof i. th~ addit;on 4iE a new tile family ., . , .. ,: :
. :.~ , .
.~ . ., , ; . . ., . ~ . , , .- .: , .
~; . .

( i ) ` -95~ 3~3 flo m-2,199 do 1~2, 1999 do n~l,100,100 if (n .eq. 1) u~l,m,l) ; ~f (n .eq. 100~
u~l,m,100) . y~:
en~o end~o enddo end a finity region---5.2 Af f inity r~gions ~næ ~ubroutine Calls The aefault requirement i~ that all tilefamilies within an affinity region de~laration mu t be in the same le~ical level as the affinity region declaration. All tile amilies within the ai~ity declaration but in another subroutine will be run by another threaa group (there~ore, possibly ~nother set of processors and without the same affinity~. The motivation is tha~ affinity region declaratio~s require ~he compilation ~ystem 60 to ~can ~11 tile families within the declaration ~nd generate summaries of the tile family ~haract~ri~tic .
Subroutine~ are not acces~ible to the ~ompil~tion system 60 and cannot be includ~d in the ~inity region information.
Howev~r, the user c~n u~e the ~intercall~
parameter of the A~INITY REGION directivs to override this requirement. The compilation ~ystem 60 will not b2 able to æc~n the tile families ~ha~ ars not at the ~ame le~ical level, and if the user does not provide deta~led in~ormation through the other . ~ .
,, -96- 2~7~3~

parameters, the ~ffinity region may not be optimally ~et up.

: An e~mple o~ the ~efault beh~vior:

c*ks~Y~ AFFINIq~r REGION(~:l,Da c~k~r~ PTILE~
dc ~.~l,n ~o ~-~1, n .
call foo () -- C311 #1: run by another threa~ gzoup enddo enddo call foo() -- #2: run by another thread group c~ksr~ PTIL~(i) dQ i81~n . . .
enddo c*ksr* END AFFI~ITY REGION

6. Runtime Lib~ary 66 Architecture 6.1 Use of ~ynchronization Runtime ~ibr~ry 66 reguires synchronization of its member threads at various points during the esecution of a Presto program. The synchroniza~ion mechanisms used are critical ~ections and barriers.
Runtime Library 66 use the pthread mute~ calls to ' ' ' ' ' ' ..

i ~ ) implement crit~cal ~ection~ ~n~ the pthread barr~er call~ to impl~ment barrier~.

6.1.1 Beginnin~ and end of a par~llQl construct ~barrier) At the beginnin~ of a ~ile famlly or parallel ~eetion, ~11 threa~s participa~ing in the parallel construct are he~ in a barrier unt~ 1 ~he threa~ group maæter ~in~hes all serial code. A~ th~
ena of the exe~ution of that con~truc , all mem~er thread~ re-en~er. The master thre~d will not execute code ollowing the parallel con~truct u~til ~11 member threads are entered the barrier.
There may be num~rous barriers declared if the program has a nestinq of parallel constructs. A
thread may be a member of several barrier groups if it i participating in ~everal nested construct~.

6.1.2 Locking ~uring the grab ~trategy (critlcal section) The grab ~trategy require~ ~11 memb~r ~hreads to update a marker da~a ~tructure to ~et the ~ext available tile. This marker data structur~ is held in a lock.

6.1.3 Locking the thread group id (cri~ical ~ection) Each new thread group i~ given a unigue id, used to identify parallel con~truct~ ~or visualization, logging, ~nd tagging output. The current group id is held in a data lock an~
incremented by each new thread group.

, I ! 98- 2 ~ 7 ~ 3 ~. ~

6.1.4 Loc~lng ~uring thread group cre~tion an~
disbanding ~crit~cal ~ection~
Runtime ~ibrary 66 create~ pthrea~s ~n~
keeps them hanging 3round ~n ~n ~le pool. When ~
threa~ group i~ crea~ed ~nd ~i~b~n~e~, thre~ must be ass~ne~ in or out o the iale pool. Thls i~
synchronized by hold~ng the number of available ~dle threads in a data lock.

6.1.5 Loc~ing during creation~destruction of team~
(critical section) Runtime ~ibrary 66 ~llows the u~er to cr~ate permanent ~eams tha~ live beyona a ~ingle parallel con~ruct. These teams ~se hsld in a global linked list. Creation ana destruction o teams ~
synchronized throu~h the lock _ pr_team_lock;

.1.6 Locking for ~tor~ge of ~tatistics information t~ritical ~ection3 ~ untime Library 66 may be configured to collect stati~ti~al performance information. This data is kept in global variables which must be written in a critical sec~ion.

6.2 Use of Thr~ads 6.2.1 Thread Teams A ~resto program starts out with one pthread, known as the program master. At ~tartu~, ~untime Library 66 creates sdditional pthreads, and keeps them in an idle pool of availabls threads. ~s the progr~m e~ecutes, Runtime Libraxy ~6 creates pthread teams to handle new parallel constructs encountereaO Thread groups may handle either one or ~ . . . . .. .
. ' ~, .
: -,:

. .

, 2~7g3~_~

mors par~llel constructs~ When teams are di~ban~e~, the pthrea~ members ~re returnea to the Runtims Library 66 ~ale pool. This idle pool ~ s designed to m~nimize t~e high co~ o~ ~threa~ creatlon.
Parallel section~ ~n~ regions: By def~ult, a new thread team i~ ~reated or each new parall~1 ~ect~on. ~t the en~ of ~he section, the threaa team is disba~de~. I the user ~peci~ies fi team for U~2 in thi~ ~on~truct, th~t team ~ usefl, an~ ~8 not disbanded at the en~ of ths construct.
T~le familie~ no ~ffinity regio~: a new threa~ team i5 create~ for each new tile family not enclosed ~n an ~finity zegion unle~s a team is specifiea in the directive. At the ena of the tile family, the new thread group i~ ~isbanded.
I a team is specified in the directi~e, that team is used and a new team i~ not created. A
user specified team is not disbanded ~t the end of the construct.
Title amilies, within affinity region~: By ~efault, a new thread group i~ created to e~ecute ~11 the tile families within an affinity region~ If a team is specifi~d with the 3ffinity region dire~tlve, that team is used and 8 neW team is not createfl . The thread team i~ kept ali~e across the tile families in order to preserve a uniform tile-> thread mappi~g across the a~ f inity regionO The motivation i~ the assumption that threads will tena to rema;n on the same processors during the li~2 of the ~hr~a~. At ~he end of each affinity region, the new thread group i5. di ~anded, unles~ it i~ a u~er-create~
~pecified team.
~ ~ingle thread may be a member of 6everal thread teams ~f parallPl constructs are nested, - . . . .
~ .
.,; . ~

-100- ~ ~3 ~ ~ s~ ~ ~

6.2.2 Creating an~ De~troying Thread~
A threa~ group ~8 compo~ed o~ a group ma~ter and O to n group slave~. ~ Preeto program ~tart~ off with sne thread, ths program ~a~tsr. As thtea~
group~ ar~ create~, ~he current master ~ecome~ the group master of the new threa~ ~roup, ~nd the ~laves are cxeate~. ~s thread yroups are disbande~, the group ~la~es are r~turned to the idle pool.
~ hi~ proces~ e~ten~ to nested parall~l constructs. ~magin~ ~ program with ~ ~eries of parallel constru~t~ nes~ed ~wo deep. At level 1, the program ma~ter creates a threa~ group ana becomes the group master. At level 2, any ~lave that encount~rs another parallel construct creates a new thread group and becomes the level 2 ~roup ma~ter. When the l~vel 2 construct ends, the level 2 thread ~roup i~
di~banded, and the group ~laves are returned. Ths level 2 group master sheds its master identity and resumes its own identity as ~ level 1 ~lave or master. When the level 1 thread group i~ aisban~e~, all group slaves are returned e~cept for the ~roup (and program) ma~ter.

6.3 Data Structure~

6.3~1 ~lobal to program Environment variables: All user-interfa~e enYironm~nt variables are æhared globals to the program and are applicable to all parallel constructs.
Statistic~ informatiQn: These measure instruction counts by ~arious cat0~0r~es are ~tored in ~hared ~lobal variables.
master_thread: i~ of program master, for UDB support. Shared global variables.

- 10 1~

_ Pr _ fp: File pointer u~e~ or output ~t en~ o program. Needea because o treatment o ~tandard out by f ortran environment. ~hared global.
_pr_t_state: Runnin~, idle, or ~ync ~tate, ~r udb suppor~. Private global~
_ pr curr_mcb_p: Pointer to current ~cb, for udb support. Private glob~l.
_ TEAM_ME~_ID: thi~ thread~ id in ~t8 current team. Prlvate global.
_ POO~ ~ ID: thi~ thr~ad'~ current i~ fQr the Runtime Library 66 iflle thread pool. Private global.
_ MY_ THREAD~ thi~ thraa~ pthread id. Private global.

~mma~y The foregoing describes an improved digital data processin~ system meeting the ~forementioned objects. Particularly, it describ~s ~n improved parallel processor that e~ecutes iterative ~e~uences by ~ividing them into su~tasks an~ allocatin~ those to the processor~. This ~ivision and allocatiQn conauote~ in such a manner as to minimize ~ata contention among the processors and tc ma~im~7e locality of data to the proce~sor~ which aGce~s that data.
Those ~killed ;n the art will appre~iate that the embodiment~ ~escribed above are e~mplary only, and that other apparatu~es and methods --including modification~, additions ~nd deletionfi --fall within the scope ~na ~pirit o~ the inYent~on.
By way of e~ample, it ~ill be appreci~ted that the functionality o~ the preproces or 60a may be incorporated into the compiler 60b itself. Thus, . . . ~ , .

.. : ' ', - .,. ~ ,;, .:

10 2 - 2 0 7 8 ~ ~ 3 eliminating the need for a ~eparate preprocese~nq s'cep .
It will ~l~o ~e appreciated that the techniques descri~ed ~bove rnay be appl~ed to massively par~llel 6y~tems ~n~ other multlproce~or ~y~ tems .
By way of ~urther e;~mple, it ~ial be appreciate~ that ~iffering dat~ structures ~torlng 'c~ling information may be u~e~. ~hat ~u~valent, but varied, proceaure~ may ~e u~ed to parallel~ze ~nd execute the iteLat~ve ~equence6. And, by way ~
further e~tampl2, that urther tiling direc~ives may be added without changing the ~pirit of the invenSion.
In view of the foregoing, what we claim 1~:

:,

Claims (70)

1. In a parallel processor digital data processing apparatus of the type having a plurality of processing units, each for executing instructions, memory means coupled to said processing units for storing at least one of data and instructions, interprocessor communication means coupled to said processing units for transferring information therebetween, the improvement for processing an iterative sequence of instructions wherein:
A. said memory means includes means for storing a tiled sequence of instructions representing the iterative sequence, and each said processing unit includes means for signalling its availability to execute said tiled sequence over a portion of an iteration space associated with said iterative sequence, said portion being referred to as a tile, B. said apparatus includes next-tile means coupled to said processing units for responding to each of at least selected such signallings by those processing units for generating a signal representing boundaries of a tile over which to execute aid tiled sequence, wherein each such tile does not overlap any other tile, and wherein all such tiles together cover said iteration space, C. said next-tile means including means for generating said boundary-representative signals so as to maximize a locality of data subject to access during execution of one or more corresponding tiles, and to minimize contention for access to such data during such execution, and D. each said processing unit including means for responding to a boundary-representative signal generate in response to signalling by that processing unit for executing said tiled sequence over the corresponding tile.

2. In a digital data processing apparatus according to claim 1, the further improvement comprising A. tile-build means for generating tile-shape signal indicative of one or more dimensions of said boundary-representative signals, in view of dependency between the corresponding tiles, B. wherein said tiled sequence includes one or more instructions for write-type access of a datum store in said memory means.
C. said memory means includes i) a plurality of memory elements, each for storing at least one of data and instructions, each said memory element being coupled to an associated one of said processing units, and ii) memory management means coupled with said memory elements for selectively transferring information therebetween, and D. said next-tile means includes mean for generating said boundary-representative signals as a function of said tile-shape signal.

3. In a parallel processor digital data processing apparatus according to claim 2, the further improvement wherein said tile-build means includes means for generating said tile-shape signal indicative of dimensions of said boundary-representative signals that minimize the number of individual datum subject to write-type access during execution of different ones of corresponding tiles.

4. In a parallel processor digital data processing apparatus according to claim 2, the further improvement wherein said tile-build means includes means for generating a tile-shape signal indicative of dimensions of said boundary-representative signals that minimize the number of individual datum subject to write-type access by plural concurrently-executing ones of the processing units.

5. In a parallel processor digital data processing apparatus according to claim 2, wherein A. first and second tiles include any of i) an instruction in said first tile that accesses for write-type access a selected datum, and a instruction in said second tile that subsequently accesses for read-type access that same datum, ii) an instruction in said first tile that accesses for read-type access a selected datum, and an instruction in said second tile that subsequently accesses for write-type access that same datum, iii) an instruction in said first tile that accesses for write-type access a selected datum, and an instruction in said second tile that subsequently accesses for write-type access that same datum, and B. said tile-build means includes means for generating a tile-shape signal indicative o dimensions of said boundary-representative signals that minimizes dependence of execution of said second tile upon execution of said first tile with respect to their respective accessing of said selected datum.

6. An apparatus according to claim 2, wherein said tile-build means includes means for generating an affinity signal representative of a tile execution sequence that minimizes a transfer of a data subject to any of read-type and write-type access during execution of successive tiles by different processing units.

7. In a digital data processing apparatus according to claim 2, the further improvement wherein said tile-build means comprises tile-shape means for generating said tile-shape signal as a function of at least one of i) an affinity signal representing a tile execution sequence that minimizes a transfer of a data subject to any of read-type or write-type access by successive executions of said tiles by different ones of said processing units, ii) a dependency direction of said tiled sequence, iii) a processing cost associated with execution of said tiled sequence, iv) a size of said iteration space, v) a number of processing units available for execution of said tiled sequence, and vi) an affinity region signal indicating that the tiled sequence is to be executed within an iteration space defined by a plurality of tile sequences.

8. In a parallel processor digital data processing apparatus according to claim 1, the further improvement wherein A. said next tile means includes means for generating successive ones of said boundary-representative signals as a function of at least one of i) a tile-shape signal indicative of one or more dimensions of said boundary-representative signals, ii) a tile-strategy signal indicative of of least one at a sequence and manner of assigning tiles to said processing units, iii) an affinity region signal indicating that the tiled sequence is to be executed within an iteration space defined by a plurality of tile sequences, B. said apparatus comprises tile-strategy means for generating said tile-strategy signal, and C. wherein said iteration space is defined by a plurality of indices.

9. In a parallel processor digital data processing apparatus according to claim 8, the further improvement wherein said tile-strategy means includes selectively actuable slice means for i) generating a tile-shape signal indicative of boundary-representative signals dividing the iteration space by a number of processing units allocated to said execution with respect to one of said indices and spanning the iteration space with respect to the others of those indices, wherein the number of respective tiles is equal to that a number of available processing units, wherein those tiles are substantially equal in size, and ii) generating a tile-strategy signal indicative of generation of the respective tile so as to cause one each of them to be executed by a respective one of the available processing units.

10. In a parallel processor digital data processing apparatus according to claim 9, the further improvement wherein said tile-strategy means includes means for actuating said slice means in instances in which i) data subject to access during execution of any such tile does not depend on that modified during execution of any other such tile, and ii) no substantial quantity of data subject to any of read-type and write-type access by one such tile is also subject to any of read-type and write-type access by another such tile.

11. In a parallel processor digital data processing apparatus according to claim 9, the further improvement comprising means for actuating said slice means in response to a user-supplied signal.

12. In a parallel processor digital data processing apparatus according to claim 8, the further improvement wherein said tile-strategy means includes selectively actuable modulo means for i) generating boundary-representative signals to divide the iteration space in fixed-length increments along all of said indices, and ii) generating a tile-strategy signal indicative of assignment of the respective tiles based on a modulus of a numerical identifier associated with each of the available processing units and a numerical identifier associated with each of the tiles.

13. In a parallel processor digital data processing apparatus according to claim 12, the further improvement wherein said tile-strategy means includes means for actuating said modulo means in instances in which i) data subject to access during execution of any such tile does not depend on that modified during execution of any other such tile, ii) no substantial quantity of data subject to any of read-type and write-type access by one such tile is also subject to any of read-type and write-type access by another such tile and iii) re-use of data subject to any of read-type or write-type access by a processing unit executing a plurality of tiles is maximized, even in the presence of a change of a size of said iteration space.

14. In a parallel processor digital data processing apparatus according to claim 12, the further improvement comprising means for actuating said modulo means in response to a user supplied signal.

15. In a parallel processor digital data processing apparatus according to claim 8, the further improvement wherein said tile-strategy means includes selectively actuable wavefront means for i) generating boundary-representative signals dividing the iteration space along one or more of said indices, wherein data subject to access during execution of at least selected tiles depends on that modified during execution of another tile, and wherein those tiles have a dependency direction, ii) generating a tile-strategy signal indicative of a sequence of generation of the respective tiles in accord with that dependency direction.

16. In a parallel processor digital data processing apparatus according to claim 15, the further improvement wherein said tile-strategy means includes means for actuating said wavefront means wherein data subject to access during execution of at least one such tile depends on that modified during execution of another such tiles.

17. In a parallel processor digital data processing apparatus according to claim 15, the further improvement comprising means for actuating said wavefront means in response to a user-supplied signal.

18. In a parallel processor digital data processing apparatus according to claim 8, the further improvement wherein said tile-strategy means includes selectively actuable modulo/wavefront means for i) generating boundary-representative signals to divide the iteration space in fixed-length increments along all of said indices, wherein data subject to access during execution of at least selected tile depends on that modified during execution of one or more other tiles, and wherein said tiles have a dependency direction, and ii) generating a tile-strategy signal indicative of assignment of the respective tiles based on a modulus of a numerical identifier associated with each of the available processing units and a numerical identifier associated with each of the tiles, and on that dependency direction.

19. In a parallel processor digital data processing apparatus according to claim 18, the further improvement wherein said tile-strategy means includes means for actuating said modulo/wavefront means in instances in which i) data subject to access during execution of at least one such tile depends on that modified during execution of one or more other such tiles, and ii) re-use of data subject to any of read-type or write-type access by a processing unit executing a plurality of tiles is maximized, even in the presence of a change of a size of said iterative space.

20. In a parallel processor digital data processing apparatus according to claim 18, the further improvement comprising means for actuating said modulo/wavefront means in response to a user-supplied signal.

21. In a parallel processor digital data processing apparatus according to claim 8, the further improvement wherein said tile-strategy means includes selectively actuable grab means for i) generating boundary-representative signals to divide the iteration space in fixed-length increments along all of said indices, and wherein data subject to access during execution of any tile does not depend on that modified during execution of any other tile, and ii) generating a tile-strategy signal indicative of a sequence of generation of the respective tiles, in response to said signalling for availability by said processing units, on first-come-first-serve basis.

22. In a parallel processor digital data processing apparatus according to claim 21, the further improvement wherein said tile-strategy means includes means for actuating said grab means i) to facilitate load-balancing among processing units executing said tiles, and in instances when ii) no substantial quantity of data is subject to any of read-type any write-type access by one such tile is also subject to any of read-type and write-type access by another such tile.

23. In a parallel processor digital data processing apparatus according to claim 21, the further improvement comprising user-responsive means for actuating said wavefront means.

24. In a parallel processor digital data processing apparatus according to claim 1, the further improvement comprising A. means for providing an affinity region signal representative of one or more iteration sequences that access the same data, B. affinity region-build means for generating one or more of i) a signal representative of an iteration space in which one or more tile sequences within the affinity region are to be executed, ii) a tile-shape signal representative of one or more dimensions of a boundary-representative signal, iii) a tile-strategy signal indicative of at least one of a sequence and manner of assigning tiles to said processing units, and iv) a team signal representative of a set of processing units by which the tiled sequences within the affinity region are to be executed to minimize a transfer of a data subject to any of read-type and write-type access during such execution.

25. In a parallel processor digital data processing apparatus according to claim 24, the further improvement wherein said affinity region-build means includes means for defining said iteration space to encompass one or more of the iteration spaces of the tile sequences in said affinity region.

26. In a compiler of the type for translating program signals representative of a program, including an iterative sequence, in source code format into a code sequence of instructions in an object code format suitable for loading for execution by a plurality of parallel digital data processing units, said compiler including dependency means for determining a dependency direction of said iterative sequence and for generating a signal representative thereof, the improvement comprising tiling means for responding to said iterative sequence in source code format for generating a tiled sequence of instructions representative thereof in objects code format, and for further responding to that iterative sequence and to zero, one or more user-specified signals for generating one or more signals representing a framework for generating boundaries for one or more tiles over which to execute said tiled sequence.

27. In a compiler according to claim 26, the further improvement comprising parallelization means for responding to instructions in said iterative sequence and to said dependency direction-representative signal for generating signals representative of one or more indexes in said iteration space over which to execute said tiled sequence.

28. In a compiler according to claim 27, the further improvement comprising A. input means for accepting signals representative of a user-specified indexing of said iteration space, and B. means coupled with said input means and said parallelization means includes means for comparing said user-specified indexing signals with those generated by said parallelization means, and for generating a notification signal in the event of at least selected disagreement thereof.

29. In a compiler according to claim 27, the further improvement comprising optimizing means coupled to said parallelization means for generating one or more signals for use at runtime in optimizing parallel execution of said tiled sequence.

30. In a compiler according to claim 29, the further improvement wherein said optimizing means comprises means for generating an affinity signal representative of a tile execution sequence that minimizes a transfer of data subject to any of read-type or write-type access during execution thereof in plural tiles.

31. In a compiler according to claim 29, the further improvement wherein said optimizing means comprises means for generating an affinity region signal representative of one or more iteration sequences that access the same data.

32. In a compiler according to claim 31, the further improvement comprising A. input means for accepting signals representative of a user-specified affinity region signal, and B. means coupled with said input means and said optimizing means includes means for comparing said user-specified affinity region signal with that generated by aid optimizing means, And for generating a notification signal in the event of at least selected disagreement thereof.

33. In a compiler according to claim 29, the further improvement wherein said optimizing means comprises means for generating a signal representative of an estimate of a processing coat associated with execution of said tiled sequence.

34. In a compiler according to claim 26, the improvement comprising call-generating means for replacing said iterative sequence in said source code format with an instruction representative of a call to a code-dispatching subroutine.

35. In a compiler according to claim 34, the improvement comprising run-time means for executing said call-representative instruction for initiating execution of said tiled sequence by said processing units.

36. In a method of operating a parallel processor digital data processing apparatus of the type having a plurality of processing units, each for executing instructions, memory means coupled to said processing units for storing at least one of data and instructions, interprocessor communication means coupled to said processing unite for transferring information therebetween, the improvement for processing an iterative sequence of instructions comprising the steps of:
A. storing a tiled sequence of instructions representing the iterative sequence, B. signalling for each processor, its availability to execute said tiled sequence over a portion of an iteration space associated with said iterative sequence, said portion being referred to as a tile, B. responding to each of at least selected such signalling by those processing units for generating a signal representative boundaries of a tile over which to execute said tiled sequence, wherein each such tile does not overlap any other tile, and wherein all such tiles together cover said iteration space, C. generating said boundary-representative signals so as to maximize a locality of data subject to access during execution of one or more corresponding tiles, and to minimize contention for access to such data during such execution, and D. each said processor responding to a boundary-representative signal generated in response to signalling by that processing unit for executing said tiled sequence over the corresponding tile.

37. In a method according to claim 36, the further improvement comprising the steps of A. generating tile-shape signal indicative of one or more dimensions of said boundary-representative signals, in view of dependency between the corresponding tiles, B. wherein said tiled sequence includes one or more instructions for write-type access of a datum stored in said memory means, C. providing a plurality of memory elements, each for storing at least one of data and instructions, each said memory element being coupled to an associated one of said processing units, and selectively transferring information between said memory elements, and D. generating said boundary-representative signals as a function of said tile-shape signal.

38. In a method according to claim 37, the further improvement comprising generating said tile-shape signal indicative of dimensions of said boundary-representative signals that minimize the number of individual datum subject to write-type access during execution of different ones of corresponding tiles.

39. In a method according to claim 37, the further improvement comprising generating a tile-shape signal indicative of dimensions of said boundary-representative signals that minimize the number of individual datum subject to write-type access by plural concurrently-executing ones of the processing units.

40. In method according to claim 37, wherein A. first and second tiles include any of that accesses for write-type access a selected datum, and a instruction in said second tile that subsequently accesses for read-type access that same datum, ii) an instruction in said first tile that accesses for read-type access a selected datum, and an instruction in said second tile that subsequently accesses for write-type access that same datum, iii) an instruction in said first tile that accesses for write-type access a selected datum, and an instruction in said second tile that subsequently accesses for write-type access that same datum, the further improvement comprising the step of B. generating a tile-shape signal indicative of dimensions of said boundary-representative signals that minimizes dependence of execution of said second tile upon execution of said first tile with respect to their respective accessing of said selected datum,

41. In a method according to claim 37, the further comprising generating an affinity signal representative of a tile execution sequence that minimizes a transfer of a data subject to any of read-type and write-type access during execution of successive tiles by different processing units.

42. In a method according to claim 37, the further improvement comprising the step of generating said tile-shape signal as a function of at least one of i) an affinity signal representing a tile execution sequence that minimizes a transfer of a data subject to any of read-type or write-type access by successive executions of said tiles by different ones of said processing units, ii) a dependency direction of said tiled sequence, iii) a processing cost associated with execution of said tiled sequence, iv) a size of said interation space, v) a number of processing units available for execution of said tiled sequence, and vi) an affinity region signal indicating that the tiled sequence is to be executed within an iteration space defined by a plurality of tile sequences.

43. In a method according to claim 36, the further improvement comprising the steps of A. generating successive ones of said boundary-representative signals as a function of at least one of i) a tile-shape signal indicative of one or more dimensions of said boundary-representative signals, ii) a tile-strategy signal indicative of at least one of a sequence and manner of assigning tiles to said processing units, iii) an affinity region signal indicating that the tiled sequence is to be executed within an iteration space defined by a plurality of tile sequences, B. generating said tile-strategy signal, and C. wherein said interation space is defined by a plurality of indices.

44. In a method according to claim 43, the further improvement comprising selectively actuable slice means for executing the steps of i) generating a tile-shape signal indicative of boundary-representative signals dividing the iteration space by a number of processing units allocated to said execution with respect to one of said indices and spanning the iteration space with respect to the others of those indices, wherein the number of respective tiles is equal to that a number of available processing units, wherein those tiles are substantially equal in size, and ii) generating a tile-strategy signal indicative of generation of the respective tiles so as to cause one each of them to be executed by a respective one of the available processing units.

45. In a method according to claim 44, the further improvement comprising the step of actuating said slice means in instances in which i) data subject to access during execution of any such tile does not depend on that modified during execution of any other such tile, and ii) no substantial quantity of data subject to any of read-type and write-type access by one subject tile is also subject to any of read-type and write-type access by another such tile.

46. In a method according to claim 44, the further improvement comprising means actuating said slice means in response to a user-supplied signal.

47. In a method according to claim 43, the further improvement comprising providing selectively actuable modulo means for executing the steps of i) generating boundary-representative signals to divide the iteration space in fixed-length increments along all of said indices, and ii) generating a tile-strategy signal indicative of assignment of the respective tiles based on a modulus of a numerical identifier associated with each of the available processing units and a numerical identifier associated with each of the tiles.

48. In a method according to claim 47, the further improvement comprising the steps of actuating said modulo means in instances in which i) data subject to access during execution of any such tile does not depend on that modified during execution of any other such tile, ii) no substantial quantity of data subject to any of read-type and write-type access by one such tile is also subject to any of read-type and write-type access by another such tile, and iii) re-use of data subject to any of read-type or write-type access by a processing unit executing a plurality of tiles is maximized, even in the presence of a change of a size of said iteration space.

49. In method according to claim 478, the further improvement comprising actuating said modulo means in response to a user-supplied signal.

50. In a method according to claim 43, the further improvement comprising providing selectively actuable wavefront means for executing the steps of i) generating boundary-representative signals dividing the iteration space along one or more of said indices, wherein data subject to access during execution of at least selected tiles depends on that modified during execution of another tile, and wherein those tiles have a dependency direction, ii) generating a tile-strategy signal indicative of a sequence of generation of the respective tiles in accord with that dependency direction.

51. In a method according to claim 50, the further improvement wherein said tile-strategy means includes means for actuating said wavefront means wherein data subject to access during execution of at least one such tile depends on that modified during execution of another such tile.

52. In method according to claim 50, the further improvement comprising actuating said wavefront means in response to a user-supplied signal.

53. In a method according to claim 43, the further improvement comprising providing selectively actuable modulo/wavefront means for executing the steps of i) generating boundary-representative signals to divide the iteration space in fixed-length increments along all of said indices, wherein data subject to access during execution of at least a selected tile depends on that modified during execution of one or more other tiles, and wherein said tiles have a dependency direction, and ii) generating a tile-strategy signal indicative of assignment of the respective tiles based on a modulus of a numerical identifier associated with each of the available processing units and a numerical identifier associated with each of the tiles, and on that dependency direction.

54. In a method according to claim 53, the further improvement comprising the steps of actuating said modulo/wavefront means in instances in which i) data subject to access during execution of at least one such tile depends on that modified during execution of one or more other such tiles, and ii) re-use of data subject to any of read-type or write-type access by a processing unit executing a plurality of tiles is maximized, even in the presence of a change of a size of said iterative space.

55. In a method according to claim 53, the further improvement comprising means for actuating said modulo/wavefront means in response to a user-supplied signal.

56. In a method according to claim 43, the further improvement comprising providing selectively actuable grab means for i) generating boundary-representative signals to divide the iteration space in fixed-length increments along all of said indices, and wherein data subject to access during execution of any tile does not depend on that modified during execution of any other tile, and ii) generating a tile-strategy signal indicative of a sequence of generation of the respective tiles, in response to said signalling for availability by said processing units, on a first-come-first-serve basis.

57. In a method according to claim 56, the further improvement wherein said tile-strategy means includes means for actuating said grab means i) to facilitate load-balancing among processing units executing said tiles, and in instances when ii) no substantial quantity of data is subject to any of read-type and write-type access by one such tile is also subject to any of read-type and write-type access by another such tile.

58. In a method according to claim 56, the further improvement comprising actuating said wavefront means in response to a user-supplied signal.

59. In a method according to claim 36, the further improvement comprising the steps of A. providing an affinity region signal representative of one or more iteration sequences that access the same data, B. generating one or more of i) a signal representative of an iteration space in which one or more tiled sequences within the affinity region are to be executed, ii) a tile-shape signal representative of one or more dimensions of a boundary-representative signal, iii) a tile-strategy signal indicative of at least one of a sequence and manner of assigning tiles to said processing units, and iv) a team signal representative of a set of processing units by which the tiled sequences within the affinity region are to be executed to minimize a transfer of a data subject to any of read-type and write-type access during such execution.

60. In method according to claim 59, the further improvement wherein said affinity region-build means includes means for defining said iteration space to encompass one or more of the iteration spaces of the tile sequences in said affinity region.

61. In a method for operating a compiler of the type for translating program signals representative of a program, including an iterative sequence, in source code format into a code sequence of instructions in an object code format suitable for loading for execution by a plurality of parallel digital data processing units, said compiler including dependency means for determining a dependency direction of said iterative sequence and for generating a signal representative thereof, the improvement comprising the steps of responding to said iterative sequence in source code format for generating a tiled sequence of instructions representative thereof in object code format, and for further responding to that iterative sequence and to zero, one or more user-specified signals for generating one or more signals representing a framework for generating boundaries for one or more tiles over which to execute said tiled sequence.

62. In a method according to claim 61, the further improvement comprising responding to instructions in said iterative sequence and to said dependency direction-representative signal for generating signals representative of one or more indexes in said iteration space over which to execute said tiled sequence.

63. In a method according to claim 62, the further improvement comprising the steps of A. accepting signals representative of a user-specified indexing of said iteration space, and B. comparing said user-specified indexing signals with those generated by said method, and for generating a notification signal in the event of at least selected disagreement thereof.

64. In a method according to claim 62, the further improvement comprising generating one or more signals for use at runtime in optimizing parallel execution of said tiled sequence.

65. In a method according to claim 64, the further improvement comprising generating an affinity signal representative of a tile execution sequence that minimizes a transfer of data subject to any of read-type or write-type access during execution thereof in plural tiles.

66. In a method according to claim 64, the further improvement wherein said optimizing means comprises means for generating an affinity region signal representative of one or more iteration sequences that access the same data.

67. In a method according to claim 66, the further improvement comprising A. accepting signals representative of a user-specified affinity region signal, and B. comparing said user-specified affinity region signal with that generated by said method, and for generating a notification signal in the event of at least selected disagreement thereof.

68. In a method according to claim 64, the further improvement comprising generating a signal representative of an estimate of a processing cost associated with execution of said tiled sequence.

69. In a method according to claim 61, the improvement comprising replacing said iterative sequence in said source code format with an instruction representative of a call to a code-dispatching subroutine.

70. In a method according to claim 69, the improvement comprising execution said call-representative instruction for initiating execution of said tiled sequence by said processing units.