512-Host Myrinet Clos Cabling

This downloadable text file is a cabling list for a 512-host Clos network. This cabling pattern is exactly the way in which the 512-host Platinum cluster at NCSA, pictured to the right, was built.

Following is the other information you'll need to interpret this cabling list.

The line-card slots in the E128 enclosure are normally numbered from top to bottom:

  M  monitoring line card
  0  port line card
  1  port line card
  ...
 15  port line card

The array of 128 ports is best described by "row," which is the slot number, 0...15, top to bottom as above, and "column," 0...7, RIGHT TO LEFT.

You will have 12 E128 switch units, 8 with line cards in the "Clos64+64" configuration, and 4 with line cards in the "Spine" configuration. The "Clos64+64" units are normally shipped with M3-SPINE line cards in slots 0...7, and M3-SW16 line cards in slots 8...15. The switches in the "Spine" configuration have M3-SPINE line cards in all 0...15 slots.

We suggest that you number the 8 Clos64+64 boxes

Clos 0, Clos 1, ..., Clos 7;

and the 4 Spine units

Spine 0, Spine 1, Spine 2, Spine 3.

The cables to hosts are all from the ports of the M3-SW16 line cards on the Clos boxes. The silkscreen label for these ports are 8...15 because these are the port numbers of the switch on these line cards. Ports 0...7 of the switches on the line cards connect to backplane switch network.

Once the hosts are attached, you can if you wish check out sub-clusters of 64 hosts before you connect the Clos boxes to each other through the Spine boxes.

There are an almost uncountable number of correct ways to do the Clos-to-Spine cabling. The spreadsheet gives you one highly systematic cabling, which is organized by numbering the cables 0...511. The way you read the spreadsheet is, for example, that Cable # 10 goes from (Spine 0, row 10, column 0) to (Clos 2, row 0, column 1). On an M3-SPINE line card, the "column" corresponds to the silkscreen port numbering, which is 0...7 from right to left.

Only in case you're curious: For the SPINE line cards, which tap directly to the backplane, the columns 0...7 correspond to backplane crossbar switches 0...7, whereas the line card or row # corresponds to the port number of that switch. (Confusing, yes, but this is the mathematical definition of the Clos network. ;-) See also this diagram (pdf file).

If you look at the first 16 cables (0...15) on the spreadsheet, they connect on the Spine end to the 16 ports of one crossbar switch, but on the Clos end are maximally spread across the Clos boxes. If you install the cables in order, you'll see the pattern and won't need to consult the spreadsheet for every cable.

Last revised: 1 July 2001