1280-Host Myrinet Clos Cabling

This downloadable text file is a cabling list for a 1280-host Clos network.

The following is the information you'll need to interpret this cabling list.

The line-card slots in the E128 enclosure are numbered from top to bottom:

  M  monitoring line card
  0  port line card
  1  port line card
  ...
 15  port line card

Each line card has 8 ports. The array of 128 ports is described by "row," which is the slot number, 0...15, top to bottom as above, and "column," either 0...7 for M3-SPINE-8F line cards, or 8...15 for M3-SW16-8F line cards (right to left). The silkscreen labels on the line cards correspond to the column.

You should have 30 E128 switch units:

• 20 with line cards in the "Clos64+64" configuration. These units are designated Clos 0, Clos 1, ..., Clos 19.
• 10 with line cards in the "Clos128" configuration. These units are designated Spine 0, Spine 1, ..., Spine 9.

You should also have 1280 host cables and 1280 switch-to-switch cables.

The "Clos64+64" units are normally shipped with M3-SPINE-8F line cards in slots 0...7, and M3-SW16-8F line cards in slots 8...15. The switches in the "Clos128" configuration have M3-SW16-8F line cards in all 0...15 slots.

The cables to hosts are all from the ports of the M3-SW16-8F line cards of the Clos units. These line cards are assumed to be in rows 8...15. Once the hosts are attached, you can if you wish check out sub-clusters of 64 hosts before you connect the Clos units to each other through the Spine units.

There are an almost uncountable number of correct ways to do the Clos-to-Spine cabling. The text file cabling list gives you one highly systematic cabling, which is organized by numbering the cables 0...1279. The way you read the cabling list is, for example, that Cable # 10 goes from (Spine 0, row 10, column 8) to (Clos 10, row 0, column 0).

If you look at the first 20 cables (0...19) in the cabling list, they connect on the Spine end to 20 ports in a top-to-bottom, right-to-left pattern on one Spine switch, but on the Clos end are maximally spread to the same port across all 20 Clos units. If you install the cables in order, you'll see the pattern and won't need to consult the cabling list for every cable.

6 July 2003