GM-2 and MX Scalability

Table of Contents

• GM-2.0.x Scalability
• GM-2.1.x Scalability
• MX Scalability

GM-2.0 Scalability

The following table lists the scalability limitations for GM-2.0 and Myrinet NICs with 2MB and 4MB of physical memory on a variety of host architectures.

4KB pages
    32-bit pointers
        ptr page product  NICS  cached PTEs    
        --- ---- -------- ----- ---------------
        32b  4kB PCI64C-2  2189  16384 (64MB)
        32b  4kB PCI64C-4  8317  32768 (128MB)
        32b  4kB PCI64C-8 20573  65536 (256MB)
        32b  4kB PCIXD-2   2876   8192 (32MB)
        32b  4kB PCIXD-4   7972  32768 (128MB)
        32b  4kB PCIXE-2   2876   8192 (32MB)
        32b  4kB PCIXE-4   7972  32768 (128MB)
    64-bit pointers
        ptr page product  NICS  cached PTEs    
        --- ---- -------- ----- ---------------
        64b  4kB PCI64C-2  1892   8192 (32MB)
        64b  4kB PCI64C-4  6988  32768 (128MB)
        64b  4kB PCI64C-8 19244  65536 (256MB)
        64b  4kB PCIXD-2   1566   8192 (32MB)
        64b  4kB PCIXD-4   6662  32768 (128MB)
        64b  4kB PCIXE-2   1566   8192 (32MB)
        64b  4kB PCIXE-4   6662  32768 (128MB)
8KB pages
    32-bit pointers
        ptr page product  NICS  cached PTEs    
        --- ---- -------- ----- ---------------
        32b  8kB PCI64C-2  2125  16384 (128MB)
        32b  8kB PCI64C-4  8253  32768 (256MB)
        32b  8kB PCI64C-8 20509  65536 (512MB)
        32b  8kB PCIXD-2   2812   8192 (64MB)
        32b  8kB PCIXD-4   7908  32768 (256MB)
        32b  8kB PCIXE-2   2812   8192 (64MB)
        32b  8kB PCIXE-4   7908  32768 (256MB)
    64-bit pointers
        ptr page product  NICS  cached PTEs    
        --- ---- -------- ----- ---------------
        64b  8kB PCI64C-2  2084   8192 (64MB)
        64b  8kB PCI64C-4  7180  32768 (256MB)
        64b  8kB PCI64C-8 19436  65536 (512MB)
        64b  8kB PCIXD-2   1758   8192 (64MB)
        64b  8kB PCIXD-4   6854  32768 (256MB)
        64b  8kB PCIXE-2   1758   8192 (64MB)
        64b  8kB PCIXE-4   6854  32768 (256MB)
16KB pages
    64-bit pointers
        ptr page product  NICS  cached PTEs    
        --- ---- -------- ----- ---------------
        64b 16kB PCI64C-2  1572   8192 (128MB)
        64b 16kB PCI64C-4  6668  32768 (512MB)
        64b 16kB PCI64C-8 18924  65536 (1024MB)
        64b 16kB PCIXD-2   1214   8192 (128MB)
        64b 16kB PCIXD-4   6310  32768 (512MB)
        64b 16kB PCIXE-2   1214   8192 (128MB)
        64b 16kB PCIXE-4   6310  32768 (512MB)

GM-2.1 Scalability

The following table lists the scalability limitations for GM-2.1 (specifically, GM-2.1.3) and Myrinet NICs with 2MB and 4MB of physical memory on a variety of host architectures.

• Note:

  The term connections refers to the number of nodes.

  The size in parentheses is the total number of cached PTEs times the page size, which is effectively how much DMAable memory can be registered with all PTEs in the cache.

32-bit pointers 4KB pages
   PCI64C-2 supports 1049 connections and 32768 PTE cache bins (64MB)
   PCI64C-4 supports 4113 connections and 65536 PTE cache bins (128MB)
   PCI64C-8 supports 10241 connections and 131072 PTE cache bins (256MB)
   PCIXD-2 supports 1156 connections and 16384 PTE cache bins (32MB)
   PCIXD-4 supports 3704 connections and 65536 PTE cache bins (128MB)
   PCIXE-2 supports 641 connections and 16384 PTE cache bins (32MB)
   PCIXE-4 supports 3189 connections and 65536 PTE cache bins (128MB)
64-bit pointers 4KB pages
   PCI64C-2 supports 900 connections and 16384 PTE cache bins (32MB)
   PCI64C-4 supports 3448 connections and 65536 PTE cache bins (128MB)
   PCI64C-8 supports 9576 connections and 131072 PTE cache bins (256MB)
   PCIXD-2 supports 760 connections and 8192 PTE cache bins (16MB)
   PCIXD-4 supports 3050 connections and 65536 PTE cache bins (128MB)
   PCIXE-2 supports 374 connections and 4096 PTE cache bins (8MB)
   PCIXE-4 supports 2535 connections and 65536 PTE cache bins (128MB)
32-bit pointers 8KB pages
   PCI64C-2 supports 8389625 connections and 32768 PTE cache bins (128MB)
   PCI64C-4 supports 4081 connections and 65536 PTE cache bins (256MB)
   PCI64C-8 supports 10209 connections and 131072 PTE cache bins (512MB)
   PCIXD-2 supports 1124 connections and 16384 PTE cache bins (64MB)
   PCIXD-4 supports 3672 connections and 65536 PTE cache bins (256MB)
   PCIXE-2 supports 609 connections and 16384 PTE cache bins (64MB)
   PCIXE-4 supports 3157 connections and 65536 PTE cache bins (256MB)
64-bit pointers 8KB pages
   PCI64C-2 supports 996 connections and 16384 PTE cache bins (64MB)
   PCI64C-4 supports 3544 connections and 65536 PTE cache bins (256MB)
   PCI64C-8 supports 9672 connections and 131072 PTE cache bins (512MB)
   PCIXD-2 supports 598 connections and 16384 PTE cache bins (64MB)
   PCIXD-4 supports 3146 connections and 65536 PTE cache bins (256MB)
   PCIXE-2 supports 341 connections and 8192 PTE cache bins (32MB)
   PCIXE-4 supports 2631 connections and 65536 PTE cache bins (256MB)
32-bit pointers 16KB pages
64-bit pointers 16KB pages
   PCI64C-2 supports 740 connections and 16384 PTE cache bins (128MB)
   PCI64C-4 supports 3288 connections and 65536 PTE cache bins (512MB)
   PCI64C-8 supports 9416 connections and 131072 PTE cache bins (1024MB)
   PCIXD-2 supports 584 connections and 8192 PTE cache bins (64MB)
   PCIXD-4 supports 2874 connections and 65536 PTE cache bins (512MB)
   PCIXE-2 supports 198 connections and 4096 PTE cache bins (32MB)
   PCIXE-4 supports 2359 connections and 65536 PTE cache bins (512MB)

MX Scalability

The scalability of MX is tunable at run-time when the MX module is loaded. Refer to the MX README for details.

• The default scalability limitation with PCIXD Myrinet interfaces is 4096 hosts. (Patrick prefers 8192 hosts for D cards.)
• The default scalability limitation with PCIXE Myrinet interfaces is 2048 hosts.

Last updated: 17 May 2006