We report iSCSI performance measurements on Myrinet 10G-PCIE-8A-C NICs with Open iSCSI and the Myri10GE driver version Beta 0.9.0pre1 using 9000 Byte (jumbo) frames and TSO enabled.

All tests were between pairs of single-core dual-processor 2.6GHz Opterons with the Tyan S2895 motherboard connected point-to-point (switchless).

---

Linux

The TCP/IP stack was tuned and the iSCSI Target and Initiator were configured according to the instructions on the Open iSCSI Myri10GE Installation web page.

The latest Linux 2.6.15 kernels from kernel.org were used (2.6.15.6 or 2.6.15.7). For testing with software header digests and data digests, Cryptographic API under Cryptographic options and CRC32c CRC algorithm in the kernel were enabled. They are the kernel options CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C respectively.

On the Target, a LUN with a nullio device with 10000 sectors was used.

Bandwidth was measured with disktest. Install disktest from the lhcs-0.4-1.i386 rpm or from the disktest source tarball. 8 threads were used by setting the -K option to disktest to 8.

CPU utilization is approximate, measured with vmstat on the iSCSI Initiator.

No data or header digest:

1K blocks:
    Write Throughput 48269243B/s (46MB/s), IOPS 47137/s (85% cpu, 11% iowait)
    Read Throughput 47060787B/s (44MB/s), IOPS 45957/s (85% cpu, 11% iowait)
64K blocks:
    Write Throughput 832770321B/s (794MB/s), IOPS 12707  (79% cpu, 20% io wait)
    Read Throughput 592615833B/s (565MB/s), IOPS 9042/s (67% cpu, 32% io wait)
256K blocks:
    Write Throughput 791133115B/s (754MB/s), IOPS 3017/s (56% cpu, 50% iowait)
    Read Throughput 736082875B/s (701MB/s), IOPS 2807/s (60% cpu, 40% iowait)

Header digest only:

1K blocks:
    Write Throughput 47369796B/s (45MB/s), IOPS 46259/s (86% cpu, 11% iowait)
    Read Throughput 46584217B/s (44MB/s), IOPS 45492/s (85% cpu, 11% iowait)
64K blocks:
    Write Throughput 816047718B/s (778MB/s), IOPS 12451/s (74% cpu, 6% i/o wait)
    Read Throughput 590332996B/s (562MB/s), IOPS 9007/s  (66% cpu, 36% i/o wait)
256K blocks:
    Write Throughput 777562794B/s (741MB/s), IOPS 2966/s (53% cpu, 47% wait)
    Read Throughput 628350429B/s (599MB/s), IOPS 2396/s (60% cpu, 40% wait)

Header + Data digest:

1K blocks:
    Write Throughput 40185787B/s (38MB/s), IOPS 39243/s (87% cpu, 7% iowait)
    Read Throughput 42824704B/s (40MB/s), IOPS 41821/s (85% cpu, 14% iowait)
64K blocks:
    Write Throughput 264632183B/s (252MB/s), IOPS 4037/s, (64% cpu, 34% iowait)
    Read Throughput 239230429B/s (228MB/s), IOPS 3650/s (66% cpu, 35% iowait)
256K blocks:
    Write Throughput 245847381B/s (234MB/s), IOPS 937/s (67% cpu, 33% iowait)
    Read Throughput 254454442B/s (242MB/s), IOPS 970/s (55% cpu, 45% iowait)