Last updated: 01 November 2009
Myri-10G
10-Gigabit Ethernet
Windows CIFS, iSCSI and AOE (ATAoE) Initiator Performance Measurements
We report three different kinds of storage performance measurements for a NAS/SAN solution for Windows: CIFS, iSCSI and AOE. While iSCSI is using TCP/IP sockets, ATAoE uses Layer-2 and is therefore not routable. For the setup Myri-10G-PCIE-8A-C NICs were connected point-to-point (switchless). Tests were between paris of dual socket, quad-core Intel Nehalem CPUs. The Myri10GE driver was using 9000 Byte (jumbo) frames.
Iometer was used for generating traffic and measuring performance for CIFS, iSCSI and AOE connected hosts.
The Initiator was running Windows Server 2008 (x64). Myri10GE for Windows (NDIS 6.x) software was whql certified with Version 1.0.2 (x64). Microsofts's iSCSI Initiator and an Open Source AOE Initiator for Windows were used to connect to the target hosts. The target disk becomes visible as a (local) disk to which Iometer can write and read from. Transactions are then forwarded over the network to the target which performs the actions.
Target Configuration
For CIFS measurements we used a Windows Server 2008 which exported (shared) a RAMDISK which was then mounted by a Windows client. For iSCSI and AOE targets we used a Ubuntu Linux server which provided both the iscsi and aoe target software (at the time of the measurement no AOE target is available for Windows). As of the time of the measurement, the latest Linux 2.6.28 kernels from kernel.org were used. For testing with software header digests and data digests, Cryptographic API under Cryptographic options and CRC32c CRC algorithm in the kernel were enabled. They are the kernel options CONFIG_CRYPTO and CONFIG_CRYPTO_CRC32C respectively.
On the Target, a LUN with a nullio device with 10000 sectors was used.
Performance measurements as reported with Iometer
CIFS
W2k8 <-> W2k8 ; CIFS
Block Size Method IOPS (Throughput) MBytes/s avg resp time (ms) cpu load
4k rd 35807 139 3.53 17.77
4k wr 54385 212 2.34 12.82
8k rd 48175 376 2.64 14.88
8k wr 57595 449 1.02 12.84
16k rd 39096 614 3.86 9.05
16k wr 55811 872 2.29 15.66
32k rd 24102 673 7.06 9.26
32k wr 33994 1059 3.71 13.74
64k rd 6326 395 19.70 6.41
64k rd 17825 1114 7.17 10.19
W2K8 <-> W2K8, iSCSI (no header, data digest)
Block Size Method IOPS (Throughput) MBytes/s avg resp time (ms) cpu load
4k rd 90642 354 1.44 10.12
4k wr 110875 435 1.16 11.34
8k rd 60290 487 2.11 8.42
8k wr 88585 692 1.44 8.8
16k rd 38096 594 3.36 8.05
16k wr 50289 785 2.54 6.84
32k rd 21102 663 6.06 7.26
32k wr 27162 848 4.71 4.74
64k rd 11417 713 11.21 6.41
64k rd 13647 853 10.42 3.54
W2k8 <-> Ubuntu ; iSCSI (no digest) [iSCSI Enterprise Target]
Block Size Method IOPS (Throughput) MBytes/s avg resp time (ms) cpu load
4k rd 41943 163 3.74 5.46
4k wr 48462 189 3.12 4.86
8k rd 40036 312 3.19 7.87
8k wr 48618 379 2.63 6.18
16k rd 20747 324 6.17 5.48
16k wr 40565 633 3.15 5.45
32k rd 16948 612 7.59 5.85
32k wr 31226 979 4.08 5.33
64k rd 15690 980 8.16 5.94
64k rd 12260 766 10.49 2.84
Config on Ubuntu:
maxRecvDataSegmentLen=262144
MaxBurstLength=16776192
Maxxmitdatasegment=262144
maxoutstandingr2t=8
InitialR2T=No
ImmediateData=Yes
a) MaxRecvDataSegmentLength - Sets the maximum data segment length that can be received. This value should be set to multiples of PAGE_SIZE. Currently the maximum supported value is 64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB.
Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 8192.
b) MaxBurstLength - Sets the maximum amount of either unsolicited or solicited data the initiator may send in a single burst. Any amount of data exceeding this value must be explicitly solicited by the target. This value should be set to multiples of PAGE_SIZE. Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 262144.
c) MaxXmitDataSegmentLength - Sets the maximum data segment length that can be sent. This value actually used is the minimum of MaxXmitDataSegmentLength and the MaxRecvDataSegmentLength announced by the initiator. It should be set to multiples of PAGE_SIZE. Currently the maximum supported value is 64 * PAGE_SIZE, e.g. 262144 if PAGE_SIZE is 4kB. Configuring too large values may lead to problems allocating sufficient memory, which in turn may lead to SCSI commands timing out at the initiator host. The default value is 8192.
d) DataDigest - If set to "CRC32C" and the initiator is configured accordingly, the integrity of an iSCSI PDU's data segment will be protected by a CRC32C checksum. The default is "None". Note that data digests are not supported during discovery sessions.
e) MaxOutstandingR2T - Controls the maximum number of data transfers the target may request at once, each of up to MaxBurstLength bytes. The default is 1.
f) InitialR2T - If set to "Yes" (default), the initiator has to wait for the target to solicit SCSI data before sending it. Setting it to "No" allows the initiator to send a burst of FirstBurstLength bytes unsolicited right after and/or (depending on the setting of ImmediateData together with the command. Thus setting it to "No" may improve performance.
g) ImmediateData - This allows the initiator to append unsolicited data to a command. To achieve better performance, this should be set to "Yes". The default is "No".
h) DataPDUInOrder - It tells initiator if data has to be sent in order. Default is "Yes", which is also recommended.
i) DataSequencerInOrder - It tells initiator if data has to be sent in order. Default is "Yes", which is also recommended.
j) HeaderDigest - If set to "CRC32C" and the initiator is configured accordingly, the integrity of an iSCSI PDU's header segments will be protected by a CRC32C checksum. The default is "None".
Note that header digests are not supported during discovery sessions.
k) Wthreads - The iSCSI target employs several threads to perform the actual block I/O to the device. Depending on your hardware and your (expected) workload, the number of these threads may be carefully adjusted. The default value of 8 should be sufficient for most purposes.
Earlier tests were between pairs of single-core dual-processor 2.6GHz Opterons with the Tyan S2895 motherboard.
Block Size Method IOPS (Throughput) MBytes/s avg resp time (ms)
4k rd 110673 432 0.01ms
wr 89372 347 1.46ms
16k rd 32833 512 3.99ms
wr 34240 535 3.80ms
32k rd 18467 577 6.81ms
wr 18651 582 6.82ms
64k rd 10015 624 12.30ms
wr 10344 648 12.52ms
ATAoE
Block Size Method IOPS (Throughput) MBytes/s avg resp time (ms)
8k rd 57490 449 0.31
wr 44496 345 0.40
16k rd 34443 536 0.52
wr 25740 403 0.69
32k rd 17456 545 1.04
wr 13782 430 1.37
64k rd 8361 521 2.11
wr 6568 410 2.75
iSCSI (no header, data digest)
Block Size Method IOPS (Throughput) MBytes/s avg resp time (ms)
8k rd 41806 326 0.43
wr 48898 380 0.37
16k rd 24864 389 0.73
wr 30759 481 0.52
32k rd 15496 484 1.17
wr 17532 548 1.04
64k rd 8768 544 2.02
wr 11232 702 1.63
iSCSI, crc32
Block Size Method IOPS (Throughput) MBytes/s avg resp time (ms)
8k rd 25079 195 0.71
wr 38600 301 0.46
16k rd 13887 217 1.29
wr 19992 313 0.89
32k rd 8493 265 2.12
wr 11961 373 1.50
64k rd 5039 314 3.57
wr 6326 395 2.85
Last updated: 01 November 2009