Legion File IO Experiments


Norm Beekwilder, Brian White, Michael Walker

August 18, 2000

We gathered Legion file IO performance data under four different scenarios. All tests measured the total bandwidth of read operations concurrently performed on one or more files. We used MPI to synchronize the reads, which yields a fairly high level of concurrency (although clock skew might affect this slightly).

 

Scenario 1: N readers, 1 file on 1 host

Notes:

We ran the 800 MHz tests on a two-node net consisting of rigel and vega, both 800 MHz intels.

Peak performance was found at N = 20, yielding 4.19 MB/sec bandwidth. If we increased N further, we might find slightly higher results, but as N increases, the bandwidth should approach its limit, since the singular host is a bottleneck.

 

Scenario 2: N readers, N files on 1 host

 

Notes:

Peak performance was found at N = 5, yielding 4.17 MB/sec bandwidth.

 

 

 

 

 

 

 

 

 

 

Scenario 3: N readers, N files on N different hosts

Notes: Peak performance was found at N = 20, yielding 30.70 MB/sec bandwidth. These numbers should continue to scale with the number of readers.

 

 

Scenario 4: 1 reader, 2d file with N components spread over N hosts

Notes: Peak performance was at N = 8 partitions, yielding a total bandwidth of 2.1 MB/sec.

 

 

Original Experiment Schedule

Scenario

Experiment

Purpose

Expectation

1

1 reader, 1 server, both on different Centurion nodes

Baseline.

I壇 expect to see around 2 MBS

1

1 reader, 1 server, both on 800 mhz machines

Need to determine the baseline difference between 400 mhz and 800 mhz

For large reads I壇 expect to be disk/network limited. The processor is so fast.

I expect around 5 MBS

1

Increase the number of readers of a single file on a single server. Do 5,10,20 readers. All on Centurion.

See what the peak rate is, and how badly performance trails off under extreme load

I知 guessing we値l peak at around 4 MBS

2

Same as above, but have as many file objects on one host as there are readers.

To see how much of the difference is due to processes versus OS bottlenecks

I知 guessing we値l peak at around 5 MBS. The underlying network and protocol processing will be the limit.

3

Multiple readers and servers. For each reader there is a different server on a different host.

Test scalability.

I知 guessing that that performance will scale with the number of readers/servers. I don稚 think that switch bandwidth will be an issue.

4

Parallel File Object performance. Specifically a single reader with 1,2,4,8,16 components. Arranged so that the 1 MB reads span them all make sure the striping is right. All component files to be on different servers.

We致e not tested this in a really long time. Performance was not great, but then Legion performance used to really be poor.

No basis for a solid prediction. I知 guessing about 50% efficiency versus straight scaling.

 

Raw data

*****************************************************************

Test1: N readers, 1 file, 1 host: Peak: 4.19 MB/sec (N=20)

******************************************************************

n = 1 (baseline case): 1.72 MB/sec

run 1:

Node 0 transferred 1.0 Mbytes in 577.7 millisecs (1.7 Mbytes/sec)

run 2:

Node 0 transferred 1.0 Mbytes in 571.7 millisecs (1.7 Mbytes/sec)

run 3:

Node 0 transferred 1.0 Mbytes in 575.9 millisecs (1.7 Mbytes/sec)

run 4:

Node 0 transferred 1.0 Mbytes in 605.5 millisecs (1.7 Mbytes/sec)

run 5:

Node 0 transferred 1.0 Mbytes in 560.1 millisecs (1.8 Mbytes/sec)

n = 1 (800 MHz case): 1.98 MB/sec

run 1:

Node 0 transferred 1.0 Mbytes in 576.3 millisecs (1.7 Mbytes/sec)

run 2:

Node 0 transferred 1.0 Mbytes in 761.2 millisecs (1.3 Mbytes/sec)

run 3:

Node 0 transferred 1.0 Mbytes in 586.4 millisecs (1.7 Mbytes/sec)

run 4:

Node 0 transferred 1.0 Mbytes in 520.0 millisecs (1.9 Mbytes/sec)

run 5:

Node 0 transferred 1.0 Mbytes in 304.3 millisecs (3.3 Mbytes/sec)

n = 5: 3.34 MB/sec

run 1:

Node 4 transferred 1.0 Mbytes in 615.9 millisecs (1.6 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 817.4 millisecs (1.2 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 1.0 seconds (978.0 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.3 seconds (813.2 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (691.1 Kbytes/sec)

Total: 3.34 MB/sec

run 2:

Node 4 transferred 1.0 Mbytes in 604.3 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 818.8 millisecs (1.2 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 1.0 seconds (977.6 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.3 seconds (808.6 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.5 seconds (690.4 Kbytes/sec)

Total: 3.34 MB/sec

run 3:

Node 4 transferred 1.0 Mbytes in 621.4 millisecs (1.6 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 817.3 millisecs (1.2 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 1.0 seconds (978.9 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.3 seconds (808.9 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (691.8 Kbytes/sec)

Total: 3.34 MB/sec

run 4:

Node 4 transferred 1.0 Mbytes in 604.7 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 822.6 millisecs (1.2 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 1.0 seconds (976.1 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.3 seconds (808.2 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (692.0 Kbytes/sec)

Total: 3.34 MB/sec

run 5:

Node 4 transferred 1.0 Mbytes in 603.6 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 825.0 millisecs (1.2 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 1.0 seconds (981.2 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.3 seconds (810.3 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (692.1 Kbytes/sec)

Total: 3.34 MB/sec

n = 10: 3.85 MB/sec

run 1:

Node 1 transferred 1.0 Mbytes in 644.4 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 845.1 millisecs (1.2 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (954.4 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 1.3 seconds (788.1 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (674.1 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.7 seconds (589.8 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.0 seconds (507.4 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.2 seconds (466.7 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 2.4 seconds (427.4 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.6 seconds (393.9 Kbytes/sec)

Total: 3.85 MB/sec

run 2:

Node 1 transferred 1.0 Mbytes in 624.5 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 845.0 millisecs (1.2 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (959.2 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 1.3 seconds (801.6 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (681.1 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.7 seconds (595.7 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.0 seconds (513.7 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.2 seconds (471.8 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 2.4 seconds (431.5 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.6 seconds (396.8 Kbytes/sec)

total: 3.85 MB/sec

run 3:

Node 1 transferred 1.0 Mbytes in 626.5 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 846.9 millisecs (1.2 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (957.6 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 1.3 seconds (801.1 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (678.0 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.7 seconds (592.3 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 1.9 seconds (527.0 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.2 seconds (476.2 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 2.4 seconds (428.8 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.6 seconds (394.3 Kbytes/sec)

Total: 3.85 MB/sec

run 4:

Node 1 transferred 1.0 Mbytes in 629.7 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 850.0 millisecs (1.2 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (956.5 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 1.3 seconds (794.0 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.5 seconds (674.9 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.7 seconds (593.4 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 1.9 seconds (530.1 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.2 seconds (476.2 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 2.4 seconds (430.4 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.6 seconds (396.2 Kbytes/sec)

Total: 3.85 MB/sec

run 5:

Node 1 transferred 1.0 Mbytes in 626.7 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 840.4 millisecs (1.2 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (952.0 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.3 seconds (794.1 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 1.5 seconds (681.0 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.7 seconds (593.2 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 1.9 seconds (526.5 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.1 seconds (476.6 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 2.4 seconds (431.4 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.6 seconds (395.8 Kbytes/sec)

Total: 3.85 MB/sec

n = 20: 4.14 MB/sec

run 1:

Node 1 transferred 1.0 Mbytes in 653.2 millisecs (1.5 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 884.4 millisecs (1.1 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (925.1 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.3 seconds (778.7 Kbytes/sec)

Node 16 transferred 1.0 Mbytes in 1.5 seconds (663.6 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.8 seconds (580.0 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 2.0 seconds (515.1 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 2.2 seconds (459.9 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 2.4 seconds (419.5 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.7 seconds (386.1 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.9 seconds (354.5 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 3.1 seconds (330.6 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 3.3 seconds (309.0 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 3.5 seconds (289.7 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 3.8 seconds (272.9 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 4.0 seconds (257.7 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 4.2 seconds (244.2 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 4.4 seconds (232.0 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 4.6 seconds (221.4 Kbytes/sec)

Node 15 transferred 1.0 Mbytes in 4.8 seconds (211.4 Kbytes/sec)

Total: 4.17 MB/sec

 

run 2:

Node 1 transferred 1.0 Mbytes in 622.6 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 835.4 millisecs (1.2 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (955.5 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.3 seconds (770.5 Kbytes/sec)

Node 16 transferred 1.0 Mbytes in 1.6 seconds (660.3 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.8 seconds (580.9 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 2.0 seconds (513.9 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 2.2 seconds (461.6 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 2.4 seconds (420.3 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.7 seconds (383.4 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.9 seconds (354.3 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 3.1 seconds (329.3 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 3.3 seconds (308.6 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 3.5 seconds (288.5 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 3.8 seconds (271.1 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 4.0 seconds (257.4 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 4.2 seconds (243.3 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 4.4 seconds (231.6 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 4.7 seconds (219.4 Kbytes/sec)

Node 15 transferred 1.0 Mbytes in 4.9 seconds (211.0 Kbytes/sec)

Total: 4.08 MB/sec

run 3:

Node 1 transferred 1.0 Mbytes in 625.0 millisecs (1.6 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 832.0 millisecs (1.2 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 1.1 seconds (966.8 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.3 seconds (793.2 Kbytes/sec)

Node 16 transferred 1.0 Mbytes in 1.5 seconds (661.1 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.8 seconds (577.1 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 2.0 seconds (513.9 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 2.2 seconds (461.5 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 2.5 seconds (417.1 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.7 seconds (386.2 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.9 seconds (355.0 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 3.1 seconds (330.0 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 3.3 seconds (309.0 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 3.5 seconds (289.0 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 3.8 seconds (271.7 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 4.0 seconds (257.0 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 4.2 seconds (243.5 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 4.4 seconds (231.8 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 4.6 seconds (220.4 Kbytes/sec)

Node 15 transferred 1.0 Mbytes in 4.8 seconds (211.3 Kbytes/sec)

Total: 4.17 MB/sec

 

run 4:

Node 1 transferred 1.0 Mbytes in 628.0 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 888.1 millisecs (1.1 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (922.8 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.3 seconds (766.9 Kbytes/sec)

Node 16 transferred 1.0 Mbytes in 1.6 seconds (657.8 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.8 seconds (577.5 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 2.0 seconds (512.3 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 2.2 seconds (456.5 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 2.5 seconds (417.4 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.7 seconds (383.2 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.9 seconds (354.6 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 3.1 seconds (329.0 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 3.3 seconds (307.0 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 3.6 seconds (285.4 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 3.8 seconds (270.7 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 4.0 seconds (256.5 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 4.2 seconds (242.6 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 4.4 seconds (230.5 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 4.7 seconds (219.7 Kbytes/sec)

Node 15 transferred 1.0 Mbytes in 4.9 seconds (209.7 Kbytes/sec)

Total: 4.08 MB/sec

run 5:

Node 1 transferred 1.0 Mbytes in 624.5 millisecs (1.6 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 889.7 millisecs (1.1 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 1.1 seconds (917.6 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.3 seconds (765.6 Kbytes/sec)

Node 16 transferred 1.0 Mbytes in 1.5 seconds (702.4 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 1.8 seconds (570.3 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 2.0 seconds (504.4 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 2.3 seconds (452.5 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 2.5 seconds (410.7 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.7 seconds (378.4 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.9 seconds (350.2 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 3.2 seconds (324.7 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 3.4 seconds (303.2 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 3.6 seconds (284.8 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 3.8 seconds (269.4 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 4.0 seconds (253.4 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 4.3 seconds (240.9 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 4.5 seconds (228.9 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 4.7 seconds (218.3 Kbytes/sec)

Node 15 transferred 1.0 Mbytes in 4.9 seconds (209.3 Kbytes/sec)

Total: 4.19 MB/sec

*****************************************************************

Test2: N readers, N files, 1 host

*****************************************************************

n = 5: 2.97 MB/sec

run 1:

Node 0 transferred 1.0 Mbytes in 976.6 millisecs (1.0 Mbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.0 seconds (1013.2 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 2.0 seconds (512.0 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 2.2 seconds (470.9 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 2.4 seconds (432.8 Kbytes/sec)

Total: 2.08 MB/sec

run 2:

Node 3 transferred 1.0 Mbytes in 4.0 seconds (254.7 Kbytes/sec)

Node 0 transferred 1.0 Mbytes in 4.4 seconds (232.0 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 4.4 seconds (230.4 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 4.6 seconds (223.7 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 4.6 seconds (222.9 Kbytes/sec)

Total: 1.09 MB/sec

run 3:

Node 0 transferred 1.0 Mbytes in 1.4 seconds (738.1 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.5 seconds (691.7 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.6 seconds (652.8 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 1.7 seconds (603.5 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 1.7 seconds (600.2 Kbytes/sec)

Total: 2.94 MB/sec

run 4:

Node 3 transferred 1.0 Mbytes in 939.2 millisecs (1.1 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 959.9 millisecs (1.0 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (955.1 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 1.1 seconds (934.3 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.2 seconds (867.5 Kbytes/sec)

Total: 4.17 MB/sec

 

run 5:

Node 3 transferred 1.0 Mbytes in 950.4 millisecs (1.1 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 953.1 millisecs (1.0 Mbytes/sec)

Node 4 transferred 1.0 Mbytes in 1.0 seconds (986.0 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 1.0 seconds (977.7 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 1.1 seconds (921.5 Kbytes/sec)

Total: 4.55 MB/sec

n = 10: 3.50 MB/sec

run 1:

Node 0 transferred 1.0 Mbytes in 3.1 seconds (333.1 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 3.6 seconds (287.6 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 3.6 seconds (285.0 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 3.6 seconds (282.2 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 4.3 seconds (237.3 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 5.2 seconds (195.7 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 5.6 seconds (183.4 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 5.6 seconds (182.5 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 5.6 seconds (182.3 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 5.6 seconds (182.8 Kbytes/sec)

Total: 1.79 MB/sec

run 2:

Node 0 transferred 1.0 Mbytes in 1.4 seconds (737.3 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 1.5 seconds (683.7 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 2.1 seconds (497.5 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.6 seconds (395.3 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 2.6 seconds (392.0 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 2.7 seconds (384.7 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 2.7 seconds (381.7 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 2.7 seconds (381.4 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.7 seconds (380.8 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 2.7 seconds (379.9 Kbytes/sec)

Total: 3.70 MB/sec

run 3:

Node 5 transferred 1.0 Mbytes in 2.4 seconds (431.9 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 2.4 seconds (428.0 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 3.2 seconds (317.1 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 3.3 seconds (314.7 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 3.3 seconds (313.9 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 3.2 seconds (315.5 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 3.3 seconds (311.1 Kbytes/sec)

Node 0 transferred 1.0 Mbytes in 3.5 seconds (295.3 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 3.5 seconds (296.5 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 3.5 seconds (295.5 Kbytes/sec)

Total: 2.86 MB/sec

run 4:

Node 5 transferred 1.0 Mbytes in 1.8 seconds (560.4 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 1.9 seconds (546.1 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 2.0 seconds (516.9 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 2.0 seconds (505.6 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 2.1 seconds (499.5 Kbytes/sec)

Node 0 transferred 1.0 Mbytes in 2.1 seconds (489.2 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 2.1 seconds (479.2 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 2.2 seconds (476.0 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.4 seconds (432.0 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 2.5 seconds (406.9 Kbytes/sec)

Total: 4 MB/sec

run 5:

Node 0 transferred 1.0 Mbytes in 2.1 seconds (492.3 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 2.1 seconds (490.1 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 3.6 seconds (287.4 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 3.7 seconds (276.6 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 4.0 seconds (259.2 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 4.1 seconds (247.4 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 4.4 seconds (230.9 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 4.4 seconds (230.4 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 4.4 seconds (230.7 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 4.4 seconds (231.2 Kbytes/sec)

Total: 2.27 MB/sec

n = 20: 1.40 MB/sec

run 1:

Node 15 transferred 1.0 Mbytes in 5.0 seconds (206.7 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 5.2 seconds (197.7 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 7.8 seconds (130.6 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 8.6 seconds (119.4 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 10.3 seconds (99.4 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 10.8 seconds (95.1 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 12.4 seconds (82.8 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 12.6 seconds (81.2 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 12.9 seconds (79.4 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 13.1 seconds (78.3 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 17.1 seconds (60.0 Kbytes/sec)

Node 0 transferred 1.0 Mbytes in 17.1 seconds (59.8 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 18.2 seconds (56.2 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 18.5 seconds (55.4 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 18.7 seconds (54.8 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 19.7 seconds (52.0 Kbytes/sec)

Node 16 transferred 1.0 Mbytes in 19.8 seconds (51.8 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 20.7 seconds (49.6 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 21.3 seconds (48.2 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 21.5 seconds (47.7 Kbytes/sec)

Total: 0.93 MB/sec

run 2:

Node 16 transferred 1.0 Mbytes in 5.9 seconds (174.3 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 6.7 seconds (152.3 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 7.1 seconds (144.4 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 7.6 seconds (134.3 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 8.6 seconds (119.6 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 8.8 seconds (115.8 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 8.9 seconds (115.5 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 9.0 seconds (114.2 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 10.3 seconds (99.8 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 12.2 seconds (83.8 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 12.4 seconds (82.8 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 12.7 seconds (80.4 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 15.0 seconds (68.1 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 16.9 seconds (60.7 Kbytes/sec)

Node 15 transferred 1.0 Mbytes in 16.8 seconds (60.8 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 17.0 seconds (60.2 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 17.1 seconds (59.9 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 21.2 seconds (48.4 Kbytes/sec)

Node 0 transferred 1.0 Mbytes in 21.3 seconds (48.2 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 22.1 seconds (46.3 Kbytes/sec)

Total: 0.91 MB/sec

run 3:

Node 15 transferred 1.0 Mbytes in 2.2 seconds (468.4 Kbytes/sec)

Node 11 transferred 1.0 Mbytes in 2.4 seconds (434.8 Kbytes/sec)

Node 1 transferred 1.0 Mbytes in 2.5 seconds (410.9 Kbytes/sec)

Node 5 transferred 1.0 Mbytes in 2.5 seconds (406.6 Kbytes/sec)

Node 16 transferred 1.0 Mbytes in 2.6 seconds (400.4 Kbytes/sec)

Node 0 transferred 1.0 Mbytes in 2.6 seconds (394.7 Kbytes/sec)

Node 12 transferred 1.0 Mbytes in 2.7 seconds (373.9 Kbytes/sec)

Node 8 transferred 1.0 Mbytes in 3.7 seconds (279.3 Kbytes/sec)

Node 7 transferred 1.0 Mbytes in 4.0 seconds (256.7 Kbytes/sec)

Node 2 transferred 1.0 Mbytes in 4.2 seconds (243.4 Kbytes/sec)

Node 3 transferred 1.0 Mbytes in 4.9 seconds (209.2 Kbytes/sec)

Node 13 transferred 1.0 Mbytes in 5.4 seconds (191.0 Kbytes/sec)

Node 10 transferred 1.0 Mbytes in 6.0 seconds (169.3 Kbytes/sec)

Node 6 transferred 1.0 Mbytes in 6.9 seconds (149.3 Kbytes/sec)

Node 17 transferred 1.0 Mbytes in 7.4 seconds (137.5 Kbytes/sec)

Node 14 transferred 1.0 Mbytes in 8.2 seconds (125.3 Kbytes/sec)

Node 18 transferred 1.0 Mbytes in 8.4 seconds (122.1 Kbytes/sec)

Node 4 transferred 1.0 Mbytes in 8.5 seconds (120.8 Kbytes/sec)

Node 9 transferred 1.0 Mbytes in 8.5 seconds (120.9 Kbytes/sec)

Node 19 transferred 1.0 Mbytes in 8.5 seconds (120.7 Kbytes/sec)

Total: 2.35 MB/sec (???)

run 4:

 

*****************************************************************

Test3: N readers, N files, N hosts

*****************************************************************

n = 5: 7.69 MB/sec

run 1:

Node 4 transferred 1.0 Mbytes in 574.6 millisecs (1.7 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 576.8 millisecs (1.7 Mbytes/sec)

Node 3 transferred 1.0 Mbytes in 576.3 millisecs (1.7 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 579.4 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 586.5 millisecs (1.7 Mbytes/sec)

Total: 8.53 MB/sec

run 2:

Node 4 transferred 1.0 Mbytes in 574.0 millisecs (1.7 Mbytes/sec)

Node 3 transferred 1.0 Mbytes in 573.8 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 576.6 millisecs (1.7 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 576.4 millisecs (1.7 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 578.1 millisecs (1.7 Mbytes/sec)

Total: 8.65 MB/sec

run 3:

Node 0 transferred 1.0 Mbytes in 564.2 millisecs (1.8 Mbytes/sec)

Node 3 transferred 1.0 Mbytes in 573.8 millisecs (1.7 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 577.5 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 588.7 millisecs (1.7 Mbytes/sec)

Node 4 transferred 1.0 Mbytes in 662.2 millisecs (1.5 Mbytes/sec)

Total: 7.55 MB/sec

run 4:

Node 3 transferred 1.0 Mbytes in 575.1 millisecs (1.7 Mbytes/sec)

Node 4 transferred 1.0 Mbytes in 596.7 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 693.1 millisecs (1.4 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 806.3 millisecs (1.2 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 827.8 millisecs (1.2 Mbytes/sec)

Total: 6.04 MB/sec

run 5:

 

n = 10

run 1:

Node 0 transferred 1.0 Mbytes in 582.8 millisecs (1.7 Mbytes/sec)

Node 8 transferred 1.0 Mbytes in 585.8 millisecs (1.7 Mbytes/sec)

Node 4 transferred 1.0 Mbytes in 581.0 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 594.7 millisecs (1.7 Mbytes/sec)

Node 9 transferred 1.0 Mbytes in 575.4 millisecs (1.7 Mbytes/sec)

Node 5 transferred 1.0 Mbytes in 584.3 millisecs (1.7 Mbytes/sec)

Node 3 transferred 1.0 Mbytes in 608.8 millisecs (1.6 Mbytes/sec)

Node 7 transferred 1.0 Mbytes in 582.6 millisecs (1.7 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 660.9 millisecs (1.5 Mbytes/sec)

Node 6 transferred 1.0 Mbytes in 647.2 millisecs (1.5 Mbytes/sec)

Total: 15.13 MB/sec

n = 20: 28.20 MB/sec

run 1:

Node 3 transferred 1.0 Mbytes in 579.2 millisecs (1.7 Mbytes/sec)

Node 16 transferred 1.0 Mbytes in 579.2 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 605.5 millisecs (1.7 Mbytes/sec)

Node 4 transferred 1.0 Mbytes in 600.3 millisecs (1.7 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 614.7 millisecs (1.6 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 621.7 millisecs (1.6 Mbytes/sec)

Node 6 transferred 1.0 Mbytes in 578.5 millisecs (1.7 Mbytes/sec)

Node 12 transferred 1.0 Mbytes in 611.6 millisecs (1.6 Mbytes/sec)

Node 13 transferred 1.0 Mbytes in 582.5 millisecs (1.7 Mbytes/sec)

Node 18 transferred 1.0 Mbytes in 585.8 millisecs (1.7 Mbytes/sec)

Node 9 transferred 1.0 Mbytes in 591.0 millisecs (1.7 Mbytes/sec)

Node 10 transferred 1.0 Mbytes in 584.4 millisecs (1.7 Mbytes/sec)

Node 11 transferred 1.0 Mbytes in 632.5 millisecs (1.6 Mbytes/sec)

Node 15 transferred 1.0 Mbytes in 632.2 millisecs (1.6 Mbytes/sec)

Node 5 transferred 1.0 Mbytes in 677.3 millisecs (1.5 Mbytes/sec)

Node 17 transferred 1.0 Mbytes in 699.4 millisecs (1.4 Mbytes/sec)

Node 8 transferred 1.0 Mbytes in 728.9 millisecs (1.4 Mbytes/sec)

Node 7 transferred 1.0 Mbytes in 708.2 millisecs (1.4 Mbytes/sec)

Node 14 transferred 1.0 Mbytes in 645.3 millisecs (1.5 Mbytes/sec)

Node 19 transferred 1.0 Mbytes in 845.3 millisecs (1.2 Mbytes/sec)

Total: 23.66 MB/sec

 

run 2:

Node 12 transferred 1.0 Mbytes in 579.7 millisecs (1.7 Mbytes/sec)

Node 16 transferred 1.0 Mbytes in 586.8 millisecs (1.7 Mbytes/sec)

Node 3 transferred 1.0 Mbytes in 604.3 millisecs (1.7 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 606.9 millisecs (1.6 Mbytes/sec)

Node 9 transferred 1.0 Mbytes in 579.5 millisecs (1.7 Mbytes/sec)

Node 10 transferred 1.0 Mbytes in 575.2 millisecs (1.7 Mbytes/sec)

Node 19 transferred 1.0 Mbytes in 574.4 millisecs (1.7 Mbytes/sec)

Node 6 transferred 1.0 Mbytes in 579.4 millisecs (1.7 Mbytes/sec)

Node 8 transferred 1.0 Mbytes in 624.8 millisecs (1.6 Mbytes/sec)

Node 5 transferred 1.0 Mbytes in 606.9 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 640.5 millisecs (1.6 Mbytes/sec)

Node 15 transferred 1.0 Mbytes in 595.3 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 656.5 millisecs (1.5 Mbytes/sec)

Node 7 transferred 1.0 Mbytes in 603.3 millisecs (1.7 Mbytes/sec)

Node 4 transferred 1.0 Mbytes in 652.9 millisecs (1.5 Mbytes/sec)

Node 11 transferred 1.0 Mbytes in 611.8 millisecs (1.6 Mbytes/sec)

Node 18 transferred 1.0 Mbytes in 627.5 millisecs (1.6 Mbytes/sec)

Node 14 transferred 1.0 Mbytes in 613.3 millisecs (1.6 Mbytes/sec)

Node 17 transferred 1.0 Mbytes in 643.9 millisecs (1.6 Mbytes/sec)

Node 13 transferred 1.0 Mbytes in 651.4 millisecs (1.5 Mbytes/sec)

Total: 30.70 MB/sec

 

run 3:

Node 16 transferred 1.0 Mbytes in 573.5 millisecs (1.7 Mbytes/sec)

Node 4 transferred 1.0 Mbytes in 583.9 millisecs (1.7 Mbytes/sec)

Node 8 transferred 1.0 Mbytes in 589.7 millisecs (1.7 Mbytes/sec)

Node 1 transferred 1.0 Mbytes in 601.7 millisecs (1.7 Mbytes/sec)

Node 12 transferred 1.0 Mbytes in 595.7 millisecs (1.7 Mbytes/sec)

Node 2 transferred 1.0 Mbytes in 604.7 millisecs (1.7 Mbytes/sec)

Node 17 transferred 1.0 Mbytes in 584.3 millisecs (1.7 Mbytes/sec)

Node 3 transferred 1.0 Mbytes in 620.3 millisecs (1.6 Mbytes/sec)

Node 9 transferred 1.0 Mbytes in 597.3 millisecs (1.7 Mbytes/sec)

Node 14 transferred 1.0 Mbytes in 584.1 millisecs (1.7 Mbytes/sec)

Node 6 transferred 1.0 Mbytes in 592.7 millisecs (1.7 Mbytes/sec)

Node 18 transferred 1.0 Mbytes in 608.2 millisecs (1.6 Mbytes/sec)

Node 0 transferred 1.0 Mbytes in 661.6 millisecs (1.5 Mbytes/sec)

Node 5 transferred 1.0 Mbytes in 626.0 millisecs (1.6 Mbytes/sec)

Node 7 transferred 1.0 Mbytes in 607.0 millisecs (1.6 Mbytes/sec)

Node 11 transferred 1.0 Mbytes in 610.3 millisecs (1.6 Mbytes/sec)

Node 19 transferred 1.0 Mbytes in 620.6 millisecs (1.6 Mbytes/sec)

Node 10 transferred 1.0 Mbytes in 645.9 millisecs (1.5 Mbytes/sec)

Node 13 transferred 1.0 Mbytes in 657.9 millisecs (1.5 Mbytes/sec)

Node 15 transferred 1.0 Mbytes in 651.8 millisecs (1.5 Mbytes/sec)

Total: 30.23 MB/sec

 

********************************************************************

Test4: 1 reader, 2d file with N components, N hosts

********************************************************************

n = 1

run 1:

Creating int file with 262144 elements and 1 partitions

Node 0 transferred 1.0 Mbytes in 728.8 millisecs (1.4 Mbytes/sec)

n = 2

run 1:

Creating int file with 262144 elements and 2 partitions

Node 0 transferred 1.0 Mbytes in 591.5 millisecs (1.7 Mbytes/sec)

n = 4

run 1:

Creating int file with 262144 elements and 4 partitions

Node 0 transferred 1.0 Mbytes in 511.5 millisecs (2.0 Mbytes/sec)

n = 8

run 1:

Creating int file with 262144 elements and 8 partitions

Node 0 transferred 1.0 Mbytes in 476.1 millisecs (2.1 Mbytes/sec)

n = 16

run1:

Creating int file with 262144 elements and 16 partitions

Node 0 transferred 1.0 Mbytes in 559.0 millisecs (1.8 Mbytes/sec)

run 2:

Creating int file with 262144 elements and 16 partitions

Node 0 transferred 1.0 Mbytes in 547.0 millisecs (1.8 Mbytes/sec)

run 3:

Creating int file with 262144 elements and 16 partitions

Node 0 transferred 1.0 Mbytes in 550.9 millisecs (1.8 Mbytes/sec)

run 4:

Creating int file with 262144 elements and 16 partitions

Node 0 transferred 1.0 Mbytes in 553.7 millisecs (1.8 Mbytes/sec)

run 5:

Creating int file with 262144 elements and 16 partitions

Node 0 transferred 1.0 Mbytes in 545.6 millisecs (1.8 Mbytes/sec)