From alan@msc.edu Mon Jan 13 13:52:44 1992 Received: from noc.msc.edu by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA00892; Mon, 13 Jan 92 13:52:44 EST Received: from af.msc.edu by noc.msc.edu (5.65/MSC/v3.0(901107)) id AA07744; Mon, 13 Jan 92 12:58:03 -0600 Received: by af.msc.edu (5.57/MSC/v3.0(901107)) id AA00521; Mon, 13 Jan 92 12:58:02 -0600 Date: Mon, 13 Jan 92 12:58:02 -0600 From: alan@msc.edu Message-Id: <9201131858.AA00521@af.msc.edu> To: mccalpin Subject: Re: Sustainable Memory Bandwidth --- Update Newsgroups: comp.arch,comp.benchmarks In-Reply-To: Organization: Minnesota Supercomputer Center, Inc. Cc: Status: R In article you write: < < Bytes MFLOPS To: mccalpin Subject: Re: Sustainable Memory Bandwidth --- Update Status: R >>You have no results for the CM-200? Given that it has a 40% faster clock, >>it ought to rank somewhere between an 4 head C90 and an 8 head C90. > >The CM-2 results posted are for an 8 MHz machine, so the CM-200 should >only be 25% faster. (It does run at 10 MHz, doesn't it?). The CM-2 clock is 7 Mhz. I checked the output in your TEST directory to make sure Alex wasn't cheating with a faster clock. (Yes, it is possible to speed up the clock to 8 mhz for margin diagnostics, but users are never supposed to run codes at that speed.) 7 * 1.4285 = 10, speedup of 42.85%. From vjs@rhyolite.wpd.sgi.com Mon Jan 13 16:44:30 1992 Received: from SGI.COM by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA01098; Mon, 13 Jan 92 16:44:30 EST Received: from rhyolite.wpd.sgi.com by sgi.sgi.com via SMTP (911016.SGI/910110.SGI) for mccalpin@perelandra.cms.udel.edu id AA08629; Mon, 13 Jan 92 13:49:48 -0800 Received: by rhyolite.wpd.sgi.com (911016.SGI/911001.SGI) for @sgi.sgi.com:mccalpin@perelandra.cms.udel.edu id AA00790; Mon, 13 Jan 92 13:49:45 -0800 Date: Mon, 13 Jan 92 13:49:45 -0800 From: vjs@rhyolite.wpd.sgi.com (Vernon Schryver) Message-Id: <9201132149.AA00790@rhyolite.wpd.sgi.com> To: mccalpin Subject: Re: Sustainable Memory Bandwidth --- Update In-Reply-To: your article News-Path: sgi!mips!zaphod.mps.ohio-state.edu!wupost!darwin.sura.net!udel!nigel.ee.udel.edu!mccalpin Status: R It might be interesting to look at some of the 486 motherboards. I recently bought some more cache for a Micronics 486-33, and wrote a quick & dirty cache buster to see whether I had the jumpering right. I found that the board uses page mode access to DRAM, so that consecutive DRAM accesses are barely slower than on-chip-cache accesses. Psuedo-random accesses spread around bigger than my guess of the page size on 1M DRAM's show all three levels of the memory hierarchy. However, it is still competative with a 4D/25 or even other IRIS's on the same dumb benchmark. The loop of the dumb benchmark is no more than x += n(i), for n an int. Your "scale" operation looks similar but uses floats. I suppose the awesomely slow speed of floating point in the *86 would probably more than compensate for the fast consecutive access to DRAM. Vernon Schryver, vjs@sgi.com From bobg@phx.mcd.mot.com Fri Jan 17 12:34:57 1992 Received: from urbana.mcd.mot.com by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA03214; Fri, 17 Jan 92 12:34:57 EST Received: from [144.191.22.87] by urbana.mcd.mot.com (5.61/1.34) id AA23946; Fri, 17 Jan 92 11:40:04 -0600 Received: from nipperx.phx.mcd.mot.com by phx.mcd.mot.com (5.61/MCDMAIL IR04 [07/10/90 20:21]/1.34) id AA02089; Fri, 17 Jan 92 10:38:36 -0700 Received: by nipper.phx.mcd.mot.com (5.61/MCDMAIL IR04 [07/23/90 15:51]/1.34) id AA06889; Fri, 17 Jan 92 10:42:04 -0700 Date: Fri, 17 Jan 92 10:42:04 -0700 From: bobg@phx.mcd.mot.com (Bob Greiner) Message-Id: <9201171742.AA06889@nipper.phx.mcd.mot.com> To: mccalpin Subject: benchmark question Status: R mccalpin@perelandra.cms.udel.edu (John D. McCalpin) - I have a question on your bandwidth-limited Mflop benchmarks. Your benchmark measures bandwidth to global memory, without allowing cache-based processors to get benefits from reusing data in local memory. I understand that this is an accurate reflection of many large dense matrix applications. Yet you apparently allow the CM-2 to access local memory. Why not measure the saxpy in the a(i) = a(i) + scalar*b(i) format? This configuration of saxpy is frequently used. This would remove the penalty on cached machines' pre-read on a(i). Alternately, why not insist that at least one of a, b, or c be on a different processor in the CM-2? This would seem to be true when using saxpy for matrix multiply or Gaussian elimination. Thank you for your attention and response. - Bob Greiner, Motorola Computer Group, bobg@phx.mcd.mot.com Personal opinion only From jws@cs.mu.OZ.AU Wed Jan 22 19:18:45 1992 Received: from mulga.cs.mu.OZ.AU by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA02469; Wed, 22 Jan 92 19:18:45 EST Received: by mulga.cs.mu.OZ.AU (5.64+1.3.1+0.50); id AA14295 Thu, 23 Jan 1992 11:24:36 +1100 (from jws) Date: Thu, 23 Jan 1992 11:24:36 +1100 From: Jeff Schultz Message-Id: <9201230024.14295@mulga.cs.mu.OZ.AU> To: mccalpin Subject: Re: HP9000/7x0 Memory Bandwidth Question Cc: jws@cs.mu.OZ.AU Newsgroups: comp.arch References: Status: R In comp.arch you write: > (It seems clear to me that >this is really a 25 MHz machine, marketing drivel notwithstanding). Each to his own. For my applications, all of them, it's 50Mhz. >My 'stream' benchmark shows a sustained performance of 43.6 MB/s for >copying 64-bit numbers. Assuming that the HP cache reads in the line >containing the destination before I write it, that means that the >memory system is doing 3/2 times this, or 65.4 MB/s. This is only 1/3 >of the rated performance. I think that you might be tripping on the next cache-line prefetch that the system controller does. After a miss has been satisfied from memory, the controller reads the next line into a buffer from where it is supplied if it's the next miss. If it's not, then the buffer's thrown away. As I remember your programs, that's an extra 8 bus cycles for each cache miss. The usual source fiddles would fix this, but I suspect that if you keep making as much noise about it as you have then HP might fix the compilers :-). Jeff Schultz From cag@hpescag.fc.hp.com Thu Jan 23 10:46:07 1992 Received: from relay.hp.com by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA03109; Thu, 23 Jan 92 10:46:07 EST Received: from hpescag.fc.hp.com by relay.hp.com with SMTP (16.6/15.5+IOS 3.13) id AA02847; Thu, 23 Jan 92 07:51:59 -0800 Received: by hpescag.fc.hp.com (16.7/15.5+IOS 3.22) id AA01209; Thu, 23 Jan 92 08:54:02 -0700 From: Craig Gleason Message-Id: <9201231554.AA01209@hpescag.fc.hp.com> Subject: 720 Memory Bandwidth To: mccalpin Date: Thu, 23 Jan 92 8:54:01 MST Mailer: Elm [revision: 66.33] Status: R >The HP9000/720 blurb that I have says that the main memory to cache >bandwidth is "200 MB/s for the duration of a 64-byte transfer". It >also says that the bus is 64 bits wide. Dividing this through says >that the bus frequency is (200 MB/s / 8 bytes/cycle) = 25 MHz, or >one half of the nominal cpu clock speed. (It seems clear to me that >this is really a 25 MHz machine, marketing drivel notwithstanding). You can check the 1991 CompCon and ICCD proceedings for papers on the processor and memory systems. There are three pieces to the path you're talking about: The DRAM <-> memory controller bus, the memory controller <-> processor bus, and the processor <-> (I/D) cache connections. The DRAM bus is actually 64 bits wide and runs at 25MHz. The memory control to processor bus is 32 bits wide and runs at 50MHz. The cache buses are 32 bits (I) and 64 bits (D). Cache writes are done two (32 bit words) per three cycles on the I-cache and one (64 bit doubleword) per two cycles on the D-cache. It really is a 50MHz machine. The weird copy-in timing was used because we optimized the caches for read timing and would have had to use faster SRAM parts to run the writes at the same frequency (so why not just run the reads faster since they tend to limit performance more than writes?). The other decisions were made based on pin counts and time to market. We were taking the processor we use in our 9000/870 multiprocessors and re-working it as a single chip workstation CPU (with about a 12 month schedule from design start to first silicon). The memory controller was designed to work with the new processor. With the two chip design, we couldn't get the wide buses of the RS/6000. It's really a question of whose applications to focus on. We took the approach that you should focus on the majority of applications that are CPU intensive and will run cache-resident most of the time. IBM took more of a high end approach and designed a really good memory connection. Their price was a 6-9 chip set and no low end box. Ours was that we lose out on the most memory intensive applications. I have a 720 on my desk, and it's blazing fast for anything I do (primarily X, vi and EE CAD). Craig Gleason From cag@hpescag.fc.hp.com Thu Jan 23 12:17:36 1992 Received: from relay.hp.com by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA03407; Thu, 23 Jan 92 12:17:36 EST Received: from hpescag.fc.hp.com by relay.hp.com with SMTP (16.6/15.5+IOS 3.13) id AA05129; Thu, 23 Jan 92 09:22:16 -0800 Received: by hpescag.fc.hp.com (16.7/15.5+IOS 3.22) id AA01269; Thu, 23 Jan 92 10:24:13 -0700 From: Craig Gleason Message-Id: <9201231724.AA01269@hpescag.fc.hp.com> Subject: Re: 720 Memory Bandwidth To: mccalpin (John D. McCalpin) Date: Thu, 23 Jan 92 10:24:13 MST In-Reply-To: <9201231552.AA03153@perelandra.cms.udel.edu>; from "John D. McCalpin" at Jan 23, 92 10:52 am Mailer: Elm [revision: 66.33] Status: R > Just out of curiosity, I have one application that runs faster than > expected on the HP. It happens to have a scalar bottleneck that is > dominated by divides (solving lots of tridiagonal systems without > factoring them first). On the IBM RS/6000, 64-bit FP divides take > about 20 cycles --- How fast are they on the Snakes? > -- They take 12 cycles, so they should be quite a bit faster, given the higher frequency. The divides will also be put in the FP execution queue, executed when the units are free and their operands are available, and can overlap with integer instructions. So if you have enough integer stuff going on, there could be quite a lot of work done while waiting for the divides to complete. Craig From sandee@sun16.scri.fsu.edu Sun Feb 2 21:25:58 1992 Received: from sun16.scri.fsu.edu by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA15140; Sun, 2 Feb 92 21:25:58 EST Received: by sun16.scri.fsu.edu (5.61/Ultrix2.4-C) id AA01566; Sun, 2 Feb 92 21:25:42 -0500 Date: Sun, 2 Feb 92 21:25:42 -0500 From: sandee@sun16.scri.fsu.edu (Daan Sandee) Message-Id: <9202030225.AA01566@sun16.scri.fsu.edu> To: mccalpin Subject: Alex timings Cc: sandee@sun16.scri.fsu.edu Status: R Alex V. sent me an agitated letter asking basically what the hell I was doing supplying lower numbers about the long-vector benchmarks. As I remember it, you told me in October that Alex was reporting faster-than- speed-of-light results, but I never saw his numbers nor did I save the results which you posted. You say Alex was exceeding SoL by 1.6 or 160 depending whether you believe the code indeed did execute the repeat loop. Charlie Grassl then pointed out that something was wrong - I can't remember seeing that either. I am now playing with the program you sent me in December, which I assume is the one Alex was using, and indeed it gives incomprehensible results ; I am still assuming it's the timer he's using which is broke. Could you provide the results he sent you, and the MFLOP rates you computed from that ? Thanks. This is what I have so far. This is for SAXPY - others are similar. Typically, Asst/Scale *times* are a bit less than Sum/SAXPY times. The first two columns are program output. The next two are my computations. Now here I was assuming that both codes (with and w/o opt) did execute the repeat count of 100 - I know the MB/s reported by the program is not counting that. The longest vectors are when using all of memory. Again, this is *Alex's* program using CM_timer_read_cm_busy. SAXPY Vector MB/s mintime bytes/PE/ ops/PE/ length clock clock Unoptimized : CM-2 (7 MHz) 4K 10000 28.0604 2.1896 3.131 0.2610 CM-2 (7 MHz) 4K 20000 50.3708 2.4395 5.627 0.4689 CM-2 (7 MHz) 8K 10000 31.1904 1.9698 1.741 0.1450 CM-2 (7 MHz) 8K 20000 56.1208 2.1896 3.131 0.2610 CM-2 (7 MHz) 8K 40000 100.7416 2.4395 5.627 0.4689 CM-2 (7 MHz) 16K 20000 62.3807 1.9698 1.741 0.1450 CM-2 (7 MHz) 16K 40000 112.2415 2.1896 3.132 0.2610 CM-2 (7 MHz) 16K 80000 201.4831 2.4395 5.622 0.4685 CM-200 (10 MHz) 8K 40000 108.1830 2.2717 4.226 0.3522 Optimized : CM-2 (7 MHz) 4K 10000 48.5427 1.2657 5.417 0.4514 CM-2 (7 MHz) 4K 20000 88.3611 1.3907 9.862 0.8218 CM-2 (7 MHz) 8K 20000 97.0858 1.2657 5.417 0.4514 CM-2 (7 MHz) 8K 40000 176.7226 1.3907 9.862 0.8218 CM-2 (7 MHz) 16K 20000 107.7191 1.1407 3.006 0.2505 CM-2 (7 MHz) 16K 40000 194.1716 1.2657 5.418 0.4515 CM-2 (7 MHz) 16K 80000 353.4454 1.3907 9.862 0.8218 CM-200 (10 MHz) 8K 20000 104.6718 1.1740 4.089 0.3408 CM-200 (10 MHz) 8K 40000 189.2043 1.2989 7.391 0.6159 CM-200 (10 MHz) 8K 80000 345.1913 1.4239 13.484 1.1237 CM-200 (10 MHz) 8K 160000 634.6840 1.5489 24.792 2.0660 There is some method in this madness. At least the timer output is positively correlated with reality. Also, factoring out machine size gives identical results. However, comparing CM2 with CM200 shows the timer screws up differently dependent on machine type. On optimization : it appears the compiler does optimize away something. For CM2 8K / N=20000 : reported estimate from wall clock unoptimized 2.1896 3.1 optimized 1.2657 0.21 Okay, so I did the obvious thing and increased the repeat count, and the time (optimized OR unoptimized) didn't change. THEREFORE, the performance I computed above is actually a factor of 100 smaller. Which is nonsense. Now *you* were in doubt whether Alex was reporting a rate 1.6 or 160 x SoL. As I see it, he was reporting either 1.6 or 0.016 x SoL. Which is still nonsense. But then, I don't know which run he reported. I will have to write a test program demonstrating the timer. From sandee@sun16.scri.fsu.edu Sun Feb 2 21:51:16 1992 Received: from sun16.scri.fsu.edu by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA15145; Sun, 2 Feb 92 21:51:16 EST Received: by sun16.scri.fsu.edu (5.61/Ultrix2.4-C) id AA01577; Sun, 2 Feb 92 21:51:03 -0500 Date: Sun, 2 Feb 92 21:51:03 -0500 From: sandee@sun16.scri.fsu.edu (Daan Sandee) Message-Id: <9202030251.AA01577@sun16.scri.fsu.edu> To: mccalpin Subject: timings (correction) Status: R Oops. Minor error. Unoptimized, repeat loop *is* executed. CM2 8K N=10000 reported for SAXPY wallclock for entire run unoptimized NL=100 1.9698 60 unoptimized NL=1000 2.7748 600 optimized NL=100 1.1407 4 optimized NL=1000 1.1407 4 (Time of one SAXPY measurement is about 1/30 of entire run, in theory.) Which makes the point even clearer - I already estimate timer time reported = 1.0 +(reality/100) Or something. It's not linear but the point is that for small t the timer reports too large values and for large t, much too small values. From sandee@sun16.scri.fsu.edu Mon Feb 3 11:46:34 1992 Received: from sun16.scri.fsu.edu by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA16045; Mon, 3 Feb 92 11:46:34 EST Received: by sun16.scri.fsu.edu (5.61/Ultrix2.4-C) id AA02154; Mon, 3 Feb 92 11:46:16 -0500 Date: Mon, 3 Feb 92 11:46:16 -0500 From: sandee@sun16.scri.fsu.edu (Daan Sandee) Message-Id: <9202031646.AA02154@sun16.scri.fsu.edu> To: mccalpin Subject: Re: alex timings Cc: sandee@sun16.scri.fsu.edu Status: R Okay, now I have what Alex provided you. (a) He was running w/o opt, because the optimizer optimizes away the code. (b) His output and my output for : CM2 16K, N=80000, SAXPY, differ by a factor of 100 in the printout. I *know* the program is wrong, so his output is actually correct. The mystery is, how he gets the right printout with the wrong program. Of course, he may have hand-corrected the output. Or, the version he sent you is not the latest one. (c) So for a 16K CM2 he gets 20148.3135 MB/s which *I* compute to be 5.622 bytes/PE/clock which is faster than the theoretical rate which *I* think is 4 bytes/PE/clock. (Those emphatic I's mean : feel free to have a different opinion). (d) I still don't know (1) what you posted, because you are posting mflops, not transfer rates ; I compute 0.4685 flops/PE/clock or 1.679 GFlops. (2) what the Cray folks were complaining about. (The bottom line, though, is that I still think the timer is screwed. But I want to get the history straight before I get back to Alex.) From sandee@sun16.scri.fsu.edu Mon Feb 3 15:51:19 1992 Received: from sun16.scri.fsu.edu by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA16234; Mon, 3 Feb 92 15:51:19 EST Received: by sun16.scri.fsu.edu (5.61/Ultrix2.4-C) id AA02350; Mon, 3 Feb 92 15:51:08 -0500 Date: Mon, 3 Feb 92 15:51:08 -0500 From: sandee@sun16.scri.fsu.edu (Daan Sandee) Message-Id: <9202032051.AA02350@sun16.scri.fsu.edu> To: mccalpin Subject: Re: alex timings Cc: sandee@sun16.scri.fsu.edu Status: RO ||>(c) So for a 16K CM2 he gets 20148.3135 MB/s which *I* compute to be ||> 5.622 bytes/PE/clock which is faster than the theoretical rate which *I* ||> think is 4 bytes/PE/clock. ||> (Those emphatic I's mean : feel free to have a different opinion). ||I have been surprised that no one has come forward with a convincing, ||definitive explanation on this.... I verified it with the local Weitek expert (Tony Kennedy) and yes, one slice per clock. Only one hardware path between memory and the Weitek. ||I believe that I stated in the last posting that the transfer rates ||were computed by || MB/s = 12 * MFLOPS ||for the SAXPY operation. So you did. Sorry. Anyway, as the Cray folks objected against xfer rate directly, it's immaterial now. (By the way : time I mflops I mflops you report compute compute Scale 0.523 239.0 261.9 ?? Sum 0.716 174.6 159.3 ?? SAXPY 0.750 333.3 333.3) ||Alex's results showed about 80 GB/s for SAXPY, and no one can explain ||how results greater than 55 GB/s can be obtained. 81 GB for summing vs. 57.344 SoL = 1.4 x SoL (not 1.6 as you have been saying). So much for history. I have now made more runs to establish clearly that the timer was bad ; with the proper timer you get xfer rates of 2 bytes/clock which is half the SoL. Why not more, I don't know. New results, 7 MHz CM2 (this was 8K but it's independent of size) : (repeat count 1000) time MB/s bytes/PE/clock ops/PE/clock Mflops Assignment 50.8104 3224.5 1.80 Scaling 50.8113 3224.5 1.80 0.1125 201.5 Summing 67.1950 3657.4 2.04 0.0825 152.4 SAXPYing 70.3214 3494.8 1.95 0.1625 291.2 For Sum and SAXPY, this is identical to what I reported before with an entirely different program. My older number for Scale (corrected for clock speed) was 209 mflops. Haven't had a chance at a CM-200 yet. From sandee@sun16.scri.fsu.edu Mon Feb 3 17:22:31 1992 Received: from sun16.scri.fsu.edu by perelandra.cms.udel.edu (5.52/890607.SGI) (for mccalpin) id AA16656; Mon, 3 Feb 92 17:22:31 EST Received: by sun16.scri.fsu.edu (5.61/Ultrix2.4-C) id AA02396; Mon, 3 Feb 92 17:22:19 -0500 Date: Mon, 3 Feb 92 17:22:19 -0500 From: sandee@sun16.scri.fsu.edu (Daan Sandee) Message-Id: <9202032222.AA02396@sun16.scri.fsu.edu> To: mccalpin Subject: Re: alex timings Cc: sandee@sun16.scri.fsu.edu Status: R ||>So much for history. I have now made more runs to establish clearly that ||>the timer was bad ; with the proper timer you get xfer rates of 2 bytes/clock ||>which is half the SoL. Why not more, I don't know. || ||Well, perhaps it is something for some of you TMC folks to work on? I guess mostly compiler, i.e. Alex. Which is why it is difficult to convince him .... ||Thanks for the stuff! Some people had complained that I put 8 MHz ||results into the table, since most machines in the field are 7 MHz.... Yes, that was an unfortunate accident that that day, the only CM2's I could get to were running 8 MHz. (Some engineer changes the clock during maintenance then doesn't change it back. Who cares, except people running benchmarks.) CM200 (8K 10 MHz repeat count 1000 N=40000) time MB/s bytes/PE/clock ops/PE/clock Mflops Assignment 31.1989 5251.4 2.05 Scaling 31.2000 5251.3 2.05 0.1282 328.2 Summing 41.4336 5931.4 2.32 0.0965 247.1 SAXPYing 43.4731 5653.2 2.21 0.1840 471.1 From uunet.UU.NET!marge!boris Fri Feb 7 12:17:42 1992 Received: from relay2.UU.NET by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA28702; Fri, 7 Feb 92 12:17:42 -0500 Received: from uunet.uu.net (via LOCALHOST.UU.NET) by relay2.UU.NET with SMTP (5.61/UUNET-internet-primary) id AA22096; Fri, 7 Feb 92 12:16:47 -0500 Received: from meikous.UUCP by uunet.uu.net with UUCP/RMAIL (queueing-rmail) id 121509.21155; Fri, 7 Feb 1992 12:15:09 EST Received: from lisa.meiko.com by marge.meiko.com (4.1/SMI-4.1) id AA14042; Fri, 7 Feb 92 10:49:28 EST Date: Fri, 7 Feb 92 10:49:28 EST From: uunet.UU.NET!marge!boris (Boris Cownie) Message-Id: <9202071549.AA14042@marge.meiko.com> To: uunet.UU.NET!uunet!perelandra.cms.udel.edu!mccalpin Subject: Meiko i860 machine memory bandwidth test Cc: boris@uunet.UU.NET Status: R John, We make a machine with multiple intel i860 nodes each with upto 32 Mbytes of memory. These talk to each other via a reconfigurable network of tranputers. 2 transputers (8 links) have shared memory with each i860 and support a through routing message passing system that does not impact the i860 memory bandwidth. I think we probably have one of the fastests i860 memory systems there is. I'm not sure that your test program is not also a measure of how smart the compiler is at optimising the vector loops. The RS6000 has some smart stuff in there and might well just wack in a single instruction for the copy. If the sparc compiles this as a loop it not really a fair test of the memory system. Anyway the results I get do correspond to the real world times on other benchmarks. If the road is down hill and the wind is behind you each i860 runs at about the speed of the R6000/530. But looks at the super times we get with the VAST vectoriser ! Meiko MK096 dual i860 board. One i860 node (16 Mbytes of memory) Portland Group Compiler pgf77 (no optimisation) Timing calibration ; t = 38.28125 clicks Assignment: Rate = 41.79592 MB/s MFLOPS = 0.0000000E+00 Scaling: Rate = 29.68116 MB/s MFLOPS = 1.855072 Summing: Rate = 47.62790 MB/s MFLOPS = 1.984496 SAXPYing: Rate = 28.71028 MB/s MFLOPS = 2.392524 pgf77 -O4 Timing calibration ; t = 11.32813 clicks Assignment: Rate = 141.2414 MB/s MFLOPS = 0.0000000E+00 Scaling: Rate = 51.84810 MB/s MFLOPS = 3.240506 Summing: Rate = 120.4706 MB/s MFLOPS = 5.019608 SAXPYing: Rate = 39.13375 MB/s MFLOPS = 3.261146 pgf77 -O4 -Mvect Timing calibration ; t = 11.32813 clicks Assignment: Rate = 141.2414 MB/s MFLOPS = 0.0000000E+00 Scaling: Rate = 48.18823 MB/s MFLOPS = 3.011765 Summing: Rate = 149.8537 MB/s MFLOPS = 6.243903 SAXPYing: Rate = 37.69325 MB/s MFLOPS = 3.141104 Greenhills compiler plus Pacific Sierra VAST vectoriser f77apx -vast -OLMA (vectorise, optimise, unroll loops, no arithmetic checks) Timing calibration ; t = 5.024004 clicks Assignment: Rate = 318.4711 MB/s MFLOPS = 0.0000000E+00 Scaling: Rate = 99.52228 MB/s MFLOPS = 6.220142 Summing: Rate = 238.8533 MB/s MFLOPS = 9.952222 SAXPYing: Rate = 103.8493 MB/s MFLOPS = 8.654112 wow this one makes it really screams ! Cheers Boris From marc@david.SACLAY.cea.fr Tue Feb 25 05:10:05 1992 Received: from pamir.inria.fr by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA10259; Tue, 25 Feb 92 05:10:05 -0500 X400-Received: by /PRMD=inria/ADMD=atlas/C=FR/; Relayed; 25 Feb 92 11:09:11+0100 X400-Received: by /PRMD=CEA/ADMD=ATLAS/C=FR/; Relayed; 25 Feb 92 11:05:05+0100 Date: 25 Feb 92 11:05:05+0100 From: SBPM Marc GINGOLD Message-Id: <9202251005.AA24015@david.saclay.cea.fr> To: mccalpin Subject: stream Cc: marc@david.SACLAY.cea.fr Status: R Some tests: ====================================== Alliant VFX80 3 processors fortran 4.3.42 -O ====================================== Double precision appears to have 16 digits of accuracy Assuming 8 bytes per DOUBLEPRECISION word -------------------------------------- Timing calibration ; time = 66.9029980897903 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assign. : 41.6932 0.3798 0.3454 0.3554 Scaling : 41.5813 0.3805 0.3463 0.3566 Summing : 43.0837 0.5513 0.5013 0.5159 SAXPYing : 43.0082 0.5516 0.5022 0.5133 -------------------------------------- Single precision appears to have 7 digits of accuracy Assuming 4 bytes per default REAL word -------------------------------------- Timing calibration ; time = 46.18700 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 23.7530 0.3049 0.3031 0.3092 Scaling : 23.7545 0.3056 0.3031 0.3096 Summing : 25.1766 0.4325 0.4290 0.4362 SAXPYing : 25.2714 0.4314 0.4274 0.4409 ====================================== IBM RS6000 320 ====================================== -------------------------------------- Double precision appears to have 16 digits of accuracy Assuming 8 bytes per DOUBLEPRECISION word -------------------------------------- Timing calibration ; time = 43.0000000000000000 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 68.5714 .0791 .0700 .0800 Scaling : 60.0000 .0800 .0800 .0800 Summing : 65.4545 .1190 .1100 .1200 SAXPYing : 65.4545 .1121 .1100 .1200 -------------------------------------- Single precision appears to have 7 digits of accuracy Assuming 4 bytes per default REAL word -------------------------------------- Timing calibration ; time = 97.00000000 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 48.0000 .1594 .1500 .1900 Scaling : 36.0000 .2262 .2000 .2400 Summing : 41.5385 .2801 .2600 .3800 SAXPYing : 43.2000 .2651 .2500 .2700 ====================================== IBM RS6000 320H ====================================== *** Liste des alias ==> Commande alias *** -------------------------------------- Double precision appears to have 16 digits of accuracy Assuming 8 bytes per DOUBLEPRECISION word -------------------------------------- Timing calibration ; time = 31.0000000000000000 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 96.0000 .0572 .0500 .0600 Scaling : 96.0000 .0591 .0500 .0600 Summing : 90.0000 .0875 .0800 .1100 SAXPYing : 90.0000 .0831 .0800 .0900 *** Liste des alias ==> Commande alias *** -------------------------------------- Single precision appears to have 7 digits of accuracy Assuming 4 bytes per default REAL word -------------------------------------- Timing calibration ; time = 52.00000000 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 60.0001 .1200 .1200 .1200 Scaling : 40.0000 .1800 .1800 .1800 Summing : 54.0001 .2010 .2000 .2100 SAXPYing : 54.0001 .2000 .2000 .2000 ====================================== IBM RS6000 530 ====================================== *** Liste des alias ==> Commande alias *** -------------------------------------- Double precision appears to have 16 digits of accuracy Assuming 8 bytes per DOUBLEPRECISION word -------------------------------------- Timing calibration ; time = 30.0000000000000000 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 120.0000 .0422 .0400 .0500 Scaling : 120.0000 .0463 .0400 .0500 Summing : 102.8571 .0700 .0700 .0700 SAXPYing : 120.0000 .0600 .0600 .0600 *** Liste des alias ==> Commande alias *** -------------------------------------- Single precision appears to have 7 digits of accuracy Assuming 4 bytes per default REAL word -------------------------------------- Timing calibration ; time = 53.00000381 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 72.0001 .1031 .1000 .1100 Scaling : 45.0000 .1671 .1600 .1700 Summing : 60.0001 .1820 .1800 .1900 SAXPYing : 60.0001 .1831 .1800 .1900 Truly Yours. From sandee@sun16.scri.fsu.edu Wed Feb 26 11:17:46 1992 Received: from sun16.scri.fsu.edu by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA11824; Wed, 26 Feb 92 11:17:46 -0500 Received: by sun16.scri.fsu.edu (5.61/Ultrix2.4-C) id AA00738; Wed, 26 Feb 92 11:15:43 -0500 Date: Wed, 26 Feb 92 11:15:43 -0500 From: sandee@sun16.scri.fsu.edu (Daan Sandee) Message-Id: <9202261615.AA00738@sun16.scri.fsu.edu> To: mccalpin Subject: Re: alex timings Cc: sandee@sun16.scri.fsu.edu Status: R FYI, I sent Alex the whole story, program, results, explanation, on Feb 2, and haven't heard from him since. Have you? Daan Sandee sandee@sun16.scri.fsu.edu Thinking Machines Corporation Supercomputer Computations Research Institute, B-186 Florida State University, Tallahassee, FL 32306-4052 (904) 644-4490 From lhe@hare.udev.cdc.com Fri Mar 6 10:49:01 1992 Received: from hare.udev.cdc.com by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA22232; Fri, 6 Mar 92 10:49:01 -0500 Received: by hare.udev.cdc.com (5.61/1.34) id AA09537; Fri, 6 Mar 92 09:47:28 -0600 From: lhe@hare.udev.cdc.com (lh erickson 913-469-9264) Message-Id: <9203061547.AA09537@hare.udev.cdc.com> Subject: Bandwidth update? To: mccalpin Date: Fri, 6 Mar 92 9:47:27 CST X-Mailer: ELM [version 2.3 PL11] Status: R John, I was intrigued by your special fortran program for determining memory bandwidth (stream_s.f etc.). Would you please email your results to: lhe@hare.udev.cdc.com Thanks in advance for your help Ron Erickson CDC Performance Analysis. From lhe@hare.udev.cdc.com Fri Mar 6 13:17:56 1992 Received: from hare.udev.cdc.com by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA22384; Fri, 6 Mar 92 13:17:56 -0500 Received: by hare.udev.cdc.com (5.61/1.34) id AA04384; Fri, 6 Mar 92 12:16:14 -0600 From: lhe@hare.udev.cdc.com (lh erickson 913-469-9264) Message-Id: <9203061816.AA04384@hare.udev.cdc.com> Subject: Re: Bandwidth update - Thanks for your prompt assistance To: mccalpin (John D. McCalpin) Date: Fri, 6 Mar 92 12:16:14 CST In-Reply-To: <9203061754.AA22301@perelandra.cms.udel.edu> from "John D. McCalpin" at "Mar 6, 92 12:54 pm" X-Mailer: ELM [version 2.3 PL11] Status: R John, Thanks for your fast response. Your numbers will help in my analysis. Ron. From cmg@magnet.cray.com Sun Mar 29 15:35:30 1992 Received: from timbuk.cray.com by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA21060; Sun, 29 Mar 92 15:35:30 -0500 Received: from sequoia.cray.com by timbuk (4.1/CRI-MX 1.6ac) id AA02853; Sun, 29 Mar 92 14:36:41 CST Received: from magnet (magnet.cray.com) by sequoia.cray.com id AA16424; 4.1/CRI-5.5; Sun, 29 Mar 92 14:36:41 CST Received: by magnet (5.64/A/UX-2.00 Matt test 1b) id AA03520; Sun, 29 Mar 92 14:37:53 CST From: cmg@magnet.cray.com (Charles Grassl) Message-Id: <9203292037.AA03520@magnet> Subject: Re: stream To: mccalpin (John D. McCalpin) Date: Sun, 29 Mar 92 14:37:52 CST In-Reply-To: <9203231452.AA13533@perelandra.cms.udel.edu>; from "John D. McCalpin" at Mar 23, 92 9:52 am X-Mailer: ELM [version 2.2 PL0] Status: R Hello John; Were you yet able to run the STREAM program on the SX-3? I am very curious about the multiple CPU runs. There is much confusion as to how the SX-3 memory system works. I have even asked NEC analysts at the IEEE Supercomputing conferences and each one gives a different description. I think that your test could shed light of this confusion. The overall NEC SX-3 CPU-memory system is represented in the figure below. (I obtained this diagram from an NEC systems analyst at the Canadian Met site.) Each vector unit has three paths to the Main Memory Interface (MMI). The width of each path is related to the number of pipe sets in each CPU. For example, a four pipe set system has paths that are four 64-bit words wide. ------------- |---| +-------+ | | | |==>| | | | | | | CPU 0 | | | | |<--| | | |=====| | +-------+ | MMU 0 | | | | |-----| | +-------+ | | | |==>| | | | | | | CPU 1 | | | | |<--| | | | | M | +-------+ ------------- | M | ------------- | I | | | | | +-------+ | | | |==>| | | | | | | CPU 0 | | | | |<--| | | |=====| | +-------+ | MMU 1 | | | | |-----| | +-------+ | | | |==>| | | | | | | CPU 1 | | | | |<--| | ------------- |---| +-------+ NEC SX-3/44R Memory Architecture. The MMI has two sets of three paths to the two individual Main Memory Units (MMUs). We do not know what is the bandwidth of these paths. The STREAM test program should be able to resolve the width of each path to memory. Regards, Charles Grassl Cray Research, Inc. From cmg@magnet.cray.com Mon Mar 30 10:00:05 1992 Received: from timbuk.cray.com by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA22035; Mon, 30 Mar 92 10:00:05 -0500 Received: from sequoia.cray.com by timbuk (4.1/CRI-MX 1.6ac) id AA05919; Mon, 30 Mar 92 09:01:19 CST Received: from magnet (magnet.cray.com) by sequoia.cray.com id AA25166; 4.1/CRI-5.5; Mon, 30 Mar 92 09:01:18 CST Received: by magnet (5.64/A/UX-2.00 Matt test 1b) id AA03807; Mon, 30 Mar 92 09:02:32 CST From: cmg@magnet.cray.com (Charles Grassl) Message-Id: <9203301502.AA03807@magnet> Subject: Re: stream To: mccalpin (John D. McCalpin) Date: Mon, 30 Mar 92 9:02:31 CST In-Reply-To: <9203301358.AA21575@perelandra.cms.udel.edu>; from "John D. McCalpin" at Mar 30, 92 8:58 am X-Mailer: ELM [version 2.2 PL0] Status: R Hello John; > It is interesting to note that no performance advantage is gained by > using 32-bit arithmetic. The halved bandwidth corresponds to identical > computational rates. This is in complete agreement with my model > timing as well. The 32-bit arithmetic (addressing?) in the SX-3 is interesting. Your data shows no speed-up in 32-bit, I presume this is because the functional units still only has 64-bit paths. Are 32-bit results simply truncated 64-bit results? Is the way the RS/6000 works? NEC has recently announced the "SX-3 R" series. It has up to 1 (or 2?) Gwords of memory. Do you know if the memory on the SX-3 is byte addressable? If it is, and the address paths and registers are 32-bit, was this changed in order to address 1 Gword on the SX-3 R? Thanks and regards, -- Charles Grassl Cray Research, Inc. (612) 683-3531 cmg@cray.com From aspgmid@cid.aes.doe.CA Sat Jun 27 22:29:11 1992 Received: from cidsv01.cid.aes.doe.ca by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA02856; Sat, 27 Jun 92 22:29:11 -0400 Return-Path: Received: by cidsv01 (5.64-bind 1.5+ida UNSUPPORTED PROTOTYPE-MX) id AA06780; Sun, 28 Jun 92 02:29:21 GMT From: Michel Dansereau Message-Id: <9206280229.AA06780@cidsv01> Subject: Re: The stream benchmark results for an SX3-44 To: aspgmid@cidsv01.cid.aes.doe.ca (aspgmid) Date: Sun, 28 Jun 92 2:29:21 GMT Cc: mccalpin In-Reply-To: ; from "aspgmid" at Feb 12, 92 6:13 pm X-Mailer: ELM [version 2.3 PL11] Status: RO Mr. McCalpin, The following message is the one that I sent a while back to contribute to your system performance database. HNSX have asked me to add the precision that this is for a 1 CPU benchmark and not 4 as could be interpreted from the entry. I would appreciate it if you could make the correction and somehow make it public. Thanks for your time, P.S. Was the SS92 competition worth your while? -- ------------------------------------------------------------------------------ Michel Dansereau | Environment Canada Tel: (514) 421-4696 | Atmospheric Environment Service (AES) Email: mdansereau@cid.aes.doe.ca | Dorval Computing Centre (CMIS) Fax: (514) 421-4703 | Systems Group (CMISS) ------------------------------------------------------------------------------ aspgmid > > > > M. McCalpin > > I ran the Stream benchmark on our SX3/44 and obtained the following > results: > > -------------------------------------- > The overhead of calling second = 2.251414E-06 seconds. > Timing calibration ; time = 9.539266917478173 hundredths of a second > Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second > --------------------------------------------------- > Function Rate (MB/s) RMS time Min time Max time > Assignment:16940.9591 0.0331 0.0331 0.0331 > Scaling :15640.6689 0.0358 0.0358 0.0359 > Summing :22436.5215 0.0376 0.0374 0.0380 > SAXPYing :21972.1612 0.0383 0.0382 0.0388 > -------------------------------------- > > Modifications: > The time required to get the "time" was evaluated and subtracted from > the timing of the loops. > > The second() function was written to take advantage of the internal > free running "tic counter" ( f = 1 / 2.9E09 ). > > > > HNSX personnel on site have approved of these figure and mentioned > that they are conservative. > > > > -- > ------------------------------------------------------------------------------ > Michel Dansereau | Environment Canada > Tel: (514) 421-4696 | Atmospheric Environment Service (AES) > Email: mdansereau@cid.aes.doe.ca | Dorval Computing Centre (CMIS) > Fax: (514) 421-4703 | Systems Group (CMISS) > ------------------------------------------------------------------------------ > From innocente@tsmi19.sissa.it Mon Jul 20 10:35:27 1992 Received: from tsmi19.sissa.it by perelandra.cms.udel.edu via SMTP (911016.SGI/911001.SGI) for mccalpin id AA16133; Mon, 20 Jul 92 10:35:27 -0400 Received: by tsmi19.sissa.it (MX V2.3) id 3601; Mon, 20 Jul 1992 16:36:50 +0100 Date: Mon, 20 Jul 1992 16:36:50 +0100 From: Roberto Innocente To: MCCALPIN Cc: innocente@tsmi19.sissa.it Message-Id: <0095DDD3.10393DB4.3601@tsmi19.sissa.it> Subject: stream_d results Status: RO I'm including stream_d results for IBM rs/6000 550 and 560. The programs were compiled with xlf fortran v 2.2 xlf -O -P -Wp,-ea2478j stream_d.f I used n = 1000000 but the results varied by more then 10% between the runs. In any case the results on the 550 were superior to the results I found in your table for the 950. regards On the rs/6000 560: -------------------------------------- Double precision appears to have 16 digits of accuracy Assuming 8 bytes per DOUBLEPRECISION word -------------------------------------- Timing calibration ; time = 44.9999988079071045 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 228.5716 .0731 .0700 .0800 Scaling : 228.5713 .0804 .0700 .0900 Summing : 240.0002 .1104 .1000 .1300 SAXPYing : 240.0002 .1082 .1000 .1200 on the rs/6000 - 550 -------------------------------------- Double precision appears to have 16 digits of accuracy Assuming 8 bytes per DOUBLEPRECISION word -------------------------------------- Timing calibration ; time = 50.9999971836805344 hundredths of a second Increase the size of the arrays if this is <30 and your clock precision is =<1/100 second --------------------------------------------------- Function Rate (MB/s) RMS time Min time Max time Assignment: 240.0002 .0724 .0600 .0800 Scaling : 240.0002 .0794 .0600 .0900 Summing : 196.3638 .1172 .1100 .1300 SAXPYing : 216.0002 .1146 .1000 .1400 -- Roberto Innocente SISSA Scuola Internazionale Superiore di Studi Avanzati Via Beirut 4 - 34014 TRIESTE - Italy Phone +39-40-3787541 Fax +39-40-3787528 Internet: innocente@sissa.it Bitnet: innocent@ITSSISSA