+- Forum (http://forum.xbian.org)
+-- Forum: Software (/forum-6.html)
+--- Forum: Testing & Experimental (/forum-21.html)
+--- Thread: Bench Testing (/thread-2606.html)
Pages: 1 2
Bench Testing - IriDium - 19th Dec, 2014 02:03 AM
With Xbian now being supported on 3 different hardware (Cubox-i, Hummingboard and of course RPi) it dawned on me that we should have a standard to work with, so that with future upgrades and possibly new hardware (Banana Pi?) we would have standard performance details for all possible configurations.
So I had a look around and came up with Sysbench.
Full details are available in the link including installing and running the CPU and I/O tests.
For the CPU use sysbench --test=cpu --cpu-max-prime=2000 run (As 20000 might take the RPi a while)
For the I/O use sysbench --test=fileio --file-total-size=1G prepare (As no one has 150G spare!!!)
For the execution use ... sysbench --test=fileio --file-total-size=1G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
To clear the file run sysbench --test=fileio --file-total-size=1G cleanup
CPU is total time
I/O is Kb/sec value
So if users could run the test on their machine, we will get a benchmark for different machines, which will assist in seeing if upgrades are faster, or if someone's machine is running slow (or faster for that matter). I don't have a basis yet for the norm, but I guess 10% either way is acceptable.
My results
Because the forum doesn't do formatting: Use
Model, Xbian Version, SD Card(speed), CPU result, I/O result
So:
Cubox-i4 Pro, RC3 (20141218), Kingston 4g(10), 23.9741s, 729.01Kb/sec
RE: Bench Testing - CurlyMo - 19th Dec, 2014 02:10 AM
Already did this once for a Dutch forum. Once single threaded, once with all supported cores.
Radxa Rock Pro - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 389.0928s
total number of events: 10000
total time taken by event execution: 389.0740
per-request statistics:
min: 37.40ms
avg: 38.91ms
max: 42.89ms
approx. 95 percentile: 42.61ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 389.0740/0.00Radxa Rock Pro - 4 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 4
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 151.7998s
total number of events: 10000
total time taken by event execution: 607.1113
per-request statistics:
min: 37.50ms
avg: 60.71ms
max: 64.93ms
approx. 95 percentile: 63.24ms
Threads fairness:
events (avg/stddev): 2500.0000/57.16
execution time (avg/stddev): 151.7778/0.01HummingBoard i2eX Dual Core - 1 Thread
Code:
root@cubox:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 604.6435s
total number of events: 10000
total time taken by event execution: 604.6313
per-request statistics:
min: 60.27ms
avg: 60.46ms
max: 88.19ms
approx. 95 percentile: 60.52ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 604.6313/0.00HummingBoard i2eX Dual Core - 2 Threads
Code:
root@cubox:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 2
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 308.9665s
total number of events: 10000
total time taken by event execution: 617.9060
per-request statistics:
min: 60.27ms
avg: 61.79ms
max: 125.25ms
approx. 95 percentile: 63.14ms
Threads fairness:
events (avg/stddev): 5000.0000/18.00
execution time (avg/stddev): 308.9530/0.01Hummingboard i1 Single Core - 1 thread
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 604.3114s
total number of events: 10000
total time taken by event execution: 604.2986
per-request statistics:
min: 60.27ms
avg: 60.43ms
max: 76.13ms
approx. 95 percentile: 60.55ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 604.2986/0.00Hummingboard i1 Single Core - 2 threads
Code:
root@cubox:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 2
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 604.6664s
total number of events: 10000
total time taken by event execution: 1209.2848
per-request statistics:
min: 107.95ms
avg: 120.93ms
max: 188.97ms
approx. 95 percentile: 130.38ms
Threads fairness:
events (avg/stddev): 5000.0000/0.00
execution time (avg/stddev): 604.6424/0.01Raspberry Pi Model B OC (arm=950, core=450, sdram=500, overvolt=2)
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 987.5324s
total number of events: 10000
total time taken by event execution: 987.4880
per-request statistics:
min: 96.75ms
avg: 98.75ms
max: 349.39ms
approx. 95 percentile: 103.51ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 987.4880/0.00Just because i can:
Raspberry Pi Model B OC (arm=950, core=450, sdram=500, overvolt=2) - 2 Threads
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 2
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 988.2666s
total number of events: 10000
total time taken by event execution: 1976.4349
per-request statistics:
min: 188.68ms
avg: 197.64ms
max: 312.59ms
approx. 95 percentile: 204.88ms
Threads fairness:
events (avg/stddev): 5000.0000/0.00
execution time (avg/stddev): 988.2174/0.00Raspberry Pi Model B No OC
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 1496.4562s
total number of events: 10000
total time taken by event execution: 1496.3830
per-request statistics:
min: 132.79ms
avg: 149.64ms
max: 367.95ms
approx. 95 percentile: 190.28ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 1496.3830/0.00Raspberry Pi Model B No OC - 2 Threads
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 2
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 1498.2080s
total number of events: 10000
total time taken by event execution: 2996.2181
per-request statistics:
min: 261.55ms
avg: 299.62ms
max: 524.10ms
approx. 95 percentile: 334.34ms
Threads fairness:
events (avg/stddev): 5000.0000/0.00
execution time (avg/stddev): 1498.1091/0.02BananaPi Pro - 1 Thread
Code:
root@bananapi:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 771.0006s
total number of events: 10000
total time taken by event execution: 770.9850
per-request statistics:
min: 76.97ms
avg: 77.10ms
max: 98.63ms
approx. 95 percentile: 77.15ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 770.9850/0.00BananaPi Pro - 2 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 2
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 389.5754s
total number of events: 10000
total time taken by event execution: 779.0987
per-request statistics:
min: 76.93ms
avg: 77.91ms
max: 104.47ms
approx. 95 percentile: 82.22ms
Threads fairness:
events (avg/stddev): 5000.0000/38.00
execution time (avg/stddev): 389.5493/0.02Raspberry Pi 2 - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 789.2328s
total number of events: 10000
total time taken by event execution: 789.2191
per-request statistics:
min: 78.45ms
avg: 78.92ms
max: 87.58ms
approx. 95 percentile: 78.99ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 789.2191/0.00Raspberry Pi 2 - 4 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 4
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 196.5245s
total number of events: 10000
total time taken by event execution: 785.9563
per-request statistics:
min: 78.35ms
avg: 78.60ms
max: 79.30ms
approx. 95 percentile: 79.01ms
Threads fairness:
events (avg/stddev): 2500.0000/7.81
execution time (avg/stddev): 196.4891/0.02Raspberry Pi 2 OC - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 707.3641s
total number of events: 10000
total time taken by event execution: 707.3520
per-request statistics:
min: 70.51ms
avg: 70.74ms
max: 73.34ms
approx. 95 percentile: 70.92ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 707.3520/0.00Raspberry Pi 2 OC - 4 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 4
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 176.6624s
total number of events: 10000
total time taken by event execution: 706.4980
per-request statistics:
min: 70.47ms
avg: 70.65ms
max: 71.41ms
approx. 95 percentile: 70.94ms
Threads fairness:
events (avg/stddev): 2500.0000/2.55
execution time (avg/stddev): 176.6245/0.03Hummingboard 2 Edge - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 608.7290s
total number of events: 10000
total time taken by event execution: 608.7171
per-request statistics:
min: 60.68ms
avg: 60.87ms
max: 65.96ms
approx. 95 percentile: 60.90ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 608.7171/0.00Hummingboard 2 Edge - 4 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 4
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 152.4677s
total number of events: 10000
total time taken by event execution: 609.7461
per-request statistics:
min: 60.67ms
avg: 60.97ms
max: 109.04ms
approx. 95 percentile: 60.90ms
Threads fairness:
events (avg/stddev): 2500.0000/1.87
execution time (avg/stddev): 152.4365/0.02ClearFog Pro - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 375.6172s
total number of events: 10000
total time taken by event execution: 375.6133
per-request statistics:
min: 37.48ms
avg: 37.56ms
max: 44.10ms
approx. 95 percentile: 37.62ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00
execution time (avg/stddev): 375.6133/0.00ClearFog Pro - 2 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 2
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 187.9854s
total number of events: 10000
total time taken by event execution: 751.8559
per-request statistics:
min: 53.03ms
avg: 75.19ms
max: 93.18ms
approx. 95 percentile: 77.63ms
Threads fairness:
events (avg/stddev): 2500.0000/2.00
execution time (avg/stddev): 187.9640/0.01Conclusion:
Code:
Intel 4460 4T : 6.9527
Intel G620T 2T : 19.5773
Intel 4460 1T : 22.6803
Intel G620T 1T : 38.3156
Radxa Rock Pro 4T : 151.7778
HummingBoard Edge Quad Core 4T : 152.4365
Raspberry Pi 2 OC 4T** : 176.6245
ClearFog Pro 2T : 187.9640
Raspberry Pi 2 No OC 4T : 196.4891
HummingBoard i2eX Dual Core 2T : 308.9530
ClearFog Pro 1T : 375.6133
Radxa Rock Pro 1T : 389.0740
BananaPi Pro 2T : 389.5493
HummingBoard i2eX Dual Core 1T : 604.6313
HummingBoard i1 Single Core 1T : 604.2986
HummingBoard i1 Single Core 2T : 604.6424
HummingBoard Edge Quad Core 1T : 608.7171
Raspberry Pi 2 OC 1T** : 707.3520
BananaPi Pro 1T : 770.9850
Raspberry Pi 2 No OC 1T : 789.2191
Raspberry Pi Model B OC 1T* : 987.4880
Raspberry Pi Model B OC 2T* : 988.2174
Raspberry Pi Model B No OC 1T : 1496.3830
Raspberry Pi Model B No OC 2T : 1498.1091** Default Turbo OC
RE: Bench Testing - IriDium - 19th Dec, 2014 02:25 AM
So now you can add to your list.
Cubox-i4 Pro 302.6553s
Threads 4!
Cubox-i4 Pro 155.7358s
RE: Bench Testing - CurlyMo - 19th Dec, 2014 02:30 AM
Better you do it in the First Post.
RE: Bench Testing - IriDium - 19th Dec, 2014 02:35 AM
No I wanted a standard benchmark that was applicable to Xbian on the 3 platforms. Unless it is now running on an Intel G620T!! Also I'm mainly interested on the Xbian version, to see if any upgrades affect the performance - for the better or the worse.
RE: Bench Testing - CurlyMo - 19th Dec, 2014 02:51 AM
G620 was to put things into perspective.
I did the test again on a Hummingboard i1 Single Core and the results are not significantly different (+/- 3 points) between now (latest XBian fully updated) and may (unusable XBian). So i think all results will still stand.
RE: Bench Testing - IriDium - 19th Dec, 2014 03:07 AM
It would seem that, not surprisingly that the hummingboard and the Cubox-i fair the same with duel threads but ramp it up to 4 and the CB wins.
So in perspective, it is NOT a fair comparison, as the thread count on multi cores machines makes a difference,
I wonder if a RPi is run with thread=8 (or 4) gives the same results as thread=1. If so, then I need to change the test to thread=max (if that option exists)
RE: Bench Testing - CurlyMo - 19th Dec, 2014 03:17 AM
No, the more threads you use on a single core machine the worse it's going to perform. So indeed, especially the cores are the thing that count most.
RE: Bench Testing - CurlyMo - 3rd Jan, 2015 01:10 AM
Just added the Radxa Rock Pro
RE: Bench Testing - mk01 - 13th Jan, 2015 05:15 PM
@IriDium,
prime number computation is the most CPU only test ever.
it is from 99.9% dependent on CPU frequency and CPU internal architecture (nr of math units, pipes in that case so CPU can do X multiplications in one clock tick even if being single core).
that's why 5000 primes it takes 90s on CortexA9(1ghz) core, 4s on i5(3.4ghz) and only 3s on i7(being only at 2ghz).
there is no involvement of memory, not even internal processor caches.
but try that :
sysbench --test=threads run
on 3.17 on RPI and 3.14 cubox, 3.10 cubox, 3.0.35 cubox. this test can make use of effective scheduler/locks handling.
and RPI on 3.17 gives ~~ as 3.14 cubox (with lower freq and inferior CPU design).
so context switching and wasted time on internal management in kernel
translated to user feel - responsiveness.
we used to use 2.4 kernels... in last century. don't remember it much but remember 3.0.35 on cubox. pain (but most Androids run it).
RE: Bench Testing - IriDium - 6th Feb, 2015 01:17 AM
Raspberry Pi 2 Single thread
Code:
pi@raspberrypi ~ $ sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 1
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 779.6814s
total number of events: 10000
total time taken by event execution: 779.6642
per-request statistics:
min: 77.94ms
avg: 77.97ms
max: 117.34ms
approx. 95 percentile: 77.96ms
Threads fairness:
events (avg/stddev): 10000.0000/0.00Raspberry Pi 2 multiple threads
Code:
pi@raspberrypi ~ $ sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
Running the test with following options:
Number of threads: 4
Doing CPU performance benchmark
Threads started!
Done.
Maximum prime number checked in CPU test: 20000
Test execution summary:
total time: 195.8783s
total number of events: 10000
total time taken by event execution: 783.3436
per-request statistics:
min: 77.95ms
avg: 78.33ms
max: 128.27ms
approx. 95 percentile: 78.73ms
Threads fairness:
events (avg/stddev): 2500.0000/9.22
execution time (avg/stddev): 195.8359/0.03RE: Bench Testing - CurlyMo - 6th Feb, 2015 01:22 AM
Why is your faster?
RE: Bench Testing - IriDium - 6th Feb, 2015 02:29 AM
No idea but I ran it in Raspbian will a full upgrade. I also deleted/removed a lot of the unwanted programs.
Try upgrading and run it again. I know there was talk of some optimization of the code, so it may have been reflected in the upgrade.
That is a bit of a difference between the two.
RE: Bench Testing - IriDium - 6th Feb, 2015 07:01 PM
Just rerun the test and the results are similar to my first run.
otal time: 195.8444s
RE: Bench Testing - CurlyMo - 8th Feb, 2015 06:30 AM
I noticed that mine was slower because i had it downclocked with XBian default OC
