Thank you for your donation


Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Bench Testing
19th Dec, 2014, 02:03 AM
Post: #1
Bench Testing
With Xbian now being supported on 3 different hardware (Cubox-i, Hummingboard and of course RPi) it dawned on me that we should have a standard to work with, so that with future upgrades and possibly new hardware (Banana Pi?) we would have standard performance details for all possible configurations.

So I had a look around and came up with Sysbench.

Full details are available in the link including installing and running the CPU and I/O tests.

For the CPU use sysbench --test=cpu --cpu-max-prime=2000 run (As 20000 might take the RPi a while)
For the I/O use sysbench --test=fileio --file-total-size=1G prepare (As no one has 150G spare!!!)
For the execution use ... sysbench --test=fileio --file-total-size=1G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run

To clear the file run sysbench --test=fileio --file-total-size=1G cleanup

CPU is total time
I/O is Kb/sec value

So if users could run the test on their machine, we will get a benchmark for different machines, which will assist in seeing if upgrades are faster, or if someone's machine is running slow (or faster for that matter). I don't have a basis yet for the norm, but I guess 10% either way is acceptable.

My results

Because the forum doesn't do formatting: Use
Model, Xbian Version, SD Card(speed), CPU result, I/O result

So:

Cubox-i4 Pro, RC3 (20141218), Kingston 4g(10), 23.9741s, 729.01Kb/sec

Please read before you post
How to post a log file

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
19th Dec, 2014, 02:10 AM
Post: #2
RE: Bench Testing
Already did this once for a Dutch forum. Once single threaded, once with all supported cores.

Radxa Rock Pro - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          389.0928s
    total number of events:              10000
    total time taken by event execution: 389.0740
    per-request statistics:
         min:                                 37.40ms
         avg:                                 38.91ms
         max:                                 42.89ms
         approx.  95 percentile:              42.61ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   389.0740/0.00

Radxa Rock Pro - 4 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          151.7998s
    total number of events:              10000
    total time taken by event execution: 607.1113
    per-request statistics:
         min:                                 37.50ms
         avg:                                 60.71ms
         max:                                 64.93ms
         approx.  95 percentile:              63.24ms

Threads fairness:
    events (avg/stddev):           2500.0000/57.16
    execution time (avg/stddev):   151.7778/0.01

HummingBoard i2eX Dual Core - 1 Thread
Code:
root@cubox:~# sysbench --test=cpu --cpu-max-prime=20000 run

sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          604.6435s
    total number of events:              10000
    total time taken by event execution: 604.6313
    per-request statistics:
         min:                                 60.27ms
         avg:                                 60.46ms
         max:                                 88.19ms
         approx.  95 percentile:              60.52ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   604.6313/0.00

HummingBoard i2eX Dual Core - 2 Threads
Code:
root@cubox:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          308.9665s
    total number of events:              10000
    total time taken by event execution: 617.9060
    per-request statistics:
         min:                                 60.27ms
         avg:                                 61.79ms
         max:                                125.25ms
         approx.  95 percentile:              63.14ms

Threads fairness:
    events (avg/stddev):           5000.0000/18.00
    execution time (avg/stddev):   308.9530/0.01

Hummingboard i1 Single Core - 1 thread
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          604.3114s
    total number of events:              10000
    total time taken by event execution: 604.2986
    per-request statistics:
         min:                                 60.27ms
         avg:                                 60.43ms
         max:                                 76.13ms
         approx.  95 percentile:              60.55ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   604.2986/0.00

Hummingboard i1 Single Core - 2 threads
Code:
root@cubox:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          604.6664s
    total number of events:              10000
    total time taken by event execution: 1209.2848
    per-request statistics:
         min:                                107.95ms
         avg:                                120.93ms
         max:                                188.97ms
         approx.  95 percentile:             130.38ms

Threads fairness:
    events (avg/stddev):           5000.0000/0.00
    execution time (avg/stddev):   604.6424/0.01

Raspberry Pi Model B OC (arm=950, core=450, sdram=500, overvolt=2)
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          987.5324s
    total number of events:              10000
    total time taken by event execution: 987.4880
    per-request statistics:
         min:                                 96.75ms
         avg:                                 98.75ms
         max:                                349.39ms
         approx.  95 percentile:             103.51ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   987.4880/0.00

Just because i can:
Raspberry Pi Model B OC (arm=950, core=450, sdram=500, overvolt=2) - 2 Threads
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          988.2666s
    total number of events:              10000
    total time taken by event execution: 1976.4349
    per-request statistics:
         min:                                188.68ms
         avg:                                197.64ms
         max:                                312.59ms
         approx.  95 percentile:             204.88ms

Threads fairness:
    events (avg/stddev):           5000.0000/0.00
    execution time (avg/stddev):   988.2174/0.00

Raspberry Pi Model B No OC
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          1496.4562s
    total number of events:              10000
    total time taken by event execution: 1496.3830
    per-request statistics:
         min:                                132.79ms
         avg:                                149.64ms
         max:                                367.95ms
         approx.  95 percentile:             190.28ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   1496.3830/0.00

Raspberry Pi Model B No OC - 2 Threads
Code:
root@pi:~# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          1498.2080s
    total number of events:              10000
    total time taken by event execution: 2996.2181
    per-request statistics:
         min:                                261.55ms
         avg:                                299.62ms
         max:                                524.10ms
         approx.  95 percentile:             334.34ms

Threads fairness:
    events (avg/stddev):           5000.0000/0.00
    execution time (avg/stddev):   1498.1091/0.02

BananaPi Pro - 1 Thread
Code:
root@bananapi:~# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          771.0006s
    total number of events:              10000
    total time taken by event execution: 770.9850
    per-request statistics:
         min:                                 76.97ms
         avg:                                 77.10ms
         max:                                 98.63ms
         approx.  95 percentile:              77.15ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   770.9850/0.00

BananaPi Pro - 2 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          389.5754s
    total number of events:              10000
    total time taken by event execution: 779.0987
    per-request statistics:
         min:                                 76.93ms
         avg:                                 77.91ms
         max:                                104.47ms
         approx.  95 percentile:              82.22ms

Threads fairness:
    events (avg/stddev):           5000.0000/38.00
    execution time (avg/stddev):   389.5493/0.02

Raspberry Pi 2 - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          789.2328s
    total number of events:              10000
    total time taken by event execution: 789.2191
    per-request statistics:
         min:                                 78.45ms
         avg:                                 78.92ms
         max:                                 87.58ms
         approx.  95 percentile:              78.99ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   789.2191/0.00

Raspberry Pi 2 - 4 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          196.5245s
    total number of events:              10000
    total time taken by event execution: 785.9563
    per-request statistics:
         min:                                 78.35ms
         avg:                                 78.60ms
         max:                                 79.30ms
         approx.  95 percentile:              79.01ms

Threads fairness:
    events (avg/stddev):           2500.0000/7.81
    execution time (avg/stddev):   196.4891/0.02

Raspberry Pi 2 OC - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          707.3641s
    total number of events:              10000
    total time taken by event execution: 707.3520
    per-request statistics:
         min:                                 70.51ms
         avg:                                 70.74ms
         max:                                 73.34ms
         approx.  95 percentile:              70.92ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   707.3520/0.00

Raspberry Pi 2 OC - 4 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          176.6624s
    total number of events:              10000
    total time taken by event execution: 706.4980
    per-request statistics:
         min:                                 70.47ms
         avg:                                 70.65ms
         max:                                 71.41ms
         approx.  95 percentile:              70.94ms

Threads fairness:
    events (avg/stddev):           2500.0000/2.55
    execution time (avg/stddev):   176.6245/0.03

Hummingboard 2 Edge - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!

Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          608.7290s
    total number of events:              10000
    total time taken by event execution: 608.7171
    per-request statistics:
         min:                                 60.68ms
         avg:                                 60.87ms
         max:                                 65.96ms
         approx.  95 percentile:              60.90ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   608.7171/0.00

Hummingboard 2 Edge - 4 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          152.4677s
    total number of events:              10000
    total time taken by event execution: 609.7461
    per-request statistics:
         min:                                 60.67ms
         avg:                                 60.97ms
         max:                                109.04ms
         approx.  95 percentile:              60.90ms

Threads fairness:
    events (avg/stddev):           2500.0000/1.87
    execution time (avg/stddev):   152.4365/0.02

ClearFog Pro - 1 Thread
Code:
# sysbench --test=cpu --cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          375.6172s
    total number of events:              10000
    total time taken by event execution: 375.6133
    per-request statistics:
         min:                                 37.48ms
         avg:                                 37.56ms
         max:                                 44.10ms
         approx.  95 percentile:              37.62ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00
    execution time (avg/stddev):   375.6133/0.00

ClearFog Pro - 2 Threads
Code:
# sysbench --test=cpu --cpu-max-prime=20000 --num-threads=2 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 2
Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          187.9854s
    total number of events:              10000
    total time taken by event execution: 751.8559
    per-request statistics:
         min:                                 53.03ms
         avg:                                 75.19ms
         max:                                 93.18ms
         approx.  95 percentile:              77.63ms

Threads fairness:
    events (avg/stddev):           2500.0000/2.00
    execution time (avg/stddev):   187.9640/0.01

Conclusion:
Code:
Intel 4460 4T                   :    6.9527
Intel G620T 2T                  :   19.5773
Intel 4460 1T                   :   22.6803
Intel G620T 1T                  :   38.3156
Radxa Rock Pro 4T               :  151.7778
HummingBoard Edge Quad Core 4T  :  152.4365
Raspberry Pi 2 OC 4T**          :  176.6245
ClearFog Pro 2T                 :  187.9640
Raspberry Pi 2 No OC 4T         :  196.4891
HummingBoard i2eX Dual Core 2T  :  308.9530
ClearFog Pro 1T                 :  375.6133
Radxa Rock Pro 1T               :  389.0740
BananaPi Pro 2T                 :  389.5493
HummingBoard i2eX Dual Core 1T  :  604.6313
HummingBoard i1 Single Core 1T  :  604.2986
HummingBoard i1 Single Core 2T  :  604.6424
HummingBoard Edge Quad Core 1T  :  608.7171
Raspberry Pi 2 OC 1T**          :  707.3520
BananaPi Pro 1T                 :  770.9850
Raspberry Pi 2 No OC 1T         :  789.2191
Raspberry Pi Model B OC 1T*     :  987.4880
Raspberry Pi Model B OC 2T*     :  988.2174
Raspberry Pi Model B No OC 1T   : 1496.3830
Raspberry Pi Model B No OC 2T   : 1498.1091
* XBian default OC
** Default Turbo OC

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
19th Dec, 2014, 02:25 AM
Post: #3
RE: Bench Testing
So now you can add to your list.

Cubox-i4 Pro 302.6553s

Threads 4!
Cubox-i4 Pro 155.7358s

Please read before you post
How to post a log file

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
19th Dec, 2014, 02:30 AM
Post: #4
RE: Bench Testing
Better you do it in the First Post.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
19th Dec, 2014, 02:35 AM
Post: #5
RE: Bench Testing
No I wanted a standard benchmark that was applicable to Xbian on the 3 platforms. Unless it is now running on an Intel G620T!! Also I'm mainly interested on the Xbian version, to see if any upgrades affect the performance - for the better or the worse.

Please read before you post
How to post a log file

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
19th Dec, 2014, 02:51 AM
Post: #6
RE: Bench Testing
G620 was to put things into perspective.

I did the test again on a Hummingboard i1 Single Core and the results are not significantly different (+/- 3 points) between now (latest XBian fully updated) and may (unusable XBian). So i think all results will still stand.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
19th Dec, 2014, 03:07 AM
Post: #7
RE: Bench Testing
It would seem that, not surprisingly that the hummingboard and the Cubox-i fair the same with duel threads but ramp it up to 4 and the CB wins.

So in perspective, it is NOT a fair comparison, as the thread count on multi cores machines makes a difference,

I wonder if a RPi is run with thread=8 (or 4) gives the same results as thread=1. If so, then I need to change the test to thread=max (if that option exists)

Please read before you post
How to post a log file

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
19th Dec, 2014, 03:17 AM
Post: #8
RE: Bench Testing
No, the more threads you use on a single core machine the worse it's going to perform. So indeed, especially the cores are the thing that count most.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
3rd Jan, 2015, 01:10 AM
Post: #9
RE: Bench Testing
Just added the Radxa Rock Pro

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
13th Jan, 2015, 05:15 PM
Post: #10
RE: Bench Testing
@IriDium,

prime number computation is the most CPU only test ever.
it is from 99.9% dependent on CPU frequency and CPU internal architecture (nr of math units, pipes in that case so CPU can do X multiplications in one clock tick even if being single core).

that's why 5000 primes it takes 90s on CortexA9(1ghz) core, 4s on i5(3.4ghz) and only 3s on i7(being only at 2ghz).
there is no involvement of memory, not even internal processor caches.

but try that :
sysbench --test=threads run

on 3.17 on RPI and 3.14 cubox, 3.10 cubox, 3.0.35 cubox. this test can make use of effective scheduler/locks handling.
and RPI on 3.17 gives ~~ as 3.14 cubox (with lower freq and inferior CPU design).

so context switching and wasted time on internal management in kernel
translated to user feel - responsiveness.

we used to use 2.4 kernels... in last century. don't remember it much but remember 3.0.35 on cubox. pain (but most Androids run it).

Please read rules and do a search before you post! . FAQs . How to post log file? . Looking for answers? Please start here
Find all posts by this user
Quote this message in a reply
6th Feb, 2015, 01:17 AM
Post: #11
RE: Bench Testing
Raspberry Pi 2 Single thread
Code:
pi@raspberrypi ~ $ sysbench --test=cpu --cpu-max-prime=20000 --num-threads=1 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          779.6814s
    total number of events:              10000
    total time taken by event execution: 779.6642
    per-request statistics:
         min:                                 77.94ms
         avg:                                 77.97ms
         max:                                117.34ms
         approx.  95 percentile:              77.96ms

Threads fairness:
    events (avg/stddev):           10000.0000/0.00

Raspberry Pi 2 multiple threads
Code:
pi@raspberrypi ~ $ sysbench --test=cpu --cpu-max-prime=20000 --num-threads=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000


Test execution summary:
    total time:                          195.8783s
    total number of events:              10000
    total time taken by event execution: 783.3436
    per-request statistics:
         min:                                 77.95ms
         avg:                                 78.33ms
         max:                                128.27ms
         approx.  95 percentile:              78.73ms

Threads fairness:
    events (avg/stddev):           2500.0000/9.22
    execution time (avg/stddev):   195.8359/0.03

Please read before you post
How to post a log file

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
6th Feb, 2015, 01:22 AM
Post: #12
RE: Bench Testing
Why is your faster? Wink

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
6th Feb, 2015, 02:29 AM
Post: #13
RE: Bench Testing
No idea but I ran it in Raspbian will a full upgrade. I also deleted/removed a lot of the unwanted programs.

Try upgrading and run it again. I know there was talk of some optimization of the code, so it may have been reflected in the upgrade.

That is a bit of a difference between the two.

Please read before you post
How to post a log file

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
6th Feb, 2015, 07:01 PM
Post: #14
RE: Bench Testing
Just rerun the test and the results are similar to my first run.

otal time: 195.8444s

Please read before you post
How to post a log file

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
8th Feb, 2015, 06:30 AM
Post: #15
RE: Bench Testing
I noticed that mine was slower because i had it downclocked with XBian default OC Tongue

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: