Forum
  • Search
  • Member List
  • Calendar
Hello There, Guest! Login Register — Login with Facebook

Preload files to memory
Thank you for your donation

Pages (2): 1 2 Next »
Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Threaded Mode | Linear Mode
Preload files to memory
5th Jan, 2013, 06:41 AM
Post: #1
namtih Offline
Tester
Posts: 269
Joined: Dec 2012
Reputation: 28
Preload files to memory
I want to share a little modification, which brought the biggest improvement in overall handling for me until now. The idea was to automatically preload some important files to the memory after the system startup - like for example the databases.

Someone called it "a poor man's trick" as I only cat the files. So here it is:
1. nano /etc/rc.local
2. Insert before "exit 0"
Code:
#make an empty file to check if the commands were executed
touch /run/xstart_`date +%F_%T`
#preload database files (about 10mb for me)
cat /home/xbian/.xbmc/userdata/Database/*.db > /dev/null
#preload commoncache plugin if installed
cat /home/xbian/.xbmc/userdata/addon_data/script.common.plugin.cache/commoncache.db > /dev/null
#preload python files for youtube-plugin
cat /home/xbian/.xbmc/addons/plugin.video.youtube/*.py > /dev/null
cat /home/xbian/.xbmc/addons/plugin.video.youtube/*.pyc > /dev/null
#another empty file to check if all commands were executed and how long it took
touch /run/xstop_`date +%F_%T`
3. Do a reboot
4. check if the "xstart*" and "xstop*" got generated
Code:
ls -al /run/x*

As I said, the system feels much smoother while browsing through the lists, infoarts and addons. Hope you will also feel an improvement.

What could be other files worth preloading?
Find all posts by this user
Quote this message in a reply
5th Jan, 2013, 06:44 AM
Post: #2
CurlyMo Offline
Registered
Posts: 3,501
Joined: Dec 2012
Reputation: 202
RE: Preload files to memory
Why not just create an addition tmpfs and rsync these files with the actual memory filesystem and the SD card?

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
5th Jan, 2013, 10:32 PM
Post: #3
namtih Offline
Tester
Posts: 269
Joined: Dec 2012
Reputation: 28
RE: Preload files to memory
An easy option would be to use the already existing tmpfs and put the files just under /run.
Do you have an example of the best/fastest rsync command for the task? For example to sync these files:
cat /home/xbian/.xbmc/userdata/Database/*.db
cat /home/xbian/.xbmc/userdata/addon_data/script.common.plugin.cache/commoncache.db
Find all posts by this user
Quote this message in a reply
6th Jan, 2013, 04:16 AM
Post: #4
namtih Offline
Tester
Posts: 269
Joined: Dec 2012
Reputation: 28
RE: Preload files to memory
So here is a second version using the rsync command. I think I will keep it for now.
If you think other files could be worth preloading, please leave a short message here. Thanks.
Code:
#Preload Files
mkdir /run/xcache
touch /run/xcache/xstart_`date +%F_%T`
rsync -q /home/xbian/.xbmc/userdata/Database/*.db /run/xcache > /dev/null
rsync -q /home/xbian/.xbmc/userdata/addon_data/script.common.plugin.cache/commoncache.db /run/xcache > /dev/null
rsync -q /home/xbian/.xbmc/addons/plugin.video.youtube/*.py /run/xcache > /dev/null
rsync -q /home/xbian/.xbmc/addons/plugin.video.youtube/*.pyc /run/xcache > /dev/null
touch /run/xcache/xstop_`date +%F_%T`

The xstart and xstop commands are only there to check how long the script runs. For me it's done in 2sec (~10MB) - so no big deal.
Find all posts by this user
Quote this message in a reply
6th Jan, 2013, 05:40 AM
Post: #5
CurlyMo Offline
Registered
Posts: 3,501
Joined: Dec 2012
Reputation: 202
RE: Preload files to memory
Of course, you should write the files back to the SD card on shutting down an reboot or else the state of the XBMC library is constantly lost.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
6th Jan, 2013, 08:19 PM
Post: #6
namtih Offline
Tester
Posts: 269
Joined: Dec 2012
Reputation: 28
RE: Preload files to memory
Sorry, but that doesn't make sense to me. Persistent data is written to the filesystem, especially with the mount option "sync". I don't edit or link the database files with the rsync or cat command, so all database files are still at their original place and fully writable.
So in my opinion all changes are written to the persistent database files and are safely stored.

Or am I wrong?
Find all posts by this user
Quote this message in a reply
6th Jan, 2013, 08:32 PM
Post: #7
CurlyMo Offline
Registered
Posts: 3,501
Joined: Dec 2012
Reputation: 202
RE: Preload files to memory
When you don't actually 'move' data around, i believe preloading doesn't make much sense...

BartOtten once had an setup in which he did something like this:
1) Create a new home folder for XBMC e.g. /home/xbian/.xbmc-pers
2) Move all data from /home/xbian/.xbmc to /home/xbian/.xbmc-pers
3) Make /home/xbian/.xbmc an tmpfs
4) On boot, copy the data from /home/xbian/.xbmc-pers to /home/xbian/.xbmc
5) On reboot or shutdown, Sync /home/xbian/.xbmc with /home/xbian/.xbmc-pers so you don't loose data.

Of course on big libraries the memory is too small to put the whole .xbmc folder in a tmpfs so you need to choose what you do and do not load into memory. The rest can become symlinks.

That's what i was suggesting.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
6th Jan, 2013, 09:57 PM
Post: #8
namtih Offline
Tester
Posts: 269
Joined: Dec 2012
Reputation: 28
RE: Preload files to memory
Of course your described method would be the top solution. But it would be so difficult to store all files safely that I can't imagine any real good and failure-safe solution for daily use.
So I think I will keep "the poor man's trick" for now. It's not perfext, but a first step Undecided

I would like to investigate the preload scenario a little bit more and I think the following tool would be helpful:
http://hoytech.com/vmtouch/
Github: https://github.com/hoytech/vmtouch

But unfortunately there is no deb package. Are you perhaps able to build one?
Find all posts by this user
Quote this message in a reply
6th Jan, 2013, 10:02 PM
Post: #9
CurlyMo Offline
Registered
Posts: 3,501
Joined: Dec 2012
Reputation: 202
RE: Preload files to memory
Here you go.


Attached File(s)
.gz  vmtouch.tar.gz (Size: 12.86 KB / Downloads: 5)

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
7th Jan, 2013, 03:19 PM
Post: #10
raspberry_pd Offline
Registered
Posts: 97
Joined: Dec 2012
Reputation: 6
RE: Preload files to memory
Hmmm, this is a very interesting and exciting thread. It's kinda disappointing to realise that the databases are not already in RAM however great to see that you guys are looking at the issue.

If you have any reasonably stable ideas you would like me to test, please let me know.
Find all posts by this user
Quote this message in a reply
7th Jan, 2013, 07:50 PM
Post: #11
namtih Offline
Tester
Posts: 269
Joined: Dec 2012
Reputation: 28
RE: Preload files to memory
(7th Jan, 2013 03:19 PM)raspberry_pd Wrote:  If you have any reasonably stable ideas you would like me to test, please let me know.

If you want, you could test the method from my first post. You don't have to install anything and it will not brake any configuration. If you don't feel any difference in the performance or don't like it, just delete the added lines from the "/etc/rc.local".

Would be interesting to hear, if there is also an performance gain for other users while browsing through the menus.
Find all posts by this user
Quote this message in a reply
8th Jan, 2013, 02:59 PM
Post: #12
raspberry_pd Offline
Registered
Posts: 97
Joined: Dec 2012
Reputation: 6
RE: Preload files to memory
(7th Jan, 2013 07:50 PM)namtih Wrote:  
(7th Jan, 2013 03:19 PM)raspberry_pd Wrote:  If you have any reasonably stable ideas you would like me to test, please let me know.

If you want, you could test the method from my first post. You don't have to install anything and it will not brake any configuration. If you don't feel any difference in the performance or don't like it, just delete the added lines from the "/etc/rc.local".

Would be interesting to hear, if there is also an performance gain for other users while browsing through the menus.

Could you provide me with a few commands with which I can test for any performance difference and get hard core metrics in return? Or is that a bit tricky? Quantitative data can be so much more concise than qualitative experience.

Either way I'll see what I can do.
Find all posts by this user
Quote this message in a reply
8th Jan, 2013, 09:58 PM
Post: #13
namtih Offline
Tester
Posts: 269
Joined: Dec 2012
Reputation: 28
RE: Preload files to memory
No sorry, can't really provide any command to create a sort of statistics.
I think you can only test it while browsing through the menus and your personal impression.
Find all posts by this user
Quote this message in a reply
8th Jan, 2013, 11:24 PM
Post: #14
brantje Offline
Tester
Posts: 305
Joined: Dec 2012
Reputation: 32
RE: Preload files to memory
What if an external mysql db is used, would this make sense?

XBian Skin Beta Testing
Clock Screensaver

If you liked my help, click on "Thanks" to show your appreciation.
Find all posts by this user
Quote this message in a reply
8th Jan, 2013, 11:30 PM
Post: #15
CurlyMo Offline
Registered
Posts: 3,501
Joined: Dec 2012
Reputation: 202
RE: Preload files to memory
No

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
« Next Oldest | Next Newest »
Pages (2): 1 2 Next »
Post Reply 


  • View a Printable Version
  • Send this Thread to a Friend
  • Subscribe to this thread
Forum Jump:

Current time: 14th May, 2025, 02:58 AM Powered By MyBB, © 2002-2025 MyBB Group.