Preload files to memory

[namtih]

I want to share a little modification, which brought the biggest improvement in overall handling for me until now. The idea was to automatically preload some important files to the memory after the system startup - like for example the databases.

Someone called it "a poor man's trick" as I only cat the files. So here it is:
1. nano /etc/rc.local
2. Insert before "exit 0"

Code:

#make an empty file to check if the commands were executed
touch /run/xstart_`date +%F_%T`
#preload database files (about 10mb for me)
cat /home/xbian/.xbmc/userdata/Database/*.db > /dev/null
#preload commoncache plugin if installed
cat /home/xbian/.xbmc/userdata/addon_data/script.common.plugin.cache/commoncache.db > /dev/null
#preload python files for youtube-plugin
cat /home/xbian/.xbmc/addons/plugin.video.youtube/*.py > /dev/null
cat /home/xbian/.xbmc/addons/plugin.video.youtube/*.pyc > /dev/null
#another empty file to check if all commands were executed and how long it took
touch /run/xstop_`date +%F_%T`

3. Do a reboot
4. check if the "xstart*" and "xstop*" got generated

Code:

ls -al /run/x*

As I said, the system feels much smoother while browsing through the lists, infoarts and addons. Hope you will also feel an improvement.

What could be other files worth preloading?

[CurlyMo]

Why not just create an addition tmpfs and rsync these files with the actual memory filesystem and the SD card?

[namtih]

An easy option would be to use the already existing tmpfs and put the files just under /run.
Do you have an example of the best/fastest rsync command for the task? For example to sync these files:
cat /home/xbian/.xbmc/userdata/Database/*.db
cat /home/xbian/.xbmc/userdata/addon_data/script.common.plugin.cache/commoncache.db

[namtih]

So here is a second version using the rsync command. I think I will keep it for now.
If you think other files could be worth preloading, please leave a short message here. Thanks.

Code:

#Preload Files
mkdir /run/xcache
touch /run/xcache/xstart_`date +%F_%T`
rsync -q /home/xbian/.xbmc/userdata/Database/*.db /run/xcache > /dev/null
rsync -q /home/xbian/.xbmc/userdata/addon_data/script.common.plugin.cache/commoncache.db /run/xcache > /dev/null
rsync -q /home/xbian/.xbmc/addons/plugin.video.youtube/*.py /run/xcache > /dev/null
rsync -q /home/xbian/.xbmc/addons/plugin.video.youtube/*.pyc /run/xcache > /dev/null
touch /run/xcache/xstop_`date +%F_%T`

The xstart and xstop commands are only there to check how long the script runs. For me it's done in 2sec (~10MB) - so no big deal.

[CurlyMo]

Of course, you should write the files back to the SD card on shutting down an reboot or else the state of the XBMC library is constantly lost.

[namtih]

Sorry, but that doesn't make sense to me. Persistent data is written to the filesystem, especially with the mount option "sync". I don't edit or link the database files with the rsync or cat command, so all database files are still at their original place and fully writable.
So in my opinion all changes are written to the persistent database files and are safely stored.

Or am I wrong?

[CurlyMo]

When you don't actually 'move' data around, i believe preloading doesn't make much sense...

BartOtten once had an setup in which he did something like this:
1) Create a new home folder for XBMC e.g. /home/xbian/.xbmc-pers
2) Move all data from /home/xbian/.xbmc to /home/xbian/.xbmc-pers
3) Make /home/xbian/.xbmc an tmpfs
4) On boot, copy the data from /home/xbian/.xbmc-pers to /home/xbian/.xbmc
5) On reboot or shutdown, Sync /home/xbian/.xbmc with /home/xbian/.xbmc-pers so you don't loose data.

Of course on big libraries the memory is too small to put the whole .xbmc folder in a tmpfs so you need to choose what you do and do not load into memory. The rest can become symlinks.

That's what i was suggesting.

[namtih]

Of course your described method would be the top solution. But it would be so difficult to store all files safely that I can't imagine any real good and failure-safe solution for daily use.
So I think I will keep "the poor man's trick" for now. It's not perfext, but a first step

I would like to investigate the preload scenario a little bit more and I think the following tool would be helpful:
http://hoytech.com/vmtouch/
Github: https://github.com/hoytech/vmtouch

But unfortunately there is no deb package. Are you perhaps able to build one?

[CurlyMo]

Here you go.

[raspberry_pd]

Hmmm, this is a very interesting and exciting thread. It's kinda disappointing to realise that the databases are not already in RAM however great to see that you guys are looking at the issue.

If you have any reasonably stable ideas you would like me to test, please let me know.

[namtih]

(7th Jan, 2013 03:19 PM)raspberry_pd Wrote:  If you have any reasonably stable ideas you would like me to test, please let me know.

If you want, you could test the method from my first post. You don't have to install anything and it will not brake any configuration. If you don't feel any difference in the performance or don't like it, just delete the added lines from the "/etc/rc.local".

Would be interesting to hear, if there is also an performance gain for other users while browsing through the menus.

[raspberry_pd]

(7th Jan, 2013 07:50 PM)namtih Wrote:  

(7th Jan, 2013 03:19 PM)raspberry_pd Wrote:  If you have any reasonably stable ideas you would like me to test, please let me know.

If you want, you could test the method from my first post. You don't have to install anything and it will not brake any configuration. If you don't feel any difference in the performance or don't like it, just delete the added lines from the "/etc/rc.local".

Would be interesting to hear, if there is also an performance gain for other users while browsing through the menus.

Could you provide me with a few commands with which I can test for any performance difference and get hard core metrics in return? Or is that a bit tricky? Quantitative data can be so much more concise than qualitative experience.

Either way I'll see what I can do.

[namtih]

No sorry, can't really provide any command to create a sort of statistics.
I think you can only test it while browsing through the menus and your personal impression.

[brantje]

What if an external mysql db is used, would this make sense?

[CurlyMo]

No