[IDEA] Dynamic priority process

[belese]

Hi,

i've done some tests to set priority in xbmc and other process in function of their utilisations

i've writted an xbmc_upstart_bridge that will send xbmc event to upstart .
events are a bit documented on github

For priority management,
xbmcplevel will be use.

it's a kind of XBMC priority profile which could take value from 0 to 5,
0, xbmc is stopped - 5 xbmc very high priority.

So, now, it doesn't do not many thing, just change the xbmc nice value, depending on it's state.

now, i'll use that defaut value :

xbmcplevel
1 : nice = 3
2 : nice = 0
3&4 : nice = -5
5 : nice = -7

to install :
(you must have git and fakeroot installed)

1. Install upstart_xbmc_bridge
   
   PHP Code:
   

   git clone https://github.com/belese/xbian-package-upstart-xbmc-bridge
cd  xbian-package-upstart-xbmc-bridge
./gen.package.sh
sudo dpkg -i xbian-package-upstart-xbmc-bridge1.0.deb 

2. Copy the upstart job
   Download attached file and copy on /etc/init/
3. Reload Upstart configuration
   sudo initctl reload-configuration
   

and that's all for now. xbmc nice value should be changed automaticaly when xbmc state change

In my TODO list :

• write priority job for download package
  but it's my first upstart script, and i'm not happy with this one, but take me time to understand all syntax.
• maybe change the cpu governor when xbmc is in use
• maybe change ionice when player is started
  

[Fred]

Nice work, I really like the idea. Have not been able to test it yet.
I think 5 levels is a bit too much. Personally I would only do two levels to keep it simple, (very) low level when screensaver is on, high level (7) when it's off. You wan't maximum smoothness when screensaver is off imo.

The latest download package already is set to lower priorities so they can run while XBMC is running. Don't really see why and how to implement dynamic priorities for this. Only thing I can think of is to higher the priority when a web-interface of one of the download packages is accessed, so this interface feels smoother. That would be really awesome but no idea how easy to implement.

[belese]

Yes 5 is maybe too much,
i'll do some test, and it's easier to remove level than add level, for now level 3 and 4 do the same.

in upstart script, it easy to react on more than one level, so 5 level could be interpreted as number level as you want.
ex :

start on xbmcplevel [12] will start when screensaver is on
start on xbmcplevel [345] will start when screensaver is off

or you can use other event (screensaver, player and library) that bridge send

For download package, the idea is create a xbmc_priority.conf like file, but when xbmc is low priority, it could set the download package higher priority and vice-versa.

Quote:Only thing I can think of is to higher the priority when a web-interface of one of the download packages is accessed

i was thinking about that too, look a bit in maraschino package, and should be easy to send an upstart event when a connection is receive, and create a upstart job that higher nice level for the process during 2 minutes (or more/less), and restore defaut priority after.

EDIT : Ater reflexion, you need to be root to send an upstart event, so this is not the solution
or by writing a script that send upstart event and add it to sudoers file

[f1vefour]

Have you tried "kill -STOP xbmcpid" and resume it with "kill -CONT xbmcpid" instead of renicing the process?

I haven't actually tried but if it worked out it would effectively pause/resume xbmc, this would save many more CPU cycles.

This would be a good implementation for those whom have no need for xbmc to update in the background.

[Fred]

I did think about that once, but when you for instance pause xbmc at screensaver start it will never be able to come out that again (because it's paused). Or you have to manually resume it again.

[belese]

@f1vefour :
no didn't try, don't know that. It can't be use for xbmc like fred explain, as xbmc is the 'Master', but could be done for other package in case xbmc take very high priority.
@Fred :
i think a found a generic way for all service that have a web interface to change priority when we access it,
without patching any of them.

The idea would be a very light weight port forwarding daemon. So you can access application with his default port, and work like before, or acces it with another 'priority' port, that will just do a port redirection, and increase nice value for x seconds/minutes

this script could be a base :
http://code.activestate.com/recipes/483730/

i'll will take a look to this next week.

[Fred]

That would be really great, looking forward to it.

[mk01]

I'm going to play with this, ... how could I miss this thread?

just general comment to niceness level. many times, it has two sides. like for example starting process of xbmc.binary. xbmc starting depends on many other tasks in the system, which are triggered, but not directly started by xbmc (so will not get it's priority). and if xbmc is too high, those processes won't finish their jobs on which xbmc is dependent. so then it will even start longer, as with priority lower.

on multicore machines this is not visible, but BCM is pain in the ass. so attention!

[belese]

>comment to niceness level

yes, idea would be more to decrease priority for service, especially xbmc when not used.
value i set is experimental and use the old default value, i see you set default xmc priority to -3 now, is it for this reason, or for better cpu share between others service?

[mk01]

(13th Jul, 2013 01:11 AM)belese Wrote:  EDIT : Ater reflexion, you need to be root to send an upstart event, so this is not the solution
or by writing a script that send upstart event and add it to sudoers file

we can easily put /sbin/initctl into sudo allowed scripts for xbian.

---

(13th Jul, 2013 07:09 PM)belese Wrote:  The idea would be a very light weight port forwarding daemon. So you can access application with his default port, and work like before, or acces it with another 'priority' port, that will just do a port redirection, and increase nice value for x seconds/minutes

for this isn't just netcounters watch enough?

---

anyhow, i did some changes to the package and aligned with package-xbmc. package-xbmc is on git already, you xbian-bridge-xbmc is at my fork.

[belese]

what do you mean by netounter watch?
if it's check network activity, could we do this for a specific port?
i've check before writing the dameon, but not find.
otherwise, i've writted the daemon, redirection work fine. will do the upstart job

and great change of course!!
just a question,
my first idea was to send xbmcplevel within a upstart script,
and let the bridge only send screensaver, player and library

i've read a big part of the great upstart book, but misunderstood ENV and EXPORT stanzas,
is it possible to export SCREENSAVER in a job that start on screensaver, and use $SCREENSAVER in another upstart job (that start after of course)?

---

after deep look on google :-)
i've found pcap, is ii something like this you mean?
it can be better than port forwarding.

[belese]

i've rewrited daemon for net connection,
i check now on low level,
here's the package :
https://github.com/belese/xbian-package-upstart-ipconnection-bridge

some default conf value can be set in beginning of dameon script

by default,
it will only emit event for tcp and upd connection, on local network.
event explnation are on github

to use in a upstart script,
it should be something like start on stanzas
ex for xbmc :

start on ipconnectionin PORT=8080 PROTOCOL=6
#do stuff here

[mk01]

(18th Jul, 2013 09:32 AM)belese Wrote:  what do you mean by netounter watch?
if it's check network activity, could we do this for a specific port?
i've check before writing the dameon, but not find.
otherwise, i've writted the daemon, redirection work fine. will do the upstart job

not only specific port, but even specific service over port - so it can not be fooled by p2p traffic on port 80 or 443.

this is all the years in kernel under netlink, counters for IPs, directions, ports, even applications which picked up the packets.

and of course on kernel level you can directly create rules, redirections, triggers etc. all the work done for 15 years.

you could have saved so much hours of work. and again, it's on kernel level, no latency, no applications, no services needed.

almost forgot, dynamic priority control of those packets and priority handling of course included ...

how could be otherwise all the routers, switches, VoIP systems could work without it. and it's all based on linux.

7 ago I was build Alcatel VoIP system for mid-size company of 250people. LINUX. all LINUX. just few lines of code licensed from CISCO.

---

(18th Jul, 2013 09:32 AM)belese Wrote:  i've read a big part of the great upstart book, but misunderstood ENV and EXPORT stanzas,
is it possible to export SCREENSAVER in a job that start on screensaver, and use $SCREENSAVER in another upstart job (that start after of course)?

env will just create VARIABLE and make it available to pre/post-start/stop runs. export, will make this VARIABLE available in all jobs triggered by starting/started/stopping/stopped upstart events of that job, including all others which will be triggered by those.

if you use env VARIABLE=1, this will happen:
- VARIABLE will be created before sh spawn, so if the sh reads (gets) the VARIABLE from other place (source of an script, standard rc scritps of env in which the sh will be spawned (user related / shell related etc)) will be overwritten.

so it's like global / extern in C (if export). if env, then its like defined inside function, or better class like in python. not seen to outsiders.

[belese]

Thanks for explanation.

netlink, and even kernel is new to me, but i've read about netlink, and my comprehension of that is

i have to open a netlink socket

and after, which way? :

send some command (which one, i read more on this)
and i receive message when connection that match on some filter is done.

or i've to polling the state of each socket?
i've try ss , and that's the info i need.

for now, i read ip paquet from a raw_socket, extract some info from header (only tcp/udp),
and send upstart event (with kind of timeout for close connection).
netlink seems better than this, but don't find easy documentation to start,
but day by day understand a bit more


for upstart export, will do some test, not sure i've understand the scope of variable.

[CurlyMo]

If you are going to use sockets, JSON objects are the bomb. Just check my wiki on how i implement it:
https://github.com/CurlyMoo/433.92-Raspberry-Pi/wiki/Code-API

Already have several C programs running and a Java GUI on top. Android App is on it's way.