+- Forum (http://forum.xbian.org)
+-- Forum: Software (/forum-6.html)
+--- Forum: Installation (/forum-16.html)
+--- Thread: Stability issues with XBian (sporadic hangs) (/thread-1296.html)
Stability issues with XBian (sporadic hangs) - Rilhas - 31st Aug, 2013 04:35 AM
Hi,
I've been using a Raspberry Pi for a long time (Model B, 256 MB). This has always worked perfectly, doing a lot of stuff (web server, subversion server, and many more, the details are shown here). It was running Raspbian, without any overclocking, nor HDMI connection.
Then I decided it was time for this Pi to work as a media server (why not??). I tried XBian 1.0 Beta 1.1, without any overclocking, and connected to the TV through HDMI. It seemed to work well, and it ran ok for the first few days. But sometimes it would hang, on a black screen, or froze with the screen it had at the time. Then it started hanging more and more often (every few hours, a day at most).
When it hangs none of the services respond. The SSH connection is immediately closed, my webserver does not accept connections, etc., but ping requests are answered. There is nothing else useful I can do to it but cycle the power.
So I started a quest to find out what might be the problem.
I swapped Pi's (for a 512 MB model) and the result was the same. I tried several diferent memory splits between CPU and GPU, but to no avail. I changed the external USB disk, but the result was the same. I changed between my two SD cards several times (formatting them anew) but the result was still the same. Changing power supplies didn't help either.
I started to suspect the problem was related to XBMC, so I turned it off and let the rest of the system and applications running. Still, it eventually hung, without any obvious relation to the moments I turned XBMC on or off. I did the same for MiniDLNA, keeping it turned off most of the time and turning it on only when needed, but it still it eventually hung without any obvious relation to turning it on or off.
Without any major leads I now suspected MiniDLNA the most, as it works terribly with the Android BubbleUPNP client and I noticed it was very very likely to get a hang while trying to browse pictures from my phone. Maybe it was something CPU-related, as a lot of "minidlna" process instances are created and the CPU gets to 100% and stays there for a long time. But it also hung while I was not using MiniDLNA, so maybe it was misbehaving even when turned off, possibly doing something I didn't know about.
Still, every test took me a lot of time, with a lot of resulting downtime, so I simply decided to go back to Raspbian. I installed everything except XBMC, and it runs perfectly, and never hangs. So I guess this (almost) rules out any hardware, USB disk, or SD card problems, and it brings me back to suspecting XBian.
To confirm the Raspbian's stability I tried viewing pictures on my phone using BubbleUPNP client streaming them from the MiniDLNA server on the Pi, and the moment I tried it the Pi became unresponsive. But it was different though, because, although it didn't seem to respond, the connections were not terminated immediately, and after waiting a while the requests got served. Running "top" in one of these instances confirmed that BubbleUPNP makes the "minidlna" processes go crazy, but the Raspbian system doesn't hang and recovers after a while.
My question is: is there any known instability in XBian that could explain these issues? It seems to me that maybe XBian is doing something (automatically) that it didn't do when I installed it for the first time (about 2 weeks ago). Maybe some automatic update is messing things up? I suspect this because it seems to be getting worse, but Raspbian working so consistently well gives me a strong conviction that the hardware is not malfunctioning nor "degrading" as time passes by. The problems are also not temperature related, all my tests were done with Pi's out of their cases and at room temperature.
Anyone would like to comment on this vague question? Has anyone experienced instability with XBian that would not show with other systems (like Raspbian)?
RE: Stability issues with XBian (sporadic hangs) - rikardo1979 - 31st Aug, 2013 04:43 AM
please can you turn on XBMC debugging an let the hang happens an get the log out after. check my signature for how to if you are not sure
-thx-
RE: Stability issues with XBian (sporadic hangs) - Rilhas - 3rd Sep, 2013 07:46 AM
(31st Aug, 2013 04:43 AM)rikardo1979 Wrote: please can you turn on XBMC debugging an let the hang happens an get the log out after. check my signature for how to if you are not sure
-thx-
Hi,
My Pi number 1 is holding stable with Raspbian, and I will have to keep it so for the next few days.
So I'm now testing exclusively with Pi number 2, SD card number 2, power supply number 2, Ethernet cable number 2, and even HDMI cable number 2! :-)
Thus far the experiment has been uneventful, Pi number 2 has been up for 3 days (unless it learned how to recover from a hangup on its own!) and this evening it has been playing music videos for 6 or 7 hours without issues (except for a stutter every now and then). This is already far better than it was before, although it has been playing media from Pi number 1 (across the network) whereas before it played from its local USB hard disk.
I will give it a few more days and, if all goes well, I will add stuff to it, to get it closer to what Pi number 1 is doing with Raspbian.
I will post meaningful results as I get them.
Thanks!
RE: Stability issues with XBian (sporadic hangs) - Rilhas - 20th Sep, 2013 05:32 AM
(3rd Sep, 2013 07:46 AM)Rilhas Wrote: I will give it a few more days and, if all goes well, I will add stuff to it, to get it closer to what Pi number 1 is doing with Raspbian.
I will post meaningful results as I get them.
Well, I've realized that I cannot do the tests I expected to do. The instability is just to unpredictable, and it affects the rest of the applications at the most inopportune times, so I keep on rolling back and rolling forward which I can't do any longer.
Suspecting the "GPU part" of XBian and XBMC (since it is very different from the non-visual use I have always made of Raspbian) I developed my own "media center" which uses omxplayer to play videos, and it hasn't caused me any problems in the 8 or 10 days it has been running. So I guess this rules out any GPU problems or conflicts (although XBMC could be doing something with the GPU that omxplayer doesn't, of course).
I'm now almost sure the problem has something to do with using the USB a lot, possibly even related to the combination and concurrency of the FTDI driver I use for the domotics application, the USB hard disk, the GSM modem, and possibly also LAN (since internally it uses USB as well), even though it doesn't cause any problems in Raspbian. However, I have no way of debugging that nor even reproducing that (I only have one domotics system).
Since I got a working solution I've given up on debugging the instabilities that originally led me to open this thread, especially since it seems they are very rare and that I may be the only person experiencing them.
Thus I'm ready to call the causes idiopathic and move on.
Thanks for your help!
RE: Stability issues with XBian (sporadic hangs) - mk01 - 21st Sep, 2013 09:19 PM
when the full setup was running, what memory utilization could be at the peak, including swaps ?
RE: Stability issues with XBian (sporadic hangs) - mk01 - 21st Sep, 2013 10:33 PM
and can you please check on the raspbian
Code:
dmesg | grep -i splitRE: Stability issues with XBian (sporadic hangs) - Rilhas - 22nd Sep, 2013 08:15 AM
(21st Sep, 2013 09:19 PM)mk01 Wrote: when the full setup was running, what memory utilization could be at the peak, including swaps ?
Hi,
The Pi I have always before used was a model B with 256MB of memory, so I'm sure "my stuff" runs on 256MB. When I tried XBian I tried it first on the 512MB model B, although it didn't seem to make any difference which Pi I used.
Anyway, over time I've always checked memory usage, but it is very hard to get exact numbers, as the amount of memory fluctuates due to variations in the web server, the SVN server, and others, and I don't know which of these ever combined to create an abnormal peak. What I've always used as an indicator that RAM is never exhausted is the fact that the 100MB swap never gets used.
Anyway, I just checked these numbers:
Terminal
pi@Rilhas-SRV ~ $ free -h
total used free shared buffers cached
Mem: 374M 363M 11M 0B 106M 189M
-/+ buffers/cache: 67M 306M
Swap: 99M 0B 99M
total used free shared buffers cached
Mem: 374M 363M 11M 0B 106M 189M
-/+ buffers/cache: 67M 306M
Swap: 99M 0B 99M
This is a typical process list when everything is running (but NFMC is not playing anything):
Terminal
pi@Rilhas-SRV ~ $ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2144 728 ? Ss 00:15 0:05 init [2]
root 2 0.0 0.0 0 0 ? S 00:15 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 00:15 0:15 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 00:15 0:00 [kworker/0:0H]
root 6 0.0 0.0 0 0 ? S 00:15 0:09 [kworker/u:0]
root 7 0.0 0.0 0 0 ? S< 00:15 0:00 [kworker/u:0H]
root 8 0.0 0.0 0 0 ? S< 00:15 0:00 [khelper]
root 9 0.0 0.0 0 0 ? S 00:15 0:00 [kdevtmpfs]
root 10 0.0 0.0 0 0 ? S< 00:15 0:00 [netns]
root 12 0.0 0.0 0 0 ? S 00:15 0:00 [bdi-default]
root 13 0.0 0.0 0 0 ? S< 00:15 0:00 [kblockd]
root 14 0.0 0.0 0 0 ? S 00:15 0:00 [khubd]
root 15 0.0 0.0 0 0 ? S< 00:15 0:00 [rpciod]
root 16 0.0 0.0 0 0 ? S 00:15 0:00 [khungtaskd]
root 17 0.0 0.0 0 0 ? S 00:15 0:01 [kswapd0]
root 18 0.0 0.0 0 0 ? S 00:15 0:00 [fsnotify_mark]
root 19 0.0 0.0 0 0 ? S< 00:15 0:00 [nfsiod]
root 20 0.0 0.0 0 0 ? S< 00:15 0:00 [crypto]
root 27 0.0 0.0 0 0 ? S< 00:15 0:00 [kthrotld]
root 28 0.0 0.0 0 0 ? S< 00:15 0:00 [VCHIQ-0]
root 29 0.0 0.0 0 0 ? S< 00:15 0:00 [VCHIQr-0]
root 30 0.0 0.0 0 0 ? S< 00:15 0:00 [VCHIQs-0]
root 31 0.0 0.0 0 0 ? S< 00:15 0:00 [iscsi_eh]
root 32 0.0 0.0 0 0 ? S< 00:15 0:00 [dwc_otg]
root 33 0.0 0.0 0 0 ? S< 00:15 0:00 [DWC Notificatio]
root 35 0.0 0.0 0 0 ? S 00:15 0:14 [mmcqd/0]
root 36 0.0 0.0 0 0 ? S< 00:15 0:00 [deferwq]
root 37 0.0 0.0 0 0 ? S 00:15 0:02 [kworker/u:2]
root 38 0.0 0.0 0 0 ? S 00:15 0:00 [scsi_eh_0]
root 39 0.0 0.0 0 0 ? S 00:15 0:23 [usb-storage]
root 40 0.0 0.0 0 0 ? S 00:15 0:00 [jbd2/mmcblk0p6-]
root 41 0.0 0.0 0 0 ? S< 00:15 0:00 [ext4-dio-unwrit]
root 157 0.0 0.3 2880 1272 ? Ss 00:15 0:00 udevd --daemon
root 288 0.0 0.2 2876 1004 ? S 00:15 0:00 udevd --daemon
root 301 0.0 0.2 2876 996 ? S 00:15 0:00 udevd --daemon
root 335 0.0 0.0 0 0 ? S 00:15 0:00 [scsi_eh_3]
root 336 0.0 0.0 0 0 ? S 00:15 0:00 [usb-storage]
root 1294 0.0 0.0 0 0 ? S 00:15 0:17 [jbd2/sda1-8]
root 1298 0.0 0.0 0 0 ? S< 00:15 0:00 [ext4-dio-unwrit]
root 1616 0.0 0.1 1744 504 ? S 00:15 0:15 /usr/sbin/ifplugd -i lo -q -f -u0 -d10 -w -I
root 1639 0.0 0.1 1744 512 ? S 00:15 0:48 /usr/sbin/ifplugd -i eth0 -q -f -u0 -d10 -w -I
root 1775 0.0 0.0 0 0 ? S 00:15 0:03 [flush-8:0]
root 1877 0.0 0.4 27968 1632 ? Sl 00:15 0:11 /usr/sbin/rsyslogd -c5
nobody 1931 0.0 0.1 2012 636 ? Ss 00:15 0:01 /usr/sbin/thd --daemon --triggers /etc/triggerhappy/triggers.d/ --socket /var/run/thd.socket --pidfile /var/run/thd.pid --user nobody /dev/input/event*
root 1967 0.0 0.2 3792 792 ? Ss 00:15 0:00 /usr/sbin/cron
104 1984 0.0 0.2 3176 1052 ? Ss 00:15 0:00 /usr/bin/dbus-daemon --system
root 2005 0.0 0.3 4256 1232 ? S 00:15 0:00 /bin/bash ./HTTP_Relay.sh
root 2025 0.2 0.7 137720 2892 ? Sl 00:15 2:45 ./HTTP_Relay_RPI_CON.exe
ntp 2058 0.0 0.4 5508 1716 ? Ss 00:15 0:16 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
root 2082 0.0 0.3 4264 1248 ? S 00:15 0:00 /bin/bash ./HTML_Server.sh
root 2093 1.6 3.7 118060 14456 ? Sl 00:15 21:44 ./HTML_Server_RPI_CON.exe
root 2094 0.0 0.2 6208 1072 ? Ss 00:15 0:03 /usr/sbin/sshd
root 2136 0.0 0.3 4264 1248 ? S 00:15 0:00 /bin/bash ./RDOMUS.sh
root 2145 0.0 0.4 5116 1596 ? S 00:15 0:00 sudo nice -n 10 ./RDOMUS_RemoteControl_Server_CON_RPI.exe
root 2146 43.8 1.2 149724 4672 ? SNl 00:15 593:52 ./RDOMUS_RemoteControl_Server_CON_RPI.exe
root 2160 0.0 0.3 4264 1248 ? S 00:15 0:00 /bin/bash ./NFMC.sh
root 2187 0.2 0.6 30404 2404 ? Sl 00:15 4:01 ./NFMC_RPI_CON.exe
root 2204 0.0 0.4 1732 1672 ? SLs 00:15 0:42 /usr/sbin/watchdog
root 2222 0.0 0.2 11620 1040 ? Ss 00:15 0:00 svnserve -d -r /RILHAS-SRV/svnrepo
root 2226 0.0 0.2 3740 804 tty1 Ss+ 00:15 0:00 /sbin/getty --noclear 38400 tty1
root 2227 0.0 0.2 3740 804 tty2 Ss+ 00:15 0:00 /sbin/getty 38400 tty2
root 2228 0.0 0.2 3740 804 tty3 Ss+ 00:15 0:00 /sbin/getty 38400 tty3
root 2229 0.0 0.2 3740 804 tty4 Ss+ 00:15 0:00 /sbin/getty 38400 tty4
root 2230 0.0 0.2 3740 804 tty5 Ss+ 00:15 0:00 /sbin/getty 38400 tty5
root 2231 0.0 0.2 3740 804 tty6 Ss+ 00:15 0:00 /sbin/getty 38400 tty6
root 2232 0.0 0.1 2060 732 ? Ss+ 00:15 0:00 /sbin/getty -L ttyAMA0 115200 vt100
root 2317 0.0 0.9 27544 3728 ? Sl 01:09 0:00 /usr/sbin/console-kit-daemon --no-daemon
root 2384 0.0 0.7 22284 2920 ? Sl 01:09 0:00 /usr/lib/policykit-1/polkitd --no-debug
root 3567 0.0 0.0 0 0 ? S< 07:01 0:00 [kworker/0:1H]
root 4314 0.0 0.8 9804 3164 ? Ss 16:16 0:01 sshd: pi [priv]
pi 4321 0.0 0.4 9804 1744 ? S 16:16 0:01 sshd: pi@notty
pi 4322 0.0 0.2 2196 800 ? Ss 16:16 0:00 /usr/lib/openssh/sftp-server
root 8039 0.0 0.8 9804 3164 ? Ss 18:38 0:01 sshd: pi [priv]
pi 8050 0.0 0.4 10008 1884 ? S 18:39 0:14 sshd: pi@notty
pi 8051 0.0 0.2 2380 1024 ? Ss 18:39 0:01 /usr/lib/openssh/sftp-server
root 9088 0.0 0.0 0 0 ? S 21:27 0:00 [kworker/0:1]
root 9170 0.1 0.8 9804 3188 ? Ss 22:38 0:00 sshd: pi [priv]
pi 9177 0.0 0.4 9804 1628 ? S 22:38 0:00 sshd: pi@pts/0
pi 9178 0.3 0.9 6720 3792 pts/0 Ss 22:38 0:02 -bash
root 9188 0.0 0.0 0 0 ? S 22:38 0:00 [kworker/0:2]
root 9197 0.0 0.0 0 0 ? S 22:44 0:00 [kworker/0:0]
pi 9207 0.0 0.2 4452 1124 pts/0 R+ 22:48 0:00 ps aux
pi@Rilhas-SRV ~ $
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2144 728 ? Ss 00:15 0:05 init [2]
root 2 0.0 0.0 0 0 ? S 00:15 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S 00:15 0:15 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< 00:15 0:00 [kworker/0:0H]
root 6 0.0 0.0 0 0 ? S 00:15 0:09 [kworker/u:0]
root 7 0.0 0.0 0 0 ? S< 00:15 0:00 [kworker/u:0H]
root 8 0.0 0.0 0 0 ? S< 00:15 0:00 [khelper]
root 9 0.0 0.0 0 0 ? S 00:15 0:00 [kdevtmpfs]
root 10 0.0 0.0 0 0 ? S< 00:15 0:00 [netns]
root 12 0.0 0.0 0 0 ? S 00:15 0:00 [bdi-default]
root 13 0.0 0.0 0 0 ? S< 00:15 0:00 [kblockd]
root 14 0.0 0.0 0 0 ? S 00:15 0:00 [khubd]
root 15 0.0 0.0 0 0 ? S< 00:15 0:00 [rpciod]
root 16 0.0 0.0 0 0 ? S 00:15 0:00 [khungtaskd]
root 17 0.0 0.0 0 0 ? S 00:15 0:01 [kswapd0]
root 18 0.0 0.0 0 0 ? S 00:15 0:00 [fsnotify_mark]
root 19 0.0 0.0 0 0 ? S< 00:15 0:00 [nfsiod]
root 20 0.0 0.0 0 0 ? S< 00:15 0:00 [crypto]
root 27 0.0 0.0 0 0 ? S< 00:15 0:00 [kthrotld]
root 28 0.0 0.0 0 0 ? S< 00:15 0:00 [VCHIQ-0]
root 29 0.0 0.0 0 0 ? S< 00:15 0:00 [VCHIQr-0]
root 30 0.0 0.0 0 0 ? S< 00:15 0:00 [VCHIQs-0]
root 31 0.0 0.0 0 0 ? S< 00:15 0:00 [iscsi_eh]
root 32 0.0 0.0 0 0 ? S< 00:15 0:00 [dwc_otg]
root 33 0.0 0.0 0 0 ? S< 00:15 0:00 [DWC Notificatio]
root 35 0.0 0.0 0 0 ? S 00:15 0:14 [mmcqd/0]
root 36 0.0 0.0 0 0 ? S< 00:15 0:00 [deferwq]
root 37 0.0 0.0 0 0 ? S 00:15 0:02 [kworker/u:2]
root 38 0.0 0.0 0 0 ? S 00:15 0:00 [scsi_eh_0]
root 39 0.0 0.0 0 0 ? S 00:15 0:23 [usb-storage]
root 40 0.0 0.0 0 0 ? S 00:15 0:00 [jbd2/mmcblk0p6-]
root 41 0.0 0.0 0 0 ? S< 00:15 0:00 [ext4-dio-unwrit]
root 157 0.0 0.3 2880 1272 ? Ss 00:15 0:00 udevd --daemon
root 288 0.0 0.2 2876 1004 ? S 00:15 0:00 udevd --daemon
root 301 0.0 0.2 2876 996 ? S 00:15 0:00 udevd --daemon
root 335 0.0 0.0 0 0 ? S 00:15 0:00 [scsi_eh_3]
root 336 0.0 0.0 0 0 ? S 00:15 0:00 [usb-storage]
root 1294 0.0 0.0 0 0 ? S 00:15 0:17 [jbd2/sda1-8]
root 1298 0.0 0.0 0 0 ? S< 00:15 0:00 [ext4-dio-unwrit]
root 1616 0.0 0.1 1744 504 ? S 00:15 0:15 /usr/sbin/ifplugd -i lo -q -f -u0 -d10 -w -I
root 1639 0.0 0.1 1744 512 ? S 00:15 0:48 /usr/sbin/ifplugd -i eth0 -q -f -u0 -d10 -w -I
root 1775 0.0 0.0 0 0 ? S 00:15 0:03 [flush-8:0]
root 1877 0.0 0.4 27968 1632 ? Sl 00:15 0:11 /usr/sbin/rsyslogd -c5
nobody 1931 0.0 0.1 2012 636 ? Ss 00:15 0:01 /usr/sbin/thd --daemon --triggers /etc/triggerhappy/triggers.d/ --socket /var/run/thd.socket --pidfile /var/run/thd.pid --user nobody /dev/input/event*
root 1967 0.0 0.2 3792 792 ? Ss 00:15 0:00 /usr/sbin/cron
104 1984 0.0 0.2 3176 1052 ? Ss 00:15 0:00 /usr/bin/dbus-daemon --system
root 2005 0.0 0.3 4256 1232 ? S 00:15 0:00 /bin/bash ./HTTP_Relay.sh
root 2025 0.2 0.7 137720 2892 ? Sl 00:15 2:45 ./HTTP_Relay_RPI_CON.exe
ntp 2058 0.0 0.4 5508 1716 ? Ss 00:15 0:16 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 102:104
root 2082 0.0 0.3 4264 1248 ? S 00:15 0:00 /bin/bash ./HTML_Server.sh
root 2093 1.6 3.7 118060 14456 ? Sl 00:15 21:44 ./HTML_Server_RPI_CON.exe
root 2094 0.0 0.2 6208 1072 ? Ss 00:15 0:03 /usr/sbin/sshd
root 2136 0.0 0.3 4264 1248 ? S 00:15 0:00 /bin/bash ./RDOMUS.sh
root 2145 0.0 0.4 5116 1596 ? S 00:15 0:00 sudo nice -n 10 ./RDOMUS_RemoteControl_Server_CON_RPI.exe
root 2146 43.8 1.2 149724 4672 ? SNl 00:15 593:52 ./RDOMUS_RemoteControl_Server_CON_RPI.exe
root 2160 0.0 0.3 4264 1248 ? S 00:15 0:00 /bin/bash ./NFMC.sh
root 2187 0.2 0.6 30404 2404 ? Sl 00:15 4:01 ./NFMC_RPI_CON.exe
root 2204 0.0 0.4 1732 1672 ? SLs 00:15 0:42 /usr/sbin/watchdog
root 2222 0.0 0.2 11620 1040 ? Ss 00:15 0:00 svnserve -d -r /RILHAS-SRV/svnrepo
root 2226 0.0 0.2 3740 804 tty1 Ss+ 00:15 0:00 /sbin/getty --noclear 38400 tty1
root 2227 0.0 0.2 3740 804 tty2 Ss+ 00:15 0:00 /sbin/getty 38400 tty2
root 2228 0.0 0.2 3740 804 tty3 Ss+ 00:15 0:00 /sbin/getty 38400 tty3
root 2229 0.0 0.2 3740 804 tty4 Ss+ 00:15 0:00 /sbin/getty 38400 tty4
root 2230 0.0 0.2 3740 804 tty5 Ss+ 00:15 0:00 /sbin/getty 38400 tty5
root 2231 0.0 0.2 3740 804 tty6 Ss+ 00:15 0:00 /sbin/getty 38400 tty6
root 2232 0.0 0.1 2060 732 ? Ss+ 00:15 0:00 /sbin/getty -L ttyAMA0 115200 vt100
root 2317 0.0 0.9 27544 3728 ? Sl 01:09 0:00 /usr/sbin/console-kit-daemon --no-daemon
root 2384 0.0 0.7 22284 2920 ? Sl 01:09 0:00 /usr/lib/policykit-1/polkitd --no-debug
root 3567 0.0 0.0 0 0 ? S< 07:01 0:00 [kworker/0:1H]
root 4314 0.0 0.8 9804 3164 ? Ss 16:16 0:01 sshd: pi [priv]
pi 4321 0.0 0.4 9804 1744 ? S 16:16 0:01 sshd: pi@notty
pi 4322 0.0 0.2 2196 800 ? Ss 16:16 0:00 /usr/lib/openssh/sftp-server
root 8039 0.0 0.8 9804 3164 ? Ss 18:38 0:01 sshd: pi [priv]
pi 8050 0.0 0.4 10008 1884 ? S 18:39 0:14 sshd: pi@notty
pi 8051 0.0 0.2 2380 1024 ? Ss 18:39 0:01 /usr/lib/openssh/sftp-server
root 9088 0.0 0.0 0 0 ? S 21:27 0:00 [kworker/0:1]
root 9170 0.1 0.8 9804 3188 ? Ss 22:38 0:00 sshd: pi [priv]
pi 9177 0.0 0.4 9804 1628 ? S 22:38 0:00 sshd: pi@pts/0
pi 9178 0.3 0.9 6720 3792 pts/0 Ss 22:38 0:02 -bash
root 9188 0.0 0.0 0 0 ? S 22:38 0:00 [kworker/0:2]
root 9197 0.0 0.0 0 0 ? S 22:44 0:00 [kworker/0:0]
pi 9207 0.0 0.2 4452 1124 pts/0 R+ 22:48 0:00 ps aux
pi@Rilhas-SRV ~ $
Next is a typical "top" result, with the domotics server always using about 40% of the CPU (the culprit is the FTDI driver for the USB chip), hence a niceness of 10 for the process:
Terminal
pi@Rilhas-SRV ~ $ top
top - 22:50:38 up 22:35, 1 user, load average: 0.76, 0.83, 0.77
Tasks: 82 total, 1 running, 81 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.1 us, 30.0 sy, 16.6 ni, 48.8 id, 0.1 wa, 0.0 hi, 2.4 si, 0.0 st
KiB Mem: 383712 total, 370964 used, 12748 free, 109336 buffers
KiB Swap: 102396 total, 0 used, 102396 free, 191908 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2146 root 30 10 146m 4672 2532 S 34.3 1.2 594:46.48 RDOMUS_RemoteCo
9209 pi 20 0 4672 1384 956 R 19.0 0.4 0:00.13 top
1 root 20 0 2144 728 620 S 0.0 0.2 0:05.51 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.05 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:15.02 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:09.66 kworker/u:0
7 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/u:0H
8 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
12 root 20 0 0 0 0 S 0.0 0.0 0:00.02 bdi-default
13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
14 root 20 0 0 0 0 S 0.0 0.0 0:00.60 khubd
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 rpciod
16 root 20 0 0 0 0 S 0.0 0.0 0:00.09 khungtaskd
17 root 20 0 0 0 0 S 0.0 0.0 0:01.24 kswapd0
18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark
19 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 nfsiod
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
27 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld
28 root 1 -19 0 0 0 S 0.0 0.0 0:00.00 VCHIQ-0
29 root 1 -19 0 0 0 S 0.0 0.0 0:00.00 VCHIQr-0
30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 VCHIQs-0
31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 iscsi_eh
32 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 dwc_otg
33 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 DWC Notificatio
35 root 20 0 0 0 0 S 0.0 0.0 0:14.47 mmcqd/0
36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq
37 root 20 0 0 0 0 S 0.0 0.0 0:02.69 kworker/u:2
38 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
39 root 20 0 0 0 0 S 0.0 0.0 0:23.97 usb-storage
40 root 20 0 0 0 0 S 0.0 0.0 0:00.91 jbd2/mmcblk0p6-
41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
157 root 20 0 2880 1272 740 S 0.0 0.3 0:00.51 udevd
288 root 20 0 2876 1004 464 S 0.0 0.3 0:00.01 udevd
301 root 20 0 2876 996 456 S 0.0 0.3 0:00.00 udevd
335 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_3
336 root 20 0 0 0 0 S 0.0 0.0 0:00.00 usb-storage
1294 root 20 0 0 0 0 S 0.0 0.0 0:17.83 jbd2/sda1-8
1298 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-dio-unwrit
1616 root 20 0 1744 504 420 S 0.0 0.1 0:15.34 ifplugd
1639 root 20 0 1744 512 420 S 0.0 0.1 0:48.10 ifplugd
1775 root 20 0 0 0 0 S 0.0 0.0 0:03.27 flush-8:0
1877 root 20 0 27968 1632 1156 S 0.0 0.4 0:11.96 rsyslogd
1931 nobody 20 0 2012 636 516 S 0.0 0.2 0:01.61 thd
1967 root 20 0 3792 792 620 S 0.0 0.2 0:00.40 cron
1984 messageb 20 0 3176 1052 764 S 0.0 0.3 0:00.13 dbus-daemon
2005 root 20 0 4256 1232 1080 S 0.0 0.3 0:00.02 HTTP_Relay.sh
2025 root 20 0 134m 2892 1264 S 0.0 0.8 2:45.30 HTTP_Relay_RPI_
2058 ntp 20 0 5508 1716 1336 S 0.0 0.4 0:16.80 ntpd
2082 root 20 0 4264 1248 1088 S 0.0 0.3 0:00.02 HTML_Server.sh
2093 root 20 0 115m 14m 3792 S 0.0 3.8 21:45.72 HTML_Server_RPI
2094 root 20 0 6208 1072 648 S 0.0 0.3 0:03.42 sshd
2136 root 20 0 4264 1248 1088 S 0.0 0.3 0:00.03 RDOMUS.sh
2145 root 20 0 5116 1596 1284 S 0.0 0.4 0:00.05 sudo
2160 root 20 0 4264 1248 1088 S 0.0 0.3 0:00.02 NFMC.sh
2187 root 20 0 30404 2404 1368 S 0.0 0.6 4:01.97 NFMC_RPI_CON.ex
2204 root -2 0 1732 1672 1352 S 0.0 0.4 0:42.48 watchdog
2222 root 20 0 11620 1040 496 S 0.0 0.3 0:00.01 svnserve
2226 root 20 0 3740 804 676 S 0.0 0.2 0:00.03 getty
2227 root 20 0 3740 804 676 S 0.0 0.2 0:00.01 getty
2228 root 20 0 3740 804 676 S 0.0 0.2 0:00.01 getty
Here is the "split" result:
Terminal
Does this give you any idea of what the problem might be?
RE: Stability issues with XBian (sporadic hangs) - mk01 - 22nd Sep, 2013 09:12 AM
nothing like to be 100% sure, but you got my attention with the USB traffic / utilization.
there seems to be few issues with the RPIs usb design causing problems starting from usb devices not found to problems with IRQ/timings ending with kernel panics - related to some usage pattern / usb devices combinations.
FIQ split should be a help there, now by default turned on with 3.10.y kernels. this wasn't the case with older branches (or months ago). XBian B1 / 1.1 wasn't using those patches - but Raspbian does use them / has patched kernels.
nevertheless it is my assumption, not proof of something, but generally speaking there are not much other relevant differences, if any.
for example there was an issue opened at github recently https://github.com/xbianonpi/xbian/issues/457 which would make it (together with you and one other) three 'reported' cases with similar symptoms I know of.
I will just tell you the same as in the comment to the issue - Beta1 branch will retire very soon. If you have reasons to go for XBMC bundled distro like XBian or RaspBMC (I have the feeling it won't be OpenELEC), give it a week or two until Beta2 is released and give it a try then.
RE: Stability issues with XBian (sporadic hangs) - Rilhas - 22nd Sep, 2013 12:25 PM
(22nd Sep, 2013 09:12 AM)mk01 Wrote: nothing like to be 100% sure, but you got my attention with the USB traffic / utilization.
there seems to be few issues with the RPIs usb design causing problems starting from usb devices not found to problems with IRQ/timings ending with kernel panics - related to some usage pattern / usb devices combinations.
FIQ split should be a help there, now by default turned on with 3.10.y kernels. this wasn't the case with older branches (or months ago). XBian B1 / 1.1 wasn't using those patches - but Raspbian does use them / has patched kernels.
nevertheless it is my assumption, not proof of something, but generally speaking there are not much other relevant differences, if any.
for example there was an issue opened at github recently https://github.com/xbianonpi/xbian/issues/457 which would make it (together with you and one other) three 'reported' cases with similar symptoms I know of.
I will just tell you the same as in the comment to the issue - Beta1 branch will retire very soon. If you have reasons to go for XBMC bundled distro like XBian or RaspBMC (I have the feeling it won't be OpenELEC), give it a week or two until Beta2 is released and give it a try then.
Your post makes sense to me on so many levels, while at the same time gets me confused about a few situations I encountered.
I started suspecting USB problems when I tried to get a composite video capture card to work with Raspbian. The capture device was not recognized in Raspbian, and various posts in various forums commented on the USB issues with the Pi and also of support being better in 3.7+ kernels.
My Raspbian is 3.6, and because XBian was 3.9 I expected it to be better at the USB issues, but the capture cards still didn't work. In fact (and that is where the first contradiction showed up), it seemed a lot worse with the capture cards I tried (detection problems, hanging, intermittent registrations and deregistrations of /dev/video0). Still, I tested about 4 capture cards on 2 different Pi's with both Raspbian and XBian, plus one of the power supplies I used turned out to be faulty, and sometimes with a USB hub and sometimes without. So I gave up trying to make sense of it all, and gave up capture cards for a while.
Then, yesterday, after the 8 or 10 days of running Raspbian and my "media center" without problems (as I mentioned in a previous post) I decided to try the capture cards in Raspbian again. I apt-updated and then apt-upgraded, and it all seemed somewhat better with the cards, but they still didn't work.
I gave up the video capture again, disconnected all the capture cards, rebooted, but - and this is where I was blown away - after a few hours the Pi hung. I turned on the TV looking for panics, but when my Pi is rebooted with HDMI matrix using another HDMI source the result is that the Pi's console is all black and no text is displayed. Since I had rebooted the Pi in these conditions the TV showed nothing, so there might be a panic that I was not able to see (only later did I think that I could have tried the composite output before restarting the Pi, but by then it was too late, I'll remember it next time).
What came to my mind right away was that the unupdated/unupgraded Raspbian (originally from NOOBS v1.2.1, I think) might have been more tolerant to the Pi's USB instabilities than both the updated/upgraded Raspbian and XBian (which is a few versions ahead of Raspbian). Maybe some change in versions after that "original" Raspbian introduced something that is affected by my setup, while previously it wasn't.
Corroborating this, after the Raspbian update/upgrade I immediately noticed something was worse, because the lights in the hallway took longer to turn on than they did before (these specific lights exchange a lot of messages with the server, so they are a noticeable and clear indicator that something changed). The FTDI USB driver in the Pi has always been the domotics bottleneck, and has always governed the reaction time of sensors and actuators, so the fact that it is now slower (2 seconds to turn on instead of the 0.2 to 0.3 seconds it took before) makes me suspect of a degradation in USB performance in the updated/upgraded version of Raspbian.
... coincidentally, turning on the lights in XBian was also slow (maybe the same 2 seconds), but I have always attributed that to the high CPU consumption of XBMC, even while sitting idle, coupled with is niceness of -3. Now I'm beginning to think that the delay in XBian was not CPU related, but USB related instead, and the theory that the new kernels are the ones causing me problems makes even more sense.
Although this theory made complete sense to me yesterday, it doesn't hold when confronted with your explanation that Raspbian had used patches and that XBian hadn't, because when updating/upgrading Raspbian the behaviour becomes closer to that of XBian (unless in case of some weird possibilities, like the updated/upgraded Raspbian not using the patches anymore).
This is all very recent, so I will still need the Pi to hang a few more times to draw any conclusions
As for the next XBian release - I'll be looking forward to it and will surely give it a try when it comes out.
Re: Stability issues with XBian (sporadic hangs) - f1vefour - 29th Sep, 2013 01:19 AM
I can tell you newer kernel revisions have regression issues occasionally, one I know of is lirc_rpi but I'm sure there are others.
RE: Stability issues with XBian (sporadic hangs) - mk01 - 29th Sep, 2013 01:33 PM
(22nd Sep, 2013 12:25 PM)Rilhas Wrote: Although this theory made complete sense to me yesterday, it doesn't hold when confronted with your explanation that Raspbian had used patches and that XBian hadn't, because when updating/upgrading Raspbian the behaviour becomes closer to that of XBian (unless in case of some weird possibilities, like the updated/upgraded Raspbian not using the patches anymore).
Rilhas,
Just to put this right - please allow some minor deviations from reality and what I said - I never really followed 3.6.y development in the core code - so finally what for me (watching forums) looks like a patch / custom hacks - can for months be in the master branch of RPI kernel tree - and later even moved to 3.8, 3.9 … And 3.10.y branch I noticed very massive (and released often) patching in general around dwc_otg and usb. Also, some parameters are now by default, what wasn't the case with kernels I remember from past.
We moved to more recent versions because of btrfs not being back ported and before 3.8.y there has been serious issues in it. On the other hand Raspbian firmware / kernel (together with RaspBMC) are closely following the git's RPI kernel/firmware repo - development and default config options as well. RaspBMC is now going to 3.10.y, Raspbian I don't know.
@f1vefour 3.9.y or 3.10.y as well? I know 3.9.y had problem with lirc_* if compiled with a different allocation model than SLAB. even CurlyMo posted the panics somewhere on the forum.
does fiq_split_enable=0 make any difference to you ?
Re: Stability issues with XBian (sporadic hangs) - f1vefour - 1st Oct, 2013 02:10 PM
In 3.9.x and 3.10.x there are issues with lirc_rpi still. This is from dmesg, when I recieve this the remote stops functioning for a random amount of time. I haven't tried the fiq kernel flag.
Code:
root@raspberry:~# uname -a
Linux raspberry 3.10.12+ #61 Fri Sep 27 19:40:13 CEST 2013 armv6l GNU/Linux
dmesg:
lirc_rpi: AIEEEE: 0 0 5249e4ac 5249e470 6e06 75004
[24020.897078] lirc_rpi: AIEEEE: 1 1 524a2d32 524a2ced 1e5d4 178e9
[24093.110976] lirc_rpi: AIEEEE: 0 0 524a2d7a 524a2d32 5200d 1e5d4
[24416.735215] lirc_rpi: AIEEEE: 1 1 524a2ebd 524a2e69 e7c07 ca111
[24459.429846] lirc_rpi: AIEEEE: 0 0 524a2ee8 524a2ebd 9cdb2 e7c07
[24920.189958] input: lircd as /devices/virtual/input/input1
[31068.958835] lirc_rpi: AIEEEE: 1 1 524a48b9 524a488b e89fc 3fda5
[31129.833571] lirc_rpi: AIEEEE: 0 0 524a48f6 524a48b9 c98dd e89fc
[31279.370015] lirc_rpi: AIEEEE: 1 1 524a498c 524a48f6 572b7 c9949
[31299.686509] lirc_rpi: AIEEEE: 0 0 524a49a0 524a498c a4479 572b7Why would SLUB/SLAB allocation matter?
RE: Stability issues with XBian (sporadic hangs) - mk01 - 1st Oct, 2013 08:48 PM
(1st Oct, 2013 02:10 PM)f1vefour Wrote: Why would SLUB/SLAB allocation matter?
exactly. should not.
but the paths of kernel hackers are hidden to me
but this one specially was ending with OOOP.