Forum

Hi,

I've been using a Raspberry Pi for a long time (Model B, 256 MB). This has always worked perfectly, doing a lot of stuff (web server, subversion server, and many more, the details are shown here). It was running Raspbian, without any overclocking, nor HDMI connection.

Then I decided it was time for this Pi to work as a media server (why not??). I tried XBian 1.0 Beta 1.1, without any overclocking, and connected to the TV through HDMI. It seemed to work well, and it ran ok for the first few days. But sometimes it would hang, on a black screen, or froze with the screen it had at the time. Then it started hanging more and more often (every few hours, a day at most).

When it hangs none of the services respond. The SSH connection is immediately closed, my webserver does not accept connections, etc., but ping requests are answered. There is nothing else useful I can do to it but cycle the power.

So I started a quest to find out what might be the problem.

I swapped Pi's (for a 512 MB model) and the result was the same. I tried several diferent memory splits between CPU and GPU, but to no avail. I changed the external USB disk, but the result was the same. I changed between my two SD cards several times (formatting them anew) but the result was still the same. Changing power supplies didn't help either.

I started to suspect the problem was related to XBMC, so I turned it off and let the rest of the system and applications running. Still, it eventually hung, without any obvious relation to the moments I turned XBMC on or off. I did the same for MiniDLNA, keeping it turned off most of the time and turning it on only when needed, but it still it eventually hung without any obvious relation to turning it on or off.

Without any major leads I now suspected MiniDLNA the most, as it works terribly with the Android BubbleUPNP client and I noticed it was very very likely to get a hang while trying to browse pictures from my phone. Maybe it was something CPU-related, as a lot of "minidlna" process instances are created and the CPU gets to 100% and stays there for a long time. But it also hung while I was not using MiniDLNA, so maybe it was misbehaving even when turned off, possibly doing something I didn't know about.

Still, every test took me a lot of time, with a lot of resulting downtime, so I simply decided to go back to Raspbian. I installed everything except XBMC, and it runs perfectly, and never hangs. So I guess this (almost) rules out any hardware, USB disk, or SD card problems, and it brings me back to suspecting XBian.

To confirm the Raspbian's stability I tried viewing pictures on my phone using BubbleUPNP client streaming them from the MiniDLNA server on the Pi, and the moment I tried it the Pi became unresponsive. But it was different though, because, although it didn't seem to respond, the connections were not terminated immediately, and after waiting a while the requests got served. Running "top" in one of these instances confirmed that BubbleUPNP makes the "minidlna" processes go crazy, but the Raspbian system doesn't hang and recovers after a while.

My question is: is there any known instability in XBian that could explain these issues? It seems to me that maybe XBian is doing something (automatically) that it didn't do when I installed it for the first time (about 2 weeks ago). Maybe some automatic update is messing things up? I suspect this because it seems to be getting worse, but Raspbian working so consistently well gives me a strong conviction that the hardware is not malfunctioning nor "degrading" as time passes by. The problems are also not temperature related, all my tests were done with Pi's out of their cases and at room temperature.

Anyone would like to comment on this vague question? Has anyone experienced instability with XBian that would not show with other systems (like Raspbian)?

please can you turn on XBMC debugging an let the hang happens an get the log out after. check my signature for how to if you are not sure
-thx-

(31st Aug, 2013 04:43 AM)rikardo1979 Wrote: [ -> ]please can you turn on XBMC debugging an let the hang happens an get the log out after. check my signature for how to if you are not sure
-thx-

Hi,

My Pi number 1 is holding stable with Raspbian, and I will have to keep it so for the next few days.

So I'm now testing exclusively with Pi number 2, SD card number 2, power supply number 2, Ethernet cable number 2, and even HDMI cable number 2! :-)

Thus far the experiment has been uneventful, Pi number 2 has been up for 3 days (unless it learned how to recover from a hangup on its own!) and this evening it has been playing music videos for 6 or 7 hours without issues (except for a stutter every now and then). This is already far better than it was before, although it has been playing media from Pi number 1 (across the network) whereas before it played from its local USB hard disk.

I will give it a few more days and, if all goes well, I will add stuff to it, to get it closer to what Pi number 1 is doing with Raspbian.

I will post meaningful results as I get them.

Thanks!

(3rd Sep, 2013 07:46 AM)Rilhas Wrote: [ -> ]I will give it a few more days and, if all goes well, I will add stuff to it, to get it closer to what Pi number 1 is doing with Raspbian.

I will post meaningful results as I get them.

Well, I've realized that I cannot do the tests I expected to do. The instability is just to unpredictable, and it affects the rest of the applications at the most inopportune times, so I keep on rolling back and rolling forward which I can't do any longer.

Suspecting the "GPU part" of XBian and XBMC (since it is very different from the non-visual use I have always made of Raspbian) I developed my own "media center" which uses omxplayer to play videos, and it hasn't caused me any problems in the 8 or 10 days it has been running. So I guess this rules out any GPU problems or conflicts (although XBMC could be doing something with the GPU that omxplayer doesn't, of course).

I'm now almost sure the problem has something to do with using the USB a lot, possibly even related to the combination and concurrency of the FTDI driver I use for the domotics application, the USB hard disk, the GSM modem, and possibly also LAN (since internally it uses USB as well), even though it doesn't cause any problems in Raspbian. However, I have no way of debugging that nor even reproducing that (I only have one domotics system).

Since I got a working solution I've given up on debugging the instabilities that originally led me to open this thread, especially since it seems they are very rare and that I may be the only person experiencing them.

Thus I'm ready to call the causes idiopathic and move on.

Thanks for your help!

when the full setup was running, what memory utilization could be at the peak, including swaps ?

and can you please check on the raspbian

(21st Sep, 2013 09:19 PM)mk01 Wrote: [ -> ]when the full setup was running, what memory utilization could be at the peak, including swaps ?

Hi,

The Pi I have always before used was a model B with 256MB of memory, so I'm sure "my stuff" runs on 256MB. When I tried XBian I tried it first on the 512MB model B, although it didn't seem to make any difference which Pi I used.

Anyway, over time I've always checked memory usage, but it is very hard to get exact numbers, as the amount of memory fluctuates due to variations in the web server, the SVN server, and others, and I don't know which of these ever combined to create an abnormal peak. What I've always used as an indicator that RAM is never exhausted is the fact that the 100MB swap never gets used.

Anyway, I just checked these numbers:


This is a typical process list when everything is running (but NFMC is not playing anything):


Next is a typical "top" result, with the domotics server always using about 40% of the CPU (the culprit is the FTDI driver for the USB chip), hence a niceness of 10 for the process:


Here is the "split" result:


Does this give you any idea of what the problem might be?

nothing like to be 100% sure, but you got my attention with the USB traffic / utilization.

there seems to be few issues with the RPIs usb design causing problems starting from usb devices not found to problems with IRQ/timings ending with kernel panics - related to some usage pattern / usb devices combinations.

FIQ split should be a help there, now by default turned on with 3.10.y kernels. this wasn't the case with older branches (or months ago). XBian B1 / 1.1 wasn't using those patches - but Raspbian does use them / has patched kernels.

nevertheless it is my assumption, not proof of something, but generally speaking there are not much other relevant differences, if any.

for example there was an issue opened at github recently https://github.com/xbianonpi/xbian/issues/457 which would make it (together with you and one other) three 'reported' cases with similar symptoms I know of.

I will just tell you the same as in the comment to the issue - Beta1 branch will retire very soon. If you have reasons to go for XBMC bundled distro like XBian or RaspBMC (I have the feeling it won't be OpenELEC), give it a week or two until Beta2 is released and give it a try then.

(22nd Sep, 2013 09:12 AM)mk01 Wrote: [ -> ]nothing like to be 100% sure, but you got my attention with the USB traffic / utilization.

there seems to be few issues with the RPIs usb design causing problems starting from usb devices not found to problems with IRQ/timings ending with kernel panics - related to some usage pattern / usb devices combinations.

FIQ split should be a help there, now by default turned on with 3.10.y kernels. this wasn't the case with older branches (or months ago). XBian B1 / 1.1 wasn't using those patches - but Raspbian does use them / has patched kernels.

nevertheless it is my assumption, not proof of something, but generally speaking there are not much other relevant differences, if any.

for example there was an issue opened at github recently https://github.com/xbianonpi/xbian/issues/457 which would make it (together with you and one other) three 'reported' cases with similar symptoms I know of.

I will just tell you the same as in the comment to the issue - Beta1 branch will retire very soon. If you have reasons to go for XBMC bundled distro like XBian or RaspBMC (I have the feeling it won't be OpenELEC), give it a week or two until Beta2 is released and give it a try then.

Your post makes sense to me on so many levels, while at the same time gets me confused about a few situations I encountered.

I started suspecting USB problems when I tried to get a composite video capture card to work with Raspbian. The capture device was not recognized in Raspbian, and various posts in various forums commented on the USB issues with the Pi and also of support being better in 3.7+ kernels.

My Raspbian is 3.6, and because XBian was 3.9 I expected it to be better at the USB issues, but the capture cards still didn't work. In fact (and that is where the first contradiction showed up), it seemed a lot worse with the capture cards I tried (detection problems, hanging, intermittent registrations and deregistrations of /dev/video0). Still, I tested about 4 capture cards on 2 different Pi's with both Raspbian and XBian, plus one of the power supplies I used turned out to be faulty, and sometimes with a USB hub and sometimes without. So I gave up trying to make sense of it all, and gave up capture cards for a while.

Then, yesterday, after the 8 or 10 days of running Raspbian and my "media center" without problems (as I mentioned in a previous post) I decided to try the capture cards in Raspbian again. I apt-updated and then apt-upgraded, and it all seemed somewhat better with the cards, but they still didn't work.

I gave up the video capture again, disconnected all the capture cards, rebooted, but - and this is where I was blown away - after a few hours the Pi hung. I turned on the TV looking for panics, but when my Pi is rebooted with HDMI matrix using another HDMI source the result is that the Pi's console is all black and no text is displayed. Since I had rebooted the Pi in these conditions the TV showed nothing, so there might be a panic that I was not able to see (only later did I think that I could have tried the composite output before restarting the Pi, but by then it was too late, I'll remember it next time).

What came to my mind right away was that the unupdated/unupgraded Raspbian (originally from NOOBS v1.2.1, I think) might have been more tolerant to the Pi's USB instabilities than both the updated/upgraded Raspbian and XBian (which is a few versions ahead of Raspbian). Maybe some change in versions after that "original" Raspbian introduced something that is affected by my setup, while previously it wasn't.

Corroborating this, after the Raspbian update/upgrade I immediately noticed something was worse, because the lights in the hallway took longer to turn on than they did before (these specific lights exchange a lot of messages with the server, so they are a noticeable and clear indicator that something changed). The FTDI USB driver in the Pi has always been the domotics bottleneck, and has always governed the reaction time of sensors and actuators, so the fact that it is now slower (2 seconds to turn on instead of the 0.2 to 0.3 seconds it took before) makes me suspect of a degradation in USB performance in the updated/upgraded version of Raspbian.

... coincidentally, turning on the lights in XBian was also slow (maybe the same 2 seconds), but I have always attributed that to the high CPU consumption of XBMC, even while sitting idle, coupled with is niceness of -3. Now I'm beginning to think that the delay in XBian was not CPU related, but USB related instead, and the theory that the new kernels are the ones causing me problems makes even more sense.

Although this theory made complete sense to me yesterday, it doesn't hold when confronted with your explanation that Raspbian had used patches and that XBian hadn't, because when updating/upgrading Raspbian the behaviour becomes closer to that of XBian (unless in case of some weird possibilities, like the updated/upgraded Raspbian not using the patches anymore).

This is all very recent, so I will still need the Pi to hang a few more times to draw any conclusions

As for the next XBian release - I'll be looking forward to it and will surely give it a try when it comes out.

I can tell you newer kernel revisions have regression issues occasionally, one I know of is lirc_rpi but I'm sure there are others.

(22nd Sep, 2013 12:25 PM)Rilhas Wrote: [ -> ]Although this theory made complete sense to me yesterday, it doesn't hold when confronted with your explanation that Raspbian had used patches and that XBian hadn't, because when updating/upgrading Raspbian the behaviour becomes closer to that of XBian (unless in case of some weird possibilities, like the updated/upgraded Raspbian not using the patches anymore).

Rilhas,

Just to put this right - please allow some minor deviations from reality and what I said - I never really followed 3.6.y development in the core code - so finally what for me (watching forums) looks like a patch / custom hacks - can for months be in the master branch of RPI kernel tree - and later even moved to 3.8, 3.9 … And 3.10.y branch I noticed very massive (and released often) patching in general around dwc_otg and usb. Also, some parameters are now by default, what wasn't the case with kernels I remember from past.

We moved to more recent versions because of btrfs not being back ported and before 3.8.y there has been serious issues in it. On the other hand Raspbian firmware / kernel (together with RaspBMC) are closely following the git's RPI kernel/firmware repo - development and default config options as well. RaspBMC is now going to 3.10.y, Raspbian I don't know.

@f1vefour 3.9.y or 3.10.y as well? I know 3.9.y had problem with lirc_* if compiled with a different allocation model than SLAB. even CurlyMo posted the panics somewhere on the forum.
does fiq_split_enable=0 make any difference to you ?

In 3.9.x and 3.10.x there are issues with lirc_rpi still. This is from dmesg, when I recieve this the remote stops functioning for a random amount of time. I haven't tried the fiq kernel flag.


Why would SLUB/SLAB allocation matter?

(1st Oct, 2013 02:10 PM)f1vefour Wrote: [ -> ]Why would SLUB/SLAB allocation matter?

exactly. should not.

but the paths of kernel hackers are hidden to me

but this one specially was ending with OOOP.