4.19.42+ kernel broke the file system, LZ4 issue, how to revert to prev kernel?

[Gyuunduh]

I'm sorry for not being able to access the log files for exact commands or error messages i've ran and had before running into problems. Also I feel I'm a bit out of me depth in here, having learnt to tinker with Linux a bit with Xbian but having no proper understanding of cmdline.txt or boot blocks, I only know how to use btrfs-auto-snapshots. Anyway here goes:

I have been running an Xbian installation at least one year old, not paying any attention to filesystem compressions or messing with them as everything has been working just fine. The latest kernel update on my Raspberry Pi 2 (4.19.42+ I guess) didn't work, apt-get upgrade gave me an error message saying: "you have LZ4 compression enabled, but this is not longer supported by kernel v4.19.42+"

The advice apt-get gave me was to run apt-get --remove-compression or apt-get --recompress or something like that that had compression in it. I dont' recall that exactly and I can't check what it was because the log files are out of my reach now.

I ran the command, and an ASCII visualization of the process showed up. It failed at 75 %, error message saying something about there not being enough space, running out of kilobytes maybe (that sounded legit, I had something like 4 GB of free space on 100+ GB SD card).

I tried running apt-get upgrade again anyway after that and this time it proceeded to unpack and install the newer kernel. But after reboot Xbian boots into recovery console, failing to mount root, telling me:
"wrong fs type, bad option, bad superblock on /dev/mmcblk0p2, missing codepage or helper program or other error." And later: "The root partition as specified in cmdline.txt or boot.src.txt is now mounted under /rootsfs", and that the directory is totally empty.

I'm pretty sure it has something to do with the kernel being upgraded to a version that doesn't support LZ4 compression but the file system still being compressed and thus being unaccessible.

Is there any way to revert to an older kernel that still supports the compression, so I can access my Xbian filesystem? Or is the problem somewhere else?

Anyway, I'd be really grateful for help, because right now it seems that I can't access any files on my Xbian.

[ Subject edit, because of exceeding max length ]

[Nachteule]

(18th May, 2019 09:04 AM)Gyuunduh Wrote:  I'm sorry for not being able to access the log files for exact commands or error messages i've ran and had before running into problems. Also I feel I'm a bit out of me depth in here, having learnt to tinker with Linux a bit with Xbian but having no proper understanding of cmdline.txt or boot blocks, I only know how to use btrfs-auto-snapshots. Anyway here goes:

I have been running an Xbian installation at least one year old, not paying any attention to filesystem compressions or messing with them as everything has been working just fine. The latest kernel update on my Raspberry Pi 2 (4.19.42+ I guess) didn't work, apt-get upgrade gave me an error message saying: "you have LZ4 compression enabled, but this is not longer supported by kernel v4.19.42+"

The advice apt-get gave me was to run apt-get --remove-compression or apt-get --recompress or something like that that had compression in it. I dont' recall that exactly and I can't check what it was because the log files are out of my reach now.

I ran the command, and an ASCII visualization of the process showed up. It failed at 75 %, error message saying something about there not being enough space, running out of kilobytes maybe (that sounded legit, I had something like 4 GB of free space on 100+ GB SD card).

I tried running apt-get upgrade again anyway after that and this time it proceeded to unpack and install the newer

That was really bad idea

Quote:kernel. But after reboot Xbian boots into recovery console, failing to mount root, telling me:
"wrong fs type, bad option, bad superblock on /dev/mmcblk0p2, missing codepage or helper program or other error." And later: "The root partition as specified in cmdline.txt or boot.src.txt is now mounted under /rootsfs", and that the directory is totally empty.

There is no way to get access to your files from recovery console, because installed kernel does not support LZ4 compression anymore. That's the bad news.

The good news is, your data is still present, what you need is an older kernel, but you can't install the kernel from recovery console.

So, one way could be install XBian on a fresh sd-card, boot from that card and install older kernel. Then you can get access to your old installation by using an usb sd-card reader plugging into an usb port of your Pi

Quote:I'm pretty sure it has something to do with the kernel being upgraded to a version that doesn't support LZ4 compression but the file system still being compressed and thus being unaccessible.

That's right. The command you run was xbian-storager compress, but for proper work you need at least 50% of free disk space. You do not have this, so command failed

Quote:Is there any way to revert to an older kernel that still supports the compression, so I can access my Xbian filesystem? Or is the problem somewhere else?

Anyway, I'd be really grateful for help, because right now it seems that I can't access any files on my Xbian.

Yes, first solution see above

Another solution would be, install XBian on a fresh sd-card, boot from that card and install older kernel. Then copy the content of this boot partition to that old boot partition and now you can boot again from your old card (backing up the boot partition first would be a good idea).

If you can boot now, you should install older kernel again and after that, you have to lock this kernel to prevent it from overwritten.

[Nachteule]

Install older kernel procedure:

1) refresh apt database

Terminal

sudo apt-get update

2) get a list of available versions

Terminal

sudo apt-cache policy linux-image-bcm2836
linux-image-bcm2836:
Installiert: 5.1.1+-1557851454
Installationskandidat: 5.1.1+-1557851454
Versionstabelle:
*** 5.1.1+-1557851454 100
100 /var/lib/dpkg/status
4.19.42+-1557939063 500
500 http://kmcubie stable/rpi2-stretch armhf Packages
4.19.34+-1555996925 500
500 http://kmcubie staging/rpi2-stretch armhf Packages
4.19.30+-1553363462 500
500 http://kmcubie staging/rpi2-stretch armhf Packages
4.19.29+-1552695630 500
500 http://kmcubie staging/rpi2-stretch armhf Packages
4.14.102+-1555993600 500
500 http://kmcubie stable/rpi2-stretch armhf Packages
4.14.102+-1550774632 500
500 http://kmcubie stable/rpi2-stretch armhf Packages
4.9.108+-1530490703 500
500 http://kmcubie stable/armv7l-stretch armhf Packages

3) Install older kernel version 4.14.102+-1550774632 for example

Terminal

sudo apt-get install linux-image-bcm2836=4.14.102+-1550774632

[Nachteule]

Lock kernel package from being updated:

Terminal

echo "linux-image-bcm2836 hold" | sudo dpkg --set-selections

Remove lock:

Terminal

echo "linux-image-bcm2836 install" | sudo dpkg --set-selections

[Gyuunduh]

(21st May, 2019 06:41 AM)Nachteule Wrote:  Lock kernel package from being updated:

Terminal

echo "linux-image-bcm2836 hold" | sudo dpkg --set-selections

Remove lock:

Terminal

echo "linux-image-bcm2836 install" | sudo dpkg --set-selections

Thank you for a really detailed and thorough answer.

I just wasn't patient enough to wait . I ran the System rescue cd -iso from an USB stick on my Windows PC, trying to back up the contents of my home folder and config files for samba, cups and such from the Xbian SD card. It turned out that most of the files were unreadable, I'm not sure if the failed xbian-storager compress corrupted those or if there was something exotic with the way xbian handles the file system and made them unreadable for system rescue cd.

So I went my way, made backups of those files I could, wiped the SD card and started off with a fresh install. :| There seems to be some problems with the fresh install, the initial configuration wizard and the Kodi xbian config both freezing when loading the list of packages installed and to be installed. Oh well, I'll start another thread for that.

And again, a million thanks for a really good answer. Whether the filesystem already been corrupted or if it could have been saved, I guess we won't know. Soon I'll have a fully working Xbian running again

[Nachteule]

(21st May, 2019 08:41 AM)Gyuunduh Wrote:  

(21st May, 2019 06:41 AM)Nachteule Wrote:  Lock kernel package from being updated:

Terminal

echo "linux-image-bcm2836 hold" | sudo dpkg --set-selections

Remove lock:

Terminal

echo "linux-image-bcm2836 install" | sudo dpkg --set-selections

Thank you for a really detailed and thorough answer.

I just wasn't patient enough to wait . I ran the System rescue cd -iso from an USB stick on my Windows PC, trying to back up the contents of my home folder and config files for samba, cups and such from the Xbian SD card. It turned out that most of the files were unreadable, I'm not sure if the failed xbian-storager compress corrupted those or if there was something exotic with the way xbian handles the file system and made them unreadable for system rescue cd.

Hmmm, I'm wondering how Rescue-CD was able to access a LZ4 compressed btrfs file system. Can't believe that this system has LZ4 compression support on board.

I'm sure, that neither Rescue-CD nor xbian-storager compress destroyed fs though. xbian-storager compress command never touches existing subvolumes until operation has been completed successfully and a reboot has been done

Quote:So I went my way, made backups of those files I could, wiped the SD card and started off with a fresh install. :| There seems to be some problems with the fresh install, the initial configuration wizard and the Kodi xbian config both freezing when loading the list of packages installed and to be installed. Oh well, I'll start another thread for that.

Does the problem still exists?

Quote:And again, a million thanks for a really good answer. Whether the filesystem already been corrupted or if it could have been saved, I guess we won't know. Soon I'll have a fully working Xbian running again

You're welcome

[Gyuunduh]

(21st May, 2019 10:21 PM)Nachteule Wrote:  

(21st May, 2019 08:41 AM)Gyuunduh Wrote:  

(21st May, 2019 06:41 AM)Nachteule Wrote:  Lock kernel package from being updated:

Terminal

echo "linux-image-bcm2836 hold" | sudo dpkg --set-selections

Remove lock:

Terminal

echo "linux-image-bcm2836 install" | sudo dpkg --set-selections

Thank you for a really detailed and thorough answer.

I just wasn't patient enough to wait . I ran the System rescue cd -iso from an USB stick on my Windows PC, trying to back up the contents of my home folder and config files for samba, cups and such from the Xbian SD card. It turned out that most of the files were unreadable, I'm not sure if the failed xbian-storager compress corrupted those or if there was something exotic with the way xbian handles the file system and made them unreadable for system rescue cd.

Hmmm, I'm wondering how Rescue-CD was able to access a LZ4 compressed btrfs file system. Can't believe that this system has LZ4 compression support on board.

I'm sure, that neither Rescue-CD nor xbian-storager compress destroyed fs though. xbian-storager compress command never touches existing subvolumes until operation has been completed successfully and a reboot has been done

Their website says that the btrfs with LZ4 compression support is there and it managed to read at least some of the files, so at least partially it worked. I don't know if all the files would've been recoverable. There could've been problems also somewhere else.

Quote:

Quote:So I went my way, made backups of those files I could, wiped the SD card and started off with a fresh install. :| There seems to be some problems with the fresh install, the initial configuration wizard and the Kodi xbian config both freezing when loading the list of packages installed and to be installed. Oh well, I'll start another thread for that.

Does the problem still exists?

The latest image had the latest kernel and all the previous data had already been lost. I guess you can say the problem of not being able to access to file system doesn't exist if the file system has been wiped

Quote:

Quote:And again, a million thanks for a really good answer. Whether the filesystem already been corrupted or if it could have been saved, I guess we won't know. Soon I'll have a fully working Xbian running again

You're welcome

[Nachteule]

Quote:Their website says that the btrfs with LZ4 compression support is there and it managed to read at least some of the files, so at least partially it worked. I don't know if all the files would've been recoverable. There could've been problems also somewhere else.

Sure, hard to say. And sd-card has been reflashed - so any further discussion about this is useless

But its interesting that rescue-cd supports lz4 compression. I couldn't find the info on their web-site, but I'm trusting you.

Today I was trying to reproduce what you did - I was wondering how it was possible to install newer kernel - and now I figured out what you probably did:

You run sudo xbian-compress and this command failed with error. But you did not reboot the system, you just returned to the command prompt. And at this state, information about the btrfs compression status was lost, because xbian-compress remounts root fs with lzo compression. And - unfortunately - the installation procedure checks current mount status of root fs only, nothing more.

So, finally, you was able to install kernel again

Quote:here seems to be some problems with the fresh install, the initial configuration wizard and the Kodi xbian config both freezing when loading the list of packages installed and to be installed. Oh well, I'll start another thread for that.

I was asking if your freezing issue is still there, not if your compression issue still exists

[Gyuunduh]

(22nd May, 2019 01:12 AM)Nachteule Wrote:  

Quote:here seems to be some problems with the fresh install, the initial configuration wizard and the Kodi xbian config both freezing when loading the list of packages installed and to be installed. Oh well, I'll start another thread for that.

I was asking if your freezing issue is still there, not if your compression issue still exists

Ah, sorry I got that wrong. Yeah, the freezing issue with that SD card still exists.

I bought a new card (and a new Raspberry Pi 3+ to replace my Pi 2, as it was on sale for 20 €) and with the new board and SD card everything works like a charm. When I try to mess around with the old Pi and SD card, the apt-get install downloads have long slowdown spikes. I have no idea what that is about, I'm just happy that the new system is working. If I had any other spare microSD cards around, I'd try to find out if the slowdown on Pi 2 are because of the sd card if there's something wrong with the board itself, but for now that remains a mystery.

[Nachteule]

Have fun with your new Pi

Quote:If I had any other spare microSD cards around, I'd try to find out if the slowdown on Pi 2 are because of the sd card if there's something wrong with the board itself, but for now that remains a mystery.

Yes indeed, it sounds like a mystery

[ikke]

Something went wrong for me too. The crypt went through, and I managed to reboot it. Then I rerun apt, which upgraded almost. Before the last reboot I noticed /boot run out of diskspace during apt upgrade (boot size is only 34MB). I removed some useless .dtb files for older raspis to make space, and did run apt install --reinstall for the kernel. It went through fine, but now it won't boot.

Now system can't find root any longer, nor does the keyboard work.

I need to dig further tomorrow, but word of warning for other updaters.

[Nachteule]

How old is your installation?

Do you really have ext4 as rootfs?

[ikke]

It's ears old, it has served very well I need to check the filesystems.

[ikke]

Well, it was 3 mins fix. Due to /boot being too small and filling up, dpkg had failed to copy commandline.txt. All good now, except that xbian should clean up the /boot a bit to make space there. 34 megs just isn't enough for all the different raspi models device tree blobs and all.

It could be the new installs already have it bigger, this is old one. Like said, it has served me well, no need for fresh installs!

And no, root is not ext4 but btrfs, it just tried to default to wrong fs due empty commandline.txt.

[Nachteule]

(24th Jul, 2019 04:49 PM)ikke Wrote:  Well, it was 3 mins fix. Due to /boot being too small and filling up, dpkg had failed to copy commandline.txt. All good now, except that xbian should clean up the /boot a bit to make space there. 34 megs just isn't enough for all the different raspi models device tree blobs and all.

In the meantime, the boot partition has been grown up to 100MB in two steps: 67MB about 3-4 years ago, 100MB some months ago

Quote:It could be the new installs already have it bigger, this is old one. Like said, it has served me well, no need for fresh installs!

And no, root is not ext4 but btrfs, it just tried to default to wrong fs due empty commandline.txt.

Yes, see above. The missing/wrong cmdline.txt was my first idea as well