btrfs and lz4

[Chris83]

(3rd Sep, 2014 09:03 PM)CurlyMo Wrote:  With all respect, don't follow the advice of f1vefour! It can actually wreck the filesystem totally.

The problem is that the kernel you are using doesn't have support for the lz4 feature in BTRFS:

Code:

#define BTRFS_FEATURE_INCOMPAT_SUPP            \
    (BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF |        \
     BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL |    \
     BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS |        \
     BTRFS_FEATURE_INCOMPAT_BIG_METADATA |        \
     BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO |        \
     BTRFS_FEATURE_INCOMPAT_RAID56 |        \
     BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF |        \
     BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA |    \
     BTRFS_FEATURE_INCOMPAT_COMPRESS_LZ4 |        \
     BTRFS_FEATURE_INCOMPAT_NO_HOLES)

The 10th feature in the list. You need to a patched kernel which linux Mint apparently doesn't provide.

I'm still backing up all my files via Samba before I try anything else (takes 24h total...).
How do I find out which Kernel has support for lz4 and which one doesn't?
Can I add compatibility myself or would I need to completely change the distribution (I read somewhere ArchLinux has LZ4 support)?
I'm using the 32bit kernel from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.16.1-utopic/
Linux Mint itself is still at 3.13.x

[CurlyMo]

Check the fs/btrfs/ctree.h in the souce for this piece of code.

[Chris83]

(3rd Sep, 2014 10:06 PM)CurlyMo Wrote:  Check the fs/btrfs/ctree.h in the souce for this piece of code.

Ok I did read up a bit more, but according to the following mail thread it doesn't look like there is going to be lz4 kernel support for brtfs anytime soon (at least outside of ARM Kernels): https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg34357.html

So that brings me to my next question: Is there a way to create a Xbian USB drive installation with zlib instead of lz4? Is there any documentation on the single steps xbian-config performs during the clone so I could do it manually step by step?

[f1vefour]

I was simply following the advice of the BTRFS wiki. I did provide a link that warned of the potential danger on the page.

Thanks for coming behind me and warning against this, I'm certainly no filesystem guru.

Apparently you know more than they do, the wiki simply says:

Quote:if and only if the kernel oops in your logs has something like this in the middle of it, or you get "open ctree failed" on a newer kernel:

? replay_one_dir_item+0xb5/0xb5 [btrfs] ? walk_log_tree+0x9c/0x19d [btrfs] ? btrfs_read_fs_root_no_radix+0x169/0x1a1 [btrfs] ? btrfs_recover_log_trees+0x195/0x29c [btrfs] ? replay_one_dir_item+0xb5/0xb5 [btrfs] ? btree_read_extent_buffer_pages+0x76/0xbc [btrfs] ? open_ctree+0xff6/0x132c [btrfs]

Newer kernels output this instead:

BTRFS: failed to read log tree BTRFS: open_ctree failed

To be fair it says nothing about it being compression related.

[Chris83]

I suppose they don't expect anyone trying to mount with something else beside zlib or lz0.
Anyway, I really don't want to build and maintain my own i386 Kernel with lz4 support (if anyone wants to give it a go, the instructions in the mailing list I posted tell you how). Hence I went a different route now and I just create two partitions on my 2TB drive, a small one with btrfs for RPi to run on it and a second big one with standard NTFS so I can easily copy and share all my media to other devices - that satisfies all my requirements.

[f1vefour]

Apparently it's depreciated or a third party patch, I haven't researched it.

Edit:
Okay so it's a third party patch, I have to wonder why was this was implemented knowing it would be impossible to mount the filesystem externally without a modified kernel.

[CurlyMo]

Because lz4 has many advantages for slow computers

[f1vefour]

I'm going to go out on a limb here and say that while the speed improvements aren't negligible I don't really notice them in my day to day XBMC usage. My 700+ movie and show library loads fairly speedy, the same as it did before BTRFS. I don't use XBMC for music so I can't compare that aspect.

Ah it's here nor there I suppose, it is in use already. I have no issues compiling the patch for my Mint system if I really need it, I have probably compiled about 3000 kernels since 1996.

[CurlyMo]

The point is that BTRFS is slower in almost all cases than EXT4. To compensate, we use techniques like LZ4 compression.

[f1vefour]

I know it was implemented not for speed or space savings but for the online snapshot capabilities. This alone makes it worthwhile.

It doesn't really matter to me that much as I keep important files on an external disk. If something goes critically wrong with the system I just start from scratch. It would be nice to be able to use the snapshots to restore from a Linux PC however, this way a nonfunctional installation could be repaired with the snapshots on the disk.

[CurlyMo]

Yes, but we created the rescue shell for that purpose

[f1vefour]

I meant if the system becomes in a state where it won't boot, not from a filesystem error but something else.

The rescue shell is awesome though

[CurlyMo]

Then you can restore the files so the rescue shell will work. They can be downloaded from the mk01 webserver.

[mk01]

the most tricky things are sometimes easiest to solve.

we provide lz4 patches - tested and build for - 3.12, 3.14, 3.15, 3.10 ...
https://github.com/xbianonpi/xbian-package-kernel/tree/master/patchesnt

then its needed to activate libkernel suppoer for lz4, lz4fast and of course appropriate compressors. all is in clasic kernel's make menuconfig.

lz4 is pre-created with XBian, once tag lz4 written (compress=lz4 used while mounting), btrfs mounter will refuse to load on kernels without the same support.

if this should be causing troubles (no possibility to compile custom kernel for other pc, or simply user doesn't want to bother with stuff) - I would recommend re-cloning to img (or disk directly) with lzo (or compress disabled) and then don't use compress=lz4 on mount.

(cmdline.txt although auto updated when needed (by xbian-update or kernel installation), there are some settings which are kept as user set them before. compress=XX is one of those.

---

(5th Sep, 2014 03:17 PM)f1vefour Wrote:  I know it was implemented not for speed or space savings but for the online snapshot capabilities. This alone makes it worthwhile.

It doesn't really matter to me that much as I keep important files on an external disk. If something goes critically wrong with the system I just start from scratch. It would be nice to be able to use the snapshots to restore from a Linux PC however, this way a nonfunctional installation could be repaired with the snapshots on the disk.

@f1vefour

as xbian-cloner using snapshot send | recv to create img or new part (as copy of local), the same way you can use it to send snap literally anywhere! now check this: (just for info, because snap stream is sent RAW, local & remote compat flags doesn't need to match !

Code:

btrfs send SNAP > filename
btrfs recv root < filename

file can be placed over net, on USB stick, printed out on printer and later retyped back probably you know ssh can be called just to execute command on remote machine ... like this

Code:

ssh xbian@xbian echo "this is printed on vts on xbian"

then a bit more hacking would be

Code:

echo "this is sent TO xbian and echoed there" | ssh xbian@xbian cat

and

Code:

cat filename | ssh xbian@xbian "cat > openme"

finally going to

Code:

btrfs send SNAP | ssh xbian@xbian "cat > fileBACKUPofSNAP"
btrfs send SNAP | ssh xbian@xbian "btrfs recv BACKUPofSNAP"

this assumes btrfs filesystem on remote system OR just save to file as in first example - but directly to remote.

just be always aware that btrfs send has no quality and structure of mountable filesystem (can't be directly loop mounted). it is just raw CoW stream (so metadata and data as they were saved to local btrfs - like magnetic tape or even the back to '70 paper punch cards (having holes)...

so it needs to be restored to btrfs again:

Code:

cat file | btrfs recv ./

(backup home function is using that way to save /home subvolume into .gz file while uing btrfs recv later for restore (all atomic on open writeable filesystem in use).

[Chris83]

*wooooosh* right over my head which button must I click? ;-)