+- Forum (http://forum.xbian.org)
+-- Forum: Software (/forum-6.html)
+--- Forum: Testing & Experimental (/forum-21.html)
+--- Thread: btrfs and lz4 (/thread-2217.html)
RE: btrfs and lz4 - Chris83 - 3rd Sep, 2014 10:03 PM
(3rd Sep, 2014 09:03 PM)CurlyMo Wrote: With all respect, don't follow the advice of f1vefour! It can actually wreck the filesystem totally.
The problem is that the kernel you are using doesn't have support for the lz4 feature in BTRFS:
The 10th feature in the list. You need to a patched kernel which linux Mint apparently doesn't provide.Code:
#define BTRFS_FEATURE_INCOMPAT_SUPP \
(BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF | \
BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL | \
BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS | \
BTRFS_FEATURE_INCOMPAT_BIG_METADATA | \
BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO | \
BTRFS_FEATURE_INCOMPAT_RAID56 | \
BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF | \
BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA | \
BTRFS_FEATURE_INCOMPAT_COMPRESS_LZ4 | \
BTRFS_FEATURE_INCOMPAT_NO_HOLES)
I'm still backing up all my files via Samba before I try anything else (takes 24h total...).
How do I find out which Kernel has support for lz4 and which one doesn't?
Can I add compatibility myself or would I need to completely change the distribution (I read somewhere ArchLinux has LZ4 support)?
I'm using the 32bit kernel from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.16.1-utopic/
Linux Mint itself is still at 3.13.x
RE: btrfs and lz4 - CurlyMo - 3rd Sep, 2014 10:06 PM
Check the fs/btrfs/ctree.h in the souce for this piece of code.
RE: btrfs and lz4 - Chris83 - 3rd Sep, 2014 11:34 PM
(3rd Sep, 2014 10:06 PM)CurlyMo Wrote: Check the fs/btrfs/ctree.h in the souce for this piece of code.
Ok I did read up a bit more, but according to the following mail thread it doesn't look like there is going to be lz4 kernel support for brtfs anytime soon (at least outside of ARM Kernels): https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg34357.html
So that brings me to my next question: Is there a way to create a Xbian USB drive installation with zlib instead of lz4? Is there any documentation on the single steps xbian-config performs during the clone so I could do it manually step by step?
Re: RE: btrfs and lz4 - f1vefour - 4th Sep, 2014 08:28 AM
I was simply following the advice of the BTRFS wiki. I did provide a link that warned of the potential danger on the page.
Thanks for coming behind me and warning against this, I'm certainly no filesystem guru.
Apparently you know more than they do, the wiki simply says:
Quote:if and only if the kernel oops in your logs has something like this in the middle of it, or you get "open ctree failed" on a newer kernel:
? replay_one_dir_item+0xb5/0xb5 [btrfs] ? walk_log_tree+0x9c/0x19d [btrfs] ? btrfs_read_fs_root_no_radix+0x169/0x1a1 [btrfs] ? btrfs_recover_log_trees+0x195/0x29c [btrfs] ? replay_one_dir_item+0xb5/0xb5 [btrfs] ? btree_read_extent_buffer_pages+0x76/0xbc [btrfs] ? open_ctree+0xff6/0x132c [btrfs]
Newer kernels output this instead:
BTRFS: failed to read log tree BTRFS: open_ctree failed
To be fair it says nothing about it being compression related.
RE: btrfs and lz4 - Chris83 - 4th Sep, 2014 08:53 AM
I suppose they don't expect anyone trying to mount with something else beside zlib or lz0.
Anyway, I really don't want to build and maintain my own i386 Kernel with lz4 support (if anyone wants to give it a go, the instructions in the mailing list I posted tell you how). Hence I went a different route now and I just create two partitions on my 2TB drive, a small one with btrfs for RPi to run on it and a second big one with standard NTFS so I can easily copy and share all my media to other devices - that satisfies all my requirements.
RE: btrfs and lz4 - f1vefour - 4th Sep, 2014 11:45 AM
Apparently it's depreciated or a third party patch, I haven't researched it.
Edit:
Okay so it's a third party patch, I have to wonder why was this was implemented knowing it would be impossible to mount the filesystem externally without a modified kernel.
RE: btrfs and lz4 - CurlyMo - 4th Sep, 2014 06:11 PM
Because lz4 has many advantages for slow computers
Re: btrfs and lz4 - f1vefour - 5th Sep, 2014 05:20 AM
I'm going to go out on a limb here and say that while the speed improvements aren't negligible I don't really notice them in my day to day XBMC usage. My 700+ movie and show library loads fairly speedy, the same as it did before BTRFS. I don't use XBMC for music so I can't compare that aspect.
Ah it's here nor there I suppose, it is in use already. I have no issues compiling the patch for my Mint system if I really need it, I have probably compiled about 3000 kernels since 1996.
RE: btrfs and lz4 - CurlyMo - 5th Sep, 2014 07:46 AM
The point is that BTRFS is slower in almost all cases than EXT4. To compensate, we use techniques like LZ4 compression.
Re: btrfs and lz4 - f1vefour - 5th Sep, 2014 03:17 PM
I know it was implemented not for speed or space savings but for the online snapshot capabilities. This alone makes it worthwhile.
It doesn't really matter to me that much as I keep important files on an external disk. If something goes critically wrong with the system I just start from scratch. It would be nice to be able to use the snapshots to restore from a Linux PC however, this way a nonfunctional installation could be repaired with the snapshots on the disk.
RE: btrfs and lz4 - CurlyMo - 5th Sep, 2014 05:53 PM
Yes, but we created the rescue shell for that purpose
RE: btrfs and lz4 - f1vefour - 5th Sep, 2014 09:52 PM
I meant if the system becomes in a state where it won't boot, not from a filesystem error but something else.
The rescue shell is awesome though
RE: btrfs and lz4 - CurlyMo - 6th Sep, 2014 06:26 AM
Then you can restore the files so the rescue shell will work. They can be downloaded from the mk01 webserver.
RE: btrfs and lz4 - mk01 - 13th Sep, 2014 12:52 AM
the most tricky things are sometimes easiest to solve.
we provide lz4 patches - tested and build for - 3.12, 3.14, 3.15, 3.10 ...
https://github.com/xbianonpi/xbian-package-kernel/tree/master/patchesnt
then its needed to activate libkernel suppoer for lz4, lz4fast and of course appropriate compressors. all is in clasic kernel's make menuconfig.
lz4 is pre-created with XBian, once tag lz4 written (compress=lz4 used while mounting), btrfs mounter will refuse to load on kernels without the same support.
if this should be causing troubles (no possibility to compile custom kernel for other pc, or simply user doesn't want to bother with stuff) - I would recommend re-cloning to img (or disk directly) with lzo (or compress disabled) and then don't use compress=lz4 on mount.
(cmdline.txt although auto updated when needed (by xbian-update or kernel installation), there are some settings which are kept as user set them before. compress=XX is one of those.
(5th Sep, 2014 03:17 PM)f1vefour Wrote: I know it was implemented not for speed or space savings but for the online snapshot capabilities. This alone makes it worthwhile.
It doesn't really matter to me that much as I keep important files on an external disk. If something goes critically wrong with the system I just start from scratch. It would be nice to be able to use the snapshots to restore from a Linux PC however, this way a nonfunctional installation could be repaired with the snapshots on the disk.
@f1vefour
as xbian-cloner using snapshot send | recv to create img or new part (as copy of local), the same way you can use it to send snap literally anywhere! now check this: (just for info, because snap stream is sent RAW, local & remote compat flags doesn't need to match !
Code:
btrfs send SNAP > filename
btrfs recv root < filenamefile can be placed over net, on USB stick, printed out on printer and later retyped back
probably you know ssh can be called just to execute command on remote machine ... like thisCode:
ssh xbian@xbian echo "this is printed on vts on xbian"then a bit more hacking would be
Code:
echo "this is sent TO xbian and echoed there" | ssh xbian@xbian catand
Code:
cat filename | ssh xbian@xbian "cat > openme"finally going to
Code:
btrfs send SNAP | ssh xbian@xbian "cat > fileBACKUPofSNAP"
btrfs send SNAP | ssh xbian@xbian "btrfs recv BACKUPofSNAP"this assumes btrfs filesystem on remote system OR just save to file as in first example - but directly to remote.
just be always aware that btrfs send has no quality and structure of mountable filesystem (can't be directly loop mounted). it is just raw CoW stream (so metadata and data as they were saved to local btrfs - like magnetic tape or even the back to '70 paper punch cards (having holes)...
so it needs to be restored to btrfs again:
Code:
cat file | btrfs recv ./(backup home function is using that way to save /home subvolume into .gz file while uing btrfs recv later for restore (all atomic on open writeable filesystem in use).
RE: btrfs and lz4 - Chris83 - 13th Sep, 2014 01:49 AM
*wooooosh* right over my head
which button must I click? ;-)