Thank you for your donation


Post Reply 
 
Thread Rating:
  • 0 Votes - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
btrfs and lz4
3rd Sep, 2014, 10:03 PM
Post: #16
RE: btrfs and lz4
(3rd Sep, 2014 09:03 PM)CurlyMo Wrote:  With all respect, don't follow the advice of f1vefour! It can actually wreck the filesystem totally.

The problem is that the kernel you are using doesn't have support for the lz4 feature in BTRFS:
Code:
#define BTRFS_FEATURE_INCOMPAT_SUPP            \
    (BTRFS_FEATURE_INCOMPAT_MIXED_BACKREF |        \
     BTRFS_FEATURE_INCOMPAT_DEFAULT_SUBVOL |    \
     BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS |        \
     BTRFS_FEATURE_INCOMPAT_BIG_METADATA |        \
     BTRFS_FEATURE_INCOMPAT_COMPRESS_LZO |        \
     BTRFS_FEATURE_INCOMPAT_RAID56 |        \
     BTRFS_FEATURE_INCOMPAT_EXTENDED_IREF |        \
     BTRFS_FEATURE_INCOMPAT_SKINNY_METADATA |    \
     BTRFS_FEATURE_INCOMPAT_COMPRESS_LZ4 |        \
     BTRFS_FEATURE_INCOMPAT_NO_HOLES)
The 10th feature in the list. You need to a patched kernel which linux Mint apparently doesn't provide.

I'm still backing up all my files via Samba before I try anything else (takes 24h total...).
How do I find out which Kernel has support for lz4 and which one doesn't?
Can I add compatibility myself or would I need to completely change the distribution (I read somewhere ArchLinux has LZ4 support)?
I'm using the 32bit kernel from here: http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.16.1-utopic/
Linux Mint itself is still at 3.13.x
Find all posts by this user
Quote this message in a reply
3rd Sep, 2014, 10:06 PM
Post: #17
RE: btrfs and lz4
Check the fs/btrfs/ctree.h in the souce for this piece of code.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
3rd Sep, 2014, 11:34 PM
Post: #18
RE: btrfs and lz4
(3rd Sep, 2014 10:06 PM)CurlyMo Wrote:  Check the fs/btrfs/ctree.h in the souce for this piece of code.

Ok I did read up a bit more, but according to the following mail thread it doesn't look like there is going to be lz4 kernel support for brtfs anytime soon (at least outside of ARM Kernels): https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg34357.html

So that brings me to my next question: Is there a way to create a Xbian USB drive installation with zlib instead of lz4? Is there any documentation on the single steps xbian-config performs during the clone so I could do it manually step by step?
Find all posts by this user
Quote this message in a reply
4th Sep, 2014, 08:28 AM (This post was last modified: 4th Sep, 2014 08:40 AM by f1vefour.)
Post: #19
Re: RE: btrfs and lz4
I was simply following the advice of the BTRFS wiki. I did provide a link that warned of the potential danger on the page.

Thanks for coming behind me and warning against this, I'm certainly no filesystem guru.

Apparently you know more than they do, the wiki simply says:

Quote:if and only if the kernel oops in your logs has something like this in the middle of it, or you get "open ctree failed" on a newer kernel:

? replay_one_dir_item+0xb5/0xb5 [btrfs] ? walk_log_tree+0x9c/0x19d [btrfs] ? btrfs_read_fs_root_no_radix+0x169/0x1a1 [btrfs] ? btrfs_recover_log_trees+0x195/0x29c [btrfs] ? replay_one_dir_item+0xb5/0xb5 [btrfs] ? btree_read_extent_buffer_pages+0x76/0xbc [btrfs] ? open_ctree+0xff6/0x132c [btrfs]

Newer kernels output this instead:

BTRFS: failed to read log tree BTRFS: open_ctree failed

To be fair it says nothing about it being compression related.

Please read rules and do a search before you post! | FAQs | How to post log file? | Can't find an answer? Contact us.
Visit this user's website Find all posts by this user
Quote this message in a reply
4th Sep, 2014, 08:53 AM
Post: #20
RE: btrfs and lz4
I suppose they don't expect anyone trying to mount with something else beside zlib or lz0.
Anyway, I really don't want to build and maintain my own i386 Kernel with lz4 support (if anyone wants to give it a go, the instructions in the mailing list I posted tell you how). Hence I went a different route now and I just create two partitions on my 2TB drive, a small one with btrfs for RPi to run on it and a second big one with standard NTFS so I can easily copy and share all my media to other devices - that satisfies all my requirements.
Find all posts by this user
Quote this message in a reply
4th Sep, 2014, 11:45 AM (This post was last modified: 4th Sep, 2014 11:54 AM by f1vefour.)
Post: #21
RE: btrfs and lz4
Apparently it's depreciated or a third party patch, I haven't researched it.

Edit:
Okay so it's a third party patch, I have to wonder why was this was implemented knowing it would be impossible to mount the filesystem externally without a modified kernel.

Please read rules and do a search before you post! | FAQs | How to post log file? | Can't find an answer? Contact us.
Visit this user's website Find all posts by this user
Quote this message in a reply
4th Sep, 2014, 06:11 PM
Post: #22
RE: btrfs and lz4
Because lz4 has many advantages for slow computers Smile

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
5th Sep, 2014, 05:20 AM
Post: #23
Re: btrfs and lz4
I'm going to go out on a limb here and say that while the speed improvements aren't negligible I don't really notice them in my day to day XBMC usage. My 700+ movie and show library loads fairly speedy, the same as it did before BTRFS. I don't use XBMC for music so I can't compare that aspect.

Ah it's here nor there I suppose, it is in use already. I have no issues compiling the patch for my Mint system if I really need it, I have probably compiled about 3000 kernels since 1996.

Please read rules and do a search before you post! | FAQs | How to post log file? | Can't find an answer? Contact us.
Visit this user's website Find all posts by this user
Quote this message in a reply
5th Sep, 2014, 07:46 AM
Post: #24
RE: btrfs and lz4
The point is that BTRFS is slower in almost all cases than EXT4. To compensate, we use techniques like LZ4 compression.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
5th Sep, 2014, 03:17 PM (This post was last modified: 5th Sep, 2014 03:18 PM by f1vefour.)
Post: #25
Re: btrfs and lz4
I know it was implemented not for speed or space savings but for the online snapshot capabilities. This alone makes it worthwhile.

It doesn't really matter to me that much as I keep important files on an external disk. If something goes critically wrong with the system I just start from scratch. It would be nice to be able to use the snapshots to restore from a Linux PC however, this way a nonfunctional installation could be repaired with the snapshots on the disk.

Please read rules and do a search before you post! | FAQs | How to post log file? | Can't find an answer? Contact us.
Visit this user's website Find all posts by this user
Quote this message in a reply
5th Sep, 2014, 05:53 PM
Post: #26
RE: btrfs and lz4
Yes, but we created the rescue shell for that purpose Smile

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
5th Sep, 2014, 09:52 PM
Post: #27
RE: btrfs and lz4
I meant if the system becomes in a state where it won't boot, not from a filesystem error but something else.

The rescue shell is awesome though Smile

Please read rules and do a search before you post! | FAQs | How to post log file? | Can't find an answer? Contact us.
Visit this user's website Find all posts by this user
Quote this message in a reply
6th Sep, 2014, 06:26 AM
Post: #28
RE: btrfs and lz4
Then you can restore the files so the rescue shell will work. They can be downloaded from the mk01 webserver.

pilight - modular domotica solution
Visit this user's website Find all posts by this user
Quote this message in a reply
13th Sep, 2014, 12:52 AM (This post was last modified: 13th Sep, 2014 01:15 AM by mk01.)
Post: #29
RE: btrfs and lz4
the most tricky things are sometimes easiest to solve.

we provide lz4 patches - tested and build for - 3.12, 3.14, 3.15, 3.10 ...
https://github.com/xbianonpi/xbian-package-kernel/tree/master/patchesnt

then its needed to activate libkernel suppoer for lz4, lz4fast and of course appropriate compressors. all is in clasic kernel's make menuconfig.

lz4 is pre-created with XBian, once tag lz4 written (compress=lz4 used while mounting), btrfs mounter will refuse to load on kernels without the same support.

if this should be causing troubles (no possibility to compile custom kernel for other pc, or simply user doesn't want to bother with stuff) - I would recommend re-cloning to img (or disk directly) with lzo (or compress disabled) and then don't use compress=lz4 on mount.

(cmdline.txt although auto updated when needed (by xbian-update or kernel installation), there are some settings which are kept as user set them before. compress=XX is one of those.

(5th Sep, 2014 03:17 PM)f1vefour Wrote:  I know it was implemented not for speed or space savings but for the online snapshot capabilities. This alone makes it worthwhile.

It doesn't really matter to me that much as I keep important files on an external disk. If something goes critically wrong with the system I just start from scratch. It would be nice to be able to use the snapshots to restore from a Linux PC however, this way a nonfunctional installation could be repaired with the snapshots on the disk.

@f1vefour

as xbian-cloner using snapshot send | recv to create img or new part (as copy of local), the same way you can use it to send snap literally anywhere! now check this: (just for info, because snap stream is sent RAW, local & remote compat flags doesn't need to match !

Code:
btrfs send SNAP > filename
btrfs recv root < filename

file can be placed over net, on USB stick, printed out on printer and later retyped back Smile probably you know ssh can be called just to execute command on remote machine ... like this
Code:
ssh xbian@xbian echo "this is printed on vts on xbian"

then a bit more hacking would be
Code:
echo "this is sent TO xbian and echoed there" | ssh xbian@xbian cat

and
Code:
cat filename | ssh xbian@xbian "cat > openme"

finally going to
Code:
btrfs send SNAP | ssh xbian@xbian "cat > fileBACKUPofSNAP"
btrfs send SNAP | ssh xbian@xbian "btrfs recv BACKUPofSNAP"

this assumes btrfs filesystem on remote system OR just save to file as in first example - but directly to remote.

just be always aware that btrfs send has no quality and structure of mountable filesystem (can't be directly loop mounted). it is just raw CoW stream (so metadata and data as they were saved to local btrfs - like magnetic tape or even the back to '70 paper punch cards (having holes)...

so it needs to be restored to btrfs again:
Code:
cat file | btrfs recv ./

(backup home function is using that way to save /home subvolume into .gz file while uing btrfs recv later for restore (all atomic on open writeable filesystem in use).

Please read rules and do a search before you post! . FAQs . How to post log file? . Looking for answers? Please start here
Find all posts by this user
Quote this message in a reply
13th Sep, 2014, 01:49 AM
Post: #30
RE: btrfs and lz4
*wooooosh* right over my head Big Grin which button must I click? ;-)
Find all posts by this user
Quote this message in a reply
Post Reply 


Forum Jump: