Forum
Development Guide? - Printable Version

+- Forum (http://forum.xbian.org)
+-- Forum: Software (/forum-6.html)
+--- Forum: Testing & Experimental (/forum-21.html)
+--- Thread: Development Guide? (/thread-1349.html)

Development Guide? - A_Porcupine - 13th Sep, 2013 08:32 AM

Hey guys,

Hopefully not posting in the wrong section! Smile

Just wondering if there's a development guide currently or even just a "Build from scratch" document for Beta 1.1?

Thanks!

RE: Development Guide? - Koenkk - 13th Sep, 2013 04:41 PM

Well basicaly XBian = raspbian + installed packages from git

RE: Development Guide? - mk01 - 1st Dec, 2013 06:25 PM

(13th Sep, 2013 08:32 AM)A_Porcupine Wrote:  Just wondering if there's a development guide currently or even just a "Build from scratch" document for Beta 1.1?

you can install raspbian from scratch by using 'debootstrap' method. you can use minbase variant with additional packages as ifupdown, upstart, netbase. this will prepare base filesystem structure and unpack .deb packages. this stage takes a looong time on RPI, you can do it from your regular PC running debian based linux distribution (Debian, Ubuntu …). you would run command

Code:
debootstrap --arch=armhf --variant=minbase --include=ifupdown,netbase,iproute,busybox,openssh-server --foreign jessie ./ http://archive.raspbian.org/raspbian

(this will grab jessie packages (successor to Wheezy). you can change jessie for "wheezy", or even "sid". sid will come after jessie)

then you need actually to do the final install step for prepared packages. so you copy all content of your folder with debootstrapped rootfs to you RPI. if you are on Beta2, you don't need additional SD card, just create sub-volume on your existing install
Code:
sudo -i
btrfs-auto-snapshot createvol jessie
btrfs-auto-snapshot createvol jessie/@

copy files over to this jessie/@ subvolume
Code:
sudo -i
mkdir /mnt/jessie
mount -o subvol=jessie/@ /dev/root /mnt/jessie
rsync -avx /my/folder/where/i/did/the/first/part/with/debootstrap /mnt/jessie/

then you chroot into this new rootfs (in half prepared stage) and finish the debootstrap install
Code:
sudo -i
chroot /mnt/jessie /debootstrap/debootstrap --second-stage

after this step, you can boot into your new clean and fresh raspbian/debian jessie installation. just change rootfsopts=subvol=root/@ in cmdline.txt to rootfsopts=subvol=jessie/@ and reboot.

then set configuration for eth0 in /etc/network/interfaces, bring up network. edit sources.list of apt by adding xbian repository and try to install packages.

the biggest issue there is not how, but install the proper packages after (to make xbmc and others run successfully without issues). last time I was doing this I tried to update all the dependencies for XBian packages to include even the core packages to allow this kind of install, but it's possible some are still missing.

so you take one day and enjoy it by testing functions (specially from xbian-package-config-shell) one by one and in case of not working, checking the code what utility is used there and is probably missing in your new system. what I remember is that the steps I wrote here will produce cca 200Mb large rootfs - looking good, having what we need. but then you starts installing XBian packages. xbian-package-config-shell is quite nasty as few dependencies will be expanded to almost 150Mb of additional packages after installation.

maybe it would be interesting sub-project to recheck/rewrite xbian-config by using as minimal toolset as possible. to create list of utils we strictly need for other packages (like perl, python, grep, sh …) and then to try to fit xbian-config to those utils.

don't forget to document missing steps/packages to allow XBian deb packages dependencies update to allow easy install from scratch. later net-installer can be easily written to provide installation over internet only by providing .img file of /boot partition - having the mini-system debootstrap capable on initram filesystem (15-20mb).

RE: Development Guide? - mk01 - 2nd Dec, 2013 01:09 AM

i just installed the packages to jessie successfully. after --second-stage finishes, exit the chroot and bind /proc /sys /dev /dev/pts /boot to the jessie dir ( /mnt/jessie in case of this guide ).
Code:
mount -o bind /dev /mnt/jessie/dev
the same for /dev/pts, /sys, /proc, /run, /boot

then chroot into it calling busybox sh applet (to have vi, less and others available).
Code:
chroot /mnt/jessie /bin/busybox sh

edit sources.list and add "deb http://xbian.brantje.com devel main", "deb http://xbian.brantje.com staging main" and "deb http://xbian.brantje.com stable main". run apt-get update and install xbian-package-kernel. this will install also initramfs tools, btrfs-tools and splash.

you have also to create xbian user and deliver the base /home/xbian files (I created deb package called xbian-package-xbianhome). this will do this for you - adduser xbian, unpack base files.

then you can reboot (change cmdline.txt as described earlier). after reboot jessie will boot. proceed to apt-get install xbian-update. reboot, xbmc will start and you have clean and fresh raspbian with XBian enhancements on it (XBian Beta2 actually).
btw:

you will probably need to install perl-modules, locales (apt-get install perl-modules, locales). later we can add this as dependency to xbian-update package.

RE: Development Guide? - Killerbee - 2nd Dec, 2013 11:55 PM

Thank you for your explanation. I definitely gonna try this.
One question:
What to do when we have a working system?
How do we create an image out of the jessie/@ subvolume?

KB

RE: Development Guide? - mk01 - 3rd Dec, 2013 01:42 AM

Beta2 contains internal tools providing this functionality. It is accessible via xbian-config in simplified form or by command line with more options:

Code:
sudo btrfs-auto-snapshot xbiancopy --img --size 500M --fs-name xbianimage /dev/root /mnt/ext.disk/xbian.img

this will create img file with two paritions, first 32mb in size as boot, containing firmware and kernel files with default configs and adapted cmdline.txt, second partition containing necessary sub volumes and default /@ snapshots
also you can run
Code:
sudo all-clean

this is helper tool removing all caches (apt, dpkg), tmp files, will reset xbmc addon settings, remove ssb keys - they get regenerated during next boot etc.

RE: Development Guide? - mk01 - 3rd Dec, 2013 06:36 AM

what is sad on this solution creating new package from scratch is that you still needs already running RPI system (at least). and if you don't want to get old doing the first half, then also faster machine running deb like distro. (just to install debootstrap and run it)

RE: Development Guide? - Killerbee - 7th Dec, 2013 11:43 PM

Just for my own backup Sleepy
to rsynch a directory from a remote (ubuntu) machine use this rsync command
Terminal

rsync -chavzP --stats user@remote.host:/path/to/debootstrap_stage1/ /mnt/jesse

Add name server before running apt-get
vi /etc/resolv.conf
change nameserver to 8.8.8.8

Add the public keys of the repos
wget -qO - http://xbian.brantje.com/xbian.gpg.key | apt-key add -

Terminal

apt-get update
apt-get install xbian-package-kernel
Reading package lists... Done
Building dependency tree... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
xbian-package-kernel : Depends: rng-tools but it is not installable
Depends: xbian-package-initramfs-tools (>= 1.0-2.3) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

please add rng-tools to the first stage of bootdebstrap.

better is to add deb http://mirrordirector.raspbian.org/raspbian/ wheezy main contrib non-free rpi to the /etc/apt/sources.list file
the file should look like this:
Code:
deb http://mirrordirector.raspbian.org/raspbian/ wheezy main contrib non-free rpi
deb http://xbian.brantje.com devel main
deb http://xbian.brantje.com staging main
deb http://xbian.brantje.com stable main

Code:
mount -o bind /dev /mnt/jessie/dev
mount -o bind /dev/pts /mnt/jessie/dev/pts
mount -o bind /sys /mnt/jessie/sys
mount -o bind /proc /mnt/jessie/proc
mount -o bind /run /mnt/jessie/run
mount -o bind /boot /mnt/jessie/boot
chroot /mnt/jessie /bin/busybox sh

apt-get install xbian-package-kernel shows this:
Terminal

/ # apt-get install xbian-package-kernel
Reading package lists... Done
Building dependency tree... Done
The following extra packages will be installed:
btrfs-tools libfreetype6 libjpeg8 liblzo2-2 rng-tools triggerhappy xbian-package-firmware xbian-package-initramfs-tools xbian-package-splash
Suggested packages:
xbian-package-kernel-headers
The following NEW packages will be installed:
btrfs-tools libfreetype6 libjpeg8 liblzo2-2 rng-tools triggerhappy xbian-package-firmware xbian-package-initramfs-tools xbian-package-kernel xbian-package-splash
0 upgraded, 10 newly installed, 0 to remove and 0 not upgraded.
Need to get 20.8 MB of archives.
After this operation, 48.2 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://xbian.brantje.com/ devel/main btrfs-tools armhf 1.0-xbian2a [1237 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ wheezy/main liblzo2-2 armhf 2.06-1 [56.0 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ wheezy/main rng-tools armhf 2-unofficial-mt.14-1 [48.7 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libfreetype6 armhf 2.4.9-1.1 [401 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libjpeg8 armhf 8d-1 [132 kB]
Get:6 http://mirrordirector.raspbian.org/raspbian/ wheezy/main triggerhappy armhf 0.3.4-2 [31.2 kB]
Get:7 http://xbian.brantje.com/ devel/main xbian-package-firmware armhf 1.4-9 [3182 kB]
Get:8 http://xbian.brantje.com/ staging/main xbian-package-splash armhf 1.0-1.1a [456 kB]
Get:9 http://xbian.brantje.com/ staging/main xbian-package-initramfs-tools armhf 1.3.2-1c [15.4 kB]
Get:10 http://xbian.brantje.com/ staging/main xbian-package-kernel armhf 1.3-6.2 [15.2 MB]
Fetched 20.8 MB in 20s (1010 kB/s)
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = "en_US.UTF-8",
LC_CTYPE = "C",
LC_TYPE = "C",
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package liblzo2-2:armhf.
(Reading database ... 7565 files and directories currently installed.)
Unpacking liblzo2-2:armhf (from .../liblzo2-2_2.06-1_armhf.deb) ...
Setting up liblzo2-2:armhf (2.06-1) ...
Selecting previously unselected package btrfs-tools.
(Reading database ... 7574 files and directories currently installed.)
Unpacking btrfs-tools (from .../btrfs-tools_1.0-xbian2a_armhf.deb) ...
Setting up btrfs-tools (1.0-xbian2a) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
Selecting previously unselected package xbian-package-firmware.
(Reading database ... 7635 files and directories currently installed.)
Unpacking xbian-package-firmware (from .../xbian-package-firmware_1.4-9_armhf.deb) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
findmnt: can't read /proc/mounts: No such file or directory
Setting up xbian-package-firmware (1.4-9) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Selecting previously unselected package rng-tools.
(Reading database ... 7850 files and directories currently installed.)
Unpacking rng-tools (from .../rng-tools_2-unofficial-mt.14-1_armhf.deb) ...
Setting up rng-tools (2-unofficial-mt.14-1) ...
Trying to create /dev/hwrng device inode...
/run/udev or .udevdb or .udev presence implies active udev. Aborting MAKEDEV invocation.
Starting Hardware RNG entropy gatherer daemon: (Hardware RNG device inode not found)
/etc/init.d/rng-tools: Cannot find a hardware RNG device to use.
invoke-rc.d: initscript rng-tools, action "start" failed.
Selecting previously unselected package libfreetype6:armhf.
(Reading database ... 7868 files and directories currently installed.)
Unpacking libfreetype6:armhf (from .../libfreetype6_2.4.9-1.1_armhf.deb) ...
Setting up libfreetype6:armhf (2.4.9-1.1) ...
Selecting previously unselected package libjpeg8:armhf.
(Reading database ... 7880 files and directories currently installed.)
Unpacking libjpeg8:armhf (from .../libjpeg8_8d-1_armhf.deb) ...
Setting up libjpeg8:armhf (8d-1) ...
Selecting previously unselected package xbian-package-splash.
(Reading database ... 7887 files and directories currently installed.)
Unpacking xbian-package-splash (from .../xbian-package-splash_1.0-1.1a_armhf.deb) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Setting up xbian-package-splash (1.0-1.1a) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Selecting previously unselected package triggerhappy.
(Reading database ... 7905 files and directories currently installed.)
Unpacking triggerhappy (from .../triggerhappy_0.3.4-2_armhf.deb) ...
Setting up triggerhappy (0.3.4-2) ...
Selecting previously unselected package xbian-package-initramfs-tools.
(Reading database ... 7919 files and directories currently installed.)
Unpacking xbian-package-initramfs-tools (from .../xbian-package-initramfs-tools_1.3.2-1c_armhf.deb) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Selecting previously unselected package xbian-package-kernel.
Unpacking xbian-package-kernel (from .../xbian-package-kernel_1.3-6.2_armhf.deb) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
findmnt: can't read /proc/mounts: No such file or directory
Setting up xbian-package-kernel (1.3-6.2) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
Setting up xbian-package-initramfs-tools (1.3.2-1c) ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
/var/lib/dpkg/info/xbian-package-initramfs-tools.postinst: line 7: /home/xbian/xbian-initramfs-update.log: No such file or directory
/var/lib/dpkg/info/xbian-package-initramfs-tools.postinst: line 9: /home/xbian/xbian-initramfs-update.log: No such file or directory
/var/lib/dpkg/info/xbian-package-initramfs-tools.postinst: line 17: /home/xbian/xbian-initramfs-update.log: No such file or directory
Processing triggers for xbian-package-initramfs-tools ...
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
findmnt: can't read /proc/mounts: No such file or directory

Installation of the kernel package went well Smile

I installed the next additional packages:
Terminal

apt-get install net-tools dhcpcd bash-completion vim apt-utils perl-modules locales apt-utils xbian-package-xbianhome

changing cmdline.txt (root for jessie)

reboot

Login as root and run:

Terminal
apt-get install xbian-update

All went well.
Auto reboot but XBMC didnot start (splash starting XBMC........)

Try to start xbmc manually terminal shows
Terminal

./xbmc.bin --standalone -fs --lircdev /run/lirc/lircd
Could not init logging classes. Permission errors on ~/.xbmc (/home/xbian/.xbmc/temp/)
ERROR: Unable to create application. Exiting

To solve this I used this command as root:
Terminal
chown -R xbian:xbian /home/xbian

Try again

Terminal
/usr/local/lib/xbmc/xbmc.bin ./xbmc.bin --standalone -fs --lircdev /run/lirc/lircd

xbmc-loaded start/running, process 9229
error reading file
calib_x_d=1280;
calib_x_fact=-1.000000;
calib_y_d=720;
calib_y_fact=-1.000000;
swap_axes=:0;
click_confines=7;error reading file
calib_x_d=1280;
calib_x_fact=-1.000000;
calib_y_d=720;
calib_y_fact=-1.000000;
swap_axes=:0;
click_confines=7;

A few more things I had to do:
Change the password of xbian user (changed it to raspberry)
Also I had to add xbian as sudousers

I don't now if this is the correct way but I added this line to the /etc/sudoers file
Code:
xbian   ALL=(ALL:ALL) ALL

reboot
WHOHOOOOO
xbmc starts and I have a working system Big Grin

One issue I have is with localisation of xbian-config-shell
Probably I need a nl_UTF8 translation file Tongue

[edit]
I had to run dpkg-reconfigure locales.
[/edit]
KB

RE: Development Guide? - Killerbee - 10th Dec, 2013 01:16 AM

@mk01

I have successfully made my first xbian image using debootstap Smile

My next challenge is to modify the script of Klaus Maria Pfeiffer found here to make it possible to create a xbian image automatically directly from Ubuntu.

KB

RE: Development Guide? - Killerbee - 11th Dec, 2013 08:17 AM

@mk01

I have successfully build a xbian image using your debootstrap method (stage 1 and stage 2) on my ubuntu laptop.
Installing the kernel and xbian-update I did on the pi.
So it's not fully automated yet but I works so far.

I noticed that the file system was re sized properly but not converted.
I manually changes some files (fstab and cmdline.txt) to keep the ext4 file system.

I was surprised that the system worked (I had to disable apt snapshots)! And surprisingly faster/more responsive than with btrfs!?!
The system is more responsive and updating of my movie library is way faster (using moviemeter plugin)!

I will further investigate this and do some more test.

KB

RE: Development Guide? - mk01 - 13th Dec, 2013 10:02 AM

@Killerbee,

yes the system works - don't be surprised. you will get some minor error msgs during updates though.

in general, ext4 is faster. btrfs wasn't picked up for speed.
(and yes, specially lib updates will struggle)

((if interested https://github.com/xbianonpi/xbian-package-initramfs/issues/1#issuecomment-15011914 ))
(10th Dec, 2013 01:16 AM)Killerbee Wrote:  My next challenge is to modify the script of Klaus Maria Pfeiffer found here to make it possible to create a xbian image automatically directly from Ubuntu.

this is still a bit complication - you need ubuntu (or debian). if you wan't to make this install option go real, do it around some kind of minimized linux image with debootstrap running on it able to perform this steps to the card it was booted from. it could run like this:

1 - 1st partition -> 64mb in size holding kernel and initramfs
2 - initramfs booted will prepare 2nd partition and run 1st stage of debootstrap (downloading packages according of users choice)
3 - reboot into 2nd partition for 2nd stage of debootstrap.

it's the same with the small difference that you ubuntu is the kernel.img and initramfs on fat partition to boot.
just to see it how it works you can copy 1:1 your fresh raspbian install, replace /var with tmpfs, remove all packages providing support for locales, man, info, timezones, *i18n*, …you could end around 100mb. which will compress under 64mb. its not effective, but it is quick to build.
or take XBians initramfs. just tried with debootstrap binary and libs with gzipped size of 7mb. you have network, fs and part utils. you don't need more.
@Killerbee

btw: very nice, I'm proud of you Smile

RE: Development Guide? - Killerbee - 16th Dec, 2013 08:52 PM

(13th Dec, 2013 10:02 AM)mk01 Wrote:  or take XBians initramfs. just tried with debootstrap binary and libs with gzipped size of 7mb. you have network, fs and part utils. you don't need more.
@Killerbee
btw: very nice, I'm proud of you Smile
@mk01
Thank you very much.
Could you please send me your specially prepared initramfs file (the one with the debootstrap binary)?
Another question:
Do you have a suggestion where to put the startupscript who will prepare the second partition and run debootstrap stage one?
KB

RE: Development Guide? - mk01 - 21st Dec, 2013 06:55 AM

@Killerbee

first try manually. boot into initramfs (keyboard hold SHIFT or "rescue" in cmdline) and try calling it manually.

and the specially prepared initramfs file is no big deal. just edit /etc/xbian-initramfs/update-initramfs.sh and anywhere you see "copy_with_libs …." put "copy_with_libs /usr/sbin/debootstrap" . then run "sudo xbian-update-initramfs". it will update initramfs by adding debootstrap binary and all linked libraries.

finally - startupscript can be located on the debootstrap minimal boot disk (initramfs) or can be always up-to-date downloaded from remote site ...