[SCRIPT] pyLoad Download Manager (with lightweight webserver, Click'n'Load, ...)

[josch]

Hey folks,

like here I want to give you another script for another download manager: pyLoad

Why another download manager?:

As it seems that "DownloadDaemon" doesn't sacrifice all users and a friend of mine asked me for
help setting up pyLoad, I decided to do the work only once and spread it with the world.

In the past I wasn't that happy with pyLoad because of the buggy client GUI and the damn slow
webinterface when running on RPi. But I did give it a new try and setted it up with the super fast
lightweight webserver called "bjoern" and deactivated the non-webinterface remote control.
Now it's running really well and there is no problem downloading and using the webinterface
even during the RPi is playing HD video files. Plus you get a lot of extras in comparison to DD,
like Click'n'Load, a bunch of auto updating plugins an so on....

I won't say more about pyLoad, just check: http://pyload.org/


What the script does:

• installing all needed dependencies for the whole installing process
• installing lightweight webserver "bjoern" via pip (https://github.com/jonashaag/bjoern)
• installing package for automatic rar archives extraction
• installing package for Click'n'Load
• installing packages for captcha recognition
• installing pyLoad v0.4.9
• calling pyLoad configuration as user "xbian"
• fixing pyLoad source to work with ALL Click'n'Load buttons
• modifying pyLoad startscript (init.d) to work as expected and to run pyLoad as user "xbian"
• adding pyLoad to xbian-config service manager, so that you can start/autostart the service via xbian-config
• optional removing all dependencies afterwards that were needed during installation
  

How to install:

  xbian_installPyLoad.zip (Size: 1.62 KB / Downloads: 345)

Download xbian_installPyLoad.zip and unpack it on to your Raspberry with Xbian running.
Make sure that your RPi is connected to the internet, as the script needs to download some packages/files.

Then run following commands via shell in the same dir where the script file is:

Terminal

chmod +x xbian_installPyLoad.sh
sudo ./xbian_installPyLoad.sh

Thats it, the script will start to guide you through the install process. If you have questions, just ask.


How to use:

Start or autostart the 'pyload' service via xbian-config. Afterwards you can just visit http://<your RPi's IP>:8000
and use your pyLoad download manager. If you've changed the standard port (8000), you have to modify the address
of course. After the first start you may have to restart pyLoad to complete a lot of plugin updates. But there will be
a notice in the webinterface and you can do the restart via the webinterface too.

If you want to make use of the Click'n'Load feature, you have to fordward the specific port (9666) to the webinterface.
So, just forward: 127.0.0.1:9666 to <your RPi's IP>:8000

For windows users here are two small batch files. One will setup the port forward, the other one will revert this.
At first you have to edit pyLoad_portforward.bat with your texteditor and place in your RPi's IP! Then run it as admin.
The port forward will stay even after rebooting until you run the pyLoad_portforward-undo.bat.

  pyLoad_portforward (windows).zip (Size: 518 bytes / Downloads: 61)


Have fun!

love and peace
josch

[tribal124]

Hi, i have one question.
I already use pyload on my Raspberry but sometimes the .rar files downloaded aren't automatically extracted even though i compile the password record in the dialog Window, and i have to extract they manually.
The extract package included in your script is the same plugin called ExtractArchives included into 0.49 pyload?
However have you some recommendations in order to correctly extract these files?

[josch]

(6th Feb, 2014 10:25 AM)tribal124 Wrote:  The extract package included in your script is the same plugin called ExtractArchives included into 0.49 pyload?

Yes! It isn't a different plugin. My script will just install the package 'unrar' via apt-get so that the ExtractArchives plugin can make use of it. (It's a needed dependency.)

(6th Feb, 2014 10:25 AM)tribal124 Wrote:  However have you some recommendations in order to correctly extract these files?

Do you see anything in the pyLoad logs when the archives don't get extracted? Or do you recognize any pattern when it does happen?


Of course you could always call unrar also manually:

Terminal

unrar x -p<YOURPW> archive.part01.rar

There is no space between -p and your password. If there is no password, drop the whole -p parameter.

[tribal124]

(6th Feb, 2014 09:50 PM)josch Wrote:  No, DownloadDaemon has it's own code to handle rar archives.

i don't understand, what do you mean with DownloadDaemon? It isn't this script intended for PyLoad?

However can you tell me where i can locate the log.txt file into my Raspberry to check what goes wrong during extraction?

[josch]

(7th Feb, 2014 12:30 AM)tribal124 Wrote:  

(6th Feb, 2014 09:50 PM)josch Wrote:  No, DownloadDaemon has it's own code to handle rar archives.

i don't understand, what do you mean with DownloadDaemon? It isn't this script intended for PyLoad?

However can you tell me where i can locate the log.txt file into my Raspberry to check what goes wrong during extraction?

Sorry dude! My DownloadDaemon thread was shown below when I was answering thats why I got distracted
and thought you were posting in the other thread. But it was just listed under "Related Threads". My fault,
seems that I was just a bit sleepy.....

So please forget this answer, it's nonsense! I've updated my post - check it again.

(7th Feb, 2014 12:30 AM)tribal124 Wrote:  However can you tell me where i can locate the log.txt file into my Raspberry to check what goes wrong during extraction?

The log files are stored at /home/xbian/.pyload/Logs
You can also view the latest logs directly via pyLoad Webinterface. Right corner, next to "Settings".

[Neutrino1986]

Awesome script Josch. Thanks for that!
Just a hint for all people running into problems when updating python2.7 while executing the script:
Login to your raspberry via SSH and use the Xbian updater to run all updates except those related to python.
Then restart your rapsberry, login via SSH again and execute the following commands:

Terminal

sed -i '/_hashlib.so/d' /var/lib/dpkg/info/python2.7.list
sed -i '/_ssl.so/d' /var/lib/dpkg/info/python2.7.list

After that you can use the Xbian updater to also update python.
See wiki for further infos regarding python update.
I highly reccomend using the Xbian updater, since I tried doing the updates manually first and ended up with XMBC not laoding correctly.

Afterwards the script works like charm!

[pix3l]

Hey,

I'm using pyLoad and I'm very satisfied with it.
Now I'm wondering if there is a better way to sort the files after the download?

What I want to do:
I'm using a 1TB external Harddrive to store my movies and tv-shows. But after the download I have to connect via ssh to my Xbian and move the files to the correct folder myself. I also have to delete the rar files by myself.
Does anybody have an idea how to get this done a bit more easier? e.g. via Webinterface or a script or by selecting a extract folder for every download separately and deleting the rar-files after download?

Thanks

[Neutrino1986]

(9th May, 2014 08:01 PM)pix3l Wrote:  Hey,

I'm using pyLoad and I'm very satisfied with it.
Now I'm wondering if there is a better way to sort the files after the download?

What I want to do:
I'm using a 1TB external Harddrive to store my movies and tv-shows. But after the download I have to connect via ssh to my Xbian and move the files to the correct folder myself. I also have to delete the rar files by myself.
Does anybody have an idea how to get this done a bit more easier? e.g. via Webinterface or a script or by selecting a extract folder for every download separately and deleting the rar-files after download?

Thanks

You can configure the extraction path and automatically delete the archives in the webinterface.
I'm not using the webinterface in English so I can't tell you the exact location of the settings:
Settings -> Addons/Plugins (?) -> ExtractArchive

Hope this helps.

[josch]

I noticed that there is a missing dependency for spidermonkey-bin (js engine) in the current repository.
Without manual installing libnspr4 too, the js engine will not run and therefore no click'n'load will be available.

If you had already run the install script with the current repository and click'n'load isn't working for you, run:

Terminal

sudo apt-get install libnspr4

to deliver the needed dependency for spidermonkey. Everything should work fine then.

I've updated the install script too.

[josch]

Just tested the script on the brand-new RC3 and it's still working without need of changes.
I've only updated the file to add RC3 to the tested versions comment.

[med]

hi here, thanks for the sricpt but i have some problems/question:

- When I download the file (directly from this forum or from a hoster*) and try to unzip it, I always got the error: "unzip: invalid zip magic 4F44213C"
(I installed unrar and unzip with sudo apt-get install)

I managed to get the file to my Xbian-System already unzip (uploaded a unzip version to a hoster* and the downloaded it with my xbian) but when i start the sricpt this error comes up:

Terminal

xbian@xbian ~/Downloads $ ./xbian_installPyLoad.sh
./xbian_installPyLoad.sh: line 1: !DOCTYPE: No such file or directory
./xbian_installPyLoad.sh: line 2: syntax error near unexpected token `newline'
./xbian_installPyLoad.sh: line 2: ` "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">'

What I'm doing wrong? Thanks for the help!
Cheers

* I downloaded the file to my pc and uploaded it to a hoster so i go a shorter link

[josch]

(19th Oct, 2014 09:19 PM)med Wrote:  

Terminal

xbian@xbian ~/Downloads $ ./xbian_installPyLoad.sh
./xbian_installPyLoad.sh: line 1: !DOCTYPE: No such file or directory
./xbian_installPyLoad.sh: line 2: syntax error near unexpected token `newline'
./xbian_installPyLoad.sh: line 2: ` "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">'

This isn't my script, it doesn't contain such things. Seems that you didn't download the script correctly.
Please check the content of your xbian_installPyLoad.sh file. It should contain the following code:

Code:

#!/bin/bash
set -e

if [ "$UID" -ne 0 ]
then
  echo -e "\033[01;37;44m Error: This script must be run as root. (sudo) \033[0m"
  exit
fi

if [ ! -f /usr/local/include/xbian-config/config/services ]
then
  echo -e "\033[01;37;44m Error: Can't find xbian-config services file. You should run this script only on XBian OS! \033[0m"
  exit
fi

userexists=false
getent passwd xbian >/dev/null 2>&1 && userexists=true
if ! $userexists
then
  echo -e "\033[01;37;44m Error: There is no user 'xbian'. \033[0m"
  exit
fi

echo -e "\033[01;37;44m This script will download and install pyLoad v0.4.9 (tested on XBian 1.0RC2+RC3) \033[0m"
echo ""
echo -e "\033[01;37;44m Following additional features will be available: \033[0m"
echo -e "\033[37;44m - super fast lightweight webserver 'bjoern'  \033[0m"
echo -e "\033[37;44m - working with Click'n'Load \033[0m"
echo -e "\033[37;44m - captcha recognition  \033[0m"
echo -e "\033[37;44m - automatic unrar archives \033[0m"
echo ""
echo -e "\033[01;37;44m Do you want to continue? [y/n] \033[0m"
read -n1 answer
answer=${answer:-n}
echo ""
if [ $answer != y ]
then
  exit
fi

echo -e "\033[37;44m Installing dependencies ... \033[0m"
apt-get install python-support python-pycurl python-crypto python2.7-dev libev-dev python-pip gcc spidermonkey-bin libnspr4 tesseract-ocr python-imaging unrar

echo -e "\033[37;44m Installing lightweight webserver 'bjoern' ... \033[0m"
pip install bjoern

echo -e "\033[37;44m Downloading pyload-cli-v0.4.9-all.deb ... \033[0m"
cd /tmp
wget http://download.pyload.org/pyload-cli-v0.4.9-all.deb

echo -e "\033[37;44m Installing pyLoad ... \033[0m"
dpkg -i pyload-cli-v0.4.9-all.deb

echo -e "\033[37;44m Calling pyLoad configuration as user 'xbian' ... \033[0m"
echo ""
echo -e "\033[01;37;44m - DON'T CHANGE YOUR CONFIG PATH!!!! (/home/xbian/.pyload) \033[0m"
echo -e "\033[01;37;44m - DISABLE REMOTE ACCESS IF YOU DON'T USE IT \033[0m"
echo -e "\033[01;37;44m - SET THE WEBINTERFACE LISTEN ADDRESS TO YOUR RASPBERRY'S IP \033[0m"
echo -e "\033[01;37;44m - DON'T FORGET TO SET 'lightweight' AS YOUR WEBSERVER!! \033[0m"
echo ""
su - xbian -c "/usr/share/pyload/pyLoadCore.py -s"

echo -e "\033[37;44m Fixing pyLoad source to work with all Click'n'Load buttons ... \033[0m"
sed -i "s@or request.environ.get('HTTP_HOST','0') == '127.0.0.1:9666':@or request.environ.get('HTTP_HOST','0') in ('127.0.0.1:9666', 'localhost:9666'):@" /usr/share/pyload/module/web/cnl_app.py

echo -e "\033[37;44m Modifying pyLoad start script to work as expected ... \033[0m"
sed -i 's@~/.pyload/pyload.pid@/home/xbian/.pyload/pyload.pid@' /etc/init.d/pyload
sed -i 's@$DAEMON --daemon@su - xbian -c "$DAEMON --configdir=/home/xbian/.pyload --daemon"@' /etc/init.d/pyload
sed -i 's@$DAEMON -q@su - xbian -c "$DAEMON -q"@' /etc/init.d/pyload

echo -e "\033[37;44m Adding pyLoad to xbian services ... \033[0m"
sh -c "echo 'pyload:/usr/bin/pyLoadCore' >> /usr/local/include/xbian-config/config/services"

echo ""
echo ""
echo -e "\033[01;37;44m Do you want to clean up some packages that were needed during installation? [y/n] \033[0m"
echo -e "\033[37;44m (You will be asked for each package and will see the changes before commiting.) \033[0m"
read -n1 answer
answer=${answer:-n}
echo ""
if [ $answer != y ]
then
  echo -e "\033[01;37;44m Finished. You can now start/autostart the service 'pyload' via xbian-config. \033[0m"
  exit
fi

echo ""
echo -e "\033[01;37;44m Do you want to remove 'python2.7-dev', 'python-pip', 'gcc' and all of their dependencies at once? [y/n] \033[0m"
echo -e "\033[37;44m (Choose no if you like to be asked separately for each package.) \033[0m"
read -n1 answer
answer=${answer:-n}
echo ""
if [ $answer = y ]
then
  apt-get purge --auto-remove python2.7-dev python-pip python2.6-minimal gcc-4.6-base libmpfr4
else
    echo ""
    echo -e "\033[37;44m Remove 'python2.7-dev' and it's dependencies? [y/n] \033[0m"
    read -n1 answer
    answer=${answer:-n}
    echo ""
    if [ $answer = y ]
    then
      apt-get purge --auto-remove python2.7-dev
    fi

    echo ""
    echo -e "\033[37;44m Remove 'python-pip' and it's dependencies? [y/n] \033[0m"
    read -n1 answer
    answer=${answer:-n}
    echo ""
    if [ $answer = y ]
    then
      apt-get purge --auto-remove python-pip python2.6-minimal
    fi

    echo ""
    echo -e "\033[37;44m Remove 'gcc' and it's dependencies? [y/n] \033[0m"
    read -n1 answer
    answer=${answer:-n}
    echo ""
    if [ $answer = y ]
    then
      apt-get purge --auto-remove gcc-4.6-base libmpfr4
    fi
fi

echo ""
echo -e "\033[01;37;44m Finished. You can now start/autostart the service 'pyload' via xbian-config. \033[0m"

You can either put this code into your file manually or download the zip file from my first post again and put it on your RPi. Then do:

Terminal

unzip xbian_installPyLoad.zip
chmod +x xbian_installPyLoad.sh
sudo ./xbian_installPyLoad.sh

IMPORTANT: DON'T USE JUST WGET TO DOWNLOAD THE FILE AS YOU WILL GET AN HTML FILE. DOWNLOAD IT ON PC AND COPY TO YOUR RPi.


Good luck!

[med]

hi,
thanks for the fast answer! My mistake was to use wget! . My next problem seems to be a bigger one, because there is something wrong with one package the sricpt needs:

Terminal

Err http://xbian.brantje.com/ stable/main linux-libc-dev armhf 3.15.8
Could not resolve 'xbian.brantje.com'
Err http://xbian.brantje.com/ stable/main unrar armhf 1:4.1.4-1
Could not resolve 'xbian.brantje.com'
Err http://mirrordirector.raspbian.org/raspbian/ wheezy/main spidermonkey-bin ar mhf 24.7.0esr-1~deb7u1+rpi1
404 Not Found
Failed to fetch http://xbian.brantje.com/pool/stable/main/l/linux-libc-dev/linux -libc-dev_3.15.8_armhf.deb Could not resolve 'xbian.brantje.com'
Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/i/icewease l/spidermonkey-bin_24.7.0esr-1~deb7u1+rpi1_armhf.deb 404 Not Found
Failed to fetch http://xbian.brantje.com/pool/stable/main/u/unrar/unrar_4.1.4-1_ armhf.deb Could not resolve 'xbian.brantje.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-mis sing?

I found a Post on this forum where they fixed the probelm in generall:
http://forum.xbian.org/thread-2516-post-24885.html#pid24885

But after following the advises there and retry the script this error occurs:

Terminal

Do you want to continue? [y/n]
y
Installing dependencies ...
Reading package lists... Done
Building dependency tree
Reading state information... Done
Package spidermonkey-bin is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'spidermonkey-bin' has no installation candidate

I think here have to be done some changes to the script
Thanks for the help!

[josch]

Seems that the spidermonkey package isn't available right now via the repository. Normally it is consistently available.
Perhaps this could be related to the other repository troubles but I do not know. I will wait with editing the script as
this may be just a temporary issue. (Tell me if not. )

The spindermonkey-bin package is necessary for the Click'n'Load feature. So please install it manually if you need this one.

You can use the install script also without installing spidermonkey, all you have to do is removing spidermonkey-bin
from line 42 or simply use the attached script.

[Neutrino1986]

I noticed some strange behavoir related to pyload auto-start and the pyload entry within xbian services overview:

1) PyLoad does not start automatically although added to autostart
2) Whenever I try to start it manually via xbian-config the interface returns to manin menu instead of loading the updated service overview. Only after leaving main menu of xbian-config the service overview is reloaded (withouth being prompted at that point) and finally starts pyload. I posted the problem here, but I relaized that it is only occuring when starting pyload. Starting VNC from service list is working fine.