gkreidl
Posts: 6345
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Speeding up youtube-dl

Tue Aug 05, 2014 12:39 pm

Update Nov. 17th:: If you have not used youtube-dl before, this is currently the best method to get a new, updated and working version, as the current version from the Raspbian repository will kill itself, if you run the upgrade command "sudo youtube-dl -U"

I'm using youtube-dl in my Minimal Kiosk Browser to extract URLs for flash based video from about 150 websites and play them with omxplayer(GUI). There are certainly other applications that do the same or you may use youtube-dl directly to download videos from youtube.com or lots of other websites.

During the last months I have noticed that youtube-dl has become slower and slower. Meanwhile it may take about 13 seconds until a video starts playing and my RPi is heavily overclocked (and I remember that it took about 7-8 seconds some time ago). It seems, that youtube-dl has become more powerful but that we have to pay a price for that with regards to speed.

Although youtube-dl is written in Python, it is distributed as a kind of fake binary: It contains a small starter and all Python files zipped together. When youtube-dl starts, it unzips the Python files and imports them.

So I wanted to test, if using youtube-dl in a more usual Python way would not make it start faster. And it does (see timing below)!

If you are using youtube-dl or plan to use it you might profit from the following guide.

You have to install git first:
sudo apt-get install git

From inside your user directory run the following commands

Code: Select all

git clone git://github.com/rg3/youtube-dl
cd /usr/bin
sudo mv youtube-dl youtube-dl-old (only if you had it installed already!)
sudo ln -s ~/youtube-dl/youtube_dl/__main__.py youtube-dl
To update youtube-dl (daily :-) ), run the following script:

Code: Select all

#!/bin/sh
file=__main__.py
maxsize=1000
if [ -d ~/youtube-dl ]
then
cd ~/youtube-dl
git pull
cd youtube_dl
actualsize=$(wc -c "$file" | cut -f 1 -d ' ')
if [ $actualsize -ge $maxsize ]
then
wget -O __main__.py https://raw.githubusercontent.com/rg3/youtube-dl/master/youtube_dl/__main__.py
chmod +x __main__.py
else
echo "main file ok"
fi
else
echo "youtube-dl from github not found"
fi
Now you can use youtube-dl as before, but you should notice that it starts faster. I've written a small test program to get the time for a simple command (--list-extractors) and for getting the URL of a youtube video:

Updated:
arm 700MHZ
original method:
9.10342597961
to list extractors
17.1481108665
to extract path

using git version:
3.49917793274
to list extractors
11.1539170742
to extract path

ARM 1000MHz / SDRAM 500 / GPU 500
original method:
5.70446586609
to list extractors
11.2198159695
to extract path

using git version:
2.11275696754
to list extractors
7.43489313126
to extract path

To open web video from Minimal Kiosk Browser I need now 3.8 seconds less on my overclocked machine. If your RPi is not overclocked, the difference is about 6 seconds!
Last edited by gkreidl on Mon Nov 17, 2014 5:23 pm, edited 6 times in total.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

gkreidl
Posts: 6345
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Speeding up youtube-dl

Mon Aug 11, 2014 9:08 am

I have updated the tutorial above, because it did only work when a version was already installed as Python distribution package (but that could not be updated). If you have a version installed as Python distribution package you must remove it before installing the github version!
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

Ivan Tham
Posts: 75
Joined: Mon Jun 24, 2013 7:06 am

Re: Speeding up youtube-dl

Sun Nov 09, 2014 8:16 am

gkreidl wrote: You have to install git first:
sudo apt-get install git

From inside your user directory run the following command
git clone git://github.com/rg3/youtube-dl
This will create a folder 'youtube-dl' in your home folder. Now run
cd /usr/bin
sudo mv youtube-dl youtube-dl-old (only if you had it installed already!)
Updated:
sudo ln -s /home/pi/youtube-dl/youtube_dl/__main__.py youtube-dl

sudo mv youtube-dl youtube-dl-old (only if you had it installed already!)
Updated:
sudo ln -s /home/pi/youtube-dl/youtube_dl/__main__.py youtube-dl

If your user name is not "pi" you have to modify the path accordingly.

You may not use
sudo youtube-dl -U
any more to update youtube-dl! Use the following instead to get the newest version
cd youtube-dl
git pull
Thanks for your explanation but the explanation is a bit too long and not newbie-friendly and adding "[code]" will improve the understanding. For example, to install youtube-dl from git:

Code: Select all

sudo apt-get -y install git
sudo apt-get -y purge youtube-dl
cd; git clone git://github.com/rg3/youtube-dl
sudo ln -s ~/youtube-dl/youtube_dl/__main__.py /usr/bin/youtube-dl
And to update:

Code: Select all

cd ~/youtube-dl
git pull

gkreidl
Posts: 6345
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Speeding up youtube-dl

Mon Nov 17, 2014 5:16 pm

They tricked me again. I recently noticed that youtube and other videos needed much longer to start than before. I checked the youtube-dl/youtube_dl directory and discovered, that the __main__.py file was a few hundred KB in size instead of just a bit more than 400 Bytes. They somehow managed to install the zipped fake binary again that is responsible for the slow loading time. I replaced it with the original script and, wow, 6 seconds instead of 12 to get a video link!

Ad this can happen again any time I wrote an update script that takes care of it. I'll put it in the 1st. post of this thread.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

otyugh
Posts: 15
Joined: Thu Jun 23, 2016 11:27 pm

Re: Speeding up youtube-dl

Fri Sep 09, 2016 11:50 pm

So !
This is a very old subject, but I think it's still is something.

$time youtube-dl -g 'https://www.youtube.com/watch?v=somevideo'
real 0m13.351s
user 0m10.730s
sys 0m0.290s

$unzip youtube-dl
$time ./youtube-dl/__main__.py -g 'https://www.youtube.com/watch?v=somevideo'
real 0m7.877s
user 0m6.170s
sys 0m0.260s

But actually, there is more to it :
I ran into this issue as well and decided to dive into the code to check it out.
I noticed that on every single execution of youtube-dl, regardless of what you're trying to do with it, it imports every single download module youtube-dl has, which in turn import their own dependencies.
By removing all imports I didn't need from this file and just leaving the 3 sites I need, I cut down the execution time from 8-9 seconds to 2 seconds.
Still a lot, even on the RasPi, but a whole lot lot less.
https://github.com/rg3/youtube-dl/issues/3029

Maybe deleting or just adding that url with "youtube.com" be only parsed by that could be a way to get an instant "youtube-dl".
I wish I knew how to hack this... Latter ! >.>

gkreidl
Posts: 6345
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Speeding up youtube-dl

Sat Sep 10, 2016 4:26 am

I don't understand why people think that youtube is the only important video website. youtube-dl supports hundreds of video websites and I wouldn't even think about removing anything.

Meanwhile the access time has been greatly reduced again in the kweb package by using the youtube-dl-server (used by omxplayerGUI, kweb, chromium-browser with kusti8's rpi-youtube extension).
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

KTWS
Posts: 75
Joined: Thu Dec 27, 2012 8:01 am

Re: Speeding up youtube-dl

Sat Sep 10, 2016 1:09 pm

If you're just wanting Youtube, try Pafy: https://github.com/mps-youtube/pafy it's faster than youtube-dl (using the non-youtube-dl dependant version).

The absolute fastest way to parse Youtube urls on the Pi is to use "uyt.pl" (http://www.ubeify.co.uk/downloads/uyt.txt), which is a stripped down heavily optimized version of Pafy. You'd need to write a wrapper for it but there's an example included in the file on how to do that.
Ubeify Desktop Media Player For Raspberry Pi Zero, 1, 2 & 3
http://www.ubeify.co.uk/

Return to “Graphics, sound and multimedia”