Python scripts are ZIP files?

[Joe Schmoe]

This has nothing to do with the R Pi.

But it does have to do with Python - a language I know next to nothing about.
But I am (and have been for a year or so) a happy user of on specific Python script - called youtube-dl.py (Google it!).

A very nice feature of the youtube-dl.py script is its "auto-update" feature - that is, if you run it (on an Internet connected machine) with the -U option, it will "contact the mother ship" and figure out if there is a new version available, and, if so, download it and replace your copy on disk with the updated version. So, I run it with -U every so often (like once a month), and usually it finds and installs a new version. Usually the file is about 160K in size.

However, today, I did this, and the resulting file was only about 40K size. This struck me as odd, although the programs seemed (in very limited testing!) to work OK. Further inspection showed that the file was no longer a script file. In fact, it looks like this:

#!//usr/local/bin/env python
PK{lots and lots of binary glop}

Aha!, says I. That PK makes it look like a ZIP file. So, I pop the script up in an editor, remove the first line, and save the result. Sure enough, "file" says it is a now a ZIP file. See:

$ unzip -v /tmp/glod
Archive: /tmp/glod
Length Method Size Ratio Date Time CRC-32 Name
-------- ------ ------- ----- ---- ---- ------ ----
24532 Defl:N 7633 69% 07-14-12 08:32 632ca8cc FileDownloader.py
97717 Defl:N 19367 80% 06-30-12 09:42 438204e9 InfoExtractors.py
6816 Defl:N 2349 66% 07-14-12 08:33 f90d7567 PostProcessor.py
20538 Defl:N 6187 70% 06-22-12 07:33 f5d9ab8b __init__.py
108 Defl:N 93 14% 04-14-12 08:02 27cad4c7 __main__.py
10007 Defl:N 3953 61% 06-22-12 07:33 9066c645 utils.py
-------- ------- --- -------
159718 39582 75% 6 files

So, I have several questions about this format change, but I'd like to hear from the python experts first. Is this common? Is this normal? What is going on here?

[jbaiter]

This is indeed possible in Python since version 2.3, albeit not very common (at least I haven't seen it very often).
You can load modules from zip-files and the interpreter will automatically decompress them for you.

This module adds the ability to import Python modules (*.py, *.py[co]) and packages from ZIP-format archives. It is usually not needed to use the zipimport module explicitly; it is automatically used by the built-in import mechanism for sys.path items that are paths to ZIP archives.

Typically, sys.path is a list of directory names as strings. This module also allows an item of sys.path to be a string naming a ZIP file archive. The ZIP archive can contain a subdirectory structure to support package imports, and a path within the archive can be specified to only import from a subdirectory. For example, the path /tmp/example.zip/lib/ would only import from the lib/ subdirectory within the archive.

Any files may be present in the ZIP archive, but only files .py and .py[co] are available for import. ZIP import of dynamic modules (.pyd, .so) is disallowed. Note that if an archive only contains .py files, Python will not attempt to modify the archive by adding the corresponding .pyc or .pyo file, meaning that if a ZIP archive doesn’t contain .pyc files, importing may be rather slow.

Using the built-in reload() function will fail if called on a module loaded from a ZIP archive; it is unlikely that reload() would be needed, since this would imply that the ZIP has been altered during runtime.

ZIP archives with an archive comment are currently not supported.

Some background
PEP 273 — Import Modules from Zip Archives: The PEP that describes this feature
zipimport — Import modules from Zip archives: Relevant entry in the Python documentation

[Joe Schmoe]

I'm curious as to why the youtube-dl.py maintainers switched to this format. Ostensibly, of course, it would be to save space (and time downloading), but who really cares about such small things these days (in either the space or time continuum) ?

You say that you haven't seen it very often. Any ideas on what would motivate someone to use it?

[Mortimer]

It could have been a mistake.

[mikerr]

Saves space and bandwidth - ok its a small file, but 3x reduction on large quantity of downloads is not to be sniffed at.

Keeps it all in a single file, and keeps idle tinkerers out

Is the code clear to read when in plain text ? they may also have gone the obfuscation route of protection.

[Joe Schmoe]

Saves space and bandwidth - ok its a small file, but 3x reduction on large quantity of downloads is not to be sniffed at.

I see what you're saying, but I somehow don't really think there's that much volume of downloading on this program. Just my gut feeling, of course - could be wrong.

Keeps it all in a single file, and keeps idle tinkerers out

See below - there is no encryption. All you have to do is unzip it and there you are.

Is the code clear to read when in plain text ? they may also have gone the obfuscation route of protection.

I did "unzip -c /tmp/glod | less" and looked at the code - it is long and complex, of course, but I don't think it is badly written - certainly not intentionally so.

BTW, not that I know anything about this stuff, but isn't it possible to "compile" Python (I think to .pyc files) and distribute that - that would be more like encryption/obfuscation than merely zipping it is.

Anyway, thanks for all the comments folks - and keep them coming. But maybe I should take this up with the youtube-dl.py maintainers as well - if I can find them/get them to respond...

[jbaiter]

BTW, not that I know anything about this stuff, but isn't it possible to "compile" Python (I think to .pyc files) and distribute that - that would be more like encryption/obfuscation than merely zipping it is.

Yup, for obfuscation purposes you can also just distribute a *pyc-file (or include that in the zip), the interpreter will be fine with that.