Julia the Language

[Heater]

I saw this text somewhere on the net about abstraction:

This possibly the worst writing on "abstraction" I have ever seen. So full of wrongness.

You noted once before that there was little mention of "strings" in the C standard. Which is true. But then there is a reason, for that. They are very simple, there is nothing much to say about them.

It's true there is not much to say about C style strings. You just have to make sure you don't lose that null on the end and be sure you do all that memory allocation/deallocation correctly. You have to be sure to use the various incredibly easy to use wrongly str** functions correctly as well.

But none of that is text in the modern world. Where unicode is the standard.

On the contrary I would say there is a huge amount to be said about text strings:
"Interesting Characters": https://www.youtube.com/watch?v=obmOsJibKtE
"Unicode in C++": https://www.youtube.com/watch?v=tOHnXt3Ycfo

Smart pointers I suppose? TBH, I've never found memory allocation/deallocation to be a huge issue. Its just programming.

Not par†ocularly smart pointers. As a certain fairly new language shows an awful lot of that can be taken care of at compile time with no extra code introduced for smart pointers.

"it's just programming" is my point though. In the same way C gets rid of all the "just programming" of constructing loops, calling functions, and so on we have to do in assembler. Which I guess you think is a good and useful thing to do. So we should be able to get rid of all that "just programming" of allocating and freeing memory and having to think about whether we have done it right or not.

In short I want to do "just programming" at a higher level of abstraction without having to be concerned of the low level details.

I very rarely have bugs here. Its all very simple. But then I have been programming in C for such a long time that the care needed to avoid bugs in this area is second nature. Same with array bound checks and similar stuff. My bugs tend to be elsewhere!

Pretty much every old C hand says something like that. Likely I have said such things in the past myself.

Sadly it is not true. Everyone makes mistakes, no matter what experience they have. Put a team together working on a large code base and you soon find all kind of weird things happening. A huge amount of time and effort goes into preventing those weird things happening. Much of which could be taken care of by the compiler. What is not to like about reducing that effort?

Anyway, the ease with which bugs can be introduced is not entirely my point. I just don't want to be "just coding" all that stuff over and over. I want to be "just coding" the problem I have in mind without any clutter in the source text or conceptually.

I'm a big fan of static error checking ...

Good. Then you will be pleased to hear there is now a compled language that does an order of magnitude or more static error checking than C or C++. More than C or C++ ever will be able to due to the nature of their design.

...and I am pleased that modern C compilers do as good a job as any other language at advanced error detection and diagnostics.

Still pretty rudimentary though. Certainly unable to detect a huge range of mistakes.

[Heater]

The attraction of Julia is not just a higher level of abstraction, but the interactivity: Julia has a read, evaluate, print loop just like Python. This means Julia can be used as an interactive calculator and learned one line at a time through experimentation.

Great. I love a REPL. I use node.js as a calculator all the time. Besides trying out little code riffs from time to time.

I'm not sold on it as being a great language learning aid. Typically a program of any size has a structure that is not easily handled in a REPL.

In many IDE's and editors one can get ones code analysed as you type and errors pointed out in real time. With a simple file watching system does code can be repeatedly compiled and run as one edits thus showing the actual results of running it.

I guess such a REPL helps for the very first baby steps into programming. As did the BASIC REPL back in the day.

[ejolson]

I guess such a REPL helps for the very first baby steps into programming. As did the BASIC REPL back in the day.

While interactive experimentation is definitely useful when first learning to program, Julia also functions as a programmable command line similar to bash that has been extended for working with vectors, matrices and other structured data. It further includes a just-in-time compiler to make things fast. Even after one becomes a skilled programmer, such an environment is useful for visualizing data and the other kinds of post-processing one might perform to understand it.

In my opinion, getting the latest stable release of Julia supported on the Pi is important both for learning and later as a working tool for many kinds of activities classified as data science and computation. My experience is that no other programming language is as easy for a beginner to get started with and then grows as well to become a high-performance tool for advanced needs.

[jalih]

While interactive experimentation is definitely useful when first learning to program, Julia also functions as a programmable command line similar to bash that has been extended for working with vectors, matrices and other structured data. It further includes a just-in-time compiler to make things fast. Even after one becomes a skilled programmer, such an environment is useful for visualizing data and the other kinds of post-processing one might perform to understand it.

With 8th programming language I use REPL all the time for testing ideas and visualizing that routines work as supposed to. I have no need for Rust like code checking features, it's a complex beast and scares me! I like prototyping as I write and try to factor code into small words (routines) that can be tested as I write. It allows programmer to be creative and productive.

I added GUI for the Sudoku solver I wrote earlier. It was really easy, fun and fast addition. Using C or Rust it would probably have taken me weeks to write.

[Heater]

Ah ha, a REPL with integrated means of drawing graphs, charts and all kind of funky graphics is a whole other world.

[ejolson]

I saw this text somewhere on the net about abstraction:

This possibly the worst writing on "abstraction" I have ever seen. So full of wrongness.

If each layer of abstraction results in a 20 percent slower performance along with a 50 percent increase in executable size, it is easy to conclude that adding layers of abstraction results in exponential growth in both aspects. In particular:

execution time = 1.2^n

and

executable size = 1.5^n.

It's not clear to me how many layers of abstraction are at work in Julia. Maybe a Julia program is just-in-time compiled by a compiler written in C++ that calls routines written in C which call subroutines written in assembler which call operating system system services written in C which again call functions written in assembler. If so then n=6.

On the other hand, if you further run Julia in a container on a virtual machine, that would add another 2 to 4 layers of abstraction and consequently n might be anywhere from 8 to 10. The worst case, as we've become familiar with during the epidemic, is

execution time = 1.2^10 = 6.2

and

executable size = 1.5^10 = 57.7.

Whether these numbers agree with reality and observation is impossible to know since Julia doesn't even run anymore. Any help with that would be appreciated.

[jahboater]

Whether these numbers agree with reality and observation is impossible to know since Julia doesn't even run anymore. Any help with that would be appreciated.

I downloaded Julia 1.6 for aarch64 and it appears to run on the Pi4, at least the repl does:

Code: Select all

pi@pi:~/julia-1.6.0/bin $ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> 2 + 2
4

julia> 

[ejolson]

Whether these numbers agree with reality and observation is impossible to know since Julia doesn't even run anymore. Any help with that would be appreciated.

I downloaded Julia 1.6 for aarch64 and it appears to run on the Pi4, at least the repl does:

Code: Select all

pi@pi:~/julia-1.6.0/bin $ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> 2 + 2
4

julia> 

This seems promising. Is that running on Raspberry Pi OS with the 64-bit kernel or the beta test of 64-bit Debian?

[Heater]

I saw this text somewhere on the net about abstraction:

This possibly the worst writing on "abstraction" I have ever seen. So full of wrongness.

If each layer of abstraction results in a 20 percent slower performance along with a 50 percent increase in executable size, it is easy to conclude that adding layers of abstraction results in exponential growth in both aspects.

I think that is the notion I most objected to there.

For a start, our word "abstraction" derives from the Latin "abstractus" meaning "drawn away". That is abstraction is about removing stuff, not adding it.

A good example would be the abstraction of a function call. One could spell out all that parameter pushing, calling and returning, reserving space for local variable, etc, in assembler. Or just abstract all that junk, draw away, into "f(x)".

Ideally a decent compiler will not produce more code to do that. Likely it can produce less. By inlining the function body or in some cases eliding the code all together.

Which is why C++ has had the notion of "zero cost abstraction" since forever. By which they mean, using some higher level, more abstract language feature should not cost more than any other way of doing the same thing.

Seems to me the notion of abstraction has been corrupted in the computer science world. Meaning something more like, we don't know how to do X or can't be bothered with it, we will let some other pile of code deal with it. Which is where you end up piling up more and more, layer on layer...

[jahboater]

This seems promising. Is that running on Raspberry Pi OS with the 64-bit kernel or the beta test of 64-bit Debian?

Raspberry Pi OS 64-bit beta.
https://downloads.raspberrypi.org/raspi ... 020-08-24/
That is 64-bit userland, which is as you say straight from Debian.

[ejolson]

This seems promising. Is that running on Raspberry Pi OS with the 64-bit kernel or the beta test of 64-bit Debian?

Raspberry Pi OS 64-bit beta.
https://downloads.raspberrypi.org/raspi ... 020-08-24/
That is 64-bit userland, which is as you say straight from Debian.

I'm still trying to build Julia 1.6.0 for 32-bit. The build seems to get further using Alpine Linux in a Singularity container, maybe because the processor type is more reliably detected compared to Raspberry Pi OS.

Since it is also important to have Mathematica which is only available in 32-bit Raspberry Pi OS, another alternative is to use the 64-bit kernel and place the 64-bit binary release of Julia in a 64-bit container. However, I'd also like to run Julia on a Pi Zero computer.

What's going on? The Raspberry Pi Foundation educational branch seems to focus on everything but Raspberry Pi hardware. Is the Raspberry Pi no longer recommended for use in science, technology, engineering and mathematics education?

[BoneFish]

I too would like to run recent versions of Julia on 32-bit Raspbian. Unfortunately, the Julia website does not currently provide up-to-date 32-bit ARM binaries. And I have had no luck compiling Julia 1.6.0 on the RPi3B+.

[ejolson]

I too would like to run recent versions of Julia on 32-bit Raspbian. Unfortunately, the Julia website does not currently provide up-to-date 32-bit ARM binaries. And I have had no luck compiling Julia 1.6.0 on the RPi3B+.

Thanks for the feedback and encouragement. Did your build break with a segmentation fault like my earlier efforts? My latest build attempt got further but ended with

Code: Select all

    LINK usr/lib/libjulia-internal.so.1.6
/usr/lib/gcc/armv6-alpine-linux-musleabihf/10.2.1/../../../../armv6-alpine-linux-musleabihf/bin/ld: /root/work/julia-1.6.0/usr/lib/libunwind.a(dyn-info-list.o):(.bss+0x0): multiple definition of `_U_dyn_info_list'; /root/work/julia-1.6.0/usr/lib/libunwind.a(Linit.o):(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:313: /root/work/julia-1.6.0/usr/lib/libjulia-internal.so.1.6] Error 1
make: *** [Makefile:76: julia-src-release] Error 2

which indicates a problem with libunwind. Is it possible that hidden link command is trying to link the same libunwind.a twice? Note this is using 32-bit Alpine Linux for ARMv6 in a Singularity container under 32-bit Raspberry Pi OS.

Does anyone know how to turn on verbose build messages when building Julia? It seems like there is a cmake hiding somewhere which is hiding the actual link command. Who thought hiding the details by default was a good idea? Will the next version of cmake display only a graphical progress bar along with a suitable selection of targeted banner advertisements instead?

[Heater]

No luck building 32 bit Julia 1.6 on my Pi 4 here. Looks like it detects my architecture wrongly, likely because I have a 64 bit kernel under my 32 bit user land:

But I did manage to build it as 64 bit using the nspawn container:

Code: Select all

pi@pi:~ $ uname -a
Linux pi 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux
pi@pi:~ $ 
pi@pi:~ $  ds64-shell
pi@debian-buster-64:~/Julia $ sudo apt-get install gfortran
pi@debian-buster-64:~/Julia $ git clone git://github.com/JuliaLang/julia.git
pi@debian-buster-64:~/Julia $ cd julia/
pi@debian-buster-64:~/Julia $ make
pi@debian-buster-64:~/Julia $ ls
pi@debian-buster-64:~/julia $ ./julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0-DEV.901 (2021-04-10)
 _/ |\__'_|_|_|\__'_|  |  Commit 008275c8d7* (0 days old master)
|__/                   |

julia> 

But now what?

I can't use Pluto, for example, to get a graphical interface because I have no idea how to get access to it's web server from my 32 bit environment where browsers are.

Code: Select all

julia> import Pluto
julia> Pluto.run()
Go to http://localhost:1235/?secret=LFdhdRDJ in your browser to start writing ~ have fun!

It runs but I can't access it in that container.

[ejolson]

No luck building 32 bit Julia 1.6 on my Pi 4 here. Looks like it detects my architecture wrongly, likely because I have a 64 bit kernel under my 32 bit user land:

But I did manage to build it as 64 bit using the nspawn container:

Code: Select all

pi@pi:~ $ uname -a
Linux pi 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux
pi@pi:~ $ 
pi@pi:~ $  ds64-shell
pi@debian-buster-64:~/Julia $ sudo apt-get install gfortran
pi@debian-buster-64:~/Julia $ git clone git://github.com/JuliaLang/julia.git
pi@debian-buster-64:~/Julia $ cd julia/
pi@debian-buster-64:~/Julia $ make
pi@debian-buster-64:~/Julia $ ls
pi@debian-buster-64:~/julia $ ./julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0-DEV.901 (2021-04-10)
 _/ |\__'_|_|_|\__'_|  |  Commit 008275c8d7* (0 days old master)
|__/                   |

julia> 

But now what?

I can't use Pluto, for example, to get a graphical interface because I have no idea how to get access to it's web server from my 32 bit environment where browsers are.

Code: Select all

julia> import Pluto
julia> Pluto.run()
Go to http://localhost:1235/?secret=LFdhdRDJ in your browser to start writing ~ have fun!

It runs but I can't access it in that container.

I think the idea behind Singularity is for native networking to just work. While the main reason is not wanting to virtualise a 200Gb/s InfiniBand interconnect, the practical result might be an ability to contact Pluto, Jupyter and other planets.

Information on how to build and install Singularity on the Pi is at

viewtopic.php?p=1843434#p1843434

[Heater]

Information on how to build and install Singularity on the Pi is at

Nooo... Don't send me down another long, dark, rabbit hole of researching and spending countless hours trying to get yet another piece of software I have never heard of before working.

Can one even return from a singularity?

I'm sure there is an easy way to bridge the networking in my nspawn container to the networking on the Pi proper. If only I could find it....

[ejolson]

I too would like to run recent versions of Julia on 32-bit Raspbian. Unfortunately, the Julia website does not currently provide up-to-date 32-bit ARM binaries. And I have had no luck compiling Julia 1.6.0 on the RPi3B+.

Thanks for the feedback and encouragement. Did your build break with a segmentation fault like my earlier efforts? My latest build attempt got further but ended with

Code: Select all

    LINK usr/lib/libjulia-internal.so.1.6
/usr/lib/gcc/armv6-alpine-linux-musleabihf/10.2.1/../../../../armv6-alpine-linux-musleabihf/bin/ld: /root/work/julia-1.6.0/usr/lib/libunwind.a(dyn-info-list.o):(.bss+0x0): multiple definition of `_U_dyn_info_list'; /root/work/julia-1.6.0/usr/lib/libunwind.a(Linit.o):(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:313: /root/work/julia-1.6.0/usr/lib/libjulia-internal.so.1.6] Error 1
make: *** [Makefile:76: julia-src-release] Error 2

which indicates a problem with libunwind. Is it possible that hidden link command is trying to link the same libunwind.a twice? Note this is using 32-bit Alpine Linux for ARMv6 in a Singularity container under 32-bit Raspberry Pi OS.

I recklessly fixed the duplicate definition by adding -Wl,-allow-multiple-definition to the link command and got a bit further. Now I'm struggling with some Python-style part of the build process involving a comedy script which tries to parse the gcc target

armv6-alpine-linux-musleabihf

and gets confused because it's not an x86 system.

Before learning how to read Python regular expressions, I decided to try building Julia 1.6.0 in a 32-bit Void Linux glibc chroot environment. Unfortunately, the resulting binary segfaults in exactly the same way as the Raspberry Pi OS build. Much to my surprise, it seems the musl C library will more likely lead to a working build.

[ejolson]

Much to my surprise, it seems the musl C library will more likely lead to a working build.

It seems I managed to create a 32-bit ARM singularity container based on Alpine Linux with Julia version 1.6.0 built inside. It runs fine on the 4B but I complains about the wrong architecture on the Pi Zero, so I must have set something wrong with the CPU target. I'll try again and then document the technique here once it's fully working.

[BoneFish]

Running "make" in the Julia 1.6.0 source directory fails to build and gives this error:

Code: Select all

donald@raspberrypi:~/julia-1.6.0 $ cat log 
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
    LINK usr/lib/libjulia-internal.so.1.6
/usr/bin/ld: ./gc.o: in function `combine_thread_gc_counts':
/home/donald/julia-1.6.0/src/gc.c:1060: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1061: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1062: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1063: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1064: undefined reference to `__atomic_load_8'
/usr/bin/ld: ./gc.o:/home/donald/julia-1.6.0/src/gc.c:1065: more undefined references to `__atomic_load_8' follow
/usr/bin/ld: ./gc.o: in function `jl_gc_collect':
/home/donald/julia-1.6.0/src/gc.c:3212: undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:313: /home/donald/julia-1.6.0/usr/lib/libjulia-internal.so.1.6] Error 1
make: *** [Makefile:76: julia-src-release] Error 2
donald@raspberrypi:~/julia-1.6.0 $

Simply running "make" works fine on a 64-bit Intel processor, but I suspect it's naive of me to think this would work on the RPi3B+.

[Heater]

I got rid of that "atomic" error by jamming "-latomic" into the compile options.

I don't recall how I did it now though.

But then it failed with something else...

[ejolson]

I got rid of that "atomic" error by jamming "-latomic" into the compile options.

I don't recall how I did it now though.

But then it failed with something else...

I tried this as well by adding -latomic with my mouse, but the resulting binary causes a segmentation fault on both 32-bit Raspberry Pi OS and the glibc-based 32-bit ARM version of Void Linux. With musl-based Alpine Linux I now have a working installation that even runs on the Pi B+ and Zero computers.

Here is a B+ test run

Code: Select all

$ uname -a
Linux snail 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
$ grep model /proc/cpuinfo
model name  : ARMv6-compatible processor rev 7 (v6l)
$ singularity version
3.7.2
$ singularity run julia160.sif 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> n=500
500

julia> A=rand(n,n);

julia> x=zeros(n);

julia> b=A*x;

julia> @time A\b;
 45.932462 seconds (2.53 M allocations: 107.392 MiB, 3.16% gc time, 98.73% compilation time)

julia> @time A\b;
  0.535017 seconds (4 allocations: 1.913 MiB)

julia> @time A\b;
  0.532816 seconds (4 allocations: 1.913 MiB)

As 32-bit ARM is now a Tier 3 platform, I suspect some popular packages such as Plots will fail to add, but there is quite a bit one can do with the built-in packages.

Note that I built the container on a 2GB Pi 4B which used 200MB of swap during the build but didn't thrash and finished with

Code: Select all

make[1]: Leaving directory '/root/work/julia-1.6.0'
real    4h 54m 25s
user    14h 56m 04s
sys 1h 37m 40s

I suspect trying to build Julia directly on a Pi Zero with less RAM would either fail or start thrashing to the point the build takes months instead of hours.

I'll describe the procedure to build a Singularity container for Julia using Alpine Linux that runs under 32-bit Raspberry Pi OS shortly. Though I don't usually post links to binary builds, I'm wondering if anyone would have an interest in this container. Should I try to upload it to Singularity hub?

[BoneFish]

I'll describe the procedure to build a Singularity container for Julia using Alpine Linux that runs under 32-bit Raspberry Pi OS shortly. Though I don't usually post links to binary builds, I'm wondering if anyone would have an interest in this container. Should I try to upload it to Singularity hub?

I don't have prior experience with Singularity containers but I would be very interested to hear the procedure!

[Gavinmc42]

I use this on 32bit versions of the VulkanScenGraph libs.
"cmake . -DCMAKE_CXX_FLAGS=-latomic"
Think I used it for other "cmake ." stuff as well.
Gone back to 32bit Raspberry Pi OS, the 64bit version does not need that cmake option.

[ejolson]

I'll describe the procedure to build a Singularity container for Julia using Alpine Linux that runs under 32-bit Raspberry Pi OS shortly. Though I don't usually post links to binary builds, I'm wondering if anyone would have an interest in this container. Should I try to upload it to Singularity hub?

I don't have prior experience with Singularity containers but I would be very interested to hear the procedure!

Singularity turns out to be very easy to install on a Raspberry Pi. This is because the Go language tool chain is well supported. I've already described the details at

viewtopic.php?p=1846641#p1846641

so I won't repeat them except for the observation that it takes less than 7 minutes to build Singularity from source on a Pi 4B.

After installing Singularity, one then needs a container with Julia in it. This presents a difficulty, because Julia is difficult to build and the current release includes no official 32-bit ARM executable. On a Pi 2B v1.2, 3B, 3B+ and 4B the best solution is to create a 64-bit container with the official 64-bit version of Julia and switch to the 64-bit kernel as discussed in

viewtopic.php?p=1849572#p1849572

However, this will not work for a B, B+, 2B v1.1 or Zero because the hardware is not capable of running 64-bit software and 64-bit emulation would be too slow.

My focus, therefore, has been to create a ARMv6 compatible Singularity container with Julia that runs on all models of Pi and all versions of 32-bit Raspberry Pi OS.

I did this using the 32-bit ARMv6 version of Alpine Linux as a base. This is the first time ever that I found building a large program was easier using the musl C library than with glibc. Since musl is targeted at small and embedded systems, my suspicion is that its 32-bit support hasn't rotted the way 32-bit support has in glibc.

After importing the standard docker image for Alpine Linux as described in

viewtopic.php?p=1847649#p1847649

which is only 5 MB, I bloated it up by installing gcc and dozens of other utilities needed to compile Julia. What I installed was based on the dependencies described at

https://github.com/JuliaLang/julia/blob ... d/build.md

though I had to add a few others that are not mentioned.

For reference the final list of installed packages was

Code: Select all

Singularity> apk info
musl
busybox
alpine-baselayout
alpine-keys
libcrypto1.1
libssl1.1
ca-certificates-bundle
libtls-standalone
ssl_client
zlib
apk-tools
scanelf
musl-utils
libc-utils
libgcc
libstdc++
binutils
libgomp
libatomic
libgphobos
gmp
isl22
mpfr4
mpc1
gcc
ncurses-terminfo-base
ncurses-libs
readline
bash
make
libacl
tar
xz-libs
xz
libgfortran
gfortran
musl-dev
libc-dev
patch
g++
m4
libbz2
perl
expat
libffi
gdbm
sqlite-libs
python2
python3
lz4-libs
zstd-libs
libarchive
ca-certificates
brotli-libs
nghttp2-libs
libcurl
rhash-libs
libuv
cmake
linux-headers
pcre2
git
perl-error
perl-git
git-perl
autoconf
automake
pkgconf
which
curl
gawk
bzip2
less
gzip
openssl
xxd
lua5.3-libs
vim
libattr
skalibs
s6-ipcserver
utmps
coreutils

Note that some of these packages were already in the base container and some came in as dependencies of the others. I have not bothered to keep track of the exact apk add commands used to configure the container.

As this post is getting long, the next post will describe the commands used to download and build Julia inside the resulting Alpine Singularity container.

[ejolson]

the next post will describe the commands used to download and build Julia inside the resulting Alpine Singularity container.

Since Singularity containers run with user-level permissions it was expedient to run the container as root while building Julia. This is because, I needed to configure an entire Alpine Linux distribution inside the sandbox as mentioned in the previous post before building and installing Julia.

Assuming alpine is the name of the directory that holds the sandbox, type

Code: Select all

$ su -
# singularity shell -w alpine

to enter the container. Since the sandbox was entered as root, the home directory of the host system will be conveniently mounted under /root in the sandbox.

Now, download Julia and unpack it with

Code: Select all

Singularity> wget https://github.com/JuliaLang/julia/releases/download/v1.6.0/julia-1.6.0.tar.gz
Singularity> tar xf julia-1.6.0.tar.gz
Singularity> cd julia-1.6.0

Configure Julia by creating the Make.user in the julia-1.6.0 directory with

Code: Select all

JULIA_CPU_TARGET := arm1176jzf-s
MCPU := arm1176jzf-s
MARCH := armv6zk
OPENBLAS_TARGET_ARCH := ARMV6
USE_BINARYBUILDER := 0
OPENBLAS_USE_THREAD := 0
prefix = /usr/local/julia-1.6.0

Note that I have disabled threads in OpenBLAS since the ARMv6 based Pi B, B+ and Zero computers have only one core.

Next, patch some of the scripts used in the build system with

Code: Select all

Singularity> patch -p1 <julia160.patch

where the contents of julia160.patch are given by

Code: Select all

*** julia-1.6.0/base/binaryplatforms.jl 2021-03-24 20:02:52.000000000 +0000
--- julia-1.6.0-good/base/binaryplatforms.jl    2021-04-11 04:21:58.706732181 +0100
***************
*** 575,582 ****
      "x86_64" => "(x86_|amd)64",
      "i686" => "i\\d86",
      "aarch64" => "(aarch64|arm64)",
      "armv7l" => "arm(v7l)?", # if we just see `arm-linux-gnueabihf`, we assume it's `armv7l`
-     "armv6l" => "armv6l",
      "powerpc64le" => "p(ower)?pc64le",
  )
  # Keep this in sync with `CPUID.ISAs_by_family`
--- 575,582 ----
      "x86_64" => "(x86_|amd)64",
      "i686" => "i\\d86",
      "aarch64" => "(aarch64|arm64)",
+     "armv6l" => "armv6(l)?", # ejo--fix armv6-alpine-linux-musleabihf
      "armv7l" => "arm(v7l)?", # if we just see `arm-linux-gnueabihf`, we assume it's `armv7l`
      "powerpc64le" => "p(ower)?pc64le",
  )
  # Keep this in sync with `CPUID.ISAs_by_family`
*** julia-1.6.0/contrib/normalize_triplet.py    2021-03-24 20:02:52.000000000 +0000
--- julia-1.6.0-good/contrib/normalize_triplet.py   2021-04-11 04:26:04.669514370 +0100
***************
*** 13,18 ****
--- 13,19 ----
      'x86_64': '(x86_|amd)64',
      'i686': "i\\d86",
      'aarch64': "(arm|aarch)64",
+     'armv6l': "armv6(l)?",   # ejo--armv6-alpine-linux-musleabihf
      'armv7l': "arm(v7l)?",
      'powerpc64le': "p(ower)?pc64le",
  }
*** julia-1.6.0/Make.inc    2021-03-24 20:02:52.000000000 +0000
--- julia-1.6.0-good/Make.inc   2021-04-11 02:34:36.805354500 +0100
***************
*** 920,926 ****
  JCFLAGS += -fsigned-char
  USE_BLAS64:=0
  OPENBLAS_DYNAMIC_ARCH:=0
! OPENBLAS_TARGET_ARCH:=ARMV7
  endif
  
  # If we are running on aarch64 (e.g. ARMv8 or ARM64), set certain options automatically
--- 920,926 ----
  JCFLAGS += -fsigned-char
  USE_BLAS64:=0
  OPENBLAS_DYNAMIC_ARCH:=0
! OPENBLAS_TARGET_ARCH:=ARMV6
  endif
  
  # If we are running on aarch64 (e.g. ARMv8 or ARM64), set certain options automatically

This patch fixes three files needed for the ARMv6 build on Alpine Linux.

Upon patching, build Julia with

Code: Select all

Singularity> export VERBOSE=1
Singularity> export LDFLAGS=-Wl,-allow-multiple-definition
Singularity> nohup time make -j4 &

After about 5 hours the build should finish. At this point test the executable by running it from the source directory as

Code: Select all

Singularity> ./julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> exit()

Assuming that went fine, install Julia into the sandbox with

Code: Select all

Singularity> make install
Singularity> cd /usr/local/bin
Singularity> ln -s ../julia-1.6.0/bin/* .

Set Julia to be the default run action by editing /singularity so it reads

Code: Select all

#!/bin/sh
exec /usr/local/bin/julia "$@"

Finally, exit the sandbox and build the container with

Code: Select all

Singularity> exit
# singularity build julia160.sif alpine

Note that no attempts have been made to reduce the size of the sandbox before converting it into a container. Thus, julia160.sif includes a full version of gcc, g++, gfortran and a slew of other things hidden inside. Mine turned out 240 MB.

The main reason I used a container for my build is because I wanted to learn about Singularity on the super-cheap cluster.

viewtopic.php?p=1247739#p1247739

Moving forward it would be nice if someone could figure out why native builds of Julia fail under the current version of 32-bit Raspberry Pi OS.