Heater
Posts: 18215
Joined: Tue Jul 17, 2012 3:02 pm

Re: Julia the Language

Sun Apr 04, 2021 2:15 am

jahboater wrote:
Sat Apr 03, 2021 10:05 pm
I saw this text somewhere on the net about abstraction:
This possibly the worst writing on "abstraction" I have ever seen. So full of wrongness.
jahboater wrote:
Sat Apr 03, 2021 10:05 pm
You noted once before that there was little mention of "strings" in the C standard. Which is true. But then there is a reason, for that. They are very simple, there is nothing much to say about them.
It's true there is not much to say about C style strings. You just have to make sure you don't lose that null on the end and be sure you do all that memory allocation/deallocation correctly. You have to be sure to use the various incredibly easy to use wrongly str** functions correctly as well.

But none of that is text in the modern world. Where unicode is the standard.

On the contrary I would say there is a huge amount to be said about text strings:
"Interesting Characters": https://www.youtube.com/watch?v=obmOsJibKtE
"Unicode in C++": https://www.youtube.com/watch?v=tOHnXt3Ycfo
jahboater wrote:
Sat Apr 03, 2021 10:05 pm
Smart pointers I suppose? TBH, I've never found memory allocation/deallocation to be a huge issue. Its just programming.
Not par†ocularly smart pointers. As a certain fairly new language shows an awful lot of that can be taken care of at compile time with no extra code introduced for smart pointers.

"it's just programming" is my point though. In the same way C gets rid of all the "just programming" of constructing loops, calling functions, and so on we have to do in assembler. Which I guess you think is a good and useful thing to do. So we should be able to get rid of all that "just programming" of allocating and freeing memory and having to think about whether we have done it right or not.

In short I want to do "just programming" at a higher level of abstraction without having to be concerned of the low level details.
jahboater wrote:
Sat Apr 03, 2021 10:05 pm
I very rarely have bugs here. Its all very simple. But then I have been programming in C for such a long time that the care needed to avoid bugs in this area is second nature. Same with array bound checks and similar stuff. My bugs tend to be elsewhere!
Pretty much every old C hand says something like that. Likely I have said such things in the past myself.

Sadly it is not true. Everyone makes mistakes, no matter what experience they have. Put a team together working on a large code base and you soon find all kind of weird things happening. A huge amount of time and effort goes into preventing those weird things happening. Much of which could be taken care of by the compiler. What is not to like about reducing that effort?

Anyway, the ease with which bugs can be introduced is not entirely my point. I just don't want to be "just coding" all that stuff over and over. I want to be "just coding" the problem I have in mind without any clutter in the source text or conceptually.
jahboater wrote:
Sat Apr 03, 2021 10:05 pm
I'm a big fan of static error checking ...
Good. Then you will be pleased to hear there is now a compled language that does an order of magnitude or more static error checking than C or C++. More than C or C++ ever will be able to due to the nature of their design.
jahboater wrote:
Sat Apr 03, 2021 10:05 pm
...and I am pleased that modern C compilers do as good a job as any other language at advanced error detection and diagnostics.
Still pretty rudimentary though. Certainly unable to detect a huge range of mistakes.
Memory in C++ is a leaky abstraction .

Heater
Posts: 18215
Joined: Tue Jul 17, 2012 3:02 pm

Re: Julia the Language

Sun Apr 04, 2021 2:30 am

ejolson wrote:
Sat Apr 03, 2021 11:01 pm
The attraction of Julia is not just a higher level of abstraction, but the interactivity: Julia has a read, evaluate, print loop just like Python. This means Julia can be used as an interactive calculator and learned one line at a time through experimentation.
Great. I love a REPL. I use node.js as a calculator all the time. Besides trying out little code riffs from time to time.

I'm not sold on it as being a great language learning aid. Typically a program of any size has a structure that is not easily handled in a REPL.

In many IDE's and editors one can get ones code analysed as you type and errors pointed out in real time. With a simple file watching system does code can be repeatedly compiled and run as one edits thus showing the actual results of running it.

I guess such a REPL helps for the very first baby steps into programming. As did the BASIC REPL back in the day.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Wed Apr 07, 2021 4:26 am

Heater wrote:
Sun Apr 04, 2021 2:30 am
I guess such a REPL helps for the very first baby steps into programming. As did the BASIC REPL back in the day.
While interactive experimentation is definitely useful when first learning to program, Julia also functions as a programmable command line similar to bash that has been extended for working with vectors, matrices and other structured data. It further includes a just-in-time compiler to make things fast. Even after one becomes a skilled programmer, such an environment is useful for visualizing data and the other kinds of post-processing one might perform to understand it.

In my opinion, getting the latest stable release of Julia supported on the Pi is important both for learning and later as a working tool for many kinds of activities classified as data science and computation. My experience is that no other programming language is as easy for a beginner to get started with and then grows as well to become a high-performance tool for advanced needs.

jalih
Posts: 178
Joined: Mon Apr 15, 2019 3:54 pm

Re: Julia the Language

Wed Apr 07, 2021 9:39 am

ejolson wrote:
Wed Apr 07, 2021 4:26 am
While interactive experimentation is definitely useful when first learning to program, Julia also functions as a programmable command line similar to bash that has been extended for working with vectors, matrices and other structured data. It further includes a just-in-time compiler to make things fast. Even after one becomes a skilled programmer, such an environment is useful for visualizing data and the other kinds of post-processing one might perform to understand it.
With 8th programming language I use REPL all the time for testing ideas and visualizing that routines work as supposed to. I have no need for Rust like code checking features, it's a complex beast and scares me! I like prototyping as I write and try to factor code into small words (routines) that can be tested as I write. It allows programmer to be creative and productive.

I added GUI for the Sudoku solver I wrote earlier. It was really easy, fun and fast addition. Using C or Rust it would probably have taken me weeks to write.
Attachments
sudoku.png
sudoku.png (34.72 KiB) Viewed 1709 times

Heater
Posts: 18215
Joined: Tue Jul 17, 2012 3:02 pm

Re: Julia the Language

Wed Apr 07, 2021 2:40 pm

Ah ha, a REPL with integrated means of drawing graphs, charts and all kind of funky graphics is a whole other world.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Thu Apr 08, 2021 3:35 pm

Heater wrote:
Sun Apr 04, 2021 2:15 am
jahboater wrote:
Sat Apr 03, 2021 10:05 pm
I saw this text somewhere on the net about abstraction:
This possibly the worst writing on "abstraction" I have ever seen. So full of wrongness.
If each layer of abstraction results in a 20 percent slower performance along with a 50 percent increase in executable size, it is easy to conclude that adding layers of abstraction results in exponential growth in both aspects. In particular:

execution time = 1.2^n

and

executable size = 1.5^n.

It's not clear to me how many layers of abstraction are at work in Julia. Maybe a Julia program is just-in-time compiled by a compiler written in C++ that calls routines written in C which call subroutines written in assembler which call operating system system services written in C which again call functions written in assembler. If so then n=6.

On the other hand, if you further run Julia in a container on a virtual machine, that would add another 2 to 4 layers of abstraction and consequently n might be anywhere from 8 to 10. The worst case, as we've become familiar with during the epidemic, is

execution time = 1.2^10 = 6.2

and

executable size = 1.5^10 = 57.7.

Whether these numbers agree with reality and observation is impossible to know since Julia doesn't even run anymore. Any help with that would be appreciated.

User avatar
jahboater
Posts: 7042
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Julia the Language

Thu Apr 08, 2021 3:42 pm

ejolson wrote:
Thu Apr 08, 2021 3:35 pm
Whether these numbers agree with reality and observation is impossible to know since Julia doesn't even run anymore. Any help with that would be appreciated.
I downloaded Julia 1.6 for aarch64 and it appears to run on the Pi4, at least the repl does:

Code: Select all

pi@pi:~/julia-1.6.0/bin $ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> 2 + 2
4

julia> 

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Thu Apr 08, 2021 3:46 pm

jahboater wrote:
Thu Apr 08, 2021 3:42 pm
ejolson wrote:
Thu Apr 08, 2021 3:35 pm
Whether these numbers agree with reality and observation is impossible to know since Julia doesn't even run anymore. Any help with that would be appreciated.
I downloaded Julia 1.6 for aarch64 and it appears to run on the Pi4, at least the repl does:

Code: Select all

pi@pi:~/julia-1.6.0/bin $ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> 2 + 2
4

julia> 
This seems promising. Is that running on Raspberry Pi OS with the 64-bit kernel or the beta test of 64-bit Debian?

Heater
Posts: 18215
Joined: Tue Jul 17, 2012 3:02 pm

Re: Julia the Language

Thu Apr 08, 2021 4:48 pm

ejolson wrote:
Thu Apr 08, 2021 3:35 pm
Heater wrote:
Sun Apr 04, 2021 2:15 am
jahboater wrote:
Sat Apr 03, 2021 10:05 pm
I saw this text somewhere on the net about abstraction:
This possibly the worst writing on "abstraction" I have ever seen. So full of wrongness.
If each layer of abstraction results in a 20 percent slower performance along with a 50 percent increase in executable size, it is easy to conclude that adding layers of abstraction results in exponential growth in both aspects.
I think that is the notion I most objected to there.

For a start, our word "abstraction" derives from the Latin "abstractus" meaning "drawn away". That is abstraction is about removing stuff, not adding it.

A good example would be the abstraction of a function call. One could spell out all that parameter pushing, calling and returning, reserving space for local variable, etc, in assembler. Or just abstract all that junk, draw away, into "f(x)".

Ideally a decent compiler will not produce more code to do that. Likely it can produce less. By inlining the function body or in some cases eliding the code all together.

Which is why C++ has had the notion of "zero cost abstraction" since forever. By which they mean, using some higher level, more abstract language feature should not cost more than any other way of doing the same thing.

Seems to me the notion of abstraction has been corrupted in the computer science world. Meaning something more like, we don't know how to do X or can't be bothered with it, we will let some other pile of code deal with it. Which is where you end up piling up more and more, layer on layer...
Memory in C++ is a leaky abstraction .

User avatar
jahboater
Posts: 7042
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Julia the Language

Thu Apr 08, 2021 6:08 pm

ejolson wrote:
Thu Apr 08, 2021 3:46 pm
This seems promising. Is that running on Raspberry Pi OS with the 64-bit kernel or the beta test of 64-bit Debian?
Raspberry Pi OS 64-bit beta.
https://downloads.raspberrypi.org/raspi ... 020-08-24/
That is 64-bit userland, which is as you say straight from Debian.

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Fri Apr 09, 2021 4:16 pm

jahboater wrote:
Thu Apr 08, 2021 6:08 pm
ejolson wrote:
Thu Apr 08, 2021 3:46 pm
This seems promising. Is that running on Raspberry Pi OS with the 64-bit kernel or the beta test of 64-bit Debian?
Raspberry Pi OS 64-bit beta.
https://downloads.raspberrypi.org/raspi ... 020-08-24/
That is 64-bit userland, which is as you say straight from Debian.
I'm still trying to build Julia 1.6.0 for 32-bit. The build seems to get further using Alpine Linux in a Singularity container, maybe because the processor type is more reliably detected compared to Raspberry Pi OS.

Since it is also important to have Mathematica which is only available in 32-bit Raspberry Pi OS, another alternative is to use the 64-bit kernel and place the 64-bit binary release of Julia in a 64-bit container. However, I'd also like to run Julia on a Pi Zero computer.

What's going on? The Raspberry Pi Foundation educational branch seems to focus on everything but Raspberry Pi hardware. Is the Raspberry Pi no longer recommended for use in science, technology, engineering and mathematics education?

BoneFish
Posts: 3
Joined: Sat Apr 10, 2021 12:26 am

Re: Julia the Language

Sat Apr 10, 2021 12:33 am

I too would like to run recent versions of Julia on 32-bit Raspbian. Unfortunately, the Julia website does not currently provide up-to-date 32-bit ARM binaries. And I have had no luck compiling Julia 1.6.0 on the RPi3B+. :cry:

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Sat Apr 10, 2021 3:59 pm

BoneFish wrote:
Sat Apr 10, 2021 12:33 am
I too would like to run recent versions of Julia on 32-bit Raspbian. Unfortunately, the Julia website does not currently provide up-to-date 32-bit ARM binaries. And I have had no luck compiling Julia 1.6.0 on the RPi3B+. :cry:
Thanks for the feedback and encouragement. Did your build break with a segmentation fault like my earlier efforts? My latest build attempt got further but ended with

Code: Select all

    LINK usr/lib/libjulia-internal.so.1.6
/usr/lib/gcc/armv6-alpine-linux-musleabihf/10.2.1/../../../../armv6-alpine-linux-musleabihf/bin/ld: /root/work/julia-1.6.0/usr/lib/libunwind.a(dyn-info-list.o):(.bss+0x0): multiple definition of `_U_dyn_info_list'; /root/work/julia-1.6.0/usr/lib/libunwind.a(Linit.o):(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:313: /root/work/julia-1.6.0/usr/lib/libjulia-internal.so.1.6] Error 1
make: *** [Makefile:76: julia-src-release] Error 2
which indicates a problem with libunwind. Is it possible that hidden link command is trying to link the same libunwind.a twice? Note this is using 32-bit Alpine Linux for ARMv6 in a Singularity container under 32-bit Raspberry Pi OS.

Does anyone know how to turn on verbose build messages when building Julia? It seems like there is a cmake hiding somewhere which is hiding the actual link command. Who thought hiding the details by default was a good idea? Will the next version of cmake display only a graphical progress bar along with a suitable selection of targeted banner advertisements instead?

Heater
Posts: 18215
Joined: Tue Jul 17, 2012 3:02 pm

Re: Julia the Language

Sat Apr 10, 2021 4:24 pm

No luck building 32 bit Julia 1.6 on my Pi 4 here. Looks like it detects my architecture wrongly, likely because I have a 64 bit kernel under my 32 bit user land:

But I did manage to build it as 64 bit using the nspawn container:

Code: Select all

pi@pi:~ $ uname -a
Linux pi 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux
pi@pi:~ $ 
pi@pi:~ $  ds64-shell
pi@debian-buster-64:~/Julia $ sudo apt-get install gfortran
pi@debian-buster-64:~/Julia $ git clone git://github.com/JuliaLang/julia.git
pi@debian-buster-64:~/Julia $ cd julia/
pi@debian-buster-64:~/Julia $ make
pi@debian-buster-64:~/Julia $ ls
pi@debian-buster-64:~/julia $ ./julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0-DEV.901 (2021-04-10)
 _/ |\__'_|_|_|\__'_|  |  Commit 008275c8d7* (0 days old master)
|__/                   |

julia> 
But now what?

I can't use Pluto, for example, to get a graphical interface because I have no idea how to get access to it's web server from my 32 bit environment where browsers are.

Code: Select all

julia> import Pluto
julia> Pluto.run()
Go to http://localhost:1235/?secret=LFdhdRDJ in your browser to start writing ~ have fun!
It runs but I can't access it in that container.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Sat Apr 10, 2021 5:10 pm

Heater wrote:
Sat Apr 10, 2021 4:24 pm
No luck building 32 bit Julia 1.6 on my Pi 4 here. Looks like it detects my architecture wrongly, likely because I have a 64 bit kernel under my 32 bit user land:

But I did manage to build it as 64 bit using the nspawn container:

Code: Select all

pi@pi:~ $ uname -a
Linux pi 5.10.17-v8+ #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux
pi@pi:~ $ 
pi@pi:~ $  ds64-shell
pi@debian-buster-64:~/Julia $ sudo apt-get install gfortran
pi@debian-buster-64:~/Julia $ git clone git://github.com/JuliaLang/julia.git
pi@debian-buster-64:~/Julia $ cd julia/
pi@debian-buster-64:~/Julia $ make
pi@debian-buster-64:~/Julia $ ls
pi@debian-buster-64:~/julia $ ./julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0-DEV.901 (2021-04-10)
 _/ |\__'_|_|_|\__'_|  |  Commit 008275c8d7* (0 days old master)
|__/                   |

julia> 
But now what?

I can't use Pluto, for example, to get a graphical interface because I have no idea how to get access to it's web server from my 32 bit environment where browsers are.

Code: Select all

julia> import Pluto
julia> Pluto.run()
Go to http://localhost:1235/?secret=LFdhdRDJ in your browser to start writing ~ have fun!
It runs but I can't access it in that container.
I think the idea behind Singularity is for native networking to just work. While the main reason is not wanting to virtualise a 200Gb/s InfiniBand interconnect, the practical result might be an ability to contact Pluto, Jupyter and other planets.

Information on how to build and install Singularity on the Pi is at

viewtopic.php?p=1843434#p1843434

Heater
Posts: 18215
Joined: Tue Jul 17, 2012 3:02 pm

Re: Julia the Language

Sat Apr 10, 2021 5:34 pm

ejolson wrote:
Sat Apr 10, 2021 5:10 pm
Information on how to build and install Singularity on the Pi is at
Nooo... Don't send me down another long, dark, rabbit hole of researching and spending countless hours trying to get yet another piece of software I have never heard of before working.

Can one even return from a singularity?

I'm sure there is an easy way to bridge the networking in my nspawn container to the networking on the Pi proper. If only I could find it....
Memory in C++ is a leaky abstraction .

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Sun Apr 11, 2021 4:01 am

ejolson wrote:
Sat Apr 10, 2021 3:59 pm
BoneFish wrote:
Sat Apr 10, 2021 12:33 am
I too would like to run recent versions of Julia on 32-bit Raspbian. Unfortunately, the Julia website does not currently provide up-to-date 32-bit ARM binaries. And I have had no luck compiling Julia 1.6.0 on the RPi3B+. :cry:
Thanks for the feedback and encouragement. Did your build break with a segmentation fault like my earlier efforts? My latest build attempt got further but ended with

Code: Select all

    LINK usr/lib/libjulia-internal.so.1.6
/usr/lib/gcc/armv6-alpine-linux-musleabihf/10.2.1/../../../../armv6-alpine-linux-musleabihf/bin/ld: /root/work/julia-1.6.0/usr/lib/libunwind.a(dyn-info-list.o):(.bss+0x0): multiple definition of `_U_dyn_info_list'; /root/work/julia-1.6.0/usr/lib/libunwind.a(Linit.o):(.bss+0x0): first defined here
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:313: /root/work/julia-1.6.0/usr/lib/libjulia-internal.so.1.6] Error 1
make: *** [Makefile:76: julia-src-release] Error 2
which indicates a problem with libunwind. Is it possible that hidden link command is trying to link the same libunwind.a twice? Note this is using 32-bit Alpine Linux for ARMv6 in a Singularity container under 32-bit Raspberry Pi OS.
I recklessly fixed the duplicate definition by adding -Wl,-allow-multiple-definition to the link command and got a bit further. Now I'm struggling with some Python-style part of the build process involving a comedy script which tries to parse the gcc target

armv6-alpine-linux-musleabihf

and gets confused because it's not an x86 system.

Before learning how to read Python regular expressions, I decided to try building Julia 1.6.0 in a 32-bit Void Linux glibc chroot environment. Unfortunately, the resulting binary segfaults in exactly the same way as the Raspberry Pi OS build. Much to my surprise, it seems the musl C library will more likely lead to a working build.
Last edited by ejolson on Sun Apr 11, 2021 6:34 pm, edited 6 times in total.

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Sun Apr 11, 2021 5:20 am

ejolson wrote:
Sun Apr 11, 2021 4:01 am
Much to my surprise, it seems the musl C library will more likely lead to a working build.
It seems I managed to create a 32-bit ARM singularity container based on Alpine Linux with Julia version 1.6.0 built inside. It runs fine on the 4B but I complains about the wrong architecture on the Pi Zero, so I must have set something wrong with the CPU target. I'll try again and then document the technique here once it's fully working.

BoneFish
Posts: 3
Joined: Sat Apr 10, 2021 12:26 am

Re: Julia the Language

Sun Apr 11, 2021 11:43 am

Running "make" in the Julia 1.6.0 source directory fails to build and gives this error:

Code: Select all

donald@raspberrypi:~/julia-1.6.0 $ cat log 
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
Warning: git information unavailable; versioning information limited
    LINK usr/lib/libjulia-internal.so.1.6
/usr/bin/ld: ./gc.o: in function `combine_thread_gc_counts':
/home/donald/julia-1.6.0/src/gc.c:1060: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1061: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1062: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1063: undefined reference to `__atomic_load_8'
/usr/bin/ld: /home/donald/julia-1.6.0/src/gc.c:1064: undefined reference to `__atomic_load_8'
/usr/bin/ld: ./gc.o:/home/donald/julia-1.6.0/src/gc.c:1065: more undefined references to `__atomic_load_8' follow
/usr/bin/ld: ./gc.o: in function `jl_gc_collect':
/home/donald/julia-1.6.0/src/gc.c:3212: undefined reference to `__atomic_fetch_add_8'
collect2: error: ld returned 1 exit status
make[1]: *** [Makefile:313: /home/donald/julia-1.6.0/usr/lib/libjulia-internal.so.1.6] Error 1
make: *** [Makefile:76: julia-src-release] Error 2
donald@raspberrypi:~/julia-1.6.0 $
Simply running "make" works fine on a 64-bit Intel processor, but I suspect it's naive of me to think this would work on the RPi3B+.

Heater
Posts: 18215
Joined: Tue Jul 17, 2012 3:02 pm

Re: Julia the Language

Sun Apr 11, 2021 2:01 pm

I got rid of that "atomic" error by jamming "-latomic" into the compile options.

I don't recall how I did it now though.

But then it failed with something else...
Memory in C++ is a leaky abstraction .

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Sun Apr 11, 2021 3:57 pm

Heater wrote:
Sun Apr 11, 2021 2:01 pm
I got rid of that "atomic" error by jamming "-latomic" into the compile options.

I don't recall how I did it now though.

But then it failed with something else...
I tried this as well by adding -latomic with my mouse, but the resulting binary causes a segmentation fault on both 32-bit Raspberry Pi OS and the glibc-based 32-bit ARM version of Void Linux. With musl-based Alpine Linux I now have a working installation that even runs on the Pi B+ and Zero computers.

Here is a B+ test run

Code: Select all

$ uname -a
Linux snail 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
$ grep model /proc/cpuinfo
model name  : ARMv6-compatible processor rev 7 (v6l)
$ singularity version
3.7.2
$ singularity run julia160.sif 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> n=500
500

julia> A=rand(n,n);

julia> x=zeros(n);

julia> b=A*x;

julia> @time A\b;
 45.932462 seconds (2.53 M allocations: 107.392 MiB, 3.16% gc time, 98.73% compilation time)

julia> @time A\b;
  0.535017 seconds (4 allocations: 1.913 MiB)

julia> @time A\b;
  0.532816 seconds (4 allocations: 1.913 MiB)
As 32-bit ARM is now a Tier 3 platform, I suspect some popular packages such as Plots will fail to add, but there is quite a bit one can do with the built-in packages.

Note that I built the container on a 2GB Pi 4B which used 200MB of swap during the build but didn't thrash and finished with

Code: Select all

make[1]: Leaving directory '/root/work/julia-1.6.0'
real    4h 54m 25s
user    14h 56m 04s
sys 1h 37m 40s
I suspect trying to build Julia directly on a Pi Zero with less RAM would either fail or start thrashing to the point the build takes months instead of hours.

I'll describe the procedure to build a Singularity container for Julia using Alpine Linux that runs under 32-bit Raspberry Pi OS shortly. Though I don't usually post links to binary builds, I'm wondering if anyone would have an interest in this container. Should I try to upload it to Singularity hub?

BoneFish
Posts: 3
Joined: Sat Apr 10, 2021 12:26 am

Re: Julia the Language

Sun Apr 11, 2021 10:54 pm

ejolson wrote:
Sun Apr 11, 2021 3:57 pm
I'll describe the procedure to build a Singularity container for Julia using Alpine Linux that runs under 32-bit Raspberry Pi OS shortly. Though I don't usually post links to binary builds, I'm wondering if anyone would have an interest in this container. Should I try to upload it to Singularity hub?

I don't have prior experience with Singularity containers but I would be very interested to hear the procedure!

User avatar
Gavinmc42
Posts: 5651
Joined: Wed Aug 28, 2013 3:31 am

Re: Julia the Language

Mon Apr 12, 2021 2:19 am

I use this on 32bit versions of the VulkanScenGraph libs.
"cmake . -DCMAKE_CXX_FLAGS=-latomic"
Think I used it for other "cmake ." stuff as well.
Gone back to 32bit Raspberry Pi OS, the 64bit version does not need that cmake option.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Mon Apr 12, 2021 3:23 am

BoneFish wrote:
Sun Apr 11, 2021 10:54 pm
ejolson wrote:
Sun Apr 11, 2021 3:57 pm
I'll describe the procedure to build a Singularity container for Julia using Alpine Linux that runs under 32-bit Raspberry Pi OS shortly. Though I don't usually post links to binary builds, I'm wondering if anyone would have an interest in this container. Should I try to upload it to Singularity hub?
I don't have prior experience with Singularity containers but I would be very interested to hear the procedure!
Singularity turns out to be very easy to install on a Raspberry Pi. This is because the Go language tool chain is well supported. I've already described the details at

viewtopic.php?p=1846641#p1846641

so I won't repeat them except for the observation that it takes less than 7 minutes to build Singularity from source on a Pi 4B.

After installing Singularity, one then needs a container with Julia in it. This presents a difficulty, because Julia is difficult to build and the current release includes no official 32-bit ARM executable. On a Pi 2B v1.2, 3B, 3B+ and 4B the best solution is to create a 64-bit container with the official 64-bit version of Julia and switch to the 64-bit kernel as discussed in

viewtopic.php?p=1849572#p1849572

However, this will not work for a B, B+, 2B v1.1 or Zero because the hardware is not capable of running 64-bit software and 64-bit emulation would be too slow.

My focus, therefore, has been to create a ARMv6 compatible Singularity container with Julia that runs on all models of Pi and all versions of 32-bit Raspberry Pi OS.

I did this using the 32-bit ARMv6 version of Alpine Linux as a base. This is the first time ever that I found building a large program was easier using the musl C library than with glibc. Since musl is targeted at small and embedded systems, my suspicion is that its 32-bit support hasn't rotted the way 32-bit support has in glibc.

After importing the standard docker image for Alpine Linux as described in

viewtopic.php?p=1847649#p1847649

which is only 5 MB, I bloated it up by installing gcc and dozens of other utilities needed to compile Julia. What I installed was based on the dependencies described at

https://github.com/JuliaLang/julia/blob ... d/build.md

though I had to add a few others that are not mentioned.

For reference the final list of installed packages was

Code: Select all

Singularity> apk info
musl
busybox
alpine-baselayout
alpine-keys
libcrypto1.1
libssl1.1
ca-certificates-bundle
libtls-standalone
ssl_client
zlib
apk-tools
scanelf
musl-utils
libc-utils
libgcc
libstdc++
binutils
libgomp
libatomic
libgphobos
gmp
isl22
mpfr4
mpc1
gcc
ncurses-terminfo-base
ncurses-libs
readline
bash
make
libacl
tar
xz-libs
xz
libgfortran
gfortran
musl-dev
libc-dev
patch
g++
m4
libbz2
perl
expat
libffi
gdbm
sqlite-libs
python2
python3
lz4-libs
zstd-libs
libarchive
ca-certificates
brotli-libs
nghttp2-libs
libcurl
rhash-libs
libuv
cmake
linux-headers
pcre2
git
perl-error
perl-git
git-perl
autoconf
automake
pkgconf
which
curl
gawk
bzip2
less
gzip
openssl
xxd
lua5.3-libs
vim
libattr
skalibs
s6-ipcserver
utmps
coreutils
Note that some of these packages were already in the base container and some came in as dependencies of the others. I have not bothered to keep track of the exact apk add commands used to configure the container.

As this post is getting long, the next post will describe the commands used to download and build Julia inside the resulting Alpine Singularity container.
Last edited by ejolson on Mon Apr 12, 2021 5:16 am, edited 5 times in total.

ejolson
Posts: 7236
Joined: Tue Mar 18, 2014 11:47 am

Re: Julia the Language

Mon Apr 12, 2021 3:51 am

ejolson wrote:
Mon Apr 12, 2021 3:23 am
the next post will describe the commands used to download and build Julia inside the resulting Alpine Singularity container.
Since Singularity containers run with user-level permissions it was expedient to run the container as root while building Julia. This is because, I needed to configure an entire Alpine Linux distribution inside the sandbox as mentioned in the previous post before building and installing Julia.

Assuming alpine is the name of the directory that holds the sandbox, type

Code: Select all

$ su -
# singularity shell -w alpine
to enter the container. Since the sandbox was entered as root, the home directory of the host system will be conveniently mounted under /root in the sandbox.

Now, download Julia and unpack it with

Code: Select all

Singularity> wget https://github.com/JuliaLang/julia/releases/download/v1.6.0/julia-1.6.0.tar.gz
Singularity> tar xf julia-1.6.0.tar.gz
Singularity> cd julia-1.6.0
Configure Julia by creating the Make.user in the julia-1.6.0 directory with

Code: Select all

JULIA_CPU_TARGET := arm1176jzf-s
MCPU := arm1176jzf-s
MARCH := armv6zk
OPENBLAS_TARGET_ARCH := ARMV6
USE_BINARYBUILDER := 0
OPENBLAS_USE_THREAD := 0
prefix = /usr/local/julia-1.6.0
Note that I have disabled threads in OpenBLAS since the ARMv6 based Pi B, B+ and Zero computers have only one core.

Next, patch some of the scripts used in the build system with

Code: Select all

Singularity> patch -p1 <julia160.patch
where the contents of julia160.patch are given by

Code: Select all

*** julia-1.6.0/base/binaryplatforms.jl 2021-03-24 20:02:52.000000000 +0000
--- julia-1.6.0-good/base/binaryplatforms.jl    2021-04-11 04:21:58.706732181 +0100
***************
*** 575,582 ****
      "x86_64" => "(x86_|amd)64",
      "i686" => "i\\d86",
      "aarch64" => "(aarch64|arm64)",
      "armv7l" => "arm(v7l)?", # if we just see `arm-linux-gnueabihf`, we assume it's `armv7l`
-     "armv6l" => "armv6l",
      "powerpc64le" => "p(ower)?pc64le",
  )
  # Keep this in sync with `CPUID.ISAs_by_family`
--- 575,582 ----
      "x86_64" => "(x86_|amd)64",
      "i686" => "i\\d86",
      "aarch64" => "(aarch64|arm64)",
+     "armv6l" => "armv6(l)?", # ejo--fix armv6-alpine-linux-musleabihf
      "armv7l" => "arm(v7l)?", # if we just see `arm-linux-gnueabihf`, we assume it's `armv7l`
      "powerpc64le" => "p(ower)?pc64le",
  )
  # Keep this in sync with `CPUID.ISAs_by_family`
*** julia-1.6.0/contrib/normalize_triplet.py    2021-03-24 20:02:52.000000000 +0000
--- julia-1.6.0-good/contrib/normalize_triplet.py   2021-04-11 04:26:04.669514370 +0100
***************
*** 13,18 ****
--- 13,19 ----
      'x86_64': '(x86_|amd)64',
      'i686': "i\\d86",
      'aarch64': "(arm|aarch)64",
+     'armv6l': "armv6(l)?",   # ejo--armv6-alpine-linux-musleabihf
      'armv7l': "arm(v7l)?",
      'powerpc64le': "p(ower)?pc64le",
  }
*** julia-1.6.0/Make.inc    2021-03-24 20:02:52.000000000 +0000
--- julia-1.6.0-good/Make.inc   2021-04-11 02:34:36.805354500 +0100
***************
*** 920,926 ****
  JCFLAGS += -fsigned-char
  USE_BLAS64:=0
  OPENBLAS_DYNAMIC_ARCH:=0
! OPENBLAS_TARGET_ARCH:=ARMV7
  endif
  
  # If we are running on aarch64 (e.g. ARMv8 or ARM64), set certain options automatically
--- 920,926 ----
  JCFLAGS += -fsigned-char
  USE_BLAS64:=0
  OPENBLAS_DYNAMIC_ARCH:=0
! OPENBLAS_TARGET_ARCH:=ARMV6
  endif
  
  # If we are running on aarch64 (e.g. ARMv8 or ARM64), set certain options automatically
This patch fixes three files needed for the ARMv6 build on Alpine Linux.

Upon patching, build Julia with

Code: Select all

Singularity> export VERBOSE=1
Singularity> export LDFLAGS=-Wl,-allow-multiple-definition
Singularity> nohup time make -j4 &
After about 5 hours the build should finish. At this point test the executable by running it from the source directory as

Code: Select all

Singularity> ./julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  
|__/                   |

julia> exit()
Assuming that went fine, install Julia into the sandbox with

Code: Select all

Singularity> make install
Singularity> cd /usr/local/bin
Singularity> ln -s ../julia-1.6.0/bin/* .
Set Julia to be the default run action by editing /singularity so it reads

Code: Select all

#!/bin/sh
exec /usr/local/bin/julia "$@"
Finally, exit the sandbox and build the container with

Code: Select all

Singularity> exit
# singularity build julia160.sif alpine
Note that no attempts have been made to reduce the size of the sandbox before converting it into a container. Thus, julia160.sif includes a full version of gcc, g++, gfortran and a slew of other things hidden inside. Mine turned out 240 MB.

The main reason I used a container for my build is because I wanted to learn about Singularity on the super-cheap cluster.

viewtopic.php?p=1247739#p1247739

Moving forward it would be nice if someone could figure out why native builds of Julia fail under the current version of 32-bit Raspberry Pi OS.
Last edited by ejolson on Tue Apr 13, 2021 6:55 pm, edited 1 time in total.

Return to “Other programming languages”