Comparing Interpreted Language Speed.

[DavidS]

Once again we are about to compare the speed of different Interpreted Programming Languages (NOT JIT Compiled Languages, only Interpreted Programming Languages).

Like last the last time we did this (back in 2012) this was provoked by a conversation in an unreleated thread, with some of the same people.

He provided source code for me to run on RISC OS under BBC BASIC for the first entry. Unfortunately it is going to take some time to get it running as it breaks the rule that all commands, statements, and directives in BBC BASIC must be upercase, as well as breaking the rules for line numbers when saved in text form. Would be nice if he provided a tokenized version (as BBC BASIC for SDL also uses the same tokenized source file format as BBC BASIC V).

Now the main point of the thread:
Lets see what completely interpreted language provides the fastest performance in a few differnt tests. We will compare them regardless of the operating System on which they run.

For test results running on BCM2835 based Raspberry Pi computers the CPU clock is expected to be 1000MHz, and the other clocks as default. Results on the RPi Zero must be specified so do to the different SDRAM and GPU clock settings.

For test results run on BCM2837 based Raspberry Pi computers the clocks are assumed to be the default settings for the particular Raspberry Pi.

So lets get to it. The challenges recommended are (this list will grow):

• Simple Fibonocci implemented using recursive algorithm (everything fits in unsigned 32-bit integers).
• Simple Fibonocci implemented using the fast algorithm (everything fits in unsigned 32-bit integers).
• Simple Prime Seive (again everything fitting in unsigned 32-bit integers).

Going to need to figure out graphics tests as well as GUI implementation speed tests for the interpreted languages.


For those that do not know, and are using a language a definition of a value that fits into a 32-bit unsigned integer is: An integer greater than or equal to 0 and less than 2 to the 32nd power, that is an integer whose value is n where n is:

Code: Select all

0 <= n < 4294967296

that is an integral value that is a natural number less than 4294967296.



Also Heater has me running an insane extremely large number Fibo test, so results for that in other interpreted languages are welcome.

EDIT 1: Corrected the error of omitting the word unsigned. Unsigned now specified where needed. Also added definition for 32-bit unsigned integer, for those that do not know and are working in a languge that provides for arbitrary percision.

[hippy]

The challenges recommended are (this list will grow):

• Simple Fibonocci implemented using recursive algorithm (everything fits in 32-bit integers).
• Simple Fibonocci implemented using the fast algorithm (everything fits in 32-bit integers).
• Simple Prime Seive (again everything fitting in 32-bit integers).

Can you define exactly what those algorithms are. And what do we do about interpreted languages where there's no real concept of "32-bit integers", where one may have no idea what the underlying interpreter may be using.

And presumably we will need some specified values to use with those algorithm.

[ejolson]

The challenges recommended are (this list will grow):

• Simple Fibonocci implemented using recursive algorithm (everything fits in 32-bit integers).
• Simple Fibonocci implemented using the fast algorithm (everything fits in 32-bit integers).
• Simple Prime Seive (again everything fitting in 32-bit integers).

Can you define exactly what those algorithms are. And what do we do about interpreted languages where there's no real concept of "32-bit integers", where one may have no idea what the underlying interpreter may be using.

And presumably we will need some specified values to use with those algorithm.

A simple prime sieve program written in C using the MIT/Cilk parallel programming extensions is
here. If you ignore the parallel procedure calls, adapting it to any other structured language should be straight forward. The same code, modified to work with OpenMP, also appears in the pichart benchmark of this Pi Pie Chart thread. I once wrote a faster sieve using wheel-factorisation and can look for that if anyone is interested.

A Fibonacci comparison of four Basic interpreters running on the Pi Zero appears here in the current continuation of the Why Avoid Basic thread. In summary, four interpreters were compared with the result that gplBasic (written by Lennart Benschop) was significantly faster. If you'll pardon me linking a copy of the graph here, the final results were

[Heater]

Why not just run that Fibo challenge entry by ejolson and Richard Russell that I gave you and tell us how it does?

Why all this dancing around and around with "32 bits" and "no JIT".

Just do it for goodness sake.

[Michiel O.]

Once again we are about to compare the speed ...

How would you define 'interpreted languages' versus 'compiled languages'?

Would you consider Java a 'compiled' language? How about C#?

[Heater]

Michiel O.

DavidS wants to only talk of interpreted languages.

But only one particular interpreted language.

The run time must not use Just In Time (JIT) compilation techniques.

But parsing into tokens is preferred. See above.

And only 32 bit numbers.

And it's essential they allow 32 bit ARM assembler instructions in their syntax.

In short, like everything else, if it is available on RISC OS it's good. Else there is a reason to reject it.

Java does not stand a chance here.

[John_Spikowski]

Is this a fibo(78) challenge for non-BIGINIT interpreters?

[Gavinmc42]

Until recently I had been ignoring the TCL/TK combo.
Too retro for me?
Turns out tcl is small enough to get embedded in small systems.

It also seem to be available on most systems.
Is it portable to RISC-OC?
TK the graphics part seems to get used by Python etc.

So Lua is in but Luajit is out?

[DavidS]

Once again we are about to compare the speed ...

How would you define 'interpreted languages' versus 'compiled languages'?

Would you consider Java a 'compiled' language? How about C#?

Ok to define the self explanitary:
An interpreted language is one for which the code is interpreted at run time in what ever source format it is stored (tokenized BASIC, text source, etc) without the source being modified (excepting possibly changing the source storage format), and without any machine language representation of the source being created.

JIT Compiled languae:
A JIT compiler produces chunks of machine language code at run time in order to increase the performance over an interpreted language.

Compiled language:
Duh.

[DavidS]

The challenges recommended are (this list will grow):

• Simple Fibonocci implemented using recursive algorithm (everything fits in 32-bit integers).
• Simple Fibonocci implemented using the fast algorithm (everything fits in 32-bit integers).
• Simple Prime Seive (again everything fitting in 32-bit integers).

Can you define exactly what those algorithms are. And what do we do about interpreted languages where there's no real concept of "32-bit integers", where one may have no idea what the underlying interpreter may be using.

And presumably we will need some specified values to use with those algorithm.

A simple prime sieve program written in C using the MIT/Cilk parallel programming extensions is
here. If you ignore the parallel procedure calls, adapting it to any other structured language should be straight forward. The same code, modified to work with OpenMP, also appears in the pichart benchmark of this Pi Pie Chart thread. I once wrote a faster sieve using wheel-factorisation and can look for that if anyone is interested.

A Fibonacci comparison of four Basic interpreters running on the Pi Zero appears here in the current continuation of the Why Avoid Basic thread. In summary, four interpreters were compared with the result that gplBasic (written by Lennart Benschop) was significantly faster. If you'll pardon me linking a copy of the graph here, the final results were

The two Fibonccci examples are traditionally run until the value produced is to large to be represented in the chosen storage format. In the case of this comparison that is beyond 2 to the 32nd power (4294967296). The algorithms are the standard entry level algorithms, will post after I finish another debate on this forum.

The Seive similarily is the standard Prime Sieve algorithm, to find all prime numbers less than 2 to the 32nd power (4294967296).

Grade school agorithms we all know. though I will add more specific descriptions as well.

[DavidS]

The challenges recommended are (this list will grow):

• Simple Fibonocci implemented using recursive algorithm (everything fits in 32-bit integers).
• Simple Fibonocci implemented using the fast algorithm (everything fits in 32-bit integers).
• Simple Prime Seive (again everything fitting in 32-bit integers).

Can you define exactly what those algorithms are. And what do we do about interpreted languages where there's no real concept of "32-bit integers", where one may have no idea what the underlying interpreter may be using.

And presumably we will need some specified values to use with those algorithm.

I will define the algorithms later today. Sorry I was a bit underprepaired with this thread. I have not even posted any other of the tests that I feel would be worthwhile in this comparison.

As to 32-bit unsigned integers, while some languages may not explicitly define such, the value range is well known, that is an integer whose value is n where n is:

Code: Select all

0 <= n < 4294967296

that is an integral value that is a natural number less than 4294967296.

Does that simplify it?

Also I do appologize for failing to mention UNSIGNED in the initial post. I will correct that now.

[Heater]

Did BBC BASIC on RISC OS get a result from Fibonacci Challenge code or is it still running?

https://github.com/ZiCog/fibo_4784969/b ... ic_bbc.bas

I am very curious to see how well the famously fast, hand coded in assembler and heavily optimized, RISC OS BBC BASIC engine performs vs the compiled from C interpreter we have from RichardRussell

The results of running the Fibonacci code above could help to finally put to rest the endless debate re: the performance of hand optimized assembler vs compiler generated code.

What is all this about the distinction between interpreters with JIT and those without?

Who on Earth would care? It reads the source, it does the work, it produces the result. Who cares if it JITs or not?

Why would anyone want to use a slow non-JITed solution when a fast JITed one is available.

[bensimmo]

Can you list the allowed interpreted languages. To many of them do fancy things other than read a line, run and instruction, read another and do that.
Many do a bit in between, python, basics, JavaScript are all a mixup iirc.
The world's too fuzzy to set limits.
32bit unsigned, is that a limit of your preferred language, why not go larger or hey go 16bit or let it choose by itself.

It's a bit more useful to be flexible and not restricted.

[Michiel O.]

To define the self explanitary: An interpreted language is one for which the code is interpreted at run time in what ever source format it is stored (tokenized BASIC, text source, etc) without the source being modified (excepting possibly changing the source storage format), and without any machine language representation of the source being created.

Ok, so if I understand you correctly, you define 'compiled' strictly as: 'compiled to machine code which is executed by the hardware CPU'. Everything else is 'interpreted'. In your view, C#, Java and Python are all interpreted languages, even if they have to go through a compilation step to produce intermediate code.

Now, what about Cython? If I put my Python program through Cython which transpiles it to a C program and then compiles it to an executable which is directly executed by the hardware CPU, does that make my Python program 'compiled' or 'interpreted'?

[Michiel O.]

Until recently I had been ignoring the TCL/TK combo. TK the graphics part seems to get used by Python etc.

Indeed. Did you know that Thonny, the Python IDE that is favoured by Raspbian, is written using TK? Thonny is a good read if you want to do some modern, tasteful GUI programming in Python.

[Heater]

Quite right Michiel O. nothing is so cut and dried.

What about when I uses LLVM to compile C++ into Javascript. Which is when run by a JS interpreter is JITed into native code?

Or compile C++ to WASM, which is a byte code, in turn interpreted by a WASM engine and also JITed.

What about when I run C++ from source using an interpreter: https://root.cern.ch/cling

And in what bizarro world is tokenized BASIC (Or anything else) source code? It's product of a lexing/or parsing step.

[hippy]

A simple prime sieve program written in C using the MIT/Cilk parallel programming extensions is here.

Thanks. It seems it would require far too much effort than I am prepared to put in.

I am not sure why such complicated algorithms are needed just for comparing respective speeds of interpretation. Simple test programs which just add numbers, concatenate strings, slice strings, access arrays, call subroutines, read and write files, should give something which allows a basic relative speed comparison to be made.

I can't see many others wanting to craft complicated programs when they probably don't particularly care if what they have chosen is faster than something else or not, though I am sure they would be interested in knowing where what they have chosen stands, what options there are.

It seems the simpler the tests the more likely people are to engage, will provide more results which should make comparisons more useful. The things I suggest to test are done by all programmers every day; very few use Fibonacci sequences, calculate primes, perform FFT's.

I personally wouldn't limit any comparison to just pure interpreters, would allow compilers, JIT interpreters, transpilers, and everything else. That seems to me to be the only way people can weigh-up what they might consider using.

As someone said, paraphrased; "C may be 100 times faster, but it's a 100 times more difficult to learn and use". Testing would show how much faster, and users can then individually judge whether that justifies whatever effort they may consider required to gain that speed advantage.

[Heater]

hippy,

Simple test programs which just add numbers, concatenate strings, slice strings, access arrays, call subroutines, read and write files, should give something which allows a basic relative speed comparison to be made.

Agreed.

...very few use Fibonacci sequences, calculate primes, perform FFT's.

Agreed.

However... If you look at the source code of such Fibonacci, prime and FFT calculators you will find that all they do is: add numbers, concatenate strings, slice strings, access arrays, call subroutines, etc. Just like any other program one might write. I mean, what else can you do in a language?

I personally wouldn't limit any comparison to just pure interpreters, would allow compilers, JIT interpreters, transpilers, and everything else. That seems to me to be the only way people can weigh-up what they might consider using.

Yes, exactly.

To answer my own question about that above. One would certainly consider using a non-JITed interpreter if it were intended to run on a machine with limited memory. For example the Espruino or JerryScript JS engines or MicroPython.

But as we are concerned with "grown up" computers here we are far from short of memory.

I don't understand the motivation for the no JIT restriction at all.

As someone said, paraphrased; "C may be 100 times faster, but it's a 100 times more difficult to learn and use".

I'm not convinced that is true.

As always, horses for course, choose the right tool for the job, jada, jada...

[Michiel O.]

nothing is so cut and dried.

Interpreted vs. compiled is very vague nowadays. It's not a useful metric for comparison. What's more interesting, I think, is dynamically typed vs. statically typed. By the way, that is something different than weak typing vs. strong typing.

[scruss]

Did BBC BASIC on RISC OS get a result from Fibonacci Challenge code or is it still running?

https://github.com/ZiCog/fibo_4784969/b ... ic_bbc.bas

It doesn't even run under BBC BASIC V on RISC OS as supplied. I get the following errors (after going through all the rigmarole of setting up RISC OS⁺):

• Doesn't seem to run on BBCSDL, Matrix Brandy or BBC BASIC V until everything's converted to upper case.
• Can't open FIBO.DAT — perhaps because of RISC OS's unusual file naming conventions?
• Out of range value assigned to HIMEM (I assume from himem = page + 20000000)
• Number too big at line 8550

I don't know enough about RISC OS to understand how to fix these. By removing the HIMEM line and fiddling with the -size option in Matrix Brandy I got it to start running, but I quit it after it had run for some minutes and done nothing.

---

⁺: while the RISC OS Open: Raspberry Pi distribution is quite well put together, you do need to assemble quite a lot of kit to get it running. Headless via wifi isn't happening for RISC OS.

• a Raspberry Pi with an Ethernet port.
• USB keyboard and three button mouse. You'll be using the middle one a lot, as it's the menu button.
• HDMI monitor
• ethernet cable
• SD/µSD card (2GB is plenty: only the first 2GB is usable by RISC OS)
• likely some way of cooling the CPU. RISC OS runs in a tight loop, so Raspberry Pis with thermal issues may throttle very quickly. I used a small desk fan pointed right at a Raspberry Pi 3 and it kept the CPU temperature below 40 °C.

Installation's the same as any other OS: use Etcher to write an image. RISC OS starts fairly quickly, and includes a browser that you'll use a lot to answer your many "how do I ___ in RISC OS?" questions. Some things I picked up on:

• To get to BASIC, hit F12 and the bottom of the screen scrolls up to reveal a console. Type BASIC at the * prompt, type QUIT to leave BASIC, and hit Return in the console to get the desktop back. *CAT inside BASIC gives a file listing. LISTO7 makes your listings prettier. There must be some way of changing directories, but I'm yet to find it.
• BBC BASIC is case sensitive. You will get the Mistake error a lot.
• RISC OS's desktop seems to be about the only way to move files about using drag and drop. There's no obvious shell with any familiar syntax.
• Files on the physical desktop seem to go away after a reboot. This is annoying if, like me, you store your current work files on the desktop.
• RISC OS uses explicit file typing to associate files with programs, just like Macs did before OS X. To get a text file into a form that BASIC will read via LOAD, drag the file onto the editor, then middle-click, choose Save As (possibly File → Save As) then choose the name and select the right file type. You can then load that file into BASIC, edit it and run it. (If you give it the right file type, you can also drag the icon that the menu presents to somewhere on the file manager and it'll save it there.)
• To shut down RISC OS, click on the big raspberry at the bottom right of the screen. In the window that opens up, middle click and select Shut Down.

RISC OS hard-locked on me for no reason when I was clicking around stuff. I hope it's more stable for you.

[Heater]

Michiel O.

What's more interesting, I think, is dynamically typed vs. statically typed. By the way, that is something different than weak typing vs. strong typing.

Yes indeed.

But oh boy, that is an entire other can of worms that could cause an endless debate...

[Heater]

scruss,

Wow, thanks for giving that a go. Shame it is being so intransigent.

I guess we could expect the ALL CAPS thing. Being rooted in the 1970's as we are here.

That "FIBO.DAT" should be "FIBO/DAT" or was it "FIBO\DAT" as DavidS explained here somewhere.

No idea about that number too big thing. That piece of code is supposed to be automatically finding the size of integers.

while the RISC OS Open: Raspberry Pi distribution is quite well put together, you do need to assemble quite a lot of kit to get it running. Headless via wifi isn't happening for RISC OS

I was considering getting a RISC OS up myself to try this, as DavidS seems to be so reluctant to, but that is all too much to take on here in the foreseeable future. I see now why David has gone so quite about it.

[hippy]

...very few use Fibonacci sequences, calculate primes, perform FFT's.

Agreed.

However... If you look at the source code of such Fibonacci, prime and FFT calculators you will find that all they do is: add numbers, concatenate strings, slice strings, access arrays, call subroutines, etc. Just like any other program one might write. I mean, what else can you do in a language?

That's true but it's also a whole lot more difficult to write, verify, debug and test. Most people won't have any clue as to how it works, any understanding of the algorithm used. That creates an obstacle to participation which simpler test programs don't have.

Plus, how representative is that of what people actually do ? That's true of any benchmark but it's often not even easy to say what a complicated algorithm's performance rests upon; is any lack of performance due to it being great for maths, great at routine calling, great for string manipulation, but rubbish at something else ? Is it useful if what it's rubbish at isn't important in other tasks ?

I can appreciate it gives an overall, combined or some average measure of performance, but that doesn't help anyone who tends to do one kind of programming. I tend to do a lot of file reading, writing and string processing, therefore have less care how slow something is at maths. For most people it's likely primarily one thing this time, some other thing the next.

Most of us gravitate to generally using one 'good on average' tool, but it would be nice to know which languages are best suited to particular types of tasks.

[Heater]

Agreed.

Using one benchmark is no good. Unless you know it is exercising things that you expect to be using a lot.

Better to have a collection of benchmarks.

You are right, if the benchmark is complex people may not be able to port it to the quirks of their run times.

I don't really much care about that. If the language systems are so different and unique that things need rewriting to run I'm not much interested in them. Give me standards like C/C++ or at least cross platform support like Python.

[bensimmo]

Benchmarks of turning LEDs on and off, how fast they can read square waves of the gpio.

Something a bit more useful to a Pi user?

Creating a rotatable object on screen?

Playing a bit of music?

None of this boring single task number crunching.
May as well just do some Seti@Home or other BOINC projects and make it worthwhile for someone?