ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Wed Aug 21, 2019 3:59 pm

scruss wrote:
Wed Aug 21, 2019 3:25 pm
Lor' lummy, are youse all still at this? Perl is Fast Enough for me, so I've not given it much more thought, except for:
  • it may be quicker to find anagrams of words all the same length, since by definition two words can only be anagrams of each other if they're the same length. Reading and classifying strings by length is a bit slow under the Unix paradigm as we don't have fixed-length records baked into the filesystem. Either way, you only have to hash far fewer words against one another even if you still need a master array for the final output order.
  • each of those comparisons of words of the same length might work quite well as separate threads.
Here's how that insane word list stacks up by word length:

Code: Select all

Length	Count	% of total
1	26	0%
2	240	0%
3	1910	0%
4	7349	2%
5	17257	4%
6	32498	8%
7	46596	11%
8	57535	13%
9	59453	14%
10	54659	13%
11	45499	11%
12	35410	8%
13	25569	6%
14	17592	4%
15	11264	3%
16	6855	2%
17	3939	1%
	… (longer words aren't significant as there are so few)
Speed isn't the primary objective, but to avoid the digital apocalypse by increasing computer literacy. In particular, with the insane British anagram challenge, any entry that obtains the correct answer wins.

I previously found 32-bit hash algorithms that are sufficient if the number of letters in the word were included as a secondary key. As it was easier to use 64-bit hashes, I never posted the code. It's possible using 32-bit hashes would make the existing C code faster when run on a 32-bit architecture; however, I was hesitant to give the zombies any more access to the encabulator.

ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Wed Aug 21, 2019 4:33 pm

hippy wrote:
Wed Aug 21, 2019 3:09 pm
"A Second Age of Personal Computing"
"A Final Fibonacci Challenge"
"Project Digital Apocalypse Not Now"
"Liberation through Computer Literacy"

Can I suggest the next change of thread title is to "Best Troll. Ever"

:lol:
There are still a number of scheduled titles coming up. In the sense of Tolkien, I do like the idea of a good troll. I wonder if your proposed title could be modified so it fit better within the forum topic "general programming chat and advice for beginners."

What do you think about "Programming Advice for the Best Troll Ever" as a coming title?

As always, thoughtful commentary and insight concerning how to liberate all the beings in Middle Earth from the oppression of digital feudalism is welcome.
Last edited by ejolson on Wed Aug 21, 2019 4:39 pm, edited 2 times in total.

User avatar
John_Spikowski
Posts: 1505
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Wed Aug 21, 2019 4:35 pm

hippy wrote:
Wed Aug 21, 2019 3:09 pm
"A Second Age of Personal Computing"
"A Final Fibonacci Challenge"
"Project Digital Apocalypse Not Now"
"Liberation through Computer Literacy"

Can I suggest the next change of thread title is to "Best Troll. Ever"

:lol:
How about 'Worthless Challenges Defined'?

User avatar
Paeryn
Posts: 2701
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: Liberation through Computer Literacy

Wed Aug 21, 2019 4:52 pm

John_Spikowski wrote:
Wed Aug 21, 2019 2:06 pm
The word list is already sorted. If you use a hash method properly the only sorting going on is the dictionary word to build the hash key.

I think the ScriptBasic approach is the most efficient but using associative arrays rather than the hash extension module is its downfall.
I already know how to do the mapping (Kira's code is very fast), the Haskell code was a quick (in terms of writing it) implementation that performs each stage separately.

I admit I was surprised when I saw how much memory it took, I thought it would be able to throw away the old lists once a stage had finished with them (as each list is only needed to link stages) but Haskell is lazy in that it doesn't evaluate anything until it is used and for some reason it is holding on to all those temporary lists created during the sort even though after there isn't (shouldn't) be anything holding references to them, something must be though.
She who travels light — forgot something.

ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Thu Aug 22, 2019 2:39 am

ejolson wrote:
Wed Aug 21, 2019 3:58 am
The canine coder said with a growl, de-mouser is just a clever way of saying cat and I'm not logging in as a cat. Consequently, there have been some delays with the 64-bit testing.
I ran some of the insane British anagram challenge winners in 64-bit mode on the Raspberry Pi 4B using this Gentoo image. The results are

Image

Numbers greater than 1 mean the program ran faster in 64-bit mode; less than 1 mean the program ran slower.

Unfortunately, the version of Node.js was too old to run iba-hash.js and the hashbrown crate was missing in Rust for iba-main.rs. Moreover, I didn't want to clean potatoes from the flux ports of the turbo encabulator again. Note also that I've not installed 8th or any of the Basic compilers and interpreters.

It is interesting that Python3 is slower on the Gentoo image compared to Raspbian. It would be 64-bit liberating if someone has the computer literacy to fix that.
Last edited by ejolson on Thu Aug 22, 2019 4:34 am, edited 1 time in total.

User avatar
Gavinmc42
Posts: 3890
Joined: Wed Aug 28, 2013 3:31 am

Re: Liberation through Computer Literacy

Thu Aug 22, 2019 4:13 am

Apart from Python, 64bit is about 10% faster.
Are they all A72 optimized compilers?
What speed difference between A53/A72/?

I would have expected a bit better than 10% with the extra registers available in 64bit mode.
Rust is looking good :D
Not bad for someone still learning Rust :lol:

Would a D version be better than C?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Thu Aug 22, 2019 4:39 am

Gavinmc42 wrote:
Thu Aug 22, 2019 4:13 am
Apart from Python, 64bit is about 10% faster.
Are they all A72 optimized compilers?
What speed difference between A53/A72/?

I would have expected a bit better than 10% with the extra registers available in 64bit mode.
Rust is looking good :D
Not bad for someone still learning Rust :lol:

Would a D version be better than C?
I believe the C and C++ codes were compiled on the respective platforms using reasonable tuning and optimization settings. I'm not so sure about Rust as I didn't explicitly specify A72 or A53 architectures in any of the compilations. The Perl and Python interpreters are whatever versions came included with the respective operating systems. That makes the low performance of Python3 on Gentoo quite mysterious. Since Gentoo is a source-based distribution, one would have expected the Python interpreter to be well tuned for the Pi 4B and faster.

User avatar
scruss
Posts: 2546
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Liberation through Computer Literacy

Thu Aug 22, 2019 2:03 pm

At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.

Gentoo isn't always the newest, fastest or best-maintained. It's quite often the crashiest distro, especially if the user doesn't know what they are doing. Building everything with -Ol33t_h4kz0r can be counterproductive, unless spending your time building and rebuilding packages is what you actually want to do with your computer.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Thu Aug 22, 2019 3:05 pm

scruss wrote:
Thu Aug 22, 2019 2:03 pm
At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.
So far the prebuilt Gentoo image appears complete, functional and crash free. That is a good point about Python versions. Does anyone know how to fix that on the Gentoo image?

The bar chart isn't meant to compare optimal performance of 64-bit to 32-bit mode, but rather what a beginning programmer might encounter using the bootable images easily available. Given the recent improvements, it would be nice to compare the same versions of the Python interpreters. I understand there is a now a 64-bit version of Manjaro Linux that is likely more up to date. However, as this thread is currently about the freedom which comes from being able to read and write computer programs, I would encourage speed comparisons of the different operating system images be performed elsewhere.

More pressing from a computer literacy point of view is whether the recently provided insane Haskell anagram finder will run on the 4GB Pi 4B and whether Haskell, by it's lazy functional nature, simply uses to much memory to be digitally liberating for users of previous Raspberry Pi models.

User avatar
John_Spikowski
Posts: 1505
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Thu Aug 22, 2019 3:36 pm

Another candidate for the mega memory class programming environment.

Memory is cheap if you have access to it.

User avatar
sakaki
Posts: 373
Joined: Sun Jul 16, 2017 1:11 pm

Re: Liberation through Computer Literacy

Thu Aug 22, 2019 5:26 pm

ejolson wrote:
Thu Aug 22, 2019 3:05 pm
scruss wrote:
Thu Aug 22, 2019 2:03 pm
At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.
So far the prebuilt Gentoo image appears complete, functional and crash free. That is a good point about Python versions. Does anyone know how to fix that on the Gentoo image?
On the 64-bit 1.5.0 Gentoo image, python 3 is v3.6.9. On Debian Buster it appears to be v3.7.3, which could well explain the difference in timings.

v3.7.4 is in the standard Gentoo ebuild tree, so I'm building it on the binhost now. I'll post again once that's done, then it should be a simple matter to install it (from the resulting binary package) and make it the default v3.

Most of the stuff on the image is pretty current as regards version, so not sure why I had python 3.7 masked back - probably an issue in early testing, most likely resolved upstream now.

best, sakaki

User avatar
Paeryn
Posts: 2701
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 12:50 am

I've improved the memory usage of my Haskell code, the culprit was the amount of ByteStrings it generated. The overheads in using them for small strings is quite big, they require 9 words + length of string each, that's 36 bytes extra per word (72 on 64-bit) and due to how they are allocated the GC can't move them to compact the heap. I've switched to normal Strings which are literally a linked list of Char, they require (1+3*length) words + 2*length bytes (Chars are UTF-16) but the GC can move them around. It's a bit slower (6.8s vs 4s) on my PC but at least the RPi can comfortably run it, it only uses up 166 MB. My RPi3 does it in 24.6s, though half that time is spent in the GC.

Code: Select all

import qualified Data.List
import qualified Data.Map.Strict as Map
import Data.Char (isAsciiLower)

printRest :: [String] -> IO ()
printRest []     = return ()
printRest [x]    = putStrLn x
printRest (x:xs) = do
  putStr x
  putStr ", "
  printRest xs

printAnagram :: [String] -> IO ()
printAnagram []     = return ()
printAnagram (x:xs) = do
  putStr x
  putStr ": "
  printRest xs

main :: IO ()
main = do
  f <- readFile "/usr/share/dict/british-english-insane"
  let wordList = filter (all isAsciiLower) (lines f)
  let hashWords = [(hw, [w]) | w <- wordList, let hw = Data.List.sort w]
  let anagrams = Map.elems $ Map.fromListWith (\ x y -> y ++ x) hashWords
  let nonUniqueWords = Data.List.filter ((> 1) . length) anagrams
  mapM_ printAnagram $ Data.List.sortOn head nonUniqueWords

Code: Select all

[email protected]:~/Programming/banagrams $ ./haskell_anagrams >insane-h +RTS -s
   1,370,853,644 bytes allocated in the heap
     550,927,724 bytes copied during GC
      76,971,508 bytes maximum residency (10 sample(s))
         111,116 bytes maximum slop
             164 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      1303 colls,     0 par    8.470s   8.476s     0.0065s    0.0096s
  Gen  1        10 colls,     0 par    3.635s   3.639s     0.3639s    1.5546s

  INIT    time    0.001s  (  0.001s elapsed)
  MUT     time   12.474s  ( 12.488s elapsed)
  GC      time   12.106s  ( 12.116s elapsed)
  EXIT    time    0.000s  (  0.000s elapsed)
  Total   time   24.581s  ( 24.605s elapsed)

  %GC     time      49.2%  (49.2% elapsed)

  Alloc rate    109,893,198 bytes per MUT second

  Productivity  50.7% of total user, 50.8% of total elapsed
She who travels light — forgot something.

ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 4:24 am

Paeryn wrote:
Sat Aug 24, 2019 12:50 am
It's a bit slower (6.8s vs 4s) on my PC but at least the RPi can comfortably run it, it only uses up 166 MB. My RPi3 does it in 24.6s, though half that time is spent in the GC.
It appears liberating after all for a Raspberry Pi owner to be literate in reading and writing Haskell programs. Running on the Raspberry Pi 4B obtains

Code: Select all

$ ghc -O3 haskgram.hs
$ time ./haskgram >haskgram.insane

real    0m10.600s
user    0m10.297s
sys 0m0.302s
$ 
$ time ./haskgram >haskgram.insane

real    0m10.589s
user    0m10.264s
sys 0m0.322s
$ time ./haskgram >haskgram.insane

real    0m10.603s
user    0m10.251s
sys 0m0.352s
$ time perl selfgrams4.pl >selfgrams4.insane

real    0m5.454s
user    0m5.232s
sys 0m0.221s
$ md5sum *.insane
addeda36a4631afc983f3bff13742e36  haskgram.insane
addeda36a4631afc983f3bff13742e36  selfgrams4.insane
The insane British anagram bar chart has been updated and now looks like

Image

Unfortunately, not all seems well with my turbo encabulator today: I tried to connect it to a gigabit switch, but it won't connect. The network lights don't even turn on.

Could one of the hydrocoptic marzlevanes be loose? Would that cause the waneshaft to fumble?

I tried three different cables known to work at gigabit speeds with other machines. After repeatedly unplugging and plugging the cables it connected once or twice, but then lost connection again when I rebooted. I finally went back to fast Ethernet.

User avatar
davidcoton
Posts: 4184
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 1:44 pm

Urgent programming challenge.

Submit code for "teboot" and the companion program "coffeboot".
Marks for adherence to @DougieLawson's spec and imagination beyond that.
Fido and Kira may develop additional variations.

Deadline for submissions:
  • teboot -- 4pm
  • coffeboot -- 11am
Signature retired

User avatar
scruss
Posts: 2546
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 5:11 pm

but tea is taken at 3 and coffee at 11 anywhere civilized. Also, Dougie's process is incorrect (by definition, the way everyone else makes tea is incorrect) and excludes critical points of doctrine such as prelactarianism vs - gasp! - postlactarianism, sugar shunning, spoon heft, and the proper observance of biscuitry.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

User avatar
DougieLawson
Posts: 36308
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 5:15 pm

You must know that pre-lactarianism is the first stage of a debilitating and ultimately fatal disease (you will be dead by age 110 if you continue on that course). Post-lactarianism is the one true way (and the only way that's tolerated in Yorkshire).

Dunking of biscuits is clearly documented by Philip Schofield. https://www.youtube.com/watch?v=L0Hf_zwJpm0
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

User avatar
Paeryn
Posts: 2701
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 8:09 pm

Kira says that tea and coffee is overrated and that pre-lactarinism is the one true way so that he can drink the milk before it is tainted by anything afterwards (and it had better be lactose free).

Personally I'm post, if there's too much milk I can't drink my tea.
She who travels light — forgot something.

ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 8:22 pm

Paeryn wrote:
Sat Aug 24, 2019 8:09 pm
Kira says that tea and coffee is overrated and that pre-lactarinism is the one true way so that he can drink the milk before it is tainted by anything afterwards (and it had better be lactose free).
I've never heard of lactose-free pre-lactarianism. The lead developer of FidoBasic prefers the biscuits.

The Pi 4B still refuses to connect to my gigabit switch. More details are here, if anyone is interested.

On a similar note, I spent yesterday helping move two servers that were noticeably heavier than Raspberry Pi only to discover the networking didn't work at the new location because the required IP numbers hadn't been requisitioned yet. How can one have liberation through computer literacy if faulty networking prevents accessing the computer?

User avatar
DougieLawson
Posts: 36308
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sat Aug 24, 2019 8:32 pm

That brings another thought to mind. Never trust a computer you can lift - regardless of its tea making abilities.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

User avatar
John_Spikowski
Posts: 1505
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sun Aug 25, 2019 1:43 am

Great to see ScriptBasic at the top of the charts.

👅

User avatar
rpdom
Posts: 15371
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Liberation through Computer Literacy

Sun Aug 25, 2019 6:48 am

DougieLawson wrote:
Sat Aug 24, 2019 8:32 pm
That brings another thought to mind. Never trust a computer you can lift - regardless of its tea making abilities.
I couldn't lift my old Unix box. It weighed around 130Kg (280lbs). I had to dismantle it every time I need to move it to a different floor. It was fine on the same floor as it was on rollers. Fortunately it was fairly modular and easy to strip down and rebuild with just two screwdrivers. Three 5.25" hard disks, one of them full height (about 3GB total storage). Two tape drives (150MB QIC and 2.3GB Exabyte). Two battery modules with 4 6Ah 12V batteries in each. Two redundant PSUs. One fan tray. One CPU card (68040 @33MHz). Two memory cards (240MB total). One serial card (with 16 micro-DB9 connectors). One network card (10Mb RJ45 and 2Mb? AUI). Some assorted metal work and covers. Then it was light enough to lift a little with some help.

List price for that spec when new would have been £75K, but it was not available as a standard option. I paid £0 for it as it was going to be scrapped. It was noisy and used a lot of power, but It was cool to be able to practice Unix stuff at home and start compiling in C. I compiled a webserver for it. Due to the ancient OS it was quite a task and ended up only supporting a single session. No threading on that system. Even some of the standard libc routines were missing and I had to hack some together to make it work. It never went on the public internet. Just as well, as the security was probably terrible, but I could access it from my 486 laptop.

ejolson
Posts: 3682
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Thu Aug 29, 2019 2:22 am

sakaki wrote:
Thu Aug 22, 2019 5:26 pm
ejolson wrote:
Thu Aug 22, 2019 3:05 pm
scruss wrote:
Thu Aug 22, 2019 2:03 pm
At least one of the Python 3 versions uses a completely different optimized library based on the installed Python version. Buster on a Raspberry Pi 4 uses the faster library; maybe the Gentoo version doesn't.
So far the prebuilt Gentoo image appears complete, functional and crash free. That is a good point about Python versions. Does anyone know how to fix that on the Gentoo image?
v3.7.4 is in the standard Gentoo ebuild tree, so I'm building it on the binhost now. I'll post again once that's done, then it should be a simple matter to install it (from the resulting binary package) and make it the default v3.
Did Python version 3.7.4 build? If so, how to I update my Gentoo image to include it?

User avatar
Gavinmc42
Posts: 3890
Joined: Wed Aug 28, 2013 3:31 am

Re: Liberation through Computer Literacy

Thu Aug 29, 2019 5:15 am

Did Python version 3.7.4 build?
I have this bookmarked
https://isshoni.org/pi64pie/
Looks like 3.7.4.-R1/2 is there in dev/lang. release candidates 1 and 2?

I tried a sudo genup and it posted an error

Code: Select all

Local copy of remote index is up-to-date and will be used.
Calculating dependencies... done!
[binary   R    ] dev-lang/python-exec-2.4.6-r1  PYTHON_TARGETS="(python3_7*)" 
[binary  NS    ] dev-lang/python-3.7.4-r2 [2.7.16, 3.5.7, 3.6.9] USE="pgo%*" 
[binary     U  ] dev-python/certifi-2019.6.16 [2018.11.29] PYTHON_TARGETS="python3_7*" 
[binary     U  ] dev-python/setuptools-41.2.0 [41.0.1] PYTHON_TARGETS="python3_7*" 
[binary   R    ] app-portage/gemato-14.1  PYTHON_TARGETS="python3_7*" 
[binary     U  ] sys-apps/portage-2.3.72 [2.3.71] PYTHON_TARGETS="python3_7*" 
[blocks B      ] <app-portage/gentoolkit-0.4.6 ("<app-portage/gentoolkit-0.4.6" is blocking sys-apps/portage-2.3.72)

 * Error: The above package list contains packages which cannot be
 * installed at the same time on the same system.

I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
sakaki
Posts: 373
Joined: Sun Jul 16, 2017 1:11 pm

Re: Liberation through Computer Literacy

Thu Aug 29, 2019 8:45 am

ejolson wrote:
Thu Aug 29, 2019 2:22 am
Did Python version 3.7.4 build? If so, how to I update my Gentoo image to include it?
Yes, I now have 3.7.3 and 3.7.4 built with pgo etc. enabled. Turned out to be more involved than I thought: 3.7 was masked upstream for arm64, so unmasking it caused the slot to be added to PYTHON_TARGETS which then caused rebuilds for everything exporting python APIs etc...
Gavinmc42 wrote:
Thu Aug 29, 2019 5:15 am
I tried a sudo genup and it posted an error
There's another step required to upgrade from v1.5.0 to v1.5.1 (which includes not only the updated python version, but also updated kernels & boot firmware, and fixes for various reported issues (glitchy audio, wifi connection problems, monitor autodetection regression etc.)).

Hoping to ship the updated image by the end of this weekend (real-world NMIs permitting), and will also post a link to the announce post, which will include v1.5.0 -> v1.5.1 upgrade instructions, here when that's done.

Apologies for the delay ><

Best, sakaki

User avatar
Gavinmc42
Posts: 3890
Joined: Wed Aug 28, 2013 3:31 am

Re: Liberation through Computer Literacy

Thu Aug 29, 2019 9:57 am

Apologies for the delay ><
No need to apologise Sakaki, we should apologise for finding bugs in your great release ;)
It is only teething troubles and we know this is a spare time thing for you.

Has it been only two months since the Pi4 has been released?
That's about 4 months ahead of my expectations of a 64bit OS :D
Gentoo64 in another 4 months?
I expect by then it will be able to do nearly ALL my PI developments.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Return to “General programming discussion”