User avatar
bensimmo
Posts: 4621
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Liberation through Computer Literacy

Tue Oct 22, 2019 6:19 pm

Heater wrote:
Tue Oct 22, 2019 12:50 am
bensimmo,
I think you are saying something like we do for out air quality project (PM2.5 sensors).

We have result in an SQL (or similar) database, they all have their GNSS location.
I have my Pi with GNSS (or phone, or Alexa) and ask it to give me the current reading.
I can grab the nearest by actual distance from my current point.
I can also grab the nearest 5 and take an average, or remove any that haven't sent data back for a while.

I first did this in Python3, cheap and easy as we don't have that many points to loop through at the moment but moved it to the server in some language (php/SQL?).

Scale it up to 1000s of point and the Python3 would be slow probably.
Yes, the challenge is kind of inspired by that geographical problem.

Rather than loop through thousands of points stored in a database why not let the database do the work:

Code: Select all

SELECT * FROM Places WHERE
    (Lat => 1.2393 AND Lat <= 1.5532) AND (Lon >= -1.8184 AND Lon <= 0.4221)
AND
    acos(sin(1.3963) * sin(Lat) + cos(1.3963) * cos(Lat) * cos(Lon - (-0.6981))) <= 0.1570;
See nice article about this here: "Finding Points Within a Distance of a Latitude/Longitude Using Bounding Coordinates" - http://janmatuschek.de/LatitudeLongitud ... SQLQueries

How fast it works will depend on your database of course. But I suspect it would be hugely faster than any home made looping through thousands of points in Python.

That sounds likes an interesting project you have there.

It wasn't slow as such, but having to transfer every point from the database to my device was not the best plan.
It is all done on the SQL database now with a call. Not sure which we are suing, that not my side of things, I think it is Maria but may be switched to something else.

I believe we now use the Spacial funtion provided ('Point' rings a bell and spherical distance, it was great working it out from my suggestion.

Interestingly, or not, it was so I could take a PiZero/Pimoroni ScrollBot with a GPS attached and show peoppe their local PM2.5 count from our sensors (or DEFRA or the Councils etc) to enthuse anyone that would listen ;-) School being the initial target.

At least that was the plan, just need to get time and a bit of money to persue partII. Getting LoRaWAN/TTN setup around the city has taken over for a bit. Not to bad for what is basically a community interest group with no money :lol:
The main part isn't to bleat on about particle counts and climate health though as that far more complicated that putting a load of sensors up and telling them a number.
But to make them think, what could they do with a free to use slow but Long Distance 'WiFi' network or just fast but short range WiFi and a few relatively cheap ESP/Ardunio/Pi devices and sensors spread around.
The only stipulation so far is all data will be stored as open-data for anyone to use, analyse and make use of.

Hopefully all the UK's money pumped into Primary and Secondard school computing will give them the skills and encouragment to try thing for themselves.

User avatar
John_Spikowski
Posts: 1614
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sat Oct 26, 2019 6:42 pm

THIS might be of interest to commercial app developers.

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sat Oct 26, 2019 7:38 pm

In my continuing search for heaven, https://www.raspberrypi.org/forums/view ... 0#p1556604, I have been assembling my collection of interesting coordinates in space.

I downloaded the US Navy Observatory CCD Astrograph Catalog (UCAC): https://www.usno.navy.mil/USNO/astromet ... -prod/ucac from the mirror given here: https://www.ap-i.net/skychart/pt/news/ucac4_catalog

There is some software in that archive to read, decode, filter that data set in Fortran, C and Python. After installing the GNU Fortran compiler I new have 113 million star records in CSV format. Nearly 20GB of it!

But damn it, now I have the irresistible urge to start plotting them some how....
Memory in C++ is a leaky abstraction .

ejolson
Posts: 5373
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Sat Oct 26, 2019 10:21 pm

Heater wrote:
Sat Oct 26, 2019 7:38 pm
now I have the irresistible urge to start plotting them some how....
What? I thought you were going to search for 5 of the nearest 10 stars to any point in outer space.

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sat Oct 26, 2019 10:40 pm

Err, yeah, that.

I originally envisioned this little challenge as just an abstract problem. A flat space of some number of grid points, a bunch of random points occupied, find the nearest occupied points to some other randomly chosen point.

The star positions only came in by way of spicing things up with some real data that looks kind of random. Then I got side tracked looking at what these astronomers have been up to in recent years...

I'm slowly getting around to pulling out a subset of that huge star position data set as our test data. It's amazing how crowded the galaxy is!

I'm going to be brutal and just map those declinations and ascensions to points on a flat grid.
Memory in C++ is a leaky abstraction .

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 3:28 am

Curious.
Memory in C++ is a leaky abstraction .

User avatar
Gavinmc42
Posts: 4526
Joined: Wed Aug 28, 2013 3:31 am

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 7:46 am

Ooh, 113 million coordinates to project onto a flat screen. I've got just the thing to do that with!
Space junk monitoring or asteroid mining?

I was impressed when Grass GIS and QGIS just worked on a Pi4.
Trying to learn OpenStreet map stuff so PM2.5 could be plotted.
I have some PM2.5 sensors and Lora feathers I want to connect to the Council LoRa network
Pi's at schools would be a good place to put them.

But Open Data and Open Gov and Open Education?
Might be easier to make my own LoRa network :lol:
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 9:46 am

Here in Helsinki they are really into the "smart city" idea and encourage citizen participation. https://helsinkismart.fi/citizen-city/

One neat project was exactly getting air quality sensors spread around the place a few years back. They tackled it in an interesting way. Basically Helsinki resident's were offered an air sensor IoT kit for a few euros, but they had to build it and get it working themselves. To that end a kind of maker space was provided where you could go and build it and learn from the guys there about the basic electronics, soldering and assembly required. From talking to the guys who ran it it seems it was very popular.

They just opened a huge and expensive new library building in the middle of the city. To my mind it's a bit short of actual books, but it has a huge area where you can go and use 3d printers, laser cutters, all kind of things. Including a proper electronics lab where you can get on with soldering and make use of scopes and such. Amazing.
https://helsinkismart.fi/portfolio-item ... ioCats=248
Memory in C++ is a leaky abstraction .

User avatar
John_Spikowski
Posts: 1614
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 9:50 am

jcyr wrote:
Sun Oct 27, 2019 12:09 am
Ooh, 113 million coordinates to project onto a flat screen. I've got just the thing to do that with!
It looks like we are back on track with absurd.

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 11:08 am

John_Spikowski,

Could you elaborate on your definition of "absurd"?

As far as I can tell it includes anything that is:
  • Just for fun.
  • Done out of simple curiosity.
  • Educational
  • Of no immediate practical use, despite it's potential for such.
In this case my simple challenge idea has inadvertently led me to researching star catalogs and finding out what astronomers get up to. All very interesting. You may think it's absurd but you would have to argue that case with the US Navy Observatory who spent ten years on the observations that went into the data set I'm looking at.

Perhaps we should just fire all those absurd, time and money wasting astronomers, cosmologists, physicists, mathematicians and other useless scientific so called "researchers".

The common theme I notice is that "absurd" seems to be anything ScriptBasic can't do very well :)
Memory in C++ is a leaky abstraction .

User avatar
John_Spikowski
Posts: 1614
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 12:00 pm

The challenge started off finding objects in a grid and evolved into mapping the universe. My initial comment was lets focus on challenges that result in code that can be reused.

The ScriptBasic project stands on its own and doesn't require my contributions to be viable.

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 1:19 pm

Ah, right. Do not worry John.

Through the awesome powers of abstraction I imagine the basic implementation of finding objects on a grid will be there. Ready to be used for whatever other purpose.

The only little twist here is where I'm proposing to get those object positions from. Might as use those stars hanging around in the galaxy and not doing much except looking pretty.

All that is needed is to map a subset of all those ascensions and declinations, in micro arc seconds, into integer grid points. On whatever size grid we end up with.

Perhaps the most expensive source of random numbers I have ever used!
Memory in C++ is a leaky abstraction .

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 5:26 pm

The story so far...

The fourth U.S. Naval Observatory CCD Astrograph Catalog (UCAC4) contains 900 files of star data in binary. Totaling 8.3GB.

There is a python script to fix up a few errors in there. Done that.

There is a dump program included, in FORTRAN, that reads the binary files and spits out ASCII files, one star record per line. I managed to build that with GNU Fortran so now I have 30GB of CSV files!

These files are named z001.asc to z900.asc. That's 900 declination zones of 0.2 degree width, with zone number 1 starting at the South Pole and increasing toward north.

I'm tempted to stuff them all into my Cockroach SQL database to see how well it copes. From there I could select a nice busy patch of sky for our challenge data. Sort of equatorialish and looking to the middle of the galaxy.
Memory in C++ is a leaky abstraction .

User avatar
John_Spikowski
Posts: 1614
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 6:56 pm

looking to the middle of the galaxy.
You will find that there is no end to the universe or was there a beginning. Get your arms around that.

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 9:19 pm

jcyr,
At least 10^6 points, I hope. It's gotta be challenging after all!
I have been thinking about that...

The data set has to be big enough to be an interesting challenge.

On the other hand I thought it would be nice to be small enough to plot in an image to see, a grid of 2048 by 2048 for example. 4 million points about.

But heck the UCAC4 star positions are only 32 bit integers in each direction. There is only about 113 million points. Why not go for the whole hog? A grid of 4 billion by 4 billion!

That should sort the men from the boys.
Memory in C++ is a leaky abstraction .

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Sun Oct 27, 2019 9:28 pm

John_Spikowski,
You will find that there is no end to the universe or was there a beginning.
How will I find that?

Not that I was even looking.

What does it even mean?

Should I cancel my dinner date at Millyways?
Memory in C++ is a leaky abstraction .

User avatar
Gavinmc42
Posts: 4526
Joined: Wed Aug 28, 2013 3:31 am

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 2:32 am

I seem to remember early drone/submarine/aircraft navigation used star positions for position fixing.
Pre GPS but still using computers to check the stars.
That would be interesting to do.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
rpdom
Posts: 17170
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 8:33 am

Heater wrote:
Sun Oct 27, 2019 9:28 pm
Should I cancel my dinner date at Millyways?
Did you go to the Big Bang Burger Bar?
Unreadable squiggle

User avatar
John_Spikowski
Posts: 1614
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 3:07 pm

The 'big bang' was nothing more than a universe fart. That theory is rapidly losing credibility.

Parallel Universe
Last edited by John_Spikowski on Mon Oct 28, 2019 4:42 pm, edited 1 time in total.

User avatar
PeterO
Posts: 5877
Joined: Sun Jul 22, 2012 4:14 pm

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 3:15 pm

Gavinmc42 wrote:
Mon Oct 28, 2019 2:32 am
I seem to remember early drone/submarine/aircraft navigation used star positions for position fixing.
Pre GPS but still using computers to check the stars.
That would be interesting to do.
https://en.wikipedia.org/wiki/Celestial_navigation
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

ejolson
Posts: 5373
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 5:35 pm

Heater wrote:
Sun Oct 27, 2019 9:46 am
To my mind it's a bit short of actual books, but it has a huge area where you can go and use 3d printers, laser cutters, all kind of things. Including a proper electronics lab where you can get on with soldering and make use of scopes and such.
Here on the liberal frontier the library was renamed as the knowledge center--all books were removed and placed on Mars in hidden shelves that only robots can access. Note that printed books are difficult to update or delete when the information contained on their pages goes out of fashion. Moreover, burning is also no longer an option, because many of the oppressive governments around the world have gone green.

Back on topic, I was talking to the lead developer of FidoBasic who told me that K9 had recently requested an extension library to search all of eternity for five of the ten nearest interesting events to a given point in 4-dimensional spacetime. Apparently business has been slow, so Dr Who is forming a partnership with Dr Emmett Brown to work as travel guides.

When I asked if two-dimensional projections of three-dimensional star charts would be enough, Fido became barking mad and growled, that's absurd, what would be a practical use for that?

Heater
Posts: 15948
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 7:37 pm

Fear not Fido. The star dogalog only has two dimensional positions on the celestial sphere.
Memory in C++ is a leaky abstraction .

User avatar
rpdom
Posts: 17170
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 8:07 pm

ejolson wrote:
Mon Oct 28, 2019 5:35 pm
Note that printed books are difficult to update or delete when the information contained on their pages goes out of fashion. Moreover, burning is also no longer an option, because many of the oppressive governments around the world have gone green.
I thought it was because no one in the civilised world knows how to set their microwave to Fahrenheit 451 as they are all marked in Celsius. :lol:
Unreadable squiggle

User avatar
Paeryn
Posts: 2966
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: Liberation through Computer Literacy

Mon Oct 28, 2019 8:22 pm

ejolson wrote:
Mon Oct 28, 2019 5:35 pm
When I asked if two-dimensional projections of three-dimensional star charts would be enough, Fido became barking mad and growled, that's absurd, what would be a practical use for that?
Kira miaowed at Fido's remark and with a gentle purr whispered "The Monarch of Pointland thinks that all stars must be figments of its own imagination. Since It is the whole universe and the only thing It can see (and having no concept of distance), all these whimsical stars must therefore be located at the exact same (and only) point."
She who travels light — forgot something.

ejolson
Posts: 5373
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Tue Oct 29, 2019 6:01 pm

Kira wrote:
Mon Oct 28, 2019 8:22 pm
The Monarch of Pointland thinks that all stars must be figments of its own imagination. Since It is the whole universe and the only thing It can see (and having no concept of distance), all these whimsical stars must therefore be located at the exact same (and only) point.
While it is easiest to search a database in which all stars have been mapped to a single zero-dimensional point, even projections onto lines or planes are a bit useless when planning travel throughout the universe. The typical time-traveling tourist needs further to work with full four-dimensional spacetime to map out a satisfying trip throughout eternity.

How could a spectator view the start of world war three, if by accident the tour arrived one million years earlier at the same spot? To avoid such absurdities, a practical search algorithm (and challenge contest entry) should likely work with four-dimensional data to find five of the ten nearest interesting events to any fixed location in spacetime.

Fido told me K9 was a bit worried: If the time-traveling tour-guide business didn't take off, the resulting failure might lead to a life of homelessness and crime. On the other paw, hopefully liberation through computer literacy can save the dog.

Return to “General programming discussion”