Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 10:38 am

bensimmo wrote:
Sat Sep 21, 2019 9:12 am
I just check and if you stay with Google earth, it can use Decimal Degrees.
So you only need to record
msg.latitude & msg.longitude

(Same as Google maps)
You don't need to split and combine.

https://support.google.com/earth/answer/148068?hl=en


EDIT simple test program for you (another one, this is python3)
EDIT2, added logging to a file for ease.

Code: Select all

#!/usr/bin/python3
import pynmea2  # NMEA GNSS splitter
import serial  # sudo pip3 install pyserial # GPIO/USB serial read
from csv import writer

GNSS_PORT = "/dev/ttyACM0"
#GNSS_PORT = "/dev/serial0"

serialPort = serial.Serial(GNSS_PORT, 9600, timeout=1.5)

with open('gpslog.csv', 'w', newline='') as f:
    data_writer = writer(f)
    data_writer.writerow([ 'latitude', 'longitude', 'altitude' ]) # data headers

    while True:
        strip = serialPort.readline()
        if strip.find(b'GGA') > 0:
            msg = pynmea2.parse(strip.decode('utf-8'))
            print( msg.latitude, msg.longitude, msg.altitude )
            data_writer.writerow([ msg.latitude, msg.longitude, msg.altitude ])
This is what I had initially and not this was not the solution.

I have to say it is fixed - and yes it was Pynmea2
Between myself an an ex-workmate we figured things out.

It would appear that pynmea2 was written by someone close to the equator, and had not anticipated someone so far from the equator might use it. this meant that a lat value increment represents something like 60% less distance here - leading to the "scaled" appearance of my coords.

We discovered that pnmea2 was doing something odd to the string as it parsed it.
The dd_to_sd function is mathematically correct, but the parsing was not.

So we decided to do the math ourselves, and amazingly it worked!
Splitting the string into an array and referencing each relevant section allowed us to do the math ourselves.

I'll be gradually phasing out pynmea2 altogether in favor of this method.

The GPS is fine, the serial interface is fine

My current "working" code, that has not been cleaned up yet is as follows:

Code: Select all

# -*- coding: utf-8 -*-
Alt = "None"
count = "0"
uSvHr = "0"
outputStr = ""
import os
import time
import serial
import pynmea2
from datetime import datetime
now = datetime.now()


def parseGPS(myStr):
    global Latitude, Longitude
    if myStr.find('GGA') > 0:
        msg = pynmea2.parse(myStr)
        Alt = (msg.altitude_units)
        time.sleep(1)
        if Alt == 'M':
            print ("GPS OK!- %s SATELITES" % (msg.num_sats))
            with open("countlog.txt", "r") as radcount:
                count = contents =radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print ("CSV FILE EMPTY!")
                    print ("ADDING COLUMN NAMES")
                    gps.write("Time,Lat,Long,Alt,Sats,Quality,CPM" + '\n')
                
                               
                Latitude = myStr.split(",")[2]
                LatNS = myStr.split(",")[3] 
                Longitude = myStr.split(",")[4]
                LongEW = myStr.split(",")[5]
                
                Latitude = float(Latitude)
                Longitude = float(Longitude)

                if LatNS == 'S':
                    Latitude = Latitude * -1
                if LongEW == 'W':
                    Longitude = Longitude * -1

                DD = int(float(Latitude)/100)
                SS = float(Latitude) - DD * 100
                Latitude = DD+SS/60
                DD = int(float(Longitude)/100)
                SS = float(Longitude) - DD * 100
                Longitude = DD + SS/60

                
                print Latitude

                print Longitude
     
                #Latitude = -(pynmea2.dm_to_sd(msg.lat))
                #Longitude = (pynmea2.dm_to_sd(msg.lon))
                gps.write(str(msg.timestamp) + "," + str(Latitude) + "," + str(Longitude) + "," + str(msg.altitude) + str(msg.altitude_units) + "," + str(msg.num_sats) + "," + str(msg.gps_qual))
                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % msg.num_sats)

serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)



while True:
    parseGPS(serialPort.readline())
    serialPort.flush()
serialPort.close()




Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 11:08 am

bensimmo wrote:
Sat Sep 21, 2019 9:02 am


I hear threading mentioned, I see no threading in the program.


You use Python3 at the command line by typing Python3 and not python
I was a little ahead of myself - threading was happening the the radcount script, that runs in the background, writing the counts to a text file.
this is then opened and sent to a variable in the above code.

re: python3 - I was just educated to this fact earlier tonight - thankyou, I will follow up and endeavor to learn.

If you see my previous post - you will see how we solved this issue.

User avatar
rpiMike
Posts: 966
Joined: Fri Aug 10, 2012 12:38 pm
Location: Cumbria, UK

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 11:47 am

We discovered that pnmea2 was doing something odd to the string as it parsed it.
The dd_to_sd function is mathematically correct, but the parsing was not.
Can you explain what the issue was here?

I'm using pynmea2 (in Cumbria,UK) and not seeing any issues.

Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 11:58 am

rpiMike wrote:
Sat Sep 21, 2019 11:47 am
We discovered that pnmea2 was doing something odd to the string as it parsed it.
The dd_to_sd function is mathematically correct, but the parsing was not.
Can you explain what the issue was here?

I'm using pynmea2 (in Cumbria,UK) and not seeing any issues.
So I can't confirm specifically "how" it happened,
other than that process of elimination led me to Pynmea2 as the culprit.

we examined the function dd_to_sd and it appeared to be code-wise and math-wise correct.
however:

Whenever we would get pynmea2 to to parse the serial data it would do something odd to it.

By odd, we "assume" that it was some sort of error pertaining to either this side of the equator, or longitudinal convergence issues
EG, long values represent shorter distances as you get further away from the equator.

This meant that while I had a nice outline of the coords in the path I had taken, they were roughly 60-70% closer together than expected.

One other crucial change we made was to flush the serial buffer after each use, as the find function was looking for the first instance of "GGA", which was at the top of the buffer, and when not flushed, remained there till the buffer was full - of which it never was due to the short duration of my initial testing.

we only discovered that when in testing we started recording coords once a second, instead of every 15.

Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 12:03 pm

I want to make this a separate post:

I would like to take the time to acknowledge those on the forum that have assisted me so far.
everyone added another piece to the puzzle.

While the final solution was in collaboration with someone outside the forums - it was the tips and hints here that heavily contributed to the final solution.

I would also like to thank everyone for being patient with my unconventional approach to coding and irritability, people aren't a strong point for me, as I'm sure many of you could relate.

So in short - Thankyou everyone.

User avatar
rpiMike
Posts: 966
Joined: Fri Aug 10, 2012 12:38 pm
Location: Cumbria, UK

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 2:03 pm

Out of interest does the following show up any anomalies ?

Ignoring any extremely minor rounding and -ve lat/lon.

Code: Select all

#!/usr/bin/env python3
import time,serial,pynmea2

def latLonToDecDegrees(val):
    DD = int(float(val)/100)
    MM = float(val) - DD * 100
    return DD + MM/60

ser = serial.Serial('/dev/ttyACM0',9600,timeout=1)

while True:
    data = ser.readline().decode().strip()
    if data.startswith('$GNGGA'):
        msg = pynmea2.parse(data)
        fields = data.split(',')
        print('data',data)
        print('fields[2],fields[4]                                    ',fields[2],fields[4])
        print('msg.lat,msg.lon                                        ',msg.lat,msg.lon)
        print('msg.latitude,msg.longitude                             ',msg.latitude,msg.longitude)
        print('pynmea2.dm_to_sd(msg.lat),pynmea2.dm_to_sd(msg.lon)    ',pynmea2.dm_to_sd(msg.lat),pynmea2.dm_to_sd(msg.lon))
        print('latLonToDecDegrees(msg.lat),latLonToDecDegrees(msg.lon)',latLonToDecDegrees(msg.lat),latLonToDecDegrees(msg.lon))
        print('')
    time.sleep(0.01)
    

Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 2:31 pm

rpiMike wrote:
Sat Sep 21, 2019 2:03 pm
Out of interest does the following show up any anomalies ?

Ignoring any extremely minor rounding and -ve lat/lon.

Code: Select all

#!/usr/bin/env python3
import time,serial,pynmea2

def latLonToDecDegrees(val):
    DD = int(float(val)/100)
    MM = float(val) - DD * 100
    return DD + MM/60

ser = serial.Serial('/dev/ttyACM0',9600,timeout=1)

while True:
    data = ser.readline().decode().strip()
    if data.startswith('$GNGGA'):
        msg = pynmea2.parse(data)
        fields = data.split(',')
        print('data',data)
        print('fields[2],fields[4]                                    ',fields[2],fields[4])
        print('msg.lat,msg.lon                                        ',msg.lat,msg.lon)
        print('msg.latitude,msg.longitude                             ',msg.latitude,msg.longitude)
        print('pynmea2.dm_to_sd(msg.lat),pynmea2.dm_to_sd(msg.lon)    ',pynmea2.dm_to_sd(msg.lat),pynmea2.dm_to_sd(msg.lon))
        print('latLonToDecDegrees(msg.lat),latLonToDecDegrees(msg.lon)',latLonToDecDegrees(msg.lat),latLonToDecDegrees(msg.lon))
        print('')
    time.sleep(0.01)
    
I'll try it out - currently around 1am here - so it might not be until tomorrow.

However I have just discovered that at 15 second intervals - the coords are either clustered again, or waay off target.
Possibly this is because I'm not clearing the buffer regularly enough.

At 1 second intervals it's fine and very accurate.

As my time is limited to survey I have just adjusted my counting script to compensate for 1 second samples.
I'll selectively edit the data afterwards - at least my coords will be accurate.

User avatar
bensimmo
Posts: 4184
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Ublox Neo 6M GPS module not changing coords.

Sat Sep 21, 2019 3:11 pm

Use the GNSS/GPS as your rate limiting device in the loop, it's 1 per second, skip 14 of them and collect teh 15th.
Set it as a variable at the start and it's easy to adjsut.


My code above works in python3 here in the UK, i'm right next to the meridian line.

You have msg.lat and msg.lon in your code (DMS), that's different to msg.latitude and msg.longitude (DD)
(dm_to_sd gives the same result as longitude/latitude, but unsigned)
so you're saying putting that into good gives the wrong result ?)

If you think it is a problem, please post to the pynmea2 github page if you can https://github.com/Knio/pynmea2
to get it fixed.

Thre is no harm making your own though.

Do have a look at moving to gpsd in the future, it's quite useful apparently.
It's always running and you just ask it for the current readings, you can also then set the Pi's clock with the GPS signal easier iirc.
(of course, when everything is working and you just want to play around)

Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Sun Sep 22, 2019 11:51 am

I have looked into GPSD, and in fact have it installed on the project as a backup, however given the initial timeframe I'm forced to work with what I know.

after the primary survey deadline is up, I'll be back to refine things, and GPSD will probably be the way to go.

the serial data has indeed become the limiter for the loop, and we have done some work in buffer flushing, which seems to have been the main source of the issue.

I replaced the battery today, only to find that the receiver stopped responding - it would seem I got sent a batch of dud batteries.
china is not making themselves look good right now.

I shoved a tiny super capacitor on it instead and I'm getting warm and hot starts now.

The meridan thing was more of an asumption - not a confirmed issue, I'm not keep to log a problem with something until I can definitively confirm that is the exact issue.

In my initial attempts i used latitude and longitude as the documentation indicated this would produce decimal degrees, it was only after this failed that I began to use lat and lon to give me the coords as seen in the serial data so that I could do the math myself - this also failed.

My problems only abated when two things happened:

1: I switched to 1 second loops (from 15)
2: I bypassed pnmea2 for splitting out and calculating the coords.

That revealed a couple of things - the code example I had used as a starting point, was looking for the most recent occurence of "GGA" - that was at the top of the buffer, that was not overflowing - leading to me getting the same or similar coords over and over again.

Pnmea2 was returning different coords, but extremely close together - but when I plotted them in something like garmin basecamp, that allows a much higher zoom level that say google earth, I could see that they were the correct shape/formation of my tracks, just in minature.
This observation led to the assumption that longitude convergence might not be factored in.

Also apologies for typos, I have MS, and latest brain lesion makes my left hand a little laggy, that means I make a lot of typos and have to slow waay down at night - really frustrating.

here's where I'm at presently: (with heavy assistance from an ex-co worker.)

Code: Select all

# -*- coding: utf-8 -*-
Alt = "None"
count = "0"
uSvHr = "0"
import os
import time
import serial
from datetime import datetime
now = datetime.now()


def parseGPS(myStr):
    global Latitude, Longitude
    if myStr.find('GGA') > 0:
        Alt = myStr.split(",")[10]
        #time.sleep(15)
        if Alt == 'M':
            print ("GPS OK!- %s SATELITES" % myStr.split(",")[7])
            with open("countlog.txt", "r") as radcount:
                count = contents =radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print ("CSV FILE EMPTY!")
                    print ("ADDING COLUMN NAMES")
                    gps.write("Time,Lat,Long,Alt,Sats,Quality,CPM" + '\n')
                
                               
                Latitude = myStr.split(",")[2]
                LatNS = myStr.split(",")[3] 
                Longitude = myStr.split(",")[4]
                LongEW = myStr.split(",")[5]
                
                Latitude = float(Latitude)
                Longitude = float(Longitude)

                if LatNS == 'S':
                    Latitude = Latitude * -1
                if LongEW == 'W':
                    Longitude = Longitude * -1

                DD = int(float(Latitude)/100)
                SS = float(Latitude) - DD * 100
                Latitude = DD+SS/60

                DD = int(float(Longitude)/100)
                SS = float(Longitude) - DD * 100
                Longitude = DD + SS/60

                gps.write(datetime.now() + "," + str(Latitude) + "," + str(Longitude) + "," + myStr.split(",")[9] + myStr.split(",")[10] + "," + myStr.split(",")[7] + "," + myStr.split(",")[6])
                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % myStr.split(",")[7])

serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)

while True:
    time.sleep(15)
    serialPort.flush()
    output = serialPort.readline()
    parseGPS(output)
    serialPort.flush()
serialPort.close()




Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Sun Sep 22, 2019 12:02 pm

For anyone taking a personal interest in this project and has lots of time to waste
- I am documenting the build on youtube.

if anyone would like their contributions acknowledged in part 6 - I'm happy to do so.


https://youtu.be/7T5gQTv4-FI Part 1
https://youtu.be/M6VdVM1Ub2c Part 2
https://youtu.be/zPE3XmHa-kM Part 3
https://youtu.be/_GTS8dqB6bY Part 4
https://youtu.be/sTpcj9tj0wI Part 5
https://youtu.be/yBfLYLGZ6o8 Part 6 (GPS errors demonstrated)
https://youtu.be/MsiX2eGemcA Part 7 (Actually scanning the area)
Last edited by Dranoweb on Thu Sep 26, 2019 12:30 pm, edited 2 times in total.

Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Mon Sep 23, 2019 2:49 am

Ok so I FINALLY have a functional code, that works at 15 second intervals, AND gives correct coords.

between all of us we got there...

here's the code at present:

Code: Select all

Alt = "None"
count = "0"
uSvHr = "0"
import os
import time
import serial
from datetime import datetime
now = datetime.now()
oldtime = time.time()


def parseGPS(myStr):
    global Latitude, Longitude
    Alt = myStr.split(",")[10]
    global oldtime
    if time.time() - oldtime > 15:
        oldtime = time.time()
        if Alt == 'M':
            print ("GPS OK!- %s SATELITES" % myStr.split(",")[7])
            with open("countlog.txt", "r") as radcount:
                count = radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print ("CSV FILE EMPTY!")
                    print ("ADDING COLUMN NAMES")
                    gps.write("Time,Lat,Long,Alt,Sats,Quality,CPM" + '\n')
                
                                
                Latitude = myStr.split(",")[2]
                LatNS = myStr.split(",")[3] 
                Longitude = myStr.split(",")[4]
                LongEW = myStr.split(",")[5]
                
                Latitude = float(Latitude)
                Longitude = float(Longitude)

                if LatNS == 'S':
                    Latitude = Latitude * -1
                if LongEW == 'W':
                    Longitude = Longitude * -1

                DD = int(float(Latitude)/100)
                SS = float(Latitude) - DD * 100
                Latitude = DD+SS/60

                DD = int(float(Longitude)/100)
                SS = float(Longitude) - DD * 100
                Longitude = DD + SS/60

                gps.write(str(datetime.now()) + "," + str(Latitude) + "," + str(Longitude) + "," + myStr.split(",")[9] + myStr.split(",")[10] + "," + myStr.split(",")[7] + "," + myStr.split(",")[6])
                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % myStr.split(",")[7])


serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)
while True:
    data = serialPort.readline()
    if data.find('GGA') > 0:
        parseGPS(data) 
serialPort.close()




Dranoweb
Posts: 147
Joined: Thu Mar 20, 2014 3:01 pm

Re: Ublox Neo 6M GPS module not changing coords.

Thu Sep 26, 2019 6:00 am

Just a followup for anyone still following this:

I made the deadline, and got to site with a few days left before my permission expired.

The site is indeed radioactive, and analysis of the data loggedis now underway.

Thankyou once again for your assistance.

Return to “Interfacing (DSI, CSI, I2C, etc.)”