Page 1 of 1

Pynmea2 output Decimal Degrees. (Solved)

Posted: Thu Sep 19, 2019 10:34 am
by Dranoweb
Hi,

I've been working on a project for over 3 months at this stage - most of that time has been spent waiting for parts.
The rest, learning that I know very little about python.
I've bolded the important parts for easy reading.

The Project: A raspberry-pi based Geiger counter with GPS logging.

The problem: Waypoints "clustering" (despite moving several kilometers, coord's change very little)

Where I went wrong:


This bit takes a few more words to explain.
However, to sum it up; I believe I'm only logging the first half of the coords.

I'm running one python script in the background to count radioactive particles via a 1 microsecond pulse, and write to a file every 5 sec
This bit works fine. (radcount.py) Not relevant to the problem, but included for context.

I'm running another python in the foreground that's much more verbose - that does most of the work.
This one does a few things:

Reads NMEA data using Pynmea2
Checks for gps fix
if there is:
opens a new file with the date in the name
creates the headers if the file didn't already exist
adds all the string values etc to a new line in said file
If not
skips over the rest, prints a message, and loops back to start

I'm having issues working out how to call the minutes and seconds portions of the coords and joining the strings to achieve decimal degrees.
After inspecting the output the first few times, and loading it into google earth, it seemed that the output of "msg.latitude and "msg.longitude" were indeed the full decimal degree output (based on length of value)

I now realize that I need to truncate the values to a number of digits and join on the minutes and seconds.

I have tried several methods to do this and all have failed, and I have sought out the documentation for pynmea2, but to a hobbyist such as myself who has not had years of intensive learning on the topic, it's a little confusing.

My current code is below:

Code: Select all

Alt = "None"
count = str(0)
uSvHr = str(0)
import os
import time
import serial
import pynmea2
from datetime import datetime
now = datetime.now()


def parseGPS(str):
    if str.find('GGA') > 0:
        msg = pynmea2.parse(str)
        Alt = (msg.altitude_units)
        time.sleep(15)
        if Alt == 'M':
            print ("GPS OK!- %s SATELITES" % (msg.num_sats))
            with open("countlog.txt", "r") as radcount:
                count = contents =radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print "CSV FILE EMPTY!"
                    print "ADDING COLUMN NAMES"
                    gps.write("Time,Lat,Long,Alt,Sats,CPM,Quality" + '\n')
                gps.write("%s,$s,$s,%s%s,%s,%s" % (msg.timestamp,msg.latitude,msg.longitude,msg.altitude,msg.a$
                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % msg.num_sats)

serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)

while True:
#    time.sleep(0.5)
    str = serialPort.readline()
    parseGPS(str)


Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:09 am
by scotty101
Pynmea already processess the NMEA message from degrees/minutes/decimal minutes to decimal degrees.

It is done by the dm_to_sd function https://github.com/Knio/pynmea2/blob/bc ... ils.py#L29

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:16 am
by Dranoweb
I'm familiar with that documentation, somewhat stuck with how to implement that function into my existing code.

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:23 am
by DougieLawson
It's not hard to do in a simple python function

Code: Select all

def decTodms(deg):
     d = int(deg)
     md = abs(deg - d) * 60
     m = int(md)
     s = (md - m) * 60
     return [d, m, s]
Or the other way round

Code: Select all

def dmsTodec(d, m, s):
    decd = d + float(m)/60 + float(s)/3600
    return decd

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:31 am
by Dranoweb
Well I took the first suggest and had an attempt , but I ran into an error

"NameError: global name 'dm_to_sd' is not defined"

in the example I'm working from, it's not defined as a global anywhere...

https://stackoverflow.com/questions/496 ... -data-gaps

Code: Select all

  GNU nano 2.7.4                                    File: gps.py

# -*- coding: utf-8 -*-
Alt = "None"
count = str(0)
uSvHr = str(0)
import os
import time
import serial
import pynmea2
from datetime import datetime
now = datetime.now()


def parseGPS(str):
    global Latitude, Longitude
    if str.find('GGA') > 0:
        msg = pynmea2.parse(str)
        Alt = (msg.altitude_units)
        time.sleep(15)
        if Alt == 'M':
            print ("GPS OK!- %s SATELITES" % (msg.num_sats))
            with open("countlog.txt", "r") as radcount:
                count = contents =radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print "CSV FILE EMPTY!"
                    print "ADDING COLUMN NAMES"
                    gps.write("Time,Lat,Long,Alt,Sats,CPM,Quality" + '\n')
                Latitude = pynmea2.dm_to_sd(msg.lat)
                Longitude = -(dm_to_sd(msg.lon))
                gps.write("%s," + Latitude + "," + Longitude + ",%s%s,%s,%s" % (msg.timestamp,msg.altitude,msg$
                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % msg.num_sats)

serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)

while True:
#    time.sleep(0.5)
    str = serialPort.readline()
    parseGPS(str)


Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:34 am
by Dranoweb
DougieLawson wrote:
Thu Sep 19, 2019 11:23 am
It's not hard to do in a simple python function

Code: Select all

def decTodms(deg):
     d = int(deg)
     md = abs(deg - d) * 60
     m = int(md)
     s = (md - m) * 60
     return [d, m, s]
Or the other way round

Code: Select all

def dmsTodec(d, m, s):
    decd = d + float(m)/60 + float(s)/3600
    return decd
The issue I have is that the NMEA data from the Neo6M gps is not in DMS...
Radther its DDMM.MMMMSS or something to that effect.

I would happily use DMS if i could work out how to get a UTF8 degree character into the string without breaking things

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:36 am
by Dranoweb
To clarify - I'm getting decimal degrees output to the file, but only the initial portion.

it's not factoring in degrees and seconds, so I have only very rough coords.

the reason I went this was is the raw values in the NMEA string are not a format that google earth will accept.

I'd be happy with any output that works with GE.

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:43 am
by DougieLawson
If it's ddmm.mmmmss you can split it with maths

You can get the degrees by int(dms/100)
subtract (degrees * 100) and you get mm.mmmmss
multiply mm.mmmmss by 10000 and the integer portion is minutes and the decimal is seconds.

What's the raw NMEA sentence you're getting? Can you get that and parse it yourself?

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:49 am
by Dranoweb
I might be most of the way with the dm_to_sd method.

I had made a typo - I'm not stuck with "TypeError: cannot concatenate 'str' and 'float' objects"

this error I am more familiar with - but I might be able to solve myself by shuffling it through another variable.

Code: Select all

# -*- coding: utf-8 -*-
Alt = "None"
count = str(0)
uSvHr = str(0)
import os
import time
import serial
import pynmea2
from datetime import datetime
now = datetime.now()


def parseGPS(str):
    global Latitude, Longitude
    if str.find('GGA') > 0:
        msg = pynmea2.parse(str)
        Alt = (msg.altitude_units)
        time.sleep(15)
        if Alt == 'M':
            print ("GPS OK!- %s SATELITES" % (msg.num_sats))
            with open("countlog.txt", "r") as radcount:
                count = contents =radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print "CSV FILE EMPTY!"
                    print "ADDING COLUMN NAMES"
                    gps.write("Time,Lat,Long,Alt,Sats,CPM,Quality" + '\n')
                Latitude = pynmea2.dm_to_sd(msg.lat)
                Longitude = -(pynmea2.dm_to_sd(msg.lon))
                gps.write("%s," + Latitude + "," + Longitude + ",%s%s,%s,%s" % (msg.timestamp,msg.altitude,msg$
                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % msg.num_sats)

serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)

while True:
#    time.sleep(0.5)
    str = serialPort.readline()
    parseGPS(str)


Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 11:52 am
by Dranoweb
DougieLawson wrote:
Thu Sep 19, 2019 11:43 am
If it's ddmm.mmmmss you can split it with maths

You can get the degrees by int(dms/100)
subtract (degrees * 100) and you get mm.mmmmss
multiply mm.mmmmss by 10000 and the integer portion is minutes and the decimal is seconds.

What's the raw NMEA sentence you're getting? Can you get that and parse it yourself?
I'll reboot so I can cat the serial output for you ..............

pretty sure everyone will know where I am after this, so I may redact this post after the deed is done.

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 12:18 pm
by DougieLawson
I usually overwrite the last three digits with zero. That moves my position about 500m away. Or switch the S to N and E to W to arrive in your antipode (somewhere in the North Atlantic).

Code: Select all

$GPGLL,3805.70???,S,14704.27???,E,115120.00,A,A*??
$GPGGA,115121.00,3805.70???,S,14704.27???,E,1,06,1.34,2.8,M,0.3,M,,*??
Is 38° 5.70???' south and 147° 4.27???' east

There's a discussion of decoding it to decimal at https://www.raspberrypi.org/forums/view ... 3#p1117895

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 12:55 pm
by Dranoweb
Yes those values appear to be correct.

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 1:13 pm
by Dranoweb
I'm basically down to one last impossible feat - concatenating a float to a string...

So I guess the question has become - Can I do that?


Code: Select all

# -*- coding: utf-8 -*-
Alt = "None"
count = str(0)
uSvHr = str(0)
strSerial = str(0)
import os
import time
import serial
import pynmea2
from datetime import datetime
now = datetime.now()


def parseGPS(str):
    global Latitude, Longitude
    if str.find('GGA') > 0:
        msg = pynmea2.parse(str)
        Alt = (msg.altitude_units)
        time.sleep(15)
        if Alt == 'M':
            print ("GPS OK!- %s SATELLITES" % (msg.num_sats))
            with open("countlog.txt", "r") as radcount:
                count = contents =radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print "CSV FILE EMPTY!"
                    print "ADDING COLUMN NAMES"
                    gps.write("Time,Lat,Long,Alt,Sats,CPM,Quality" + '\n')
                Latitude = pynmea2.dm_to_sd(msg.lat)
                Longitude = -(pynmea2.dm_to_sd(msg.lon))
                gps.write("%s," + Latitude + "," + Longitude + ",%s%s,%s,%s" % (msg.timestamp,msg.altitude,msg.altitude_units,msg.num_sats,msg.gps_qual))

                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % msg.num_sats)

serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)

while True:
    str = serialPort.readline()
    parseGPS(str)

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 1:24 pm
by DougieLawson
If you convert everthing to a string then concatenation is trivial.

That gets easier if you convert your float to a decimal
https://docs.python.org/3/library/decimal.html

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 1:42 pm
by Dranoweb
This was my inital thought,

however I'm using a function of pynmea2 called "dm_to_sd"
and that hands me a float value - which is exactly what I want.

but I'm coming from vb6 programming and don't quite know how to do that here

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 1:51 pm
by DougieLawson
What happens if you simply wrap a

Code: Select all

myString = STR(myFloating)
function round your float variable?

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 2:12 pm
by Dranoweb
I have attempted that in various forms, and it still throws an error about being unable to concatenate a string and a float.

Originally I found that the tutorial I had followed had used "str" as a variable - and that kinda screwed things up.

that being fixed now, I can print the floats, and I can use the above method without error but the value seems to remain as a float.

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 2:47 pm
by scotty101
If you want to create a string, don't concatenate use the string formatting functions in python.

Code: Select all

myString = "{0} is a float converted to a string, {1} is a string".format(123.456,'A string')
or the older fashioned method

Code: Select all

myString = "%f is a float converted to a string, %s is a string" % (123.456,'A string')

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 2:47 pm
by bensimmo
I'm a bit lost in what format you want
but

For example, latitude and longitude properties exist as helpers to access the geographic coordinates as python floats (DD, "decimal degrees") instead of the DDDMM.MMMM ("Degrees, minutes, seconds") format used in the NMEA protocol. latitude_minutes, latitude_seconds, longitude_minutes, and longitude_seconds are also supported and allow easy creation of differently formatted location strings.

Code: Select all

print( msg.latitude )
-19.4840833333

print( msg.longitude )
24.1751

print( '%02d°%07.4f′' % (msg.latitude, msg.latitude_minutes) )
'-19°29.0450′'

print( '%02d°%02d′%07.4f″' % (msg.latitude, msg.latitude_minutes, msg.latitude_seconds) )
"-19°29′02.7000″"

Re: Pynmea2 output Decimal Degrees.

Posted: Thu Sep 19, 2019 2:48 pm
by Dranoweb
well I got it in the end.

consulted an ex-coworker who's much more up to date with this stuff and together we worked out a solution.

here's what I we ended up with:

Code: Select all

# -*- coding: utf-8 -*-
Alt = "None"
count = "0"
uSvHr = "0"
import os
import time
import serial
import pynmea2
from datetime import datetime
now = datetime.now()


def parseGPS(myStr):
    global Latitude, Longitude
    if myStr.find('GGA') > 0:
        msg = pynmea2.parse(myStr)
        Alt = (msg.altitude_units)
        time.sleep(15)
        if Alt == 'M':
            print ("GPS OK!- %s SATELITES" % (msg.num_sats))
            with open("countlog.txt", "r") as radcount:
                count = contents =radcount.readline()
                print ("COUNTS PER MIN: " + count)
                print ("uSv/Hr " + uSvHr)
            with open("radlog" + now.strftime("%d-%m-%y") + ".csv", "a") as gps:
                if (os.stat("radlog" + now.strftime("%d-%m-%y") + ".csv").st_size == 0):  #check if the file i$
                    print "CSV FILE EMPTY!"
                    print "ADDING COLUMN NAMES"
                    gps.write("Time,Lat,Long,Alt,Sats,CPM,Quality" + '\n')
                Latitude = -(pynmea2.dm_to_sd(msg.lat))
                Longitude = (pynmea2.dm_to_sd(msg.lon))
                gps.write(str(msg.timestamp) + "," + str(Latitude) + "," + str(Longitude) + "," + str(msg.altitude) + str(msg.altitude_units) + "," + str(msg.num_sats) + "," + str(msg.gps_qual))
                gps.write(",%s" % (count) + '\n')
        else:
            print ("NO GPS FIX! - LOGGING PAUSED. %s SATELITES" % msg.num_sats)

serialPort = serial.Serial("/dev/ttyAMA0", 9600, timeout=0.5)

while True:
#    time.sleep(0.5)
    outputStr = serialPort.readline()
    parseGPS(outputStr)



Re: Pynmea2 output Decimal Degrees. (Solved)

Posted: Fri Sep 20, 2019 1:33 pm
by DougieLawson

Code: Select all

def parseGPS(myStr):
    global Latitude, Longitude
    if myStr.find('GGA') > 0:
        msg = pynmea2.parse(myStr)
        Alt = (msg.altitude_units)
        time.sleep(15)
        if Alt == 'M':
I don't know what you're trying to achieve with that prolog code in your parseGPS() function. If you want to be sure you've got a good fix your should parse $GPGGA for the byte after your lat/long fix

https://www.gpsinformation.org/dale/nmea.htm#GGA

$GPGGA,123519,4807.038,N,01131.000,E,1,08,0.9,545.4,M,46.9,M,,*47

Fix quality:
  • 0 = invalid
  • 1 = GPS fix (SPS)
  • 2 = DGPS fix
  • 3 = PPS fix
  • 4 = Real Time Kinematic
  • 5 = Float RTK
  • 6 = estimated (dead reckoning) (2.3 feature)
  • 7 = Manual input mode
  • 8 = Simulation mode