I now have serial data on Minicom (after setting the correct serial port and baud rate).
Bloxter still can’t see the GPS.
The software that wants to use the GPS uses a library “easygps”, but the easygps library has no method “read” which is required by other methods, so it always fails.
Note the raw data received in Thonny’s terminal window. It’s horrible data because the GPS isn’t’ in a window and has no satellite lock.
What appears to be happening is that the GPS read is expecting a very specific format of a “GPGGA” sentence which isn’t happening. I added a line where it prints the raw data received, prior to filtering, and you can see I’m getting data. Crummy data because I’m nowhere near a window with the GPS, but I’m getting data.
What I need to do is generalize the data filtering so that any additional characters, (the "b’ " in this case), don’t contaminate the filtering process.
Viz.:
Capture a raw string.
Look within the first ten characters for the string “GPGGA”.
If that string is found, strip off anything before the first “G”, and return it as a valid string.
The problem is that the easygps library is broken as it is missing its fundamental “read” method.
Viz.:
This is the equivelent Python code generated by Bloxter when it attempts to read the GPS sensor.
import easygopigo3 as easy
import time
import easygps
gpg = easy.EasyGoPiGo3()
gps_sensor = easygps.GPS()
gps_sensor.read() # take an initial reading
# start
while True:
print(gps_sensor.get_latitude())
time.sleep(0.2)
time.sleep (1)
print(gps_sensor.get_longitude())
time.sleep(0.2)
time.sleep (1)
time.sleep(0.05) # slowdown
This is the easygpg.py library file:
(Note the absence of a “read” method.)
import serial, time, sys
import re
from multiprocessing import Process, Queue
en_debug = False
def debug(in_str):
if en_debug:
print(in_str)
patterns=["\$GPGGA",
"[0-9]+\.[0-9]+", # timestamp hhmmss.ss
"[0-9]+.[0-9]+", # latitude of position
"[NS]", # North or South
"[0-9]+.[0-9]+", # longitude of position
"[EW]", # East or West
"[012]", # GPS Quality Indicator
"[0-9]+", # Number of satellites
"[0-9]+.[0-9]+", # horizontal dilution of precision x.x
"[0-9]+\.[0-9]+" # altitude x.x
]
class GPS():
def __init__(self,port='/dev/ttyAMA0',baud=9600,timeout=0, delay_to_die=10):
self.ser = serial.Serial(port, baud, timeout=timeout)
self.ser.flush()
self.raw_line = ""
self.gga = []
self.validation =[] # contains compiled regex
self.time_to_die = time.time()
self.delay_to_die = delay_to_die # how long do we keep a value before requesting another read
self.threaded_access = None
self.queue = None
# compile regex once to use later
for i in range(len(patterns)-1):
self.validation.append(re.compile(patterns[i]))
self.clean_data()
def clean_data(self):
'''
clean_data:
ensures that all relevant GPS data is set to either empty string
or -1.0, or -1, depending on appropriate type
This occurs right after initialisation or
after 50 attemps to reach GPS
'''
self.timestamp = ""
self.lat = -1.0 # degrees minutes and decimals of minute
self.NS = ""
self.lon = -1.0
self.EW = ""
self.quality = -1
self.satellites = -1
self.altitude = -1.0
self.latitude = -1.0 #degrees and decimals
self.longitude = -1.0
self.fancylat = "" #
def _threaded_read(self, q):
'''
Start a thread to query the GPS as this can loop up to 50 times when the GPS is unable
to get a proper reading. As soon as a reading is valid, exit background thread.
An exit code of 0 means we have a valid reading.
An exit code of 1 means we cannot get a reading after 50 reads, or an empty line
'''
debug ("starting thread")
for i in range(50):
time.sleep(0.5)
self.raw_line = self.ser.readline()
# an empty line means the GPS isnt ready to send data
if (self.raw_line == b""):
exit(1)
try:
self.line = self.raw_line.decode('utf-8')
self.line = self.line.strip()
except: #leftover from Python2
self.line = self.raw_line
debug(f"DATA #{i}: {self.line}")
if self.validate(self.line):
q.put (self.line)
self.ser.reset_input_buffer()
exit(0)
# else:
# print ("invalid line")
debug ("Giving up")
exit (1)
def control_panel_read(self):
'''
Starts a non-blocking background thread that attempts 50 times at most to get valid data from GPS
If a thread is found to be already running, then just exit
It's expected that control_panel_read() is being called repeatitively
and that a reading will caught on the following call.
If valid data is not found, then clean up data in GPS instance
'''
return_value = 0
if self.queue == None:
self.queue = Queue()
if self.threaded_access == None:
# start the background process
debug ("starting first process")
self.threaded_access = Process(target=self._threaded_read, args=(self.queue,))
self.threaded_access.start()
if self.threaded_access.is_alive() is False:
# query background process
if self.threaded_access.exitcode == 0:
if self.queue.empty() is False:
gga = self.queue.get()
self.parse_gga(gga)
debug ("FOUND: ")
debug (self.gga)
self.time_to_die = time.time()
return 0
else:
# we got an exit code indicating failure
debug("cleaning up")
self.clean_data()
return_value = 1
# start the next background process
debug("starting next process")
self.threaded_access = Process(target=self._threaded_read, args=(self.queue,))
self.threaded_access.start()
else:
# debug("nothing to do but wait")
return_value = 2
return return_value
def validate(self, in_line):
'''
Runs regex validation on a GPGAA sentence.
Returns False if the sentence is mangled
Return True if everything is all right and sets internal
class members.
'''
if in_line == "":
return False
if in_line[:6] != "$GPGGA":
return False
self.gga = in_line.split(",")
# debug("in validate():")
# debug (self.gga)
#Sometimes multiple GPS data packets come into the stream. Take the data only after the last '$GPGGA' is seen
try:
ind=self.gga.index('$GPGGA',5,len(self.gga))
self.gga=self.gga[ind:]
except ValueError:
pass
if len(self.gga) != 15:
debug ("Failed: wrong number of parameters ")
debug (self.gga)
return False
for i in range(len(self.validation)-1):
if len(self.gga[i]) == 0:
debug (f"Failed: empty string {i}")
self.ser.flush()
self.ser.reset_input_buffer()
return False
test = self.validation[i].match(self.gga[i])
if test == None:
debug (f"Failed: wrong format on parameter {i}: {self.gga[i]}")
return False
else:
# debug("Passed %d"%i)
pass
self.parse_gga(self.gga)
return True
def parse_gga(self, in_gga):
# in_gga may be a str or a list. We need it in a list
try:
self.gga = in_gga.split(",")
except:
pass
try:
self.timestamp = self.gga[1]
self.lat = float(self.gga[2])
self.NS = self.gga[3]
self.lon = float(self.gga[4])
self.EW = self.gga[5]
self.quality = int(self.gga[6])
self.satellites = int(self.gga[7])
self.altitude = float(self.gga[9])
self.latitude = self.lat // 100 + self.lat % 100 / 60
if self.NS == "S":
self.latitude = - self.latitude
self.longitude = self.lon // 100 + self.lon % 100 / 60
if self.EW == "W":
self.longitude = -self.longitude
except ValueError as e:
debug( "FAILED: invalid value")
debug( e)
def get_latitude(self):
if time.time() - self.time_to_die > self.delay_to_die:
self.read()
return round(self.latitude,6)
def get_longitude(self):
if time.time() - self.time_to_die > self.delay_to_die:
self.read()
return round(self.longitude,6)
def convert(self, coord_in):
degrees = int(coord_in)
# print(f"Degrees: {degrees}")
minutes = int((coord_in - degrees) * 60 )
# print(f"Minutes: {minutes}")
seconds = round((coord_in - degrees - (minutes/60)) * 3600,2)
# print(f"Seconds: {seconds}")
return f"{degrees}°{minutes}'{seconds:{2}.2f}"
def get_latitude_str(self):
if time.time() - self.time_to_die > self.delay_to_die:
self.read()
return self.convert(abs(self.latitude))+self.NS
def get_longitude_str(self):
if time.time() - self.time_to_die > self.delay_to_die:
self.read()
return self.convert(abs(self.longitude))+self.EW
def get_utc0_time(self):
if time.time() - self.time_to_die > self.delay_to_die:
self.read()
return f"{self.timestamp[:2]}:{self.timestamp[2:4]}:{self.timestamp[4:6]}"
def get_nb_satellites(self):
if time.time() - self.time_to_die > self.delay_to_die:
self.read()
return self.satellites
if __name__ =="__main__":
print("STARTING GPS")
gps = GPS()
while True:
time.sleep(5)
in_data = gps.read()
# print (f"read: {in_data}")
print(f"lat: {gps.lat}, N/S: {gps.NS}, long: {gps.lon}, E/W: {gps.EW}, nb satellites: {gps.satellites}, google: {gps.convert(abs(gps.latitude))+gps.EW}, {gps.convert(abs(gps.longitude))+gps.NS} ")
assert(gps.convert(45.504103)=="45°30'14.77")
assert(gps.convert(73.806683)=="73°48'24.06")
This is the grovegps library file gps_reading.py
Note that this works, (but doesn’t parse properly), and has a “read” method.
#!/usr/bin/env python
# This example demonstrates how to use the Grove GPS sensor with the GoPiGo3.
# Connect the GPS sensor to the SERIAL port on the GoPiGo3. This port
# is on the right hand side of the GoPiGo3.
from __future__ import print_function
import serial, time, sys
import re
en_debug = False
def debug(in_str):
if en_debug:
print(in_str)
patterns=["$GPGGA",
"/[0-9]{6}\.[0-9]{2}/", # timestamp hhmmss.ss
"/[0-9]{4}.[0-9]{2,/}", # latitude of position
"/[NS]", # North or South
"/[0-9]{4}.[0-9]{2}", # longitude of position
"/[EW]", # East or West
"/[012]", # GPS Quality Indicator
"/[0-9]+", # Number of satellites
"/./", # horizontal dilution of precision x.x
"/[0-9]+\.[0-9]*/" # altitude x.x
]
class GROVEGPS():
def __init__(self,port='/dev/serial0',baud=9600,timeout=0):
self.ser = serial.Serial(port,baud,timeout=timeout)
self.ser.flush()
self.raw_line = ""
self.gga = []
self.validation =[] # contains compiled regex
# compile regex once to use later
for i in range(len(patterns)-1):
self.validation.append(re.compile(patterns[i]))
self.clean_data()
# self.get_date() # attempt to gete date from GPS.
def clean_data(self):
'''
clean_data:
ensures that all relevant GPS data is set to either empty string
or -1.0, or -1, depending on appropriate type
This occurs right after initialisation or
after 50 attemps to reach GPS
'''
self.timestamp = ""
self.lat = -1.0 # degrees minutes and decimals of minute
self.NS = ""
self.lon = -1.0
self.EW = ""
self.quality = -1
self.satellites = -1
self.altitude = -1.0
self.latitude = -1.0 #degrees and decimals
self.longitude = -1.0
self.fancylat = "" #
def get_date(self):
'''
attempt to get date from GPS data. So far no luck. GPS does
not seem to send date sentence at all
function is unfinished
'''
valid = False
for i in range(50):
time.sleep(0.5)
print (i)
self.raw_line = self.ser.readline().strip()
if self.raw_line[:6] == "GPZDA": # found date line!
print ("")
print (self.raw_line)
def read(self):
'''
Attempts 50 times at most to get valid data from GPS
Returns as soon as valid data is found
If valid data is not found, then clean up data in GPS instance
'''
valid = False
for i in range(50):
time.sleep(0.5)
self.raw_line = self.ser.readline().strip()
print(self.raw_line)
if self.validate(self.raw_line):
valid = True
break;
if valid:
return self.gga
else:
print("Invalid/no data received")
self.clean_data()
return []
def validate(self,in_line):
'''
Runs regex validation on a GPGAA sentence.
Returns False if the sentence is mangled
Return True if everything is all right and sets internal
class members.
'''
if in_line == "":
return False
if in_line[:6] != "$GPGGA":
return False
self.gga = in_line.split(",")
debug (self.gga)
#Sometimes multiple GPS data packets come into the stream. Take the data only after the last '$GPGGA' is seen
try:
ind=self.gga.index('$GPGGA',5,len(self.gga))
self.gga=self.gga[ind:]
except ValueError:
pass
if len(self.gga) != 15:
debug ("Failed: wrong number of parameters ")
debug (self.gga)
return False
for i in range(len(self.validation)-1):
if len(self.gga[i]) == 0:
debug ("Failed: empty string %d"%i)
return False
test = self.validation[i].match(self.gga[i])
if test == False:
debug ("Failed: wrong format on parameter %d"%i)
return False
else:
debug("Passed %d"%i)
try:
self.timestamp = self.gga[1]
self.lat = float(self.gga[2])
self.NS = self.gga[3]
self.lon = float(self.gga[4])
self.EW = self.gga[5]
self.quality = int(self.gga[6])
self.satellites = int(self.gga[7])
self.altitude = float(self.gga[9])
self.latitude = self.lat // 100 + self.lat % 100 / 60
if self.NS == "S":
self.latitude = - self.latitude
self.longitude = self.lon // 100 + self.lon % 100 / 60
if self.EW == "W":
self.longitude = -self.longitude
except ValueError:
debug( "FAILED: invalid value")
return True
if __name__ =="__main__":
gps = GROVEGPS()
print("Reading GPS sensor for location . . . ")
print("If you are not seeing coordinates appear, your sensor needs to be")
print("outside to detect GPS satellites.")
while True:
print("Reading GPS sensor for location . . . ")
time.sleep(2)
in_data = gps.read()
if in_data != []:
# Print out the GPS coordinates in Google Coodinates.
# You should be able to paste these into maps.google.com
# and see your location.
print ("Lat: {} North/South: {} Long: {} East/West: {}".format(gps.lat, gps.NS, gps.lon,gps.EW))
print ("Expanded Lat: {} Expanded Long: {}".format(gps.latitude, gps.longitude))
What I suspect happened is that, originally, Bloxter was intended to run as a part of DexterOS and be compatible with the GrovePi - and that you would use GrovePi methods on an earlier GoPiGo.
The easygps library appears to be a half-finished attempt to port the Grove library over to the GoPiGo. I am going to try renaming methods in the Grove version to conform to what the GoPiGo wants and see what happens.
Additional notes:
There are also easypicamera and easyusb libraries in the egg file for the gps. IMHO, I think the easygps should be part of easysensors.
The program as written is designed to do two things (AFAIK):
If it finds a valid GPGGA sentence, it parses it and prints the result.
If it does not find a GPGGA sentence, or it is mal-formed, (such as it would be with no satellite lock), it doesn’t print anything.
In my code, I added a print statement for the raw data as received so that I could determine if I was getting any data at all.
P.S.
My GPS module stopped getting satellites at all, though my cell phone could see dozens of them. I don’t know what happened. Maybe the ceramic antenna cracked when I dropped it, or the GPS module’s receiver section died?
I have another on the way for tomorrow and I will see what happens.
As far as I understood from the NEMA specification document, the NEMA sentences are plain-vanilla ASCII text, not utf-8, and they are just text strings instead of “byte data” - that you could send to a serial printer or terminal if you want. (And I did, miniterm.)
I wonder why python thinks it’s byte data instead of just a string of characters?
gps_read.py is its own, stand-alone, routine that interfaces with the serial port directly.
My eventual idea, (once I verify things), is to “rewrite” the easygps library by renaming gps_read and renaming some of the internal methods and renaming the class to “GPS” instead of “GROVEGPS” so that it matches the expectations of the rest of the system.
In fact, if you look at the original easygps code, you will notice that Robert Lucian mentioned that he couldn’t get the read routines working.
AFAIR, someone else (Anna?) finished off the Grove GPS routines.
I will re-check the manual tomorrow but I believe it specifies ASCII expecting it to be used on resource constrained systems.
AFAIK UTF-8 is a two-byte encoding standard since 8 bits can only represent 256 char’s and the characters from 0x00 to 0x1f are control characters. 0x20 through 0x40 are punctuation and the digits 0-9.
Unicode “ASCII” looks like this:
(ABCDEFG)
0028 0041 0042 0043 0044 0045 0046 0047 0029
0x28 is the open parentheses, 0x29 is the closing parentheses, and 0x41 through 0x47 are the capital letters A through G.
That’s why older, non-unicode aware text editors would choke on UTF-8 encoded text.
I know, (or highly suspect), that the GoPiGo3 is still in use in educational circles.
Agreed that the GPG code needs a bit of love, especially since it looks like several GPG projects got cut off at the knees when MR took over, not really knowing what they had.
(Which, IMHO, is why some of the GPG stuff looks like a fuster-cluck on steroids - it got decapitated.)
If I can demonstrate that:
Various non-Grove GPS units will work if wired correctly.
Note that the non-Grove, non-Seeed versions are much less expensive. ($7 compared to almost $30.)[1]
The GoPiGo O/S routines, including Bloxter, can run the stuff natively.
The GPG3 can effectively use a GPS sensor without jumping through hoops.
The only requirement is that it communicate using NEMA GPS sentences, serial, 5v, at 9600 8N1, which is quite common. Most of the ones I see online run on either 3.3v or 5v, and even if run at 3.3v, it seems that most can handle TTL voltage levels.
It should be noted that a number of the GPS modules also support a packed-binary U-Blocks format too. Though interesting, and potentially more efficient as it exposes additional features and capabilities, I’m going to leave that programming task as “an exercise for the student”.
Re: Standards? We don’t need no stinkin’ standards!
The question of what is, (and is not), a GoPiGo “standard” was raised, and this is my considered opinion as to what constitutes a GoPiGo “standard” library.
My definition of a “standard” library for the GoPiGo robot:
It comes built-in with the “standard” operating system release. (Currently GoPiGo O/S 3.0.3.)
It’s used by other GoPiGo functionality that’s a part of the standard O/S. (i.e. Bloxter, et.al.)
It is prefixed with “easy” as in “easygopigo”, “easysensors” and “easygps” because the “easy” libraries, (found within the various “easy” egg files), are the usual API entry points for access to the GoPiGo features and functions.
It might not be included with the externally downloaded versions for things like Ubuntu, (though it should), and a major goal is to make sure that any updates are included in any downloaded versions.
Examples are just that - examples - indicating usage information or providing additional details.
IMHO, examples provide guidance into either how to use existing functionality or provide a way to understand how particular functionality works “under the hood” if you need to specifically modify it or “grow your own”.
The gps_readings.py example shows exactly how to read GPS data from the serial port and parse it into useful values.
Note that we’ve also used the GoPiGo library files as API examples and a view into the underlying “nuts and bolts”, so the line between an “example” and a “library” has become somewhat blurred.
The “standard” GPS library:
If you look at the easygps library file (which I unpacked from the associated egg file), and the gps_readings.py file, you will notice that they are substantially similar, with the exception of the base class name and - maybe - the names of some of the methods.
My plan is to use the gps_readings file to replace the easygps library by renaming various parts to match the easygps library’s expectations. Once this is done it can be used as a drop-in replacement for the original easygps library that’s currently broken.
========================================
Potential enhancements:
Add a “read_raw” method that will return the most recent sentence received. This would allow the user to read a raw GPS data sentence if necessary. Because of the way GPS data is received as continuing bursts of serial data, (separated by one second pauses), any particular “read_raw” call may not return a specific sentence - or sentences in any particular order - since it would simply grab the most recent sentence received and return it.
Add a “read_sentence(“name”)” method that would take as its argument a particular sentence name, (GPxxx) as a string, and it would return the next instance of that sentence as raw data.
Add a “read_frame” that would return an entire data frame, (defined as the entirety of the data transmitted between 1-second pauses). I am not sure exactly how to organize it, especially since different GPS units can return either a larger or smaller subset of the NEMA GPS sentences.
The idea behind these methods is to provide the ability to do custom post-processing if the programmer wants to do so, or access features of a particular GPS module that may not be present in other GPS modules.
Another possible method to organize the libraries would be to divide the libraries into two parts - gps and easygps - with the gps library providing raw sentence data, sentences, or data frames and the easygps library provides methods for extracting things like UTC/local time, Lat/Lon, velocity, heading, altitude, etc. (Note the relationship between the gopigo and easygopigo libraries.)
========================================
My current plan is to reformulate gps_readings into a new easygps library and potentially amplify it at a later date.
Other modifications can be done on a time-available, (interest available ), basis.
What say ye?
P.S.
I don’t know why, but my GPS module is working again. . . .
It appears that the RF radiation from the robot, especially if I have an external SSD plugged in, “jams” the GPS receiver, causing it to loose its satellite lock.
If the robot is using a SD card, the minimum distance is a couple of centimeters, depending on the placement of the antenna.
If the robot is using an external SSD, (like the 500gb Seagate external SSD’s I have), the minimum distance is somewhere between 30 and 40 cm measured from the edge of the robot and the SSD.
It appears that emitted RF interference plays a huge role in the robot’s ability to use a GPS.
More research is needed.
Apparently there are still a few flies in the ointment:
It appears that later versions of Python interpret serial data as “byte” objects instead of “character” objects.
The original code, (perhaps intended for Python 2?), doesn’t seem to anticipate the need to convert to Unicode, which appears to be a requirement for Python 3.
Python assumes that any character data will be in UTF-8 Unicode - the concept of “raw ASCII” appears to be alien to Python.
Encoding data into UTF-8 has a fundamental assumption that the data is a “printable” ASCII character. If it’s not a printable ASCII character, it throws an exception.
You can override this by telling it to either “ignore” or “replace” the offending data.
“ignore” sends the offending bytes through unchanged.
“replace” substitutes a “replacement” character, (a diamond-shaped question mark that represents the offending character).
(i.e.str(serial_data, encoding="utf-8", errors="ignore")
The gps_readings routine is looking for a sentence type that these boards don’t provide.
I am planning to try a different, (more common), sentence type as the sync character.
So far I have not been able to get the gps_readings routine to sync and actually process data.
I now have two (slightly) different types of GPS module that I can work with and see if either one is easier to sync up.
I am still waiting for the Grove type GPS module to arrive.
I was able to get the gps_reading.py routine to work with the U-Bloks GPS modules by making the following modifications:
Change the target “validate” string from $GPZDA to $GPGGA
Change the read statement from: self.raw_line = self.ser.readline().strip()
to: self.raw_line = str(self.ser.readline().strip(),encoding='utf-8', errors='ignore')
It produces output that is reasonable and repeatable.
Reading GPS sensor for location . . .
If you are not seeing coordinates appear, your sensor needs to be
outside to detect GPS satellites.
Reading GPS sensor for location . . .
Lat: 5533.04882 North/South: N Long: 3734.9303 East/West: E
Expanded Lat: 55.55081366666666 Expanded Long: 37.58217166666667
Reading GPS sensor for location . . .
Lat: 5533.04895 North/South: N Long: 3734.93032 East/West: E
Expanded Lat: 55.55081583333334 Expanded Long: 37.582172
Reading GPS sensor for location . . .
Lat: 5533.04903 North/South: N Long: 3734.93034 East/West: E
Expanded Lat: 55.55081716666667 Expanded Long: 37.58217233333333
Reading GPS sensor for location . . .
Lat: 5533.04917 North/South: N Long: 3734.93039 East/West: E
Expanded Lat: 55.5508195 Expanded Long: 37.582173166666664
Reading GPS sensor for location . . .
Lat: 5533.04936 North/South: N Long: 3734.93056 East/West: E
Expanded Lat: 55.55082266666667 Expanded Long: 37.582176
Reading GPS sensor for location . . .
Now that this is done, I can begin trying to find out why the easygps.py code doesn’t appear to have a “read” routine, how to provide it, and where to go next.