Multithreading in Python with the GrovePi+

Archives GrovePi
,
1

Hi Robert,

I have been trying over the weekend to create a function that has a continious loop. This loop counts the number of pulses created by created by for testing pressing a switch, that will be replace by Anemometer, which has a magnetic switch that when closed, to produces a pulse, that can be read by the grovepi.digitalRead…

So far I have a scipt that loops and counts the number of pulses over 2 seconds, then resets to 0.Pulse_Counter.py (1.5 KB)

My problem is that I need to run in the background as fuction and continously runs, and outsputs the total number of pulses counted, so the maths can be done to convert to speed (mph, etc).

Have you any ideas how this can be done, I have been trying to get this to work without any luck. Pulse_Counterv1.py (1.7 KB)

Most of the other blocks are about finished.

Are you or a colleague able to help me solve this?
Many Thanks Michael

2

I’m not a colleague but I do have a suggestion. You need to use threads. When you run your pulse counter program it is running in it’s on thread. Any functions you run are on that same thread. However it is possible to run a function in another thread. For more info take a look at the excellent and extensive documentation for it in python.
https://docs.python.org/3/library/threading.html

Also check out a nice tutorial on it.
https://www.tutorialspoint.com/python/python_multithreading.htm

3

Hi @whimickrh,

You need to separate the main process and your best bet is to use threads.

Also, thank you @graykevinb for providing those excellent resources. I think that documentation is the best option for anyone who desires to learn multithreading. Also it’s a really good resource for learning anything about Python. So I’d hold onto that one.

Here’s a very simplistic template of what you want to achieve.

import threading

output_function_variable = None
thread_stopper = threading.Event()

def function_needed_for_parallelization():
    # use variables that're already defined
    global output_function_variable
    global thread_stopper

    while thread_stopper.is_set() is False:
        # .. do some coding

        # set the output_function_variable to some value you want to use in your main program
        # the idea is to continuously assign a value to your variable so it can be used in main function
        output_function_variable = x


def Main():
    global output_function_variable
    global thread_object

    # create a thread object which will run function_needed_for_parallelization on start
    thread_object = threading.Thread(target = function_needed_for_parallelization)
    # launch the function_needed_for_parallelization function
    thread_object.start()

    # do your parallel code
    # and use here the output_function_variable
    # like the following
    print(output_function_variable)

    # set the thread stopper -> will cause the while loop to end
    # in function_needed_for_parallelization
    thread_stopper.set()
    # wait for function_needed_for_parallelization to stop
    thread_object.join()

It ain’t the best architecture for it, but it’s going to do the job and it’s a great way to start learning multithreading.
So, try to understand this template code in conjunction with the resources @graykevinb pointed out and see how far you can go.

Finally, I’m suggesting you to move on to Python 3, as Python 2 is going to become obsolete in the near future.

Thank you!

4

Thankyou both for your replies, they were very helpfull. I have had a play with multi Threading. With some success, however I have not been able get the total pulsecount produced over a period of seconds without using,
return pulsecount which causes the thread or function to exit
The loop I am using is below.
Have you any suggestions how to get around this problem?

Many Thanks Michael

class ThreadingExample(object):
	"""
	Threading example class
	The run() method will be started and it will run in the background
	until the application exits.
	"""

	def __init__(self):
		"""
		Constructor
		:type interval: int
		"""
		thread = threading.Thread(target=self.run, args=())
		# thread.daemon = True                            # Daemonize thread
		thread.start()                                  # Start the execution

	def run(self):
		global trigger, pulse, pulsepersecond
		""" Method that runs forever """
		while True:
			endTime = datetime.datetime.now() + datetime.timedelta(seconds=2)
			# this is the end of the 1 #second loop in seconds
			while datetime.datetime.now() <= endTime:
				# pulsepersecond = 0
				if grovepi.digitalRead(Anemometer) == 1:
					# print "pulse"
					if trigger == 0 and (grovepi.digitalRead(Anemometer) == 1):
						# this stops the switch being read more than once
						pulse = pulse + 1
						trigger = 1
						# print "Counting Pulses", pulse
						sleep(0.032)
					else:
						trigger = 0
						pulsepersecond = pulse
						# print "Pulsepersecond:", pulsepersecond
						# divided by 2 because the read switch open and closes twice per revolution
			print "Pulsepersecond:", pulsepersecond
			pulse = 0
			return pulsepersecond


	def main(self):
		# thread = threading.Thread(target=self.run, args=())
		global record_no
		# digitalWrite(blue_led, 1)

		while True:
			try:
				#if threading.enumerate() == 1:
					#thread.start()
				pulsepersecond = self.run()

			except KeyboardInterrupt as e:
				print"ERROR"
				exit(0)
			else:
				print "loop-Pulsepersecond:", pulsepersecond

			time.sleep(2)

if __name__ == "__main__":
	ThreadingExample().main()
5

Hi,
threading is a bit tricky to wrap your head around at first. :slight_smile:
You basically launch a second independent loop. You need to figure out how to get both loops running (which you managed) but you also need to have a way to stop both of them (or your program will get stuck waiting for the secondary thread).
I have edited your code into a working example. Unfortunately I don’t have a GrovePi so I had to simplify that piece.

multiThreading.py (1.5 KB)

Hopefully this helps you along!

-Frost

6

Hi @whimickrh,

I think using classes and initializing a thread within one is not the best option for us - nor deriving the Thread class and creating a custom-made Thread class.

For the moment, starting a thread the ol’ fashioned way is just going to do the job, plus it’s going to help you better understand how it does work.

Here’s the example I’ve created with the template from my previous post, but this time it’s going to measure the number of digital pulses on D5 port.

import threading
import time
import grovepi

thread_stopper = threading.Event()
anemometer_pin = 5
pulsepersecond = 0

def anemometerMonitor():

	# use global because pulsepersecond is going to be modified
	global pulsepersecond
	measure_time = 2 # measured in seconds
	trigger = 0
	wait_time = 0.032 # measured in seconds

	grovepi.pinMode(anemometer_pin, "INPUT")

	while thread_stopper.is_set() is False:

		current_pulses = 0
		pulsepersecond = 0
		start_time = time.time()

		while time.time() - start_time < measure_time:
			if grovepi.digitalRead(anemometer_pin) == 1:
				if trigger == 0:
					current_pulses += 1
					trigger = 1
					time.sleep(wait_time)
				else:
					trigger = 0
					pulsepersecond = current_pulses

	print("thread stopping")

def Main():
	# use global because we're going to set the event
	global thread_stopper
	global pulsepersecond

	# create a thread object which will run anemometerMonitor on start
	thread_object = threading.Thread(target = anemometerMonitor)
	# launch the anemometerMonitor function
	thread_object.start()

	while True:
		print("Pulses per second : {}".format(pulsepersecond))
		time.sleep(1)

	# set the thread stopper -> will cause the while loop to end
	# in anemometerMonitor
	thread_stopper.set()

if __name__ == "__main__":
	try:
		Main()

	except KeyboardInterrupt:
		print("CTRL-C pressed")
		thread_stopper.set()

	except IOError:
		print("An IO Error occured")
		thread_stopper.set()

I have tested it with a switch button and it seems to be measuring the number of button presses okay.
Please give it a try and try to understand how it was done.

Thank you!

7

Thankyou all for taking the time to write scrpts, I will try them out tomorrow and let you know my results.
I also have been playing, and you last script worked OK(ish). I am fortunate in that I have two Raspberry Pi3,s with a GrovePi mounted on top and are running the same versions of Raspbian jessie, and up to date Grove Software. Sadly I had problems with Raspbian for Robots, in that I was unable to install some extra packages. Which i am not bothering at thia present time.
One thing issue I do have is I am not able to run GrovePi with Python 3. so for now I will stick to Python 2 for now, and get it working on that.
Regarding the programming the windspeed pulse counter I am using my second GrovePi that has a grovepush button switch to simulate the anemometer as a test bed. I have been able to assemble a scipt for testing on my other RPI that is going the be used as my main weather station and had all the sensors connected. Which are connect as listed below.

--------------------------BOARD AND SENSOR SET UP---------------------------

rev = GPIO.RPI_REVISION
if rev == 2 or rev == 3:
bus = smbus.SMBus(1)
else:
bus = smbus.SMBus(0)

GPIO.setwarnings(False)

12C SENSORS

ds3231 = SDL_DS3231.SDL_DS3231(1, 0x68) # RTC

bmp280 = BMP280.BMP280() # GROVE 12C TEMPERATURE-HUMID-BARO

bmp280 = BMP280.BMP280()
hdc = HDC1000()
hdc.Config()

GROVE SENSORS

Anemometer = 5 # Pin 5 is D5 Port. ----- Anemometer
Rain_Tipper = 6 # Pin 6 is D6 Port. ----- Rain_Tipper
Light_Sensor = 14 # Pin 14 is A0 Port. Grove - Light_Sensor
Water_Sensor = 15 # Pin 15 is A1 Port. Grove - Water_Sensor
Moisture_Sensor = 16 # Pin 16 is A2 Port. Grove - Moisture_Sensor

grovepi.pinMode(Anemometer, “INPUT”) # Anemometer
grovepi.pinMode(Rain_Tipper, “INPUT”) # Rain Tipper
grovepi.pinMode(Light_Sensor, “INPUT”) # Grove - Light Sensor
grovepi.pinMode(Water_Sensor, “INPUT”) # Grove - Water Sensor
grovepi.pinMode(Moisture_Sensor, “INPUT”) # Grove - Moisture Sensor

dust_ppm = dust_sensor_lib.Dust_Sensor()

GROVE LEDS AND GROVE DIGITAL

blue_led = 4
flash_led = 3
digitalWrite(flash_led, 0)
digitalWrite(blue_led, 0)

adc = Adafruit_ADS1x15.ADS1115() # ADC ADS1115
GAIN = 1 # 5volts
value = 0
voltageValue = 0

For now the dust sensor is disconnected as that is causing me problems, again I will look at this later, after everything else is working.

Thanks for your help, please keep the suggestions coming, as I hope it may encourage others the try and build a weather station.

Michael

8

Here is the weather staiton script so which has been working, how ever I have not been able to exit it .

Raspi_Met_v4.09.py (9.7 KB)

9

Hi @whimickrh,

I see a couple of mistakes in your code that need to be corrected.

Let’s take this fragment of code:

def wind_direction():
	global degrees, bearing, voltageValue
	""" Method that runs forever """
	
	while True:
		endTime = datetime.datetime.now() + datetime.timedelta(seconds=1)
		# this is the end of the 1 #second loop in seconds
		while datetime.datetime.now() <= endTime:
			value = adc.read_adc(1, gain=GAIN, data_rate=250)
			time.sleep(0.1)

Since this function is threaded, you also need to have an event stopper here.
What is going to happen if there isn’t a trigger/stopper?
Well, when you hit CTRL-C the main thread will exit, except for this thread, because it doesn’t know that the main exited - from this perspective, threads are blind and don’t know when another thread exits. You need to give them a trigger.

Let’s analyse another fragment:

def Main():
	digitalWrite(blue_led, 1)
	# use global because we're going to set the event
	global thread_object, pulsepersecond, record_no

	# create a thread object which will run anemometerMonitor on start
	thread_object = threading.Thread(target = anemometerMonitor)
	thread_object = threading.Thread(target = wind_direction)
	# launch the anemometerMonitor function
	thread_object.start()

Here, you assign to the same variable 2 thread instances.
If you do this, the first one you’ve assigned to will get “overwritten” by the second, so this means that only the 2nd gets launched, whereas the 1st one is lost.
You need separate variables for each thread object.

Here is a good suggestion: https://www.tutorialspoint.com/python/
It’s great you already have a project to experiment on, so take this tutorial and get through all of it.
The idea is to experiment as much as you can - and you have the chance with this weather station project.

If there’s something that’s unbearable or too hard to understand, please reach out here and describe your situation, just as you have done so far.

Thank you!

10

Hi Robert,

Thanks for your help I already use tutorialspoint.com and stack exchange for help, and spent many an hour reading through posts and tutorials.
And have found them a great help, but saying that they only solve some of my problems.
And do not ask for help unless I’m stuck.

I have made some adjustments to the threading code as below:
global thread_anemometer, thread_wind_direction, record_no

# create a thread object which will run anemometerMonitor on start
thread_anemometer = threading.Thread(target = anemometerMonitor)
thread_wind_direction = threading.Thread(target=wind_direction)

# launch the anemometer & wind_direction function
thread_anemometer.start()
thread_wind_direction.start()

The problem now is that one of these threads stops after a few loops, and I have not been able to find the reason why, or re-start the thread again.

I have not been able to detect any errors, which making it have to solve.

And so far despite my searching for a solution, and hours of playing, I have not been able to solve the problem. Believe or not I am slowly getting there, its can be very hard going and sometimes the progress is so. I have tested each of the blocks separately and they appear to work.

Can any one help.
Many thanks

11

Hey @whimickrh, just wanted to touch base and see if you had solved this or figured out what wasn’t working with multithreading here?

12

Hi Robert and John,

I think I have managed to set the multithreading working after many hours of research and playing. Has run though out today, with no problems so far.

Please take a look at the script and comment.

I have not used python class as I have found them hard to understand, dispite spending some time playing and researching.
Would a class improve the script?

Raspi_Met_Beta_v1.19g.py (14.4 KB)

Many Thanks Michael

13

Hi Robert and John,

Just thought I would post my latest two GrovePi Weather Station Scripts.
Both create a WeatherDisplay CSV and one will output to Weewx

At this moment both seem to work, however they some times throw a threading script error, which I am looking into.

Any help would be gratefully recieved.
Michael.

Raspi_Met_WD.py (15.0 KB)
Raspi_Met_WDWX.py (17.3 KB)

15

Hi All,
When I run the above scripts I ben getting this error.

Jun 18 02:44:32 RaspiDexmet1 weewx[1244]: engine: Shutting down StdReport thread
Jun 18 02:44:34 RaspiDexmet1 weewx[1244]: cheetahgenerator: Generated 14 files for report StandardReport in 2.57 seconds
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]: imagegenerator: Generated 13 images for StandardReport in 1.38 seconds
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]: copygenerator: copied 0 files to /home/pi/WxRam
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]: engine: Caught unrecoverable exception in engine:
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****  can't start new thread
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****  Traceback (most recent call last):
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****    File "/home/weewx/bin/weewx/engine.py", line 871, in main
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****      engine.run()
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****    File "/home/weewx/bin/weewx/engine.py", line 187, in run
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****      for packet in self.console.genLoopPackets():
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****    File "/home/weewx/bin/user/Raspi_Met_WDWX.py", line 468, in genLoopPackets
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****      thread_wind_gust.start()
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****    File "/usr/lib/python2.7/threading.py", line 745, in start
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****      _start_new_thread(self.__bootstrap, ())
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****  error: can't start new thread
Jun 18 02:44:36 RaspiDexmet1 weewx[1244]:     ****  Exiting.

After some searching it would appear that after a period of time the scripts seem to have to many threads running at once.

Any idea how I could solve this problem?
could the return statement be the causing the wind_speed and wind_gust functions to exit without closing the threads.

Michael

16

Hi @whimickrh,

I’ve looked over your code and the can't start new thread error is almost certainly due to the fact that there are way to many threads running within your python process (I’m referring to the process that’s created when Raspbi_met_WDWX.py is launched).

There’s a limit as to how many threads can be run within a single process, but that number is pretty high and is also dependent on the system you’re running the process on.

I’d suggest to rethink the code’s architecture, since this is where the original problem came from.

Here are a couple of reasons for rethinking the architecture :

  • There’s a “leakage” somewhere which leads to the creation of lots of threads - which is the actual issue.

  • Inside the code, the wind_gust function is started by a thread and after that there’s an instruction that’s calling the wind_gust function again - and this is happening within the try-except block.
    The function which calls this wind_gust function again is wind_speed and that shouldn’t happen - this can lead to a myriad of complications.

  • Too many global variables I think. The issue is that too many global variables can slowly lead to a spaghetti code.
    Structured code is the way to go and this usually means concepts such as object programming and modularity.

I hope I’ve helped you understand the issues.

Again, if there are other issues you want to discuss about or you didn’t understand something I’ve said, please don’t hesitate to ask me.

Thank you!

17

Thankyou for taking the time.

Sadly I have not had any expierience of object programming or modularity, I still learning. However I will do some research.

As a guide would lokking at some of the GrovePi driver scripts give me some clues,I have also found a website that tries to explain the above.

I will not post any more scripts for now until I had a play.
I would grateful for any examples or web pages,you could suggest

Many Thanks

18

Hi All

I sorry for my silence recently, I have been tweaking my weather station scripts, and I hope to post my latest efforts in the next few days.

Michael

19

Hi @whimickrh,

And we haven’t forgotten you.
I’m happy to hear you’ve been working on your station and we’re curious to see how far you’ve got.

Thank you!

20

HI Robert and everone,

Here is my latest attempt at a GrovePi Weather station.
It uses in two scripts.

  1. Raspi_met_wd_v7.py Which to main Weather Station

  2. Wind_Rain1.py This is a module and runs the, anemometer wind vane and rain tipper gauge, these plug in to a Switchdoc lab Grove Weatherboard, and use standard Grove able to connect to the GrovePi ports
    .
    You may also need to use, python Adafruit Python ADS1x15, python Adafruit Python GPIO Library, and Python BMP280. And some Switchdoc libraries

I have one small problem in that the timer in the two scripts causes this error and can effect the timing. Which does not stop the scripts form working.
[Errno 22] Invalid argument Traceback (most recent call last):
File “/home/pi/Raspi_met_wd_v7.py”, line 318, in the_loop time.sleep(dif_time)
IOError: [Errno 22] Invalid argument.

Perhaps some could help me solve this error.

I would be very grateful for any adjustment or amendments, and perhaps a new thread could be created for those who may be interested.

Raspi_met_wd_v7.py (12.6 KB)
Wind_Rain1.py (4.1 KB)

Many Thanks
Michael

21

Hi @whimickrh,

Within the following section, in Raspbi_med_wd_v7.py file, at line 318:


start_loop = time.time()
met_out()
record_no = record_no + 1
#time.sleep(loop_time)
finish_loop = time.time()
loop_dif = finish_loop - start_loop
dif_time = loop_time - loop_dif
time.sleep(dif_time)

You need to make sure dif_time is a positive float, otherwise, time.sleep will throw an IOError [Errno 22].

Please let us know if this solves your issue.

Thank you!