How to set a custom Python path system wide that sticks

GoPiGo OS, DexterOS and Bloxter
1

Greetings!

Background:
I am attempting to do some experiments with my GoPiGo3 robot, attempting to add a mutex to the SPI communication routines. I want to do this because I would like to use a GPIO Waveshare e-Paper display that uses a separate SPI channel that the GoPiGo controller doesn’t use.[1]

There appears to be a conflict/collision between the two devices, so I want to try a mutex to force them to cooperate.

The robot is using a Pi-4 running Raspberry Pi O/S Buster.

I am using Thonny as my primary programming IDE.

Note that Buster is a hard requirement for this robot’s libraries and code and it cannot be changed. (I say this because every time I say “Buster” folks say “Upgrade to Bookworm!!” and that is not possible at the present time.)

There is an egg file located at
/home/pi/.local/lib/python3.7/site-packages/gopigo3-1.3.2.1-py3.7.egg.

This is the system default library path for the special GoPiGo3 function libraries.

I want it to use the gopigo3 library files that I have copied from that egg file into /home/pi/Test_Libraries/, entirely ignoring the existing egg file.

The reason for this is I want to specify my experimental files that I will edit to create a mutex wrapper around the existing SPI code. In order to do this, I have to be able to guarantee that my local python libraries are being used instead of the default libraries within the egg file.

Issue:
I want to force the python path to somewhere else for certain libraries and it doesn’t seem to work.

Looking around the internet, I have done the following based on suggestions from Stack Overflow and others:

Set the python path within a terminal window:

pi@CharlineWiFi:~ $ export PYTHONPATH=${PYTHONPATH}:/home/pi/Test_Libraries/easygopigo3.py
pi@CharlineWiFi:~ $ export PYTHONPATH=${PYTHONPATH}:/home/pi/Test_Libraries/easysensors.py
pi@CharlineWiFi:~ $ export PYTHONPATH=${PYTHONPATH}:/home/pi/Test_Libraries/gopigo3.py

pi@CharlineWiFi:~ $ echo ${PYTHONPATH}
:/home/pi/Test_Libraries/:/home/pi/Test_Libraries/easygopigo3.py:/home/pi/Test_Libraries/easysensors.py:/home/pi/Test_Libraries/gopigo3.py
pi@CharlineWiFi:~ $

Restarting Thonny I see the following in the RPL shell:

Python 3.7.3 (/usr/bin/python3)
>>> import easygopigo3 as easy
>>> print(str(easy))
<module 'easygopigo3' from '/home/pi/.local/lib/python3.7/site-packages/gopigo3-1.3.2.1-py3.7.egg/easygopigo3.py'>
>>>

I edited the .pypath file as follows:

/usr/local/lib/python2.7/dist-packages/gopigo3-1.2.0-py2.7.egg
/usr/local/lib/python2.7/dist-packages/gopigo3-1.2.0-py2.7.egg
/home/pi/Test_Libraries/*
/home/pi/Test_Libraries/easygopigo3.py
/home/pi/Test_Libraries/easysensors.py
/home/pi/Test_Libraries/gopigo3.py

Then I added the following to the end of the .bashrc file:

# export python path
export PYTHONPATH=${PYTHONPATH}:/home/pi/Test_Libraries/:/home/pi/Test_Libraries/easygopigo3.py:/home/pi/Test_Libraries/easysensors.py:/home/pi/Test_Libraries/gopigo3.py

I rebooted the robot and then executed the following in a terminal:

pi@CharlineWiFi:~ $ echo ${PYTHONPATH}
:/home/pi/Test_Libraries/:/home/pi/Test_Libraries/easygopigo3.py:/home/pi/Test_Libraries/easysensors.py:/home/pi/Test_Libraries/gopigo3.py
pi@CharlineWiFi:~ $

. . . and it returns the desired path.

However, when I open Thonny and run:

import easygopigo3 as easy
print(str(easy))

I get the following response in the RPL window:

Python 3.7.3 (/usr/bin/python3)
>>> %Run Python_Path.py
<module 'easygopigo3' from '/home/pi/.local/lib/python3.7/site-packages/gopigo3-1.3.2.1-py3.7.egg/easygopigo3.py'>
>>>

Note that this is still the default system path to the library .egg file instead of my custom installation of these libraries.

Request:
How do I:

  1. Set the python path for my libraries at the system level so every robot function uses these libraries?
  2. Set the python path so that Thonny uses the desired path when I am testing code in Thonny?

Bonus question:
Is it possible to completely stop and restart all the robot functions without rebooting?

Thanks in advance for any help you can provide.

========== Footnotes: ==========

  1. The GoPiGo3 controller board is set to use chip-select-1 (CS-1) and the Waveshare e_Paper display is set to use chip-select-0 (CS-0)
2 Likes
2

I agree chasing Python paths is a nightmare, so … what I do:

  • either put any changed library files in the execution folder
    or
  • explicitly point to a “my useful Python scripts” folder:
import os

sys.path.insert(1,"/home/pi/GoPi5Go/plib/")         # puts my lib before all dist package folders
from myEasyGoPiGo3 import EasyGoPiGo3          # located in  /home/pi/GoPi5Go/plib/

Otherwise you have to build your own egg file with your libraries and install it. Python will search local libs first before searching dist-package libs. This is too painful for me, although for my GoPiGo3 API on Bookworm, I actually backed up the DI files I needed to change, changed the files, and then ran the sudo setup.py install which made my (totally tested before do this) version appear as the official version to every application on the system.

2 Likes
3

I discovered that making the changes I made, adding to the system path, adding to .pypath, .bashrc, and .profile cause a terminal window execution to return the correct path. (Oops! I need to test it with sudo!)

However, any IDE I try always uses the original path to the egg file instead of my libraries.

2 Likes
4

Another reason to be explicit in the program - it always works, and the dependency is obvious, not hidden off somewhere:

2 Likes
5

Absolute gospel truth.

However. . . .

Since the base code is buried in system level libraries that are run by the robot’s controller code, that’s not feasible.  That’s why I redirect to my test libraries globally - I don’t know what piece of code hidden where might use a feature I don’t control.

Also I have ONE place - the /etc/profile file - to manage the global system path to these libraries.  I make one change, in one file, and everything’s done.

Unfortunately I don’t know where that other path is set.  I’m probably going to have to do a global search in every file in the filesystem to find that string.

P.S.
It appears that the files in the user’s home directory do NOT need to be modified if I add my new path to /etc/profile.

2 Likes
6

It is feasible, (really very few files identified by following the DI import statements) - but you are doing fine your way.

2 Likes
7

Also true.

However I like the idea of a “global switch”, kinda’ like an “ifdef ‘DEBUG’” that provides a single point switch for everything.

Right now I’m doing a grep for that path string on the entire mortal file-system and, (as expected), it’s taking forever!

1 Like
8

Correct, existing files never need to be modified with any of the path solutions. I find documenting the fact that an imported file/object has been changed is useful to remember what files I changed.

2 Likes
9

I use this often:

#!/bin/bash

# FILE:  what_uses.sh
# USAGE:  ./what_uses.sh "pattern"

if [ "$#" -ne 1 ] ;
	then echo "Usage:  ./what_uses.sh \"import xyz\" "
	exit
fi
echo "Searching to find what Python files use \"$1\" "
grep -r --include "*.py" "$1" .
2 Likes
10

Great!

Except that this string probably isn’t in a Python file.

Or did I misunderstand something?

1 Like
11

I think you might be searching for a static string that does not exist - Python dynamically builds the PythonPath with modifiable rules. That is why I like to use the explicit “put my folder first” directive in the script.

Many IDEs run their own Python environment, so even more so the “put my folder first” directive in the script is effective to ensure you know where the imports are coming from.

2 Likes
12

I ran

grep -Rnw '/' -e '/home/pi/.local/lib/python3.7/site-packages' > ./Grep_Files_With_Path.txt

from a root prompt, redirecting output to a file.  It’s been running for hours and is still chewing.

1 Like
13

OK, I tried it your way.

So long as I do the path insertion before I import easygopigo3, it works.

Honestly, I did not know you could “do things” before importing something.  I always thought you had to import everything first before doing stuff.

2 Likes