Wanted: A good SPI mutex

[jimrh]

One additional question:

Your sample routines contain USER = [xxx] statements and I am assuming that they are simply labels to indicate what’s happening.

I also am assuming that both of the sample programs and the class instance are running as, and are owned by, the “pi” user.

I am also assuming that any file created by the pi user is modifiable by the pi user.

I also assume that the mutex works because “user 2” cannot open the mutex for write access when “user 1” has it.

Question:
Why?

Both processes are owned by the pi user so they should both have write access to the same file, right?

However, it appears that the second process, despite being owned by “pi”, cannot write to an open file it owns.

Possible answer:
open([path/filename], w) returns a file handle, (a numerical pointer to the open file instance), as in FILE_HANDLE = open([path/filename], w) and you access the file using something like write(FILE_HANDLE, bytes).

If a second process, or even a different place in the same process, tries to “open” the same file again via FILE_HANDLE2 = open([path/filename], w), since the file is already open and has an assigned file handle, it can’t be re-opened for write a second time unless the first instance has been formally closed via close(FILE_HANDLE).

Am I correct?

Additional question:

I know that variables can exist within a local function/class scope and multiple instances of the same variable can exist due to scoping rules.

Can there be files with local scope?

Class(Test_Class, [etc.])
    [various statements]

    def(method, [xxx])
         HANDLE =open([path/filename], w)
         (do something with it)
    [more stuff]

(end of class definition)

---

Process_1

Instance = Test_Class([parameters])
    [do something using the file]
etc

---

Process_2 (which can run in parallel with process 1)

Instance_2 = Test_Class([parameters])
    [do something using the file]
etc

Are the two file instances local to the class instances or do they reference the same object if [path/file] are identical?

My assumption is that files exist independently of the object creating them, but are “owned” by the creating object.  Therefore the second class instance, if trying to open the same path/filename in an exclusive way (for write) while the first class instance has it open for write will fail and possibly throw an exception.

[jimrh]

@cyclicalobsessive

(Note this is a multi-processing mutex. It does not protect multi-threaded where multiple threads run under the same process.)

Translation:
If I try to grab the lock more than once within a particular [process|program|thread|etc.], it will succeed.

However if I have two separate [process|program|thread|etc.]. . .

. . . the calling [whatever] has a unique ID that is not shared by any different [whatever] that may want to use the lock file - and the different [whatever] will fail to get the lock.

Does failing to get the lock throw an exception or simply fail silently w/o a file handle or a non-zero return code?

[. . . .]
throw
    grab_lockfile([path/filename], w)
    [more stuff]

catch
    print("oops! it didn't work!!")
    [more stuff]

or more like

#  if "grab_lockfile" fails to get the lock,
# it returns a non-zero value which evaluates as "true".
if (grab_lockfile([path/filename, w)):
    then:
        print("oopsie!  Someone else has it!")
        [more stuff]
    else:
        print("I got it!  I got it!!")
        [more stuff]
fi

[jimrh]

@cyclicalobsessive

For the example mutex, both processes open the file for write access. Both get write permission when they open the file, but at some point one of the mutex instances changes the [shared|exclusive] flag preventing the other process from using the write permission it has.

Doesn’t that happen automagically behind the scenes?

My reading of your sample mutex code does not have any reference to “chage the [x] to [y]” and I assumed that exclusive use was an intrinsic attribute of lock files.

[cyclicalobsessive]

(deleted by author that cannot delete for 23 hours)

[jimrh]

I’m beginning to feel like Costello in Who’s on first?

1. A lock file can be successfully opened for write by more than one independent program object.  (i.e. Two entirely independent programs can successfully open the lock file for write.)

2. However at some point in time, (this is what confuses me), the second process does a “something” that fails, and that’s what tells it that it has to wait.

Let me see if I have this right:

• Handle = open(path/filename, w) # Always succeeds and returns a|the same file handle.
• However, fcntl.flock(self.Handle, fcntl.LOCK_EX | fcntl.LOCK_NB) attempts to change some attribute of the file specified by “handle” and only the first one gets it.  Subsequent attempts by different programs always fail.

Because of the asynchronous nature of messing with files/handles, and the way the kernel may or may not queue requests, it’s possible that process2 might execute the “fcntl” statement first, beating process1 to the punch, even if process1 started first, right?

Is it possible for the “fcntl” requests to collide?  (i.e. Happen at the identically same time?)  Or is that intrinsically handled by the kernel, and the kernel forces each operation to be atomic, exclusive unto itself?  (I am assuming "yes the kernel somehow queues the I/O requests and prevents them from colliding so two programs cannot execute the exact same file command at the identically exact same time.)

Then again, that asks the question:  Does the kernel force file operations to be atomic?  (i.e. two requests to open the same file cannot interleave with parts of one request being mixed with parts of another.)  I am also assuming that a low-level kernel function like write_bytes(sector, bytes) is executed more like write_bytes(sector, bytes, atomic).  Therefore, absent a mutex, two different programs can execute the same thing, but not absolutely simultaneously.  (i.e. The data will not be interleaved and blended together, but one can overwrite the other unless prevented somehow.)

Am I understanding this correctly?

[jimrh]

OK, now that we have the python path issue solved. . .

Given:

1. The existence of mutex code somewhere in the GoPiGo libraries.

2. The GoPiGo library creates a lockfile named “LOCKFILE” in /home/pi/.

3. I have additional code located somewhere else, doing something else, and I incorporate some mutex code into it.

4. It also wants to create a lockfile called “LOCKFILE” in /home/pi.

5. These two code objects are not synchronized, and - in fact - have no idea that the other program(s) are even running.

Question:

1. Will this work?
   Is the mere fact that the lockfile is given the same name and is located in the same directory sufficient?

2. Is this an appropriate use for a mutex/lockfile?

[cyclicalobsessive]

@jimrh

• Will this work?
  Is the mere fact that the lockfile is given the same name and is located in the same directory sufficient?
• Is this an appropriate use for a mutex/lockfile?

yes, yes…exactly the purpose and usage

[jimrh]

One last wrinkle:

It appears that there is already a system lockfile directory located at /run/lock, and dexter libraries appear to use it for i2c mutexes.

It also appears to be owned by root:root.

It is also annoying to have to make a special lockfile directory/place just for the waveshare code.

Aside from running the code via sudo, is there anything else I can do? (Guess: Stinks being me.)

[jimrh]

@jimrh

It is also annoying to have to make a special lockfile directory/place just for the waveshare code.

The big problem I want to avoid is creating and managing a mutex in a dozen different places, managing filenames and everything for anything that thinks about using SPI.

My thought is creating a library file that contains a “SPI_Mutex” class that would manage the whole thing for me - from placing the lock-file in the correct location to managing the lock and arbitrating the access.

With this I could include SPI_Mutex as mutex and then use the methods, (for example mutex.get and mutex.release), in each of the various things I want to have a mutex for the SPI buss.

Would you give me a good class template that I can use?

[cyclicalobsessive]

@jimrh

The big problem I want to avoid is creating and managing a mutex in a dozen different places, …
Would you give me a good … template that I can use?

OK - I’ve learned a bit through this - guess what - DI already has you covered, we just had to find out how!

The GoPiGo3 API has a class called DI_Mutex that allows passing a name parameter to do everything for us.

The pattern for every user program (located anywhere on the GoPiGo3):

from di_mutex import DI_Mutex

spi_mutex = DI_Mutex("SPI")


try:
    spi_mutex.acquire()
    # do protected SPI access stuff here
finally:
    spi_mutex.release()

I built two examples for testing:

wget https://github.com/slowrunner/GoPi5Go/raw/main/systests/mutex/di_mutex_user.py
chmod +x di_mutex_user.py

wget https://github.com/slowrunner/GoPi5Go/raw/main/systests/mutex/d2_mutex_user.py
chmod +x d2_mutex_user.py


In Terminal 1  $ ./di_mutex_user.py
In Terminal 2  $ ./d2_mutex_user.py

Note acquire times never overlap,
and the mutex file /run/lock/DI_Mutex_SPI will be used for inter-process mutual exclusion

On my system the DI_Mutex is located:

$ unzip -l /usr/local/lib/python3.7/dist-packages/Dexter_AutoDetection_and_I2C_Mutex-0.0.0-py3.7.egg | grep di_mutex
     1700  2020-11-10 13:08   di_mutex.py

PS. I tried to clean up this thread by deleting all my off-track learning posts but got:

[jimrh]

@cyclicalobsessive

PS. I tried to clean up this thread by deleting all my off-track learning posts but got:

I really hate doing that unless the posts are 99% angst and another 10% frustration talking, as I believe the process that gets you to the destination is as important as the destination itself.  There’s usually a lot of irreplaceable experimental knowledge along the way!

And You deleted all the example classes!!!  I wanted to copy them!!!  (Growl, growl, hiss, hiss)

Wasn’t it Einstein who said “I’ve learned much more from my failures than I ever did from my successes.”

[jimrh]

@cyclicalobsessive

@jimrh

The big problem I want to avoid is creating and managing a mutex in a dozen different places, …
Would you give me a good … template that I can use?

OK - I’ve learned a bit through this - guess what - DI already has you covered, we just had to find out how!

The GoPiGo3 API has a class called DI_Mutex that allows passing a name parameter to do everything for us.

The pattern for every user program (located anywhere on the GoPiGo3):

Oh, and P.S.

Great find!

I’m sorry that I can’t give that posting five or ten likes as it is absolutely the correct solution, (Qualifier: Within the context of the GoPiGo robot, but that’s the requirement anyway so it’s all good!)

Thanks SO MUCH for all your help and effort you’ve put into to this project.  And the education you’ve given me along the way!

[cyclicalobsessive]

@jimrh

You deleted all the example classes!!! I wanted to copy them!!!

You mean the several “doesn’t work” and the one “hard” way to do it on the GoPiGo3? No one should ever look at those. One of the principles of learning is never learn wrong stuff, and another is don’t learn from bad examples.

If you want an example - read the beautiful code the amazing DI folks wrote (on your robot at):
/home/pi/Dexter/lib/Dexter/RFR_Tools/miscellaneous/di_mutex.py

and on Github at: https://github.com/DexterInd/RFR_Tools/blob/master/miscellaneous/di_mutex.py

It is pure Python and Linux, no Dexter dependancies, and will run on any robot using the:

• put it somewhere in the Python path

or

• put it somewhere and modify the Python path with:

import os
os.path.insert(1,'/path_to_di_mutex.py/')

[jimrh]

I found the following thread:

Use_mutex, "safe", multi-task and multi-processing

Continuing here - the mutex and the EasyXXX “safe” methods discussion that hijacked GoPiGo Line Follower Assembly Instructions out-of-date: The EasyGoPiGo3 and Easy<SensorX> classes pass down a “use_mutex = True” instantiation for “safe” access methods to some shared resources. I have made it a practice to always write my GoPiGo3 Python programs using the EasyGoPiGo3 class with “use_mutex = True” even if the program functionality does not use multi-tasking or multi-processing. This way, if I …

. . . and there was discussion of making the use of the mutex the default.

Question:
Aside from the “pedagogical benefit” of making an inadvertent fool out of the unsuspecting student, is there any real disadvantage to having the mutex on full-time?

[cyclicalobsessive]

@jimrh

is there any real disadvantage to having the mutex on full-time?

Yes, it increases the risk that the mutex lock file will get locked by a zombie, and require a ghost-buster visit.

[jimrh]

Ref:  The following quotes:

@cyclicalobsessive

I have made it a practice to always write my GoPiGo3 Python programs using the EasyGoPiGo3 class with “use_mutex = True” even if the program functionality does not use multi-tasking or multi-processing.

. . . and then

@cyclicalobsessive

@jimrh

is there any real disadvantage to having the mutex on full-time?

Yes, it increases the risk that the mutex lock file will get locked by a zombie, and require a ghost-buster visit.

Does this mean you do this, but accept the risk that you may end up hung somewhere if the mutex gets deadlocked^[1]?

If that’s true, how often do you experience this, if at all?

========== Footnotes ==========

1. That is, the risk is POSSIBLE though UNLIKELY, as in “going outside” has various risks, (getting hit by lightning^[2], an errant airplane landing on your head, getting bitten by a biker-gang of badgers, etc.), though the probability may be quite small.

2. That actually happened to an uncle of mine many years ago.
   He was out playing golf at a golf course in Norfolk, Va., with a bunch of his buddies on a warm and sunny day, and he was using an iron to get from the fairway to the green.  When he swung the club it got struck by lightning, (from who-knows-where), and it struck him dead on the spot.  Nobody knows for sure, but the suspicion is “heat lightning”.  (i.e.  A “bolt from the blue” as noted below.)

While the phrase “heat lightning” stems from a false belief, a “bolt from the blue” is a true and serious threat.

Lightning can strike dozens of [miles] away from a thunderstorm, particularly when it’s a robust thunderstorm that’s thriving on the instability of a steamy day. There are countless cases of people who were injured or killed by lightning thrown off by distant thunderstorms.

Ref:
https://www.theweathernetwork.com/ca/news/article/lightning-safety-tips

[cyclicalobsessive]

@jimrh

Does this mean you do this, but accept the risk that you may end up hung somewhere if the mutex gets deadlocked[1]?

If that’s true, how often do you experience this, if at all?

Yes I have to use mutex=True, and even one step further for programs which need read access to the GoPiGo3 but will never issue any effector commands, I created a no_init_easygopigo3.py which allows the initialization option (use_mutex=True, no_init=True) to turn off resetting the GoPiGo3 default speed to 300. Most of my programs set the default to 150 DPI, so if a second program that just wants to read the battery resets the speed to 300, the first program’s commands can be changed mid-motion.

Both Carl and Dave set use_mutex=True and roughly once every 4 to 9 months I have to reboot to clear the I2C error 5 of unknown cause.

Don’t know if you recall, but I had to create an I2C mutex protected BNO055 DI EasyIMU class - for some reason the DI IMU did not get the use_mutex=True option when they added the Software I2C with three channel I2C mutex (global, AD1, and AD2 connected I2C).

Dave no longer has a DI IMU (requiring clock stretching), and I have removed his DI Distance Sensor in light of having a 360 degree LIDAR). I have not seen an I2C error 5 on Dave since.

I do not think I have ever seen the mutex zombie locked, but Carl’s health checker does not test for it.

@jimrh

distant thunderstorms.

It is “daily thunderstorm season” here, so much of my days are spent counting the speed of sound seconds before venturing out of the house. We have “armour piercing lightning” roll through that did not read the book on Faraday Shields.

[cyclicalobsessive]

@cyclicalobsessive

Don’t know if you recall, but I had to create an I2C mutex protected BNO055 DI EasyIMU class - for some reason the DI IMU did not get the use_mutex=True option when they added the Software I2C with three channel I2C mutex (global, AD1, and AD2 connected I2C).

Dave no longer has a DI IMU (requiring clock stretching), and I have removed his DI Distance Sensor in light of having a 360 degree LIDAR). I have not seen an I2C error 5 on Dave since.

prior updated with this after you saw it.

[jimrh]

@cyclicalobsessive

It is “daily thunderstorm season” here, so much of my days are spent counting the speed of sound seconds before venturing out of the house. We have “armour piercing lightning” roll through that did not read the book on Faraday Shields.

When I was “a wee lad” growing up in the southeast corner of Virginia, there was one steadfast, engraved in depleted Uranium, don’t even THINK of disobeying rule:

• Thunder stops EVERYTHING and everyone immediately goes indoors.
  Even if it’s not raining, if you hear thunder - that’s it.

On the other hand, a summer shower, sans thunder, was a treat because it was God’s sprinkler we kids used to cool off in the 90°+ heat.

[jimrh]

@cyclicalobsessive

@jimrh

The big problem I want to avoid is creating and managing a mutex in a dozen different places, …
Would you give me a good … template that I can use?

OK - I’ve learned a bit through this - guess what - DI already has you covered, we just had to find out how!

The GoPiGo3 API has a class called DI_Mutex that allows passing a name parameter to do everything for us.

Great idea, except for one tiny fly in the ointment:

The resultant filename would be “DI_Mutex_[name]”

What I want is a generic SPI mutex that will be suitable for anyone’s code - that I could push to Waveshare or Adafruit, or [etc.].

What I did was to make a copy of di_mutex and rename it to spi_mutex, with the following changes:

class SPI_Mutex(object):
    """ Generic SPI mutex """

    def __init__(self, loop_time = 0.0001):
        """ Initialize """

        self.Filename = "/run/lock/SPI_Mutex"
        self.LoopTime = loop_time
        self.Handle = None

1. I changed the name from DI_Mutex to SPI_Mutex.
2. I removed the “name” parameter since it is automatically “SPI_Mutex”
3. I changed the descriptive text.

Hopefully this will work and create a generic SPI_Mutex lock.

Viz,:

from spi_mutex import SPI_Mutex

spi_mutex = SPI_Mutex


try:
    spi_mutex.acquire()
    # do protected SPI access stuff here
finally:
    spi_mutex.release()

Yes, I know that the DI_Mutex class already exists and - for my very specific GoPiGo application I can use it.

However, I am thinking of a generalized mutex that anyone can use without all the references to Dexter Industries.

Attached:
Entire library code of the proposed spi_mutex library.  Please examine.
spi_mutex.py (2.1 KB)