Did you see my posting on the setup for the raspberry pi camera up above?
https://forum.dexterindustries.com/t/autonomous-driving-gopigo3-bachelor-project/8685/55
There are a few cute settings in the current config file that might be useful - the only caveat is to keep the aspect ratio at 4:3 if you want to use the entire field of view.
Okay thatās nice and realistic anecdote 
Yes, I have seen it but havenāt tested it yet! Stillā¦ very interesting and great input, thanks!
Is there any chance on finding a data sheet for the motors of the gopigo? I need basic information about them for my thesis and there are no information at all.
Most of the hardware is remarkably well documented on the Dexter Industries GitHub page.
I donāt remember if there is a BOM for the motor assemblies, but they are, essentially, standard robot gear-motors that you can buy anywhere with a magnet wheel and a hall-sensor PCB attached.
The encoders are made of:
GitHub is your friend.
Anyone else would end up looking there to answer this question.
Also the have schematics for just about everything there.
What prevents you from referencing the entire robot instead of every individual component?
Alright guys, time for an updateā¦ So I was trying to get this lane detection algorithm from github working but there are two main problems. firstly, the raspberry pi 3 is not powerful enough to make this algorithm go in real time and secondly, the positioning of my camera is pretty bad. If the gopigo is at an turn I only āseeā the outer lane and not the innerā¦ So I am not really sure if i can make this algorithm be dumber and easier to use or If i should use something elseā¦
Till the histogram it worked quite goodā¦ But since starting the lane search everything just failed hardā¦
This is the current code:
Do you have the camera set up to use the entire field-of-view?Ā Depending on the video mode you set up, itās really easy to get a much more limited FOV than you want.
Messing with the camera and trying to figure out how to make the latest rpi_camera updates work with GoPiGo O/S, I discovered two things:
You can get fancier cameras with higher resolution and a wider FOV, but I donāt know if you have time for that.
Another option is to get an add-on lens kit, like they sell for smartphones. For example, if you add a fish-eye lens adapter it will increase the FOV, but will add spherical distortion.Ā If you can compensate for spherical distortion in software, that might be a way to do it.
Is your Raspberry Pi throttling because of voltage or heat?Ā @cyclicalobsessive has some nifty scripts that show exactly that kind of problem.
Raspberry Pi-4 boards are not that expensive, comparatively.Ā Assuming that the software can handle it, you want to go with the largest amount of RAM you can get as RAM = speed.Ā Another thought that I just had, make sure you assign enough RAM to the GPU so it can help you.
Also, donāt skimp on the SD card.Ā Spend the extra dinero and get the best and fastest cards you can - A1 rated as a minimum.
Another speed boost is to forgo the SD card and go directly to a small USB SSD drive like the 500GB Seagate Expansion/One-Touch SSDās, as they blast past SD cards in performance.Ā Here in Russia they are the equivalent of about $80 US.Ā The 1T version is about $120 - $150, but you probably donāt need that much space.
Another tip is to turn on TRIM as that will help the SD card/SSD last longer and run faster.Ā Chances are, if you have a really good, name brand, SD card, TRIM will work right out of the box without any additional setup.Ā If you use a SSD, you will probably have to enable discard/trim in software.Ā You can enable automatic TRIM on the SD card/SSD by starting the fstrim.timer service.Ā You can find more information here:
https://www.jeffgeerling.com/blog/2020/enabling-trim-on-external-ssd-on-raspberry-pi
What say ye?
Very good questionā¦ Not really sureā¦ how to check that. All I know is that, I am streaming the video feed with ācv2.VideoCapture(-1)ā 
Yes, I am using 640x480 as a compromise between image quality and complexity.
Yeah, there is no time to get more hardware. This hardware need to be enough. Everything else needs to be software.
Not sure, but I tested the code and it was extremely slowā¦ Like a couple of seconds till the camera recognized a new scene.
The main problem was and probably is the delivery bottleneck of Pis right now. So I dont think I could get a Pi4, even if I wanted.
Youāll have to do what I did, do a web search on raspberrypi.orgās documentation pages for the camera stuff.Ā Get yourself a cup of tea and some cookies when you study it - it took me awhile to figure it out.
@cyclicalobsessive has a couple of interesting routines that tell you if itās throttling or not, and what the battery voltage is.
Do you have the link to his GitHub repo?Ā I think itās named āslowrunnerā
The two routines you are looking for are called āprint_voltages.pyā and āthrottled.shā and they would be in the āCarlā project - or something like that.
I donāt know how proper it would be for me to distribute his code without his permission, so Iām hoping he hops on soon and gives you his blessing to use his code.
==================
Seriously, we can theorize about cameras, FOV, Raspberry Piās that are way undervoltage or smokinā hot - but until we get some real data about where the actual bottlenecks are, itās all smoke and mirrors.
One thing you can do is āhtopā - it gives you a nice, colorized, representation of your systemās status, (processor load, memory used, swap used, etc.), and like regular top, it shows you who the hogs are and what theyāre spinning onĀ Processor?Ā Disk I/O?Ā Thrashing swap?
You canāt clear the bottleneck until you know what it is.
Some questions and thoughts:
Your program is compute bound - only using one core of the Raspberry Pi. My single vs multiprocess lane finding experiment on the Pi3B showed a 2.5x speed improvement (20FPS vs 8FPS) when allowing asynchronous computation. If the 9 histograms are the bottleneck, perhaps multi-threading (or multi-processing) that routine could use the three other idle cores.
Using MultiProcessing for 640x480 image results in 2.5x higher frame rate (20 vs 8 fps)
Using MultiProcessing for 320x240 image results in (only) 20-30% higher frame rate (28 vs ~23)
Utilizing the processing result of the single-threaded is much easier,
(multi needs interprocess result messaging)
Find_lane_in( frame) method needs a computation only mode, (no image result)
to further minimize computation time and improve performance for robot navigation
I need to adjust the functions after histogram first, because it doesnāt recognize anything right now. These functions are pretty hard to understand, so this would need quite some time, I guessā¦
I think the problem with that is that I have a continuous lane line on the right side but the lane line in the middle isnāt continuousā¦ So if I just take lines in which I am searching for the lanesā¦ then there will be lines that dont find the lane on the left.
Yeah, I have that in my mind but I donāt have experience with multi-threading at all. If I have a algorithm working that finds the lanes but is just too slowā¦ Then I can implement the multi-threading approach.
The more I think about itā¦ the more I am sure that I need to look for a different algorithm. Like I said, in turns the camera doesnāt see the whole lane and I think this algorithm is not made for thatā¦
I am trying to go back to the basicsā¦ and improve the visibility of the lanesā¦
At the moment I am using the functions canny and threshold and combining the images to get a good result on the lanesā¦
note: I lowered the threshold, that the combination image is better understood.
I am thinking about other color or filter based images, but not sure which one is best.
So the next step is to extract the information of the lanes and visualize them in the original imageā¦
After that I am warping the image and then the clusterf**k starts again and i need to calculate the curvage information
Little update about the whole project. The lane detection so far in a decent state. I figured that the problem of lane detection with turns is the gathering of the lines from the hough transformation. In Turns this merging doesnāt work because itās not gonna be a straight line afterwardsā¦ So I decided to skip the merging part and voila the lanes are foundā¦ No pretty, but itās fineā¦ I focus on the writing part in the next weeks. But I have still a couple of things I need to fix:
Lane detection: The result of the lane detection needs to be some sort of information for the steering. I was thinking about the seperating idea, that I had a couple of weeks agoā¦ Maybe I can divide the roi in a couple of sections and use a histogram on them to detect the middle of this particular section. After that I can connect the midpoints with a line and I have something like a trajectory. Problem here is that, like I said earlier, when I drive a turn the camera loses one lane. So I thought a work around. In the bottom of the screen I know the width of the lane in pixels of the picture and in real life of the track. I made a small function that calculates the width of the lane for a certain height. So If the camera loses one lane I can use this ādefaultā width and estimate the middle.
Obstacle detection: I tried to use a haar cascade for detecting cars. I works quiet fine. Of course itās not robust at all and pretty inaccurate, but this will do. On the pc I tried a DNN with trained data to detect cars and itās slow as f**k, but still fun to try. So the goal here is to write a small programm that lets the gopigo drive forward while capturing the camera data and use the haar cascade to detect a model car infront. Then I have two options. First, I draw the bounding box and if the bottom edge of the bounding box is below a certain threshold I stop. Second, I try to build a distance estimation for a single cam. I found two github pages about this topic. And if the distance is below a certain thresholdā¦ againā¦ stop.
Lane keeping: IIIIIF I can get the information from lane detection about the steering, I can calulate the steering angle for the gopigo. I tried the gopigo function steer which seems promising. I havenāt tried the spin-right, spin-left or target_reached - functions. Maybe those could work as well.
Soooā¦ Conclusionā¦ I am finally out of the writerās block and I try to write as fast and as good as I can that I still have time to complete this tasks above. If all goes to plan in about one month the thesis is written, the gopigo is driving by himself, I am happy, exhausted and a freaking engineer!!
Congratulations. You put a lot of work into this.
/K
Hello everyone! Quick updateā¦ I am trying to extract any information of steering out of the imageā¦ But I just canāt seem to get it down.
The code so far does this: (Current code is linked on the bottom!)
Now the tricky partā¦ My approach was the following:
EDIT: After thinking about it last night, I realized that I calculated the angles wrongā¦ Maybe I can get this approach working after allā¦ Still the problem in curved lanes is this thereā¦
And this approach is not working at all so far. Here are a few output images:
First the result of the Hough Transformationā¦ Some would say this is lane ādetectionā. Works fineā¦ Not perfect, but fine I guess
(1)
This is the warp image of the combination of canny and thresholdā¦ The lanes get displayed pretty good. Huge problem is that If I get near a turn like in picture 3 the result of the warping is very badā¦ and this is killing the histogram approach ā¦ Any Ideas how to make this work differently?
(2) Wheel is around 30cm away from the turn
(3) Wheel is around 15cm away from the turn
(4) This is a piece of the birdeye image with the histogram of itā¦ The averaging works fineā¦ Maybe with a offset the middle gets found
The current code:
https://www.codepile.net/pile/Y6WmZzgP
Small update, big impact:
Obstacle Detection: I implemented a distance estimation with a comparison of a reference image . Itās not super accurate and the distance alters the whole time, but stillā¦ the distance is estimated. I reused my code to record a drive-by video to get the camera footage and drive at the same time. At a certain limit of distance the gopigo stop. So, I count that as a win!
Lane Detection: I fixed the bad calculation of the angle this morning and I kept going. I have coded something like a trajectory on the birdeye-view image and transported it back into the driving perspectiveā¦ Also not pretty, but wellā¦ itās a trajectory . After fixing a few more bugs I managed to get a steering output, that is calculated through the histogram averages like I showed yesterday. After that the lane detection part ended and I created a new file called lane keeping
Lane Keeping: I used the steering output to calculate a movementā¦ Not that easy to be honestā¦ Itās very flawed so far, BUT itās the first time the gopigo is driving based on the camera and the calculations. I managed the drive a āstraightā line and even a left turnā¦ Of course driving repeatedly in the oncoming traffic but heyā¦ nobody is perfect, right? Somehow the right turns donāt seem to work properlyā¦ I also noticed that I probably made a mistake with the steeringā¦ Because I donāt consider the middle of the footage as a baseā¦ Stillā¦ my boy is driving for the first timeā¦ calculations are a little off, but he did drive the lanes he was supposed tooā¦ at least most of the timesā¦ One thing I noticed and predicted was that he turns are executed a little too earlyā¦ Donāt really sure how to fix thatā¦ I thought about some sort of a dead time controllerā¦ Has anybody any experience with that?
By the wayā¦ @cyclicalobsessive , you talked about the advantages of multiprocessing. I am interested in this kind of optimization now. Any tips on how to implement that bad boy quickly?
There are many sites to introduce and contrast the two architectures such as this one
I have some simple examples of each architecture with one of the several possible data transfer/messaging and synchronization methods.
multi-threaded:
multi-processed:
It sounds like youāre kicking ass and taking names!
This isnāt a trivial project, and youāre doing a lot of fundamental research on the GoPiGo - which is always valuable.
A serious, big, BIG 
Thank you very much! I am trying my best to get this project up and running. There has been a lot of research and countless hours of coding and writing so far. The next four weeks is the completion of it all. Deadline is the seventh of february with a little presentation before that. And if all goes to plan I should get a replay from the admissions office for my masters application for next march. I keep you guys posted! If I complete this thing, it wouldnāt have been possible without you guys! Thank youā¦ again!
