My "New" Remote Camera Robot Project

Update:

@superbam
@cyclicalobsessive
@KeithW

I am curious how you calculated your turns.

I am working on how to calculate the ratio of the wheel speeds to get the degree of turn I desire based on how far forward the joystick is pressed and how far to the side it is pushed.

(Right now, I am ignoring sign which is forward/back and left/right since once I get a smooth turn based on the two axes, I can always flip the sign for the other direction.)

My current calculations are based on the following:

  • Assume the joystick is being pushed forward, so the y-axis value is increasing.
    • It’s actually increasing in the negative direction, but I am ignoring sign for the time being)
  • Assume a x-axis deflection between zero and one.  (centered to all the way to one side.
  • Y-axis further forward, (closer to 1), means faster travel up to the limit set by max_speed. (actual_speed = |max_speed * y_axis|)
  • As the x-axis is moved further away from center, the ratio of the speeds of the outside wheel to the inside wheel should become greater.  (i.e. The inside wheel should become slower as the x-axis is deflected more.

I came up with one formula where the inside wheel was a fraction of max_speed, which didn’t work.

I have modified the formula as follows:
(Note: I haven’t tried this on Charlie yet.)

actual_speed = max_speed * abs(y_axis)

percent_speed = int(((y_axis / x_axis) / 2 ) * 100)

Where

  • actual_speed is the forward speed if x_axis = 0 and is the speed of the outside wheel during a turn.
  • percent_speed is the percentage of actual_speed that the inside wheel gets.
  • “int” is a function that I will probably have to write so that both positive and negative numbers round away from zero to the nearest integer.
    • +1.5 => +2
    • -1.5 => -2
      As it is, the rounding methods available in Python round negative numbers toward zero.
    • +1.5 => +2
    • -1.5 => -1
    • -0.5 => ZERO!

Sigh. . . .

I actually don’t. I just use the ROS functionality.

Sorry.
/K

1 Like

Update:  (Failure and an idea)

Issue:
The “normal” way the original remote camera robot project sent data to the robot was a combination of an event driven interface and a short timer to keep from flooding the internet connection.  This was possible because the virtual joystick library, (nipple.js), used mouse and keyboard activity events to regulate the volume of data sent.  (i.e. Data was sent ONLY  if there was something useful to send.)

Unfortunately that cannot be done with a real joystick because the current implementation of the gamepad interface does NOT  generate generalized activity events - the software has to continually poll the joystick.  This results in either the server being flooded with data, (even if limited by a brief timer), or if a activity polling loop is used, the browser stalls.

I attempted to mitigate this by using a web-worker thread of execution to poll the joystick and only send useful events to the main thread.  Unfortunately, that does not work because web-worker threads cannot access the “window” attribute of the page and therefore cannot interact with the joystick directly.

I have an idea that might work:

  1. Abandon the web-worker attack on the problem.  (At least for the time being.  I may return to that if true activity events are implemented.)

  2. Allow the current code to work the way it does - flooding the server with POST data messages.  However, instead of allowing them every time the animation loop is run, I will try wrapping the “send_data” routine in a test for the joystick’s time_stamp attribute - has it changed or not?

    • If the time_stamp attribute HAS changed, that means something happened and I will allow send_data to send the current data-frame to the robot.
    • If not, there’s nothing interesting to send and I’ll just skip the send_data routine.
    • Keyboard events, (for moving the robot’s head), don’t cause changes in the joystick’s time_stamp.  Therefore, if there is a keypress to transmit to the robot, I’ll just dump some bogus value into the time_stamp so that the send_data process triggers.

Let’s see if that helps.  Hopefully I will be able to generate the functional equivalent of gamepad activity events without bogging down the browser or making joystick response sluggish.

2 Likes

Update:

Problem solved!

The browser now only sends data when there is something interesting to send - and is much more responsive since it doesn’t stall in a wait-loop.

The robot motion logic itself still needs considerable help.

  • The motion isn’t as smooth and controllable as I’d like, and it sometimes sticks “running”, but these should not be insurmountable problems.

The big problem is that, right now, it only works in Chrome because Firefox won’t fire joystick events unless you’re within a secure context - and I don’t know how to negotiate a secure context even with a self-signed certificate.

It is very likely that Chrome will follow Firefox’s lead very soon and I won’t be able to continue development.  :face_with_symbols_over_mouth:

2 Likes

I have posted issues on both Mozilla’s site and the W3C group’s GitHub page about requiring a secure context to use a gamepad/joystick within a browser.

Unfortunately, the move to restrict everything possible to a secure context has a lot of momentum and the people who develop the relevant specifications seem to believe that a secure context is a panacea that will solve all the Internet’s problems - and that’s just not true.

Requiring a secure context to use a hardware device to prevent fingerprinting is not the correct solution because those sites that wish to fingerprint you, the Google’s, Meta’s, (facebook), and the rest of the skum-sites that want to follow you around already have secure sites!  Therefore, requiring a secure site does absolutely nothing to protect you from fingerprinting.

I have submitted objections to this on both Mozilla Developpment and the W3C GitHub page.

Viz.:

Anyone who wishes to comment on either of those sites objecting to this pointless solution that actually makes things worse, is welcome to do so.

What say ye?

2 Likes

Update:

Re: The requirement for a “secure context”.  (See previous message.)

  • I discovered that the domain “gopigo3.com” was available, so I “bought” it.
    • Along with the domain, I purchased a simple e-mail plan so I could receive e-mail at the xxx@gopigo3.com address.
       
  • I have also purchased a simple site certificate from Comodo.
    • Unfortunately, I cannot provision the certificate until the e-mail becomes active which might take a couple of days.  I am in contact with the support people at Dirctnic and if it’s not working by tomorrow, I will open a support ticket.
       
  • I have sent an e-mail to the support/sales people at Comodo asking how to solve the problem of requiring a secure context to use just about any hardware device, (aside from a mouse or keyboard), within a browser.
    • The “obvious” solution of using a self-signed certificate for the robot is not as simple as it appears.  (I tried that.)
    • Using a self-signed certificate - at least with Firefox - is a horrible user experience.
    • Within Firefox the user has to traverse no less than three warning screens, each scarier than the one before, predicting doom, destruction, the end of the world, along with incurable dandruff and hideously bad breath.  “It should be intuitively obvious” that this is a less than stellar user experience, virtually guaranteed to scare away most everyone.

Re:  Joysticks and gamepads.

  • I purchased two different gamepads.
    • One from “Ritmix”, ($8 US), which is hideous.
      • The controls are not linear - the axes traverse the entire range in each direction with less that a third of their travel.
      • It has a flimsy feel, the buttons stick, and it generally gives the impression of being a cheap piece of :poop:.
      • The default game mode is some strange controller mode where everything, (including the joystick axes), maps as a button.  There is a secondary mode where the gamepad acts as a gamepad, but it is not the default and has to be selected each time the gamepad is plugged in.
      • Attempting to reach the ritmixrussia.ru web, (to see if there is calibration software), site throws several “bogus site” warnings claiming that the site is fraudulent and infested with Trojans.  Malwarebytes absolutely refused to allow access to the site without a specific exception added.  I decided not to go there. . . .
         
    • The second one was from Logitech, (about $25 US), which has much better control, the joystick axes behave as one would expect, and it’s default game mode is as a “standard” gamepad.
      • The gamepad “feel” is much more solid, the button action is firm and positive, it defaults to a standard gamepad format, and the general impression is that of a solidly built device.
         
    • “Da’ bitch part”, (as they said in Blazing Saddles), is that the gamepad button mapping is totally different than the button mapping for my joystick.  (i.e. The left-hand trigger on a gamepad is button 7 instead of button 0 and the right-hand trigger is button 8.)
      • I suspect that as the project nears completion, I will have to add the capability to define the mapping between the actual joystick buttons and the logical buttons within the software either via a JSON or XML configuration file.
      • I’d really like to use XML as that will allow me to define one parameter per line, as opposed to the “everything on one line” JSON format.
1 Like

is inure to such warnings and will simply see this as just another existential threat to tip-toe away from.

Same effect, but to think a warning, one, two or three, will induce fear is to give too much credit to humanity.

Quite ingenious approach you have discovered. I applaud your tenacity.

One thing I don’t understand is how you getting a real cert will be useful to the next person wanting to use a remote machine connected joystick?

Additionally, what advantages does your remote machine browser connected joystick have over a “Wireless USB Gamepad” connected to the GoPiGo3?

2 Likes

It’s a dirty, rotten job; but someone has to do it.

Actually, you can run stuff on “localhost” and that’s considered secure - but to make it work in my case would require re-mapping localhost to a different IP address, which I hesitate to do.  I suspect it would be roughly the same as remapping /dev/null - disaster would ensue.

1 Like

I’m still trying to figure that out too, unless the private key ships with the robot.  Hopefully the W3C people get wise to reality, but I’m not holding my breath.

They don’t seem to understand that placing the joystick/gamepad behind a certificate paywall won’t solve the problem because the people who want to do the fingerprinting already have certificates.  All it accomplishes is to make things difficult for everyone else.

The only other choice is to actually write a gamepad driver, (like nipple.js), that runs outside the context of the browser that a browser script can attach to or include, and completely avoids the gamepad browser API.  But, as I mentioned in my objections, that’s like what the developer’s had to do back in the 1980’s when they had to capture the INT-19, (or whatever), vector to capture and control mouse inputs.  I really do not want to do that, as it’s more trouble than it’s worth and it would be extremely joystick specific.  (Not to mention requiring a software signing certificate and Microsoft approval before it could load a kernel driver in Windows.)  I just don’t know how to pound it through their thick skulls that requiring a certificate to use the joystick is the WRONG answer.

What the certificate DOES do is it helps me avoid five minutes or so of navigating warning pages when I want to continue testing the robot.

It allows me to be somewhere else.

A wireless, (bluetooth), gamepad requires me to be in the immediate vicinity of the robot as the range is limited.

With my setup, I can be in front of my computer with a browser open, and the robot can be anywhere my WiFi reaches.  Or, with a dynamic DNS arrangement, I can be anywhere and my robot can be anywhere else.  If I were to attach a small 4G cellular card to the robot, and set the robot up for dynamic DNS, I could access it from anywhere I might happen to be.  (i.e. under the house chasing a skunk out, while I am far, far away!  I could have used the example of bomb sniffing but the skunk is a more likely scenario. :wink:)

=====================

Update:

One of the big issues with using SSL with the Remote Camera Robot code is that there are TWO data streams:

  • The web data stream which is the “plain vanilla” web traffic.
  • The video data stream which shows the real-time video from the robot - which is being streamed from a different web server, (socketserver), on the 'bot.

The problem with that is the SSL connection only binds to the main web connection and not the video connection - and socketserver doesn’t support https connections.

One suggestion is nginx, configured as a “reverse proxy” which is like NAT traversal on a router.  If this works the way they say it will, I can set up nginx as the https endpoint and have the un-encrypted streams sent to the existing infrastructure.

Let’s see what happens.

1 Like

What do the DARPA underground search challenge bots use?

Perhaps you need to create a “remote robot command application” as a go between for the joystick and web posts, with that application running a video display browser in an app window? A lot more complex in one way, but perhaps a lot simpler than trying to put the application in the browser?

2 Likes

I have no idea what they used, or that this even existed.  Maybe these folks had deep enough pockets to afford the kind of development effort that this would require?

In order to avoid capturing the low-level USB/Joystick communications, I’m using the gamepad interface in the browser.

I know I’ve said this before, but what really burns my biscuits is how completely disconnected from reality the W3C people seem to be.

That could be done, but that would mean re-inventing the wheel for every platform that a gamepad/joystick would be used on, with all the code-signing and driver development efforts that would require.  IMHO it would be simpler to find a way to get a certificate to pass - or turn lead into gold, or get the world’s leaders to play nice in the sandbox, or. . . .

One way that I have been thinking about is to write some routine, like nipple.js or whatever, that captures joystick inputs outside the browser context (the way nipple.js captures mouse events), and then send them to the browser as mouse and/or keyboard messages - or some other kind of message that the browser could use that is not dependent on the gamepad API.

However, that in and of itself would be a non-trivial task as each platform, (Windows, Mac, Linux, Android, iPad, etc.) handles the joystick at the API level differently.  Despite my best efforts, I have not yet found a use_joystick application or module that I can use cross-platform.

2 Likes

I thought they used ROS
/K

2 Likes

(The most used remote robot control, most versitile, and quite possibly easier than cracking SSL safeguards - somewhat at least.)

2 Likes

Well. . . .

I have nginx running as a reverse proxy on Charlie and it’s serving up the web stuff just fine - no certificate errors and the joystick works like a charm in both firefox and chrome.

The only nut left to crack is how to let the insecure video stream out at the same time.  Supposedly, nginx can handle redirects to more than one server instance at the same time.

===================

Update:

Success!

I forgot one tiny detail, to specify that the incoming connection for the video connection was ALSO secure - it has to be because the connection to the base URL is secure.  (Silly me, that pesky “ssl” parameter!)

Even though nginx re-directs the incoming port request (at 5001) to an insecure connection (at port 5002 - they have to be different!), I need to specify that the incoming request from the browser will also be secure.

I now have a secure connection, binding at the standard ssl port 443, that includes both the joystick and video components of the web site.

Next steps: Figure out why I am getting bizarre results from some of my calculations!

2 Likes

Additional update:

I created a question, and then updated it with what I discovered about securing connections to the GoPiGo robot.

You can read about it here:

I am going to post details in a separate thread.

2 Likes

More progress to report!

Issue:
Aside from the actual movement calculations, (which I need to refine), I was experiencing two very puzzling problems:

  1. The x_axis value was being returned as a two-decimal-place precision number and the y_axis value was being returned as the string representation of a two-decimal-place precision number.  (i.e. The difference between 0.00 and “0.00”.)

  2. The primary trigger, once pressed, would never appear to return to zero once released.  The “on screen” value always faithfully followed the button action but the data sent to the server never changed unless something else happened - like a key-press.

Problem #1:
While researching ways to debug JavaScript in the browser in realtime, I ran across one tasty little tidbit:

Apparently there is a “magic word” for JavaScript, (like the #! in shell scripts), “// @ts-check”.  If you put this on the first line of a JavaScript file, it “magically” enables additional type-checking and linting of the file in browsers and in VS Code - and they become really picky about good programming style - like declaring variables instead of just using them, and - if necessary - creating them of the correct type before use.

That little tidbit helped me find the first problem.

Here is a code snippet that describes where the x and y axes get assigned their values:

gopigo3_joystick.x_axis = Number.parseFloat((jsdata.axes[0]).toFixed(2));
[linter error] >>>gopigo3_joystick.y_axis = Number.parseFloat(jsdata.axes[1]).toFixed(2);

. . . and I’ve been staring at that code for the entire week, getting nothing but a headache to show for my troubles.

Enabling the advanced type checking flagged the second line as trying to assign a string to “gopygo3_joystick.y_axis” - but the first line was just fine.

If you look carefully, you will notice that the second line is missing a pair of outer parenthesis.  It doesn’t throw an error, it just “casts” the value as a string instead of a number.  It wasn’t until after I set the theme in VS Code to “high contrast dark” and used the extra type checking that I saw this subtle mistake.

Problem #2:

Here is the relevant section of the code:

function is_something_happening(old_time, gopigo3_joystick) {
    if (gopigo3_joystick.trigger_1 == 1 || gopigo3_joystick.head_enable == 1) {
        if (old_time != Number.parseFloat((gopigo3_joystick.time_stamp).toFixed())) {
            send_data(gopigo3_joystick)
            old_time = gopigo3_joystick.time_stamp
        }
    }
    return;
}

The purpose of this block of code is to regulate and limit the data being sent to the server so that only “something interesting” is sent.  The definition of “something interesting” is:

  • A key-press.  Key-presses generate their own event and trigger their own data being sent.
  • A joystick event, but ONLY if an enabling trigger is pressed.
    • A “joystick event” is indicated by a change in the time_stamp, so I capture an initial time_stamp value when the joystick is first initialized.
    • “Enabling trigger” = trigger_1 or head_enable.
    • This keeps the robot from running around if someone accidentally hits the joystick.

Since you folks are much smarter than I could ever hope to be, I am sure you already see the problem.  It took me weeks to find it. . .

The definition of an “interesting” event depends on an enabling trigger being pressed - but what happens when the trigger is released?  The trigger’s value is no longer "1", and as a consequence the trigger-release action is never captured!

The solution is to capture:

  • If the trigger is pressed.  (Trigger = 1)
  • If the trigger’s state changes from “1” to “0”

I am banging my head on the floor in total and abject humiliation.

2 Likes

Another update:

Using what I have learned about debugging withing the browser, I have (finally) achieved the major milestone of having the browser:

  • Communicate accurate information to the robot.
  • Communicate it in a timely manner without stalling the browser in a tight loop.
  • And do this by only communicating if there is something interesting to say - to avoid flooding the connection with useless and repetitive messages.

To effectively accomplish this, I had to create the equivelent of a “joystick activity event” that will only send information if there is something of interest to send, and remain silent otherwise.

I implemented this with the following code:

First:  The important data structures used by the script.

These structures are essentially Python dictionaries in JavaScript form.

//  Formal definition of "gopigo3_joystick"
//  gopigo3_joystick is the structure that contains all the joystick elements of interest to the GoPiGo robot
//  This collects them together in one place so they can be used, changed, monitored, and ultimately
//  transmitted to the robot as a serialized parameter string.
var gopigo3_joystick = {
    controller_status: 'Disconnected',
    motion_state: 'Waiting for Joystick',
    angle_dir: 'None',
    time_stamp: 0,  // a large integer that, (sometimes), becomes a float (shrug shoulders)
    x_axis: 0.00,  //  x-axis < 0, joystick pushed left - x-axis > 0, joystick pushed right
    y_axis: 0.00,  // y-axis < 0, joystick pushed forward - y-axis > 0 , joystick pullled back
    head_x_axis: 0.00,  //  head x and y axes mirror the joystick x and y axes
    head_y_axis: 0.00,  //  if the pinky-switch, (head motion enable), is pressed
    force: 0.00,  //  force is the absolute value of the y-axis deflection
    trigger_1: 0,   // Partial primary trigger press (motion enabled)
    trigger_2: 0,   // Full primary trigger press  (faster speed - not yet implemented)
    head_enable: 0  // Pinky-switch press  (enable joystick to move head)
};

//  Formal definition of old_event_context
//  old_event_context allows values for old_time_stamp and old_trigger_state to persist across functon calls
var old_event_context = {
    old_time_stamp: 0,
    old_trigger_state: 0
};

The old_event_context structure is used because JavaScript cannot pass more than one item back in a return statement unless it one “object” that contains multiple values.

Later on in the code, the main gopigo3_joystick structure is used to capture the important attributes of the joystick in question and make them available in a way that can be sent to the robot.

is_something_happening is my implementation of a “joystick event”, set up such that data is sent only if there is something that needs to be sent.

//  is_someting_happening is my attempt to create a joystick "event" when "something interesting" happens.
//  "someting interesting" = any joystick movement if either enabling trigger is pressed
//  or, if a previously pressed trigger has been released.
//  Otherwise, no data should be sent.
function is_something_happening() {
    if (gopigo3_joystick.trigger_1 == 1 || gopigo3_joystick.head_enable == 1) {  //  Has an enabling trigger event happened?
        if (old_event_context.old_time_stamp != Number.parseFloat((gopigo3_joystick.time_stamp).toFixed())) {  // and, has the timestamp changed?
            send_data()  //  then, send data to the robot and. . .
            old_event_context.old_time_stamp = Number.parseFloat((gopigo3_joystick.time_stamp).toFixed())  //  Save current time_stamp to compare to future changes
            old_event_context.old_trigger_state = 1  // record trigger was pressed
        }
    }
    else if (old_event_context.old_trigger_state == 1) {  // current trigger value MUST be zero here - old_trigger_state = 1 means a trigger has changed
        send_data()  // send the trigger release event
        old_event_context.old_time_stamp = gopigo3_joystick.time_stamp  //  Save current time_stamp to compare to future changes
        old_event_context.old_trigger_state = 0  // record the fact that the trigger was released
    }  //  else. . . there's nothing interesting to say so we just return.
    return;
}

I took the opportunity to “refactor”, (reorganize and make variable usage more consistent), while I was doing this.

Having done all of this, the robot is now provably getting correct data, only when necessary, by examining the received data and verifying that the data is both correct and that the timestamp only changes when there is something to say.

Next Steps:
Improve the robot’s motion.

Corollary question:
Are there updated API documents? All the stuff I see dates from the early pre-Cambrian epoch.

1 Like

Since the API docs are auto-generated from the code, I tend to check the code first.

At the bottom of the GoPiGo3 read the docs dot io API page, the commit matches the current master GoPiGo3 commit even though the copyright states 2018. Perhaps the API docs are up to date as well.

2 Likes

. . . but they are incomplete.

the egg file that sources the gopigo3 library files is labeled “gopigo3-1.3.0-py3.7” and is located at /home/pi/.local/lib/python3.7/site-packages/ - and I am assuming the version is 1.3.0.  How do I determine if this is the most current version?

@cleoqc
What is the best way to verify and/or update the particular GoPiGo3 library files contained within the GoPiGo O/S itself ?  Since they’re not standard packages, using aptitude, (apt/apt-get), to update them won’t work.  I hesitate to just blindly curl a set of files in, as I don’t know if they get installed in the correct places, or if they are the correct ones to use with the GoPiGo O/S.

2 Likes

While messing with the robot speed control routines in my project, I have made an interesting discovery:

Viz.:
If set_motor_dps speed is set less than about 20, the wheels move in a series of jerks timed about once a second, with the motion per jerk being proportional to the speed value.  After 20 dps, the wheel motion is more-or-less smooth.[1]

Viz.:  (executed from the python 3.n shell in Thonny)

Python 3.7.3 (/usr/bin/python3)
>>> import gopigo3
>>> from easygopigo3 import EasyGoPiGo3
>>> gopigo = EasyGoPiGo3()
>>> gopigo.set_motor_dps(gopigo.MOTOR_RIGHT, 3)
>>> gopigo.set_motor_dps(gopigo.MOTOR_RIGHT, 10)
>>> gopigo.set_motor_dps(gopigo.MOTOR_LEFT, 10)
>>> gopigo.set_motor_dps(gopigo.MOTOR_RIGHT, 20)
>>> gopigo.set_motor_dps(gopigo.MOTOR_LEFT, 20)
>>> gopigo.set_motor_dps(gopigo.MOTOR_LEFT, 15)
>>> gopigo.set_motor_dps(gopigo.MOTOR_RIGHT, 15)
>>> gopigo.stop()
>>> gopigo.set_motor_dps(gopigo.MOTOR_RIGHT, 20)
>>> gopigo.set_motor_dps(gopigo.MOTOR_LEFT, 20)
>>> gopigo.stop()
>>> 

Additional important findings concerning robot speed control:

Use of the set_speed() category of commands, (as well as the functions that depend on it; i.e. forward(), backward(), left(), right(), (etc)), simultaneously with the set_motor_dps() category of commands, (set_motor_dps(), etc.), results in unpredictable behavior as it appears that these two categories of commands are, (or should be), mutually exclusive.

Apparently the robot tries to satisfy both the set_speed() and the set_motor_dps() commands simultaneously, instead of allowing the last command entered to supersede any previous robot motion command.[2]

Example:

Originally I used set_speed() along with forward() or backward() for motion directly forward or backward; whereas for turns I used set_motor_dps() with each wheel receiving a calculated speed value.  The result of this was that the transition from turning to straight travel was “buggy” at best with sudden speed changes, or the robot not responding to changes in speed at all.  (i.e. The transition from turning to straight motion was accompanied by a sudden burst of speed - or a slow-speed turn continued as slow speed motion regardless of the joystick position.)

When I started using set_motor_dps() for all motion, (with directly ahead/behind using the same speed value for both motors), the control action became much smoother.

Has anyone else experienced this?

@cleoqc
Is this expected behavior?

===============

One detail I am still researching is what appears to be “command caching” or “command queuing”.

If I do something for longer than about 30 seconds, the instructions get “stored up” somewhere and after I release the joystick, things keep playing for a non-trivial amount of time afterwards.

I don’t know if this is happening within the Python server on the 'bot or within the browser.

Any ideas?

-----------------------
Footnotes:

[1]  Speeds up to 15 dps experienced jerkiness. Speeds of 20 dps or greater did not jerk.  I did not experiment to determine the exact value that jerking stopped at.

[2]  IMHO, this is a bug.
These two categories of commands should automatically supersede each other.  Either that or there should be a documented way, (aside from physically stopping the robot or restarting the code), for one set of commands to relinquish command authority to the other set.

1 Like