I had looked up the option of using an optical sensor used in the mouse for tracking movement a while back for one of the projects. It does look like people have used it in the past and there are a lot of resources available for it. Sparkfun used to sell a kit for it here. But they stopped selling it because the sensors disappeared. The sensors are available in China on Aliexpress if you want then, though you'll have to get the PCB;s done and also get the optical element from China.
The big problem that everyone faced with these was that it was very much dependent on the surface you use it on for accurate readings similar to a mouse and it is prone to bad readings if the surface changes a lot.