Speedo a no go.

Make sure you don't have any tight bends in the speedo cable. If it's not coming straight out of speedometer it could cause it to slightly bind and cause your issue.
Sounds like the only real way to fix it would be to replace the cable and housing. I had kept the original housing and just replaced the actual cable to make it easier (and also because the new housing didn't have the clip) but I guess to be sure I should just replace the entire thing.