Am I the only having trouble here?

How much of that are users who haven't updated their java since, well, ever?

That's a fair question, but it doesn't apply here.
I have cleared cache
Reset browsers to default with no add on running
Checked, uninstalled and reinstalled the latest Java
Run multiple Malware and Virus scans
Used three different browsers
Different computers at different locations
Used browsers in safe mode.

Nothing ever made a difference, and the servers response times have always seem pretty good with a ping.
Tracert has shown some slow servers along the way here and there but nothing major.
It most times does seem worse in the eves out here, but the pings and trace routes don't seem to reflect that.
This is why I feel it's in the software itself.