Am I the only having trouble here?

well, one answer would be that the site itself is quite busy in the evenings -- which wouldn't be reflected in the tracerts --

At the same time -- the servers are way overkill for what we need so there should be no problem there. In addition, as I have said before - I am seeing none of this. I'll get a little sluggish at times if I use IE -- but, its IE so that's expected.

Yep, and I have been watching this for quite some time.
Especially when it comes up again.
I don't think it's a time of day or amount of users issue though.

I'd bet it's something simple, but so is picking up a big diamond off the ground.
The diamond just has to be laying there for a person to see it first.:D
Either that or it's something you have zero control over.