Should I replace this caliper?
yes, that is common on aftermarket hubs. There was a post here about that.
Print