Should I replace this caliper?

yes, that is common on aftermarket hubs. There was a post here about that.