Should I replace this caliper?

Dust boots, yes in time.

You can get new ones if you Google or post a wanted add here.

Get innertube glue...lol or just rtv over it for now.