RV2 Clutch Slipping

Okay so here is what is happening and what I believe the issue to be. When I am at a idle, the RPMs are low, if the AC clutch engages it squeels intermittenatly. I thought it was the bearing but I then realized it does NOT happen ALL the time. I can hear at regular running RPMs the clutch slips as it engages but just barely.

So I am thinking maybe the surface of the clutch is causing the problem. I had it apart the other day and cleaned it pretty well put it back together and it seems to work a bit better. My question is can I take it back apart and make sure ALL surfaces are clean and then use a red scuff pad and rough up the surface or is that a bad idea?