Am I wrong ?

Perhaps the payroll for the mentioned was less than the others (painter) for the work needed or thought to be needed. Hard to guess from the sidelines just trying to answer your question from another perspective.

He is being paid for productivity, profitability and performance from leadership. When these fall his job is in jeopardy. I will not and no more than you can not afford to have someone steal the food off of my plate as it were.

Unless your disability is part of a workers comp claim through your employer it is not a concern to him unless it compromises the above productivity. Your ability to handle the work load expected vs actual. Sad but true.
The way he spoke to you was marginally out of line and there are laws to protect the disabled that govern that.

I had to change professions after 27 years of wrenching body could not maintain, back, wrists and knees.... I am doing better now then I was, both psychically and monetarily. Perhaps take this opportunity to look into what options you have towards doing that.