will a bigger cam kill the torque?

It's all relative. Liek they said.. You have a mild cam now, so going to another mild cam is a lateral step, not forwards or backwards in terms of performance. Very generally, the larger a given cam lobe, the more the two (intake and exh) lobes will have to overlap in order to open and close at the right times. The overlap of the lobes is what can negatively affect low end response and torque. But in smaller cams, there's very little or no overlap anyway.