Can motors experience detonation in neutral?

If the compression ratio is high enough you can get detonation whenever. Idle, higher rpms, whenever.

Although, if you have detonation at idle you're WAY off on the compression for whatever fuel you're trying to run. Or the tune/timing.