If you remove all practical limits, then displacement doesn't matter if all you want is to hit a pre-determined HP number. But if 'max' HP is the goal, then max size is an absolute condition.
HP is work. A 512 is 22% larger than a 400. So the 400 needs to turn 22% more revs to match the HP potential. But like @33IMP says: why not turn the 512 22% faster again? This gets completely circular in a hurry. The 512 will do more at a lower rpm - which is a good thing because reality sucks compared to thought experiments.
So what it comes down to is that since we're all crippled by an RPM limit which is the result of spring and valve technology. As a result, in order to obtain more power in an acceptable RPM range, we increase size. No matter HOW you increase size, it will increase HP output because the torque potential at most operating points goes up. If you increase size without increasing flow potential, the HP increase is crippled, but torque will move up in all places where flow can keep up.
