My Wife is going to drive me crazy about this.

Yea, they just rely on sucking air through a wet pad and blowing it into the house.
Problem is, that air speed through the wet pads is what makes the air cold so you have to let air into the house for them to work.
With the windows closed the air doesn't move, so it doesn't get cool.
Perfect is where when the amount of air that goes out of the windows and door screens is the same amount the cooler can blow in.

And you are correct that some have no idea what a swamp cooler is because the area where they live has too much humidity for the air to evaporate the water making the air cold.
During monsoons here is a good example, because it gets real humid even though it might be 90 degrees so all you get is sticky.:D
Generally though, it's pretty dry here (average 6-10%) and why we like the swamp cooler because it ads humidity to the air in the house.

When we had our swamp cooler in Colorado I would turn on the cooler then open a couple windows on the other end of the house just enough to hold a piece of toilet paper on the screen. Seemed to get maximum draw that way.