"Pulsing" voltage, RPM drop with load.

When your car is running, the alternator is the main source of power, until it's output voltage drops below the voltage of the battery.
When the engine is at idle, the rotational speed of the alternator may not be sufficient to create a higher output voltage than the battery. This causes a discharge condition, which you have noted on the ammeter as well as the dim headlights.
There are two common causes for the conditions you described. First, the alternator is not capable of properly charging the battery at idle speeds. Usually, this means you need a better alternator. Second, the condition of the battery and all related cabling also comes into play. If the battery is weak, it adds additional load on the charging system at all times, but it's particularly evident at idle. The fact that you can load the engine to drop 100 rpm simply by turning on the lights makes me suspect the battery. Also, you mentioned it drops to around 11 volts... another sign that the battery is junk.
Still, I'd make 100% sure the cables that connect the battery to the car, and the output of the alternator to the battery are clean, tight, and of adequate gauge for the current needed. Lot's of times a charging system looks suspect when the problem is a corroded or loose cable.

If this was my car, I would load test the charging system, and the battery. If one or both were not 100%, I'd replace them. If your battery is is top condition, you won't likely need a high performance alternator, unless you do a lot of low speed driving, and have a lot of idle time with moderate to heavy electrical loads.