A One Sided Argument
Riding with one pannier doesn't change the feel or balance of riding at all. Often times, I have the pannier full of gear, laptop and clothes and my riding is never affected. When I'm pedaling or standing at a stoplight with the bike between my legs, everything balances just fine.
The only caveat to having one loaded pannier is when I'm walking the bike through the building. With my hands on the bars and all the weight in the back (without straddling the bike), it does tend to sweep out to the side. I've got to be a little more cautious, but it isn't really an issue at all.
So if you're running panniers for the sake of plurality, try sitting one aside. There's no reason to use two if you don't need the cargo capacity. It seems like a simple thing, but lots of people (me included) have a tendency to use two even with out a real need. And using one at a time will cause less wear and tear to your set. You'll have a lighter load and longer lasting panniers if you go with the one sided argument.
Get out there.