I think this thread is all about miscommunication. "Scroll up" and "scroll down" are just confusing the matter. Just like the rest of the iPhone, the contacts follow the motion of your finger (sliding your finger down, makes the contacts go down). If F is at the top of my screen and i touch it and swipe down, F moves toward the bottom of the screen.
You could see where people would say the contacts are "scrolling down". But, if you were on a website and wanted to achieve the same thing, some might say you are scrolling up on the page. To make things more confusing, if you click the right side of the contacts(near the small letters) and then slide your finger, the reverse motion is noticed. Sliding your figer up makes the contacts slide down.
I would say that it is very unlikely that we are having different experiences...