The mute switch on the side above the volume buttons will only mute system sounds and ringers ("effects" for short), not media (music, video, podcasts, anything inside the web browser, etc.). And there are some apps & games that play their sounds as "media" rather than "effects".
When you have flipped the switch and are still hearing sound, that's basically the "media" kind of sound which has a different volume control. If the app doesn't give you a mute button or volume slider, you need to use the volume buttons on the phone to turn those sounds down while they're playing.
If you want to change the volume of the ringer separately, that's a setting in the sounds section. There is also the "Change with buttons" switch that makes things interesting. If you flip that to ON and you press the volume buttons while nothing is playing media, it'll change effects volume. If you flip it to OFF, the effects volume will stay at the setting you pick on that page, and then the volume buttons will only affect the media volume.
Thanks for the long reply. My point is that I expect a mute button to do that - turn off the speaker completely, as all other mute buttons in the known universe do. If I want replay a video with sound OFF I naturally go for the MUTE button. Apple does this kind of thing all the time. I have been using macs for over twenty years and teaching in Mac labs for 15, and I still can't get used to the topsy turvy world of Apple. Remember the puck mouse? Single clicking? And let's not forget the latest clanger - monochrome icons. And then there's the skins on Game Center, Find my Friends and iCal. Oh, and MobileMe syncing - delete all contacts? Of course. I'll do that for you right now.