I got my my first Mac almost two years ago. All my previous Windows machines had been desktops and I really wanted the portability option of a notebook. I had the money at the time to splurge a little so I went with the mid-level 15" MacBook Pro with a faster processor and HDD. If I had to do it over I would make the same decision, but with that being said, I think I've used my Mac as a laptop only about 1%-2% of the time I've had it. Docked to a 22" monitor, external keyboard and a Magic Trackpad, my MBP happily operates in desktop mode most of the time.
Ok, I know you're asking about the Mac mini so enough about my notebook. The Mac mini has actually popped up on my radar recently for a couple of reasons. One is that I foresee myself using my MBP in laptop mode more frequently. Docking and undocking the notebook isn't that big of a deal, but it would be nice to just have a permanent desktop setup and not have to go through the hassle. Another reason is that my girlfriend and I will be moving in together sometime next year and it would be nice if we each had access to a computer at any given time (that and I would like to end her suffering through her current Windows experience.

) Since I already have the monitor, keyboard and trackpad, the mini seems like a no-brainer as a plug-n-play desktop. Sure, an iMac would be nice, but I don't really need everything it offers above and beyond what I can get from the mini. And for the price, well that's a no-brainer too.
The super compact size along with the pre-existing monitor (your TV) seem ideal for your situation. All you'd need is a wireless keyboard and mouse/trackpad (highly recommend the Magic Trackpad) and you're set. As far as cons go, the only ones I can think of are that there isn't an optical drive, built in HDD or SSD size is limited, and processor speed is limited. External optical and other drives are readily available and cheap, so that's no biggie, and if your a power user you're probably not looking at the mini in the first place.
I say go for it!