I don't know what software can work on a Mac, but if you have Windows, Slysoft AnyDVD (runs as a task to decode any DVD), and CloneDVD Mobile (Converts DVDs to multiple file formats). Relatively cheap, and you could run these on Parallels. Then just import into iTunes.
No need to run Windows in the mac to convert video, there are many options.
Quicktime Player should be able to export anything it can play on both platforms to iTunes compatible formats.
mPlayer, VLC, ffmpeg2 should all have binaries for every platform and I think all three convert (the 1st two are cross platform open source players).
And there are various single platform programs for each.
But video conversion is best found at it's own specific forums, as it's discussed in much more detail elsewhere already... no need to duplicate here the huge amounts of discussion already happening elsewhere.