Just because it passes the test doesn't mean that it will work well in all of the browsers either. I've experienced that first hand. One of the problems with our blog is that the posts themselves have a lot of issues since we have to use Live Writer in HTML mode because XHTML still has some issues. So there are 250+ errors on our homepage because of that. Not to mention that the more complicated a site gets the harder it is to abide by all of the W3C rules.
In short I would say to fix any easy errors that you may see, but don't waste time trying to make everything perfect. Just test it in all the major browsers. We always test in Firefox, Opera, IE, Konqueror, and Safari. And we normally test the current version of the browser and the last major version.