I'm a huge fan of the Vibrams -- they've actually become my primary day-hike footwear the last couple of years. I've got three pair of them now.
They need to fit fairly snug, but not to the point where your longest toe is at all compressed. Depending on the model, a size that seems just slightly tight at first will likely be a perfect fit after break-in.
Different models of the Vibrams have a slightly different fit, so if you can, it's a really good idea to find a place where you can actually try on the specific model you're thinking about buying.
And enjoy them ... they're great! Despite the funny looks you sometimes get.
Mark