My face exactly.
Because let's just talk about how misinformed one has to be to assume that most women are not fans of Lord of the Rings
because they are women. The reality being sex - male or female - has nothing to do with one's attraction to the fantasy genre, or Lord of the Rings.
That being said, the guy in the OP is probably a male who is not a Lord of the Rings fan. Being male is separate from his like or dislike for the series. Therefore he wouldn't be attracted to ladies who like the series - it's not a shared passion. Guys who like Lord of the Rings usually dig ladies who like it too, because it is a common interest - common interests usually being what brings people together.
So there, that's my bit, as a lady and Lord of the Rings lover. I hate the idea that ladies who like Tolkien are somehow "rare" or "nerdy." It's a gross misrepresentation of the sex.