Link has lived in a kokiri forest, has lived on an island in the vast seas, has lived with is gradpa near the citadel of Hyrule, has lived with an uncle in a little house near the castle of Hyrule.
And those Links where all male(most likely in all zeldas), if there is a game where Link is born female and Zelda born as a male, then maybe the argument of this topic would be valid.
Link is not a woman. If you think his voice sounds girly, you've gone and hurt Noboyuki Hiyama's feelings. Hiyama is the (male) voice actor who has been Link's voice from Ocarina of Time on up. Also, in the June/July issue of Nintendo Power, the artist of the Zelda games was interviewed. Direct quote: "Link is a handsome, strong, masculine boy. He isn't feminine, so I had to avoid drawing him to look feminine in my drawings."