What about olive oil? Virgin olive oil has no previous oil experience, but regular non-virgin olive oil has probably never had 'intercourse' either. Another example is the term 'virgin territory' meaning land that has been unclaimed or unsettled, wilderness. Another whole gender-political minefield in that term: claimed, conquered, used. It would be silly to think these aren't statements uttered by a patriarchal society, but it would be even sillier to keep repeating them.
I think the term is broader in its roots, and broader in its current use, than a purely heterosexual narrow definition.
Take another word with a similar meaning for example - maiden. Maiden technically means a young woman who hasn't had sex, and later (or perhaps all along) was therefore extrapolated to all young women who were presumed to have not had sex. I've also read in a number of sources that in the hebrew version of the bible Mary is not actually referred to as a virgin but as a young woman - which is the same word as virgin - but by context alone (ie giving birth) one would infer that in her case it clearly just means young woman rather than virgin.
I think the best question we could ask out of all this, is why is it important to create some invisible arbitrary line where on one side stand those who haven't had sex. and on the other everyone else.
