This song is about a transvestite. According to a 2004 piece in Rolling Stone
magazine, it may have been inspired by the famous transvestite Candy Darling, who Kinks lead singer Ray Davies allegedly dated for a brief time. This is the same Candy mentioned in Lou Reed's "Walk On The Wild Side
" ("Candy came from out on the island, in the backroom she was everybody's darling").
The other possibility, which is stated in The Kinks: The Official Biography
, is that Ray Davies wrote the lyrics after their manager got drunk at a club and started dancing with what he thought was a woman. Toward the end of the night, his stubble started showing, but their manager was too tanked to notice. (thanks, Martin - Rostock, Germany)