"The late George Woolnough" is Peter's grandfather, who was the saddler in the Australian town of Tenterfield. He "worked on High Street, and lived on Manners", Manners being another street name. There is a library dedicated to him.
"The son of George Woolnough" - Peter's father - took to the bottle, and shot himself, inspiring the lines "something was wrong, and it's easier to drink than go crazy" and "George had no answers about why a son ever has need of a gun."
"The grandson of George" is Peter, who changed his name from Woolnough to Allen when he became part of "The Allen Brothers" with Chris Bell. He's "been all 'round the world and lives no special place," compared to a line from another song, "no matter how far or how wide I roam, I still call Australia home."
The "girl with an interesting face" is Liza Minnelli, who Allen married in 1967.