In this song Kurt Wagner of Lambchop riffs on the word "stand." He admitted to The Village Voice that this track was a songwriting exercise. Wagner explained: "I wanted to try to use one word, and then, 'What word?' And for some reason 'stand' became the word. For me there's lots of minor, interesting reasons why I like that. I mean, I sit all the time and play and whatever. That was the starting point."
The lyrics for "Mary, Did You Know?" were written by Christian singer and comedian Mark Lowry, after his pastor asked him to write a Christmas musical for their church. Southern gospel musician Buddy Greene later added music to his words.