R.E.M. guitarist Peter Buck explained to Q magazine in 1992: "There was an old guy in my neighborhood who would be out gardening at 2 a.m. in his suit and tie. I'd see him when I was out trying to get a beer at the Magic Mart or somewhere. It's basically a metaphor for the uselessness of everything."
Drummer Bill Berry remembers this as being the band's first layered guitar track. He recalled in the liner notes for Part Lies, Part Heart, Part Truth, Part Garbage 1982-2011: "We were novices to the studio and overdubbing capabilities of multi-track tape. Despite his youth at the time, [producer] Mitch Easter had managed to assemble a fine collection of guitars. Pete seemed to be having a blast adding guitar tracks with various 6-stringed instruments and for good reason. It sounded cool. Listen for his psychedelic touch on electric sitar during the two quiet breakdowns."
Michael Stipe noted this as "our first real song, all the ones before it were us counting to ten, back and forth."
In the 2006 compilation And I Feel Fine... The Best of the I.R.S Years 1982-1987, Berry recalled how he first heard the title phrase: "We were driving at night after a show (I don't remember where), and I was at the wheel of our old car, with a rental trailer in tow. One of my three passengers aimed a directive at me. Rather than inform me of his desire to evacuate his bladder, he instead suggested that I pull over so that he might engage in the task of roadside 'night gardening.' To four guys in their early twenties this was a glaring catalyst for a new song."