American alternative rock band Pixies were a major influence on Cage the Elephant's sophomore album, Thank You, Happy Birthday and on this amped-up song, vocalist Matt Shultz breaks out his best Frank Black yell to let off steam. Shultz recalled to MTV News that at the time he was writing for the album, "I had just broken up with my girlfriend of five years, and I'd drive around until like 3 or 4 in the morning listening to Doolittle. It was my therapy, I guess."
When Shultz wrote the song he was hanging out with Tiger Merritt and other members of Kentucky Psychedelic rock band Morning Teleportation. He told Spinner UK how his pals inspired him to pen this tune. "Tiger's the type of guy that every time he touches an instrument, it makes you jealous so I wrote it like on the spot. They had all these new songs and I said, 'Yeah, I have a new song I'm working on right now,' so I just picked up the guitar and started playing it and that's how that song came about pretty much."
Kacey Musgraves originally offered "Follow Your Arrow" to her friend Katy Perry. However, Perry thought Musgraves should record the song herself as it seemed, "like something that you would totally say."