Vampire Weekend first started working on this song with the subject matter of worrying about life rushing past during the early days of the band at Columbia University in New York City. Vocalist Ezra Koenig told NME: "We first started writing it in college. The chorus feels like it has an important line for the album. 'Though we live on the US dollar. You and me, we got our own sense of time.' Two people enjoying their youth, but maybe there's the first seeds of discontent popping up..."
The band had the song in their back pocket for a long time before they found an album where it fitted. Ezra Koenig told Q magazine: "Our leftovers tend to be snippets, a beat, an idea, a lyric. Sometimes they just didn't have enough time. 'Hannah Hunt's' roots go back seven years but this is the LP it belonged on. So things get discarded but never discarded eternally. You never know when it will make sense."
Vampire Weekend recorded an uncompleted version of the song for their Contra album. According to the band's Rostam Batmanglij, "it sounded kinda like 'Say It Right' by Nelly Furtado."
In The Beatles "When I'm 64," Paul McCartney asks a woman if she'll still be there for him when he's 64. In 2006, he got his answer when shortly before his 64th birthday, he and Heather Mills separated.
Kacey Musgraves originally offered "Follow Your Arrow" to her friend Katy Perry. However, Perry thought Musgraves should record the song herself as it seemed, "like something that you would totally say."