Wainwright told the Sun newspaper why she named the album after a line from this song: "I thought it was funny. I write phrases like that which are almost nonsensical or a bit off the wall. I find it provocative. I was very nervous about naming the album. I don't even like naming songs until I absolutely have to. I call everything 'the something' song. Bleeding All Over was obviously going to be the title track and I thought it was really obvious and really menstrual. It's clear my music and my person is bleeding all over people so I just wanted to give them a break. I didn't want it to be me on the cover with my wrists split open. They were pressurizing me to come with a title and I kept coming back to I Know You're Married But I've Got Feelings Too. So I thought, 'F--k it, we'll go with it.'"
Wainwright told Uncutmagazine February 2008 about the album: "It's less singer-songwriter. Martha Wainwright was all navel-gazing songs about me and my family. The new songs are more demented, and more concerned for other people."
"It's the End of the World As We Know It (And I Feel Fine)" was inspired by a dream where Michael Stipe conjured up images of people with the initials L.B.: Lester Bangs, Leonid Breshnev, Lenny Bruce and Leonard Bernstein.