Wainwright told the Sun newspaper why she named the album after a line from this song: "I thought it was funny. I write phrases like that which are almost nonsensical or a bit off the wall. I find it provocative. I was very nervous about naming the album. I don't even like naming songs until I absolutely have to. I call everything 'the something' song. Bleeding All Over was obviously going to be the title track and I thought it was really obvious and really menstrual. It's clear my music and my person is bleeding all over people so I just wanted to give them a break. I didn't want it to be me on the cover with my wrists split open. They were pressurizing me to come with a title and I kept coming back to I Know You're Married But I've Got Feelings Too. So I thought, 'F--k it, we'll go with it.'"
Wainwright told Uncutmagazine February 2008 about the album: "It's less singer-songwriter. Martha Wainwright was all navel-gazing songs about me and my family. The new songs are more demented, and more concerned for other people."
There was a rumor in the Soviet Union that The Beatles had secretly visited the U.S.S.R. and given a private concert for the children of top Communist party members. They believed the song "Back In The U.S.S.R." was written because of the concert. Actually, some fans still believe so.