The idea for the song came from Eric Stewart saying to his wife that if he kept saying "I Love You" over and over, it wouldn't mean anything even though he did love her. He recalled to The Guardian
"I met this gorgeous girl called Gloria at Halifax town hall. I was 18. She was 16. Three years later, we got married. A few years after that, Gloria told me: 'You don't say 'I love you' much anymore.' I told her that, if I said it all the time, it would sound glib. But I started wondering how I could say it without using those actual words. So 'I'm not in love' became a rhetorical conversation with myself – and then a song.
I wrote the lyrics in a couple of days. The line, 'I keep your picture up on the wall, it hides a nasty stain' was about the crack in my bedroom wall at my parents' house in Manchester. I'd put a photograph of Gloria over it. When I took the song to the band, they said: 'I'm not in love'? What the f--k is that? You can't say that!' But Graham Gouldman, our bass-player and chord-master, agreed to work on it with me. We both liked The Girl From Ipanema
, so we gave it a similar bossa nova style. Then Kevin Godley, our drummer, said it was crap.
We were about to scrap it and wipe the tape but, as I walked around the studio, I heard the secretary singing it and the window-cleaner whistling it. I knew we had a tune: we just hadn't captured it properly. Kevin suggested doing it again, but with banks of voices. I thought that meant hiring a choir, but Lol Creme, our keyboard player, said we could do it using tape loops."