According to Rich Podolsky's book
Don Kirshner: The Man with the Golden Ear, Larry Kolber was a veteran with journalism experience, having written for
Stars and Stripes (yes, the same newspaper mentioned in the film
Full Metal Jacket!). But his post-veteran career found him as a whiskey salesman - an unpredictable twist. While talking with a fellow veteran, he learned by chance that he was a songwriter. Kolber was amazed that people actually earned money just for writing songs, and decided to try his hand at it. Through his veteran friend, he met Don Kirshner of Aldon Music. Kolber's first song for Kirshner was "Forget Me Not," a minor hit for the Kalin Twins.
Kolber was having lunch at a cafe across the street from Aldon Music when he literally jotted down the lyrics to "I Love How You Love Me" on a napkin! Seven minutes later, he was back across the street looking for somebody to set it to music. Barry Mann happened to be in the Aldon offices just at that moment. Tony Orlando was originally slated to sing it, but Phil Spector happened to have dropped by and asked for the song for one of his girl groups. Kolber was disappointed, thinking that he'd lost a shot at fame without Orlando's voice. Kirshner and Spector were acquainted by common partners, the legendary songwriting team of Jerry Leiber and Mike Stoller, so Kirshner let him have it.
The sisters Paris rehearsed the song at home, then recorded with Spector in Hollywood, at Goldstar Studios at the corner of Santa Monica and Vine. They reported that Spector was difficult to work with.