The title refers to the song structure as it was originally written: "A", "B", and "C" refer to different sections of the song:
A = Verse
B = Chorus
C = Bridge
So the song went verse, chorus, verse, bridge, verse, chorus, but it ended up having a different structure - "ABACAB" was just a way for them to remember how it was formed when they started working on it.
When it came time to title the song and the album, they wanted something a bit abstract, and "Abacab" worked because it had no meaning. The album art was created to look like an abstract painting to go with this theme, which indicated that this would not be typical of Genesis' previous work. "It wasn't going to be goblins and fairies," Mike Rutherford