In an audio interview with vocalist Ronnie James Dio he explained that this song is about a man who is addicted to the Bible Black, a book of sinister scriptures, which goes on to destroy him.
This was one of the first songs written for the album. Dio explained that it established a tone for the rest of the LP. "When you start off with a blockbuster like that, it makes the rest of the album so much easier because it gives you a benchmark to measure the other songs against."
Guitarist Tony Iommi admitted to Billboard magazine that the song's animated music video rubbed him the wrong way. He explained: "I think it' bloody awful, to be honest. It looks like Casper the Ghost or something. We didn't have anything to do with it; that was sort of just presented to us, really. I'd have thought it could be done better than that, but that's just my opinion, and I'm just one of four."
In an interview by Justin Donnelly posted on his MySpace site, bassist Geezer Butler explained the album title: "Everybody was asking me if I had any ideas for the title of the album. So I got a few titles together, and that was the one that everybody seemed to like. So you can say that I was the one responsible for coming up with The Devil You Know. The meaning behind the title is really quite simple. Everybody still calls us and sees us as Black Sabbath. So even though we call ourselves Heaven & Hell, we're still very much the devil you know in the sense that we're still Black Sabbath. That's the story behind that."
The album artwork is adapted from a painting by Norwegian artist Per Qyvind Haagensen's entitled "Satan." Butler told Justin Donnelly that the artwork doesn't sit comfortably with him. He explained: "The record company found that image for us. I think Iommi and Dio quite liked it. I wasn't that happy with it, but then what can I say? It's a majority rules within this band, and I was outvoted It's a bit too satanic looking for me, but it is what it is."
The artwork features the addition of the numbers 25 and 41, which Butler was responsible for on either side of the band's familiar winged demon logo on the front cover. Butler explained to Justin Donnelly: "We've had the winged demon logo for a long time, and the record company wanted to use that logo once again on the new album. So I thought that rather than do the same thing once again, I decided that we needed to do something a little different. So I came up with a verse from the bible. It's from the gospel according to Matthew, chapter twenty-five, verse forty-one. The verse goes, 'Then He will also say to those on His left, 'Depart from Me, accursed ones, into the eternal fire which has been prepared for the devil and his angels!' It's about those who sit at the left hand side of God who are cast down into hell. That's all."