This song is about the small-town rumor mill, where everyone knows each other's business, creating a toxic environment. The town in this case is Woodstock, New York, which The Band called home for years. Woodstock was subject to extraordinary scrutiny and innuendo because it was home to Bob Dylan, Van Morrison, Janis Joplin and many of far less renown who came there to be part of the scene.
The Band had three lead vocalists, and they used them all on this track. In order, Rick Danko, Levon Helm and Richard Manuel sing. Like most of their songs, it was written by guitarist Robbie Robertson.