In this song, the "Dirt" represents the people who are always trying to bring you down and are jealous of your success. Jay-Z expounds on his accomplishments and lavish lifestyle, and notes that these disparaging types are nothing more than dirt that he must brush off his shoulder.
After this was released, "Get that dirt off your shoulder" became a common catch phrase. The phrase can be accompanied by a gesture of casually wiping your shoulder, which became popular among athletes after making a big play.
Suggestion credit: Donovan Berry - El Dorado, AR
In this song, Jay-Z mentions the "40/40 Club," which is a restaurant he owns. He also mentions ESPN in that line, which is a sports network he enjoys and has interviewed him.