Kathmandu (or as Stevens spelled it, Katmandu) is the capital of Nepal. In the '60s, many Americans were fascinated with Eastern orthodoxy and saw places like Kathmandu as beacons of enlightenment. Many young people made the trek, and some high-profile musicians mentioned the city in songs where it was used as a metaphoric Eden, which is what Stevens did here as he sings about finding refuge from Western civilization in a simpler, peaceful place.
You'll find plenty of reports claiming that Stevens wrote this song in Kathmandu (a common rumor has him writing it in a tea house in the Asan Tole section), but there is no evidence that he made this journey. These tales are often found in tourist guides with an interest in drumming up business; Mick Jagger, Janis Joplin, Jimi Hendrix and Bob Marley have also been cited as visitors to the city in various apocryphal stories (note how documented The Beatles trip to India was - a rock star couldn't vanish to Nepal without some hard evidence).
Stevens was certainly there in spirit, however, as Bob Seger would be in 1975 when he released his own song about Kathmandu
, also spelled without the H.