The harbour is the main focus of the village, particularly early each morning when the fishing boats return with their catch and the daily fish market gets underway. Many of the picturesque houses around the harbour are of traditional Fife design with red pantile or grey-slate roofs with crow-step gables and have been restored by the National Trust for Scotland.
Interesting buildings in the village include Kellie Lodge, a late 16th century town house that was once the residence of the earls of Kellie, the parish church that dates from the same period and the attached ruins of the 12th century Augustinian priory. In the cliff face beneath the priory, St Fillan's Cave once housed the eponymous saint and is now regarded as a shrine to him.
The East Neuk is particularly rich in castles and mansions and two of the more interesting north of Pittenweem, Balcaskie House and Kellie Castle.
Pittenweem also hosts a thriving arts festival each August.
