A village has been on the site of Castleton since at least 1086, when it was mentioned in the Domesday Book. The settlement was linked to Peveril Castle which overlooks the town.

Peveril Castle gets its name from William Peverel who was granted Royal Manors of the Peak shortly after the Norman Conquest. It is suggested that he may have been the illegitimate son of William I. The castle was constructed to overlook the Hope Valley and Cave Dale, protecting the valleys. The castle was of traditional construction, with a central square keep and curtain walls.

Castleton continued to thrive, and prospered from Lead Mining, with the Odin Mine lying close to the village. It is the oldest documented mine in Derbyshire and one of the oldest in England.