The houses of Rastenberg were clustered around the castle mound and the inhabitants sought shelter. Shard finds confirmed the assumption of a prehistoric rampart castle at this location.

Raspenburg Castle was probably built on the castle hill in the years 1070-1078. There is only contradictory information about the history of the castle from the first decades. It remains unclear whether the builders of the castle were the sons of Louis the Bearded, Louis the Knight or his brother Heinrich Raspe (i.e. the Rough). The owners of Raspenburg Castle are attested from 1252. In the first written record of Rastenberg from this year, two brothers, «Heinrich and Albert von Raspenberck», are mentioned.

The castle was located close to the „Via Regia“, the Erfurt-Naumburg trade route, and over time degenerated into a nasty nest of robber barons. It was therefore destroyed in 1321 by Landgrave Friedrich (the Bitten) with the help of merchants from Mühlhausen and Erfurt.

Today, only the remains of the keep can still be seen. It measures 32 m in circumference and has a 2.5 m thick wall.