The castle and the imperial palace were first mentioned in documents by King Otto I in 967 and 974. From this time until well into the 13th century, many rulers found accommodation in the imperial palace and castle. The most significant historical event took place on March 8, 1198, when Frederick Barbarossa's son Philip of Swabia was elected king. The citizens of Mühlhausen, who were striving for more autonomy, had the castle completely torn down in 1256. Some of the stones from a round tower were used to extend the choir of St. Mary's Church, and some of the last remains of the castle in the ground were removed and reused when a local brewery built beer cellars in 1851. The former fortress is commemorated by nearby street names and the shopping center that opened in 1998 Castle gallery. It took a while for the king to forgive the citizens for the violent destruction of the castle in 1256.

Finally, in 1310, Henry VII donated the site to the nearby Cistercian monastery of Volkenroda, which had to cede it to the town of Mühlhausen eight years later. Near the former castle was the Jewish cemetery, which was mentioned in documents as early as 1417 and may have existed before 1349. The cemetery was closed in 1871 and its gravestones moved to the new Jewish cemetery to make room for the widening of the road connecting the old town with the newly opened railroad station.