In 1208, the Counts of Schwarzburg received the former imperial city of Saalfeld as a pledge. However, the tower castle, whose name "Hoher Schwarm" was first documented in 1593, was hardly used by the Schwarzburgs as a residence; instead, it was home to their castle servants, members of the lower nobility.

Around 1300, the castle was rebuilt, the remains of which are today's castle ruins. At the end of the 14th century, Saalfeld was sold to Margrave Frederick IV of Meissen. The governors of the tightly organized Saxon administration moved in.

200 years later, the castle lost its function as a defensive structure. The following legend is still told today about the name "Hoher Schwarm": A priest let a white dove with a silver bell fly. It was supposed to determine the future location of the castle by choosing its resting place. The dove settled on an oak tree on the banks of the Saale. When the mighty tree was felled, a swarm of bees flew high into the air from its trunk. This is how the name "Hoher Schwarm" is said to have originated.

The building has been in ruins since the 16th century.