After the great town fire in 1517, the knight Georg von Holbach acquired the site of the fire and had today's small castle built in 1521-1522 using part of the still standing foundation walls of the previous building and a defense tower of the town wall.

The building, formerly known as the „Schlösschen“, is a richly structured building in the early Renaissance style. Characteristic are the seven richly structured four-zone decorative gables, which elegantly break up the façades. At the south-west corner of the courtyard side of the building is an oriel whose parapet is ornamentally decorated with stylized, late Gothic tracery. The richly structured portal on the south side with its pointed arch and canopy capitals is still in the late Gothic style, while the use of seating niches, formed by enlarged bases and the omission of pilasters, are already part of the Renaissance design principle.

In 2006, the gardens at Kitzerstein Castle were redesigned and are now home to the town's music school.