The church building "Zu unserer lieben Frauen" was erected in its present external form in 1587, destroyed down to the brickwork in 1634 and rebuilt between 1648 and 1651. There are still many traces from that time, such as fire remains and fire-preserved food, as well as a coin hoard and, a few years ago, even a buried contemporary just behind the archway at the entrance to the castle complex.

The church is located on the castle hill and owes its isolated position in part to artificial rock work. The deep ditch on the north side is a manually created neck ditch. It separates the terrain from the adjacent hill. The staircase is on the south-west side.

The historic fortified church in Walldorf burned down on April 3, 2012. The organ and altar were completely destroyed. The roof of the nave collapsed and the bell tower was in serious danger of collapsing. It was rebuilt in the following years and reopened in 2019.