Pottery finds in the area show that the castle of the settlement of Symi has been inhabited since prehistoric times. This natural fortress was crested by walls, whose earliest phase dates from Classical Antiquity; it was repaired several times, and it seems that it was used by the locals until the 18th century. Some of the later work on the castle was carried out by the Knights of St. John: the arms of grand master Emery d ‘Amboise overlook the southern entrance, and those of grand masters Jacques de Milly and Pere Ramon Zacosta are mounted on the south wall of the present church of Megali Panagia (Great Virgin) inside the castle. In the older church of Megali Panagia were once kept the archives and valuables of the monasteries, alongside the firmans (royal charters) of the privileges accorded to the island by Ottoman sultans. During World War II, Megali Panagia was destroyed along with a substantial part of the castle. For this reason, the nearby church of St George was renovated and renamed after it.
In addition to the present Megali Panagia, there are other churches within the castle, such as Panagia Portiani, Panagia Kyra and one more, dedicated to Archangel Michael of Tharri. Τhe castle was surrounded by a settlement, traces of which still survive.