Before the end of the 12th century, the Lusignan family carried out new construction work on the rocky outcrop.
On the first platform a fortified stone outer wall was built. The chapel of Saint Peter was incorporated into one corner of the fortification, near the cliff edge. On the approach to the chapel, there are a few surviving blocks of stone from the wall, a tower and a gate. This passage lead via a walkway towards the old motte and bailey keep.
The chapel's chancel was originally semi-circular. However, when a section of the cliff collapsed the chancel had to be rebuilt as a quarter circle due to the limited space that remained.
Also in the 12th century, the Lusignan family had a Romanesque keep built to the north. Even though only one side has survived, it is possible to imagine how imposing that would have been over the site. Laid out in a quadrangle, it measured 16 metres long by 8 metres wide. It rose to a height of four stories and was protected by semi-cylindrical buttresses, like those of the castles at Niort and Chauvigny.
The castle we can see today gradually grew up around the Romanesque keep, with curtain walls and towers connected by parapet walkways. The northern entrance is flanked by two towers, the prison tower and the Saint Marie tower. The second of these towers contains a castle chapel of the same name. To the western side, the Amallec tower dominates passage over the Anglin. This imposing onion-domed tower protected the eastern curtain wall, which was the most exposed to attack. To the south-east, the tower gateway was probably used to reach the outer wall of the chapel of Saint Peter, via a drawbridge walkway over the English Trench.
The fortifications would eventually cover the entire rocky outcrop and measure 300 metres in length.