The castle underwent significant changes during the 15th century. The Bishop of Poitiers, Hugues de Combarel, transformed the Romanesque keep into a dwelling fit for a 15th century bishop. He had wide windows knocked into the walls on various sides of the building and had the upper floors redone. A tower complete with spiral staircase provided access to each of the four levels. A doorway and protective bartizan were added to provide access to the building from the interior courtyard.
In the main courtyard, on the edge of the cliff, the bishop also had a second dwelling built comprising two rooms per floor and a tower complete with spiral staircase. A small balcony-like passageway dominated the cliff.
To adapt to the new weapons of siege and the increasingly common use of artillery, Hugues de Combarel had the walls and towers fortified. The thickness of the walls was increased, and new firing slits which would accommodate cannons were installed.
He hung the family coats of arms at various points around the castle.
The fortress suffered significant damage during the religious wars of the 16th century and the war of the Fronde in the 17th century. The bishops abandoned Angles and Chauvigny in favour of their castle at Dissay. The fortress at Angles was poorly maintained and was eventually abandoned in 1708 after which time it gradually fell into ruin. It was declared a national asset during the Revolution and served as a public quarry.
The castle was declared a Historic Monument in 1926 and works were carried out on several occasion to preserve the ruins. Since 1986, the castle has belonged to Angles-sur-Anglin Local Authority.
It is now possible to visit the fortress and admire the magnificent views from the cliff tops.