In the middle of the 15th century, Ségur and the Vicounty of Limoges were placed into the hands of the d’Albret family, when Alain d’Albret married Jean de l’Aigle’s niece. The couple settled down in Ségur where their first son was born : Jean, the future king of Navarre.
Afterwards, Ségur slowly lost its strategic role as a stronghold and seigneurial residence. In the 16th century, the d’Albret family sold their castle to their faithful servants, the Pérusses, who had become lords of Les Cars. They lived in Les Cars castle : a site that could easily be reorganized with new gardens and artillery boulevards, unlike the hilly and small area around Ségur.
During the following century, the Pérusses gave Ségur to the Hautefort family, who lived elsewhere, in a castle in Dordogne. After the French Revolution, the Hautefort family sold the castle to one of their farmers, Gabriel Dumas-Lavareille, who had become the town’s mayor.