As a result of these invasions, by the High Middle Ages Agen had been reduced to a cluster of fortified islands known as "castra".
However, the new millenium heralded the beginning of a period of prosperity, which would last into the middle of the 14th century. 
Agen became part of the new Dukedom of Gascony, which from 1032 was incorporated into the State of Aquitaine.
Aquitaine's fate would be closely linked to that of its beautiful Duchess, Eleanor, who became Queen of France in 1137, then Queen of England in 1152. And for the next 300 years, Agen was destined to move perpetually backwards and forwards between the Duke of Aquitaine (also the King of England) and the King of France, with the Count of Toulouse playing a walk-on part.
Eleanor's son, Richard the Lionheart, gave his sister in marriage to the Count of Toulouse and the County of Agen was her dowry. 


Under the Counts of Toulouse in the 13th century, at the time of the Albigensian Wars, Agen, like Albi, Toulouse and Carcassonne, became a Cathar stronghold, but in 1251, it was retaken by Alphonse de Poitiers and on his death reverted to the King of France.
However, political expediency led the King of France to hand the County of Agen back to Edward the First of England and so once again Agen had a new King. 
This did not prevent further warfare between the two countries between 1292 and 1298 and even when peace returned, the town of Agen was vulnerable to attack by neighbouring robber barons. 
The imperative of the 14th and 15th centuries was, therefore, to construct a new town rampart, which would enclose a 60 hectare (150 acre) site, six times the size of the previous town. 


The Hundred Years' War began in 1324 with an incident between the French fortified village of St.Sardos and the English fortress of Montpezat, both now in the Lot-et-Garonne Department. The war fluctuated back and forth over the region, with local robber barons changing sides at will. 
The inevitable accompaniments of war - famine and disease - were never far away and in 1347 the town was devastated by bubonic plague. 
Agen was to change hands three times during the war and it was only with the final departure of the English from the Aquitaine in 1453 that the area became definitively French. 
From the 1440s onward, new settlers began to arrive from all over France and the town of Agen started to rebuild.