Age-related memory decline has been associated with a faulty regulation of the hypothalamus-pituitary-adrenal axis (HPA-axis). The aim of this study was to investigate whether the magnitude of the stress-induced cortisol increase is related to memory performance when memory is measured in non-stressful conditions. To do so, declarative and working memory performance were measured in 31 men and 35 women between 55 and 77 years of age. On a different day, the magnitude of their cortisol response to acute psychosocial stress was measured. The relationship between the cortisol response and memory performance was U shaped: a low cortisol response to stress was related to poorer declarative and working memory performance, whereas those who did not increase their cortisol levels and those who had the largest cortisol increase had better declarative and working memory capabilities. Sex did not moderate these relationships. These results suggest that a low cortisol response to stress could reflect a defective HPA-axis response to stressors that is accompanied by poorer memory performance. Conversely, a high cortisol response seems to reflect a correct functioning of the HPA-axis and may protect against memory deficits in the later stages of human life.