Brain imaging studies have shown that slow and progressive cerebral atrophy characterized the development of Alzheimer's Disease (AD). Despite a large number of studies dedicated to AD, key questions about the lifespan evolution of AD biomarkers remain open. When does the AD model diverge from the normal aging model? What is the lifespan trajectory of imaging biomarkers for AD? How do the trajectories of biomarkers in AD differ from normal aging? To answer these questions, we proposed an innovative way by inferring brain structure model across the entire lifespan using a massive number of MRI (N = 4329). We compared the normal model based on 2944 control subjects with the pathological model based on 3262 patients (AD + Mild cognitive Impaired subjects) older than 55 years and controls younger than 55 years. Our study provides evidences of early divergence of the AD models from the normal aging trajectory before 40 years for the hippocampus, followed by the lateral ventricles and the amygdala around 40 years. Moreover, our lifespan model reveals the evolution of these biomarkers and suggests close abnormality evolution for the hippocampus and the amygdala, whereas trajectory of ventricular enlargement appears to follow an inverted U-shape. Finally, our models indicate that medial temporal lobe atrophy and ventricular enlargement are two mid-life physiopathological events characterizing AD brain.