Potato virus Y (PVY) is an important plant pathogen, whose host range includes economically important crops such as potato, tobacco, tomato, and pepper. PVY presents three main strains (PVYO, PVYN and PVYC) and several recombinant forms. PVY has a worldwide distribution, yet the mechanisms that promote and maintain its population structure and genetic diversity are still unclear. In this study, we used a pool of 77 complete PVY genomes from isolates collected worldwide. After removing the effect of recombination in our data set, we used Bayesian techniques to study the influence of geography and host species in both PVY population structure and dynamics. We have also performed selection and covariation analyses to identify evolutionarily relevant amino acid residues. Our results show that both geographic and host-driven adaptations explain PVY diversification. Furthermore, purifying selection is the main force driving PVY evolution, although some indications of positive selection accounted for the diversification of the different strains. Interestingly, the analysis of P3N-PIPO, a recently described gene in potyviruses, seems to show a variable length among the isolates analyzed, and this variability is explained, in part, by host-driven adaptation.