While people are flocking to regions, others aren't leaving