They take core ice samples that are miles deep and look at the different sections. During an ice age the oxygen molecules are different (the reason is complicated, but it happens). They weigh more than oxygen at a normal temperature and have 18 neutrons instead of the normal 16 for most oxygen we find today.
Another way would be to look at deep rock samples and examine things like pollen and other environment things that change during an ice age. It is less accurate than the oxygen method, but can work to confirm things.