Hi jayne,
Post #2 here along with post #29 here may give you some insights into what may be happening. Temperature regulation on a mattress is usually a combination of factors that work together either for or against good temperature regulation.
The things to look at would be the top layers of foam (not all latex mattresses have latex in the top layer), the mattress quilting and ticking (synthetic fibers and fabrics and particularly polyester will be warmer than natural materials that are more breathable and wick moisture better), the mattress protector you are using (the thin semi breathable membrane type is less breathable and can sleep warmer), your sheets and bedding (although it seems that these are fine) and the toppers you are using. Natural fibers would normally be cooler than synthetic fibers (like the polyester in the down alternative topper). How far you are sinking in to all the layers will also make a difference because the more you sink in the more insulating material there is around you.
What type of mattress protector are you using?
Have you considered a wool topper? Wool is one of the most temperature regulating materials there is.
Phoenix