Hi jege41,
Some of the newer memory foams are much more breathable than the older versions and FXI in partiular has been making higher airflow memory foams for quite a few years (starting with Aerus). In the end though … all foam is still an insulator to differing degrees and the type of foam you use is just one of the many factors that in combination with each other control the sleeping temperature of a mattress. Post #2 here has more information about the factors that control temperature regulation. In most cases even memory foam can be 'cool enough" for most people if all the other factors are “cool enough”.
All the layers that are over your memory foam will affect temperature including the mattress cover, any cover on the topper, the down quilt, the type of mattress protector you are using, your sheets, and of course the bedding which is over you.
If you are using the down duvet underneath you I would agree with your wife and remove it. Down is great over the body where it can stay lofted and has more airflow but not as good when it is compressed under the body. Many down comforters also contain feathers which are not as breathable (although they are more resilient than down) and they can also retain moisture. It can also isolate the memory foam from your body heat and slow down its response as well. “Down on Down” can also be particularly warm.
You can also read about some of the different types of mattress protectors here and if you have one of the thinner membrane types then they are less breathable than the other types and could warm up the mattress as well. If you are using polyester sheets then cotton or other natural fibers or even viscose fibers can also help cool things down. Linen is particularly cooling and durable.
So I would look at every layer that is in between the memory foam and you and see if any of them could be contributing to your sleeping temperature. You could also add a stretchy cotton or bamboo cover to the topper which may help as well. I would consider changing any of these these before considering the unknown variable of adding a topper and completely changing the feel and performance of the mattress.
Phoenix