Hi gregtames,
Just to verify - the topper is flat and level when you put it on the floor, correct? Next you need to eliminate the possibility that the frame and support layers are not uneven. Can you see any sagging under the mattress? What you describe seems to point to a worn out mattress with virtual impressions (foam broken down but not visible to the naked eye). You can read what @Phoenix has to say on virtual impressions here
~ Basilio