Sometimes even 1/2" can make a difference.
I would recommend a 1" super low ILD latex layer. Memory foam may work for you but for me, even 1" of it causes my lower back to hurt. If you do go with memory foam, go with a higher density layer such as a 5lb or higher density.
For latex I forget who it is - someone here can tell you - but some online site sells something like 18ILD latex. You might even want to go 2" with latex but I wouldn't go 2" with memory foam.