I like a firm mattress, and my wife likes a little more padding than I do. I was thinking about starting conservatively and add layers to get the right feel.  I personally like the pu foam core mattresses with latex on top better than the full latex mattresses that we have tried. 

1" 20 ILD latex
1" 32 ILD latex
2" hd36 hq
3" lux hq

Is 2" of latex just too little? (We have a 3" sensus topper that we don't use because it feels like we are being swallowed by it.)

Also, we currently have a feather bed and plush mattress pad that can be part of our formula.

I have a 2" latex topper of 20 ILD and it is good. 

It is quite supportive but has significant give.  I wouldn't consider it to be too soft, but since that is what I want I am buying an additional 2" super soft conventional foam topper to put on top of the latex.

But I find latex to be a pretty good product.