This paper investigates the reduction of backscatter radar cross section (RCS) for cavities embedded in the ground plane. It is established by placing a multilayered radar absorbing material (RAM) at the bottom of the cavities. The minimization of the backscatter RCS is formulated as a bound constrained optimization problem. A new vertical mode expansion method is developed to solve the scattering problem, and it is integrated into the gradient projection method to obtain the optimal composite coating. Numerical examples show that the method we developed is efficient and the backscatter RCS can be significantly reduced.