During road construction on problematic soil, selected soil is brought for replacement of existing soil to a certain depth. how can an engineer determine this depth is my question
It will depend on the properties (mainly rigidity) you are looking for. You want to provide a platform that can resist engine loads during the construction without any permanent deformations (sags and bumps). You should replace the thickness so that the new rigid material layer will be able to distribute engine loads (stresses) that could be resisted by to the bad layer. A simple model or recommendations provided by existing manuals (depending on the existing soil type and the treatment method) would be able to give you the answer.
Please note that thickness of pavement depends on traffic loading created by trucks for next 20 years (if you are planning for 20 year design life). Basic soils tests such as Atterberg limits, grading, CBR test, and field moisture content test will help you to make judgement on parameters required to run CIRCLY based pavement modelling. Otherwise, there are design charts based on traffic loading and CBR of subgrade, subbase and base. If you have no access to the above, my gut feeling or thumb rule is the depth to be replaced should be within 450mm to 1000mm because minimum base thickness of 150mm of CBR value of 80, subbase thickness of 150mm of CBR value of 60 and 150mm selective material of CBR value of 15 and subgrade CBR of minimum of 3 are required.
Please note that problematic soils are expansive soils (please check swelling potential), collapsible soils (when water added) and residual soils (which comes from decomposed weathered rocks and have more fine content and higher plasticity). Please refer to Fredlund, et al (2012) on unsaturated soils mechanics in engineering practice.
For replacement of expansive soils in road construction, you need to know the depth of expansive soil layer and its swelling potential. For thin layer and shallow depth expansive soil it is recommended to remove it and to be replaced with selected granular soils. In case that expansive soil layer is thick and deep, you can remove part of the soil to a certain depth depending on swelling pressure of this soil, then replace it with a layer of selected granular soil that exerts an overburden pressure to resist the swelling pressure of expansive soil.
Using of trial pits will identify the depths of the clay soil which has less CBR. So you need to remove or improve the CBR by adding rocks and compacting. If the dept is more than 1 m it becomes expensive for a long road to remove and dispose and hence the need to stabilize.