For compressible flows because the airfoil is curved there is a transverse pressure gradient near the surface. In such cases it is necessery to compute the theoretical (ideal) velocity profile first (by taking total pressure equal to stagnation pressure of the inflow and using isentropic flow equations to get velocity which is not a constant value as in case of a flat plate flow). In the next step you need to find a point at which the velocity in your flow is reaching e.g. 99.5% of this theoretical (ideal) profile. This way you will find your boundary layer edge on a convex surface and be able to compute for example integral parameters, such as displacement or momentum thickness.
For compressible flows because the airfoil is curved there is a transverse pressure gradient near the surface. In such cases it is necessery to compute the theoretical (ideal) velocity profile first (by taking total pressure equal to stagnation pressure of the inflow and using isentropic flow equations to get velocity which is not a constant value as in case of a flat plate flow). In the next step you need to find a point at which the velocity in your flow is reaching e.g. 99.5% of this theoretical (ideal) profile. This way you will find your boundary layer edge on a convex surface and be able to compute for example integral parameters, such as displacement or momentum thickness.
You may find the velocity distribution in the boundary layer by experimental or numerical methods. By definition, the boundary layer thickness is the position where u/U = 0.99. However, in both the methods above, it is difficult to have a smooth boundary line. So, you may reduce the definition to be less than 0.99, i.e 0.98 or lesser until you have the smooth one. It is the same case when you want to find the boundary layer thickness by CFD. But if your calculation is accurate then you will find a smooth boundary layer line thickness by using a definition of near u/U = 0.99.
I can recommend you take a look at a book by Alan Pope called Basic Wing and Airfoil Theory. There are specific chapter in this book explaining how to calculate the velocity at any point on an airfoil, both thick and thin. Starting on page 122, chapter 7.3 you'll find first the discussion as it applies to a thin airfoil. This will not be a constant, and most likely it will vary with the wing aspect ratio. Also Abbot & Doenhoff treats this subject in their book Theory of wing sections. Also G.D. McBain has an excellent book called Theory of Lift, that might be very helpful. Good luck.