Is there a mathematical equation or formula to find the extinction coefficient or absorption coefficient of a thin layer based on transmittance or from the refractive index of the material?
What kind of thin layer do you have? Coated on a substrate or just a very thin slab such as a plastic sheet? The surfaces of your thin layer are crucial to calculate the extinction coefficient too. You have to make clear.
If you know the complex refractive index (n-jk) of the material at a wavelength "lambda", you can use the imaginary part "k", also called extinction coefficient, to calculate the absorption coefficient (A) using the formula "A=4*pi*k/lambda". Or alternatively, if you have the transmission spectrum (Pout/Pin) for the normal incidence, you can use the following equation to calculate the absorption coefficient (A):
Pout/Pin=exp(-At); where t represents the thickness of the layer.
I would not use the Beer-Lambert law to calculate the absorption coefficient, not even for very thin layers. It basically assumes that light has no wave properties i.e. interference effects are excluded. Absorption (and absorptance, which is 1-R-T) is depending on the electric field intensity, which is strongly altered by interference.
Article The electric field standing wave effect in infrared transfle...
Article The Electric Field Standing Wave Effect in Infrared Transmis...
Article Employing Theories Far beyond Their Limits-The Case of the (...
Even for very thin films, Beer-Lambert does not work properly: Article Comparative study of optical analysis methods for thin films
The correct way to go is to determine the optical constants of the film, see e.g. Article Removing interference-based effects from the infrared transf...
Or, if you have the index of refraction function, you can calculate the index of absorption function and from the latter the absorption coefficient. A simple possibility to apply Kramers-Kronig is explained in here: Article Improving Poor Man's Kramers-Kronig analysis and Kramers-Kro...