I am working in control radiant flux of LEDs using a PWM controller. For this purpose I will use a MOSFET as a switch between the PWM chip and the LEDs. In this case should my mosfet operate in linear or saturation mode?
Perhaps there some confusion regarding the terms here. The BJTs operate in 'saturation mode' when used as switch, usual definition being forward biasing condition of base-collector junction However, the 'saturation mode' for for MOSFETs means essentially current limiting mode, in 'switch mode' MOSFET is usually ohmic = 'linear mode'. Usually, PWM is used combined with switch mode.
However, one should note that LEDs should be current not voltage controlled. The easy way to implement it is using PWM controlled switch mode MOSFET and current limiting resistor. More sophisticated schemes may use PWM smoothing by using conductors, capacitors.
PWM controlled MOSFET based current source can also be considered but usually is more complicated and less efficient.
Nikolay Pavlov thank you a lot, it was very helpful. Just one more question, the PWM will control my VGS, could this change the operation mode? How can I ensure that the MOSFET will always be in linear mode?
The MOSFET datasheet has Drain-Source Current vs Drain-Source Voltage curves for given VGS, and knowing expected LED current you will see expected Drain-Sorce Voltage drop. Usually the MOSFET is chosen to be large enough so Drian-Source Voltage is small e.g.