Changing the value using a variable resistor (which in turn is converted to a value using ADC), the duty cycle of PWM changes but the waveform is unstable. It constanly starts vibrating to and fro in the x axis
1). Your variable resistor range is too small, and the ADC value is too large (resolution).. so that, small change on variable resistor would affect to large ADC conversion values.
2) Your resolution of PWM is too small, so the result after calculating the ADC values make it not stable or vibrating.
You may check and observe :
The ADC input and ADC values, use it and compute by your hand to feed the PWM duty cycle. So you are sure that if it is a problem or not. If necessary, use filter before feeding to ADC Conversion.