I am not sure of what I am going to write because I am afraid to miss some data, however the equation you wrote is similar to elongation of a bar under axial load when non linear elasticity occurs. The constitutive function D(u) will govern the stress strain relationship
stress = D(u) strain
read in your case
stress = D(u) u'(x)
so D(u) is the slope of stress strain relationship, so that when this slope become almost null or negative the Newton Raphson does not converge any more. (to follow instable path requires ad hoc numerical strategy say arch-length or similar)
From the pure mathematical stand point I guess that you have to check the coercivity of the 'strain energy operator' I mean 1/2 D(u) u'(x)^2 must be locally positive (for all u and everywhere with respect to x). Finally the loss of coercivity could means that the bar collapsed or buckled (buckling is more related to non linearity when D depends on F that is slightly different to your case as I read it). In this latter case D become syngular when F approaches an eigenvalue of D.