Lanthanides are always difficult to handle, especially due to the (necessary) relativistic corrections. I've always been a Turbomole user so I can't be too Dmol3 specific, but here are some strategies:

-start with the smaller basis set out of your options (I suppose you already knew that)

-As far as I can see, Dmol3 also has an SCF damping option in addition to the smearing you mentioned (see "Damping and Convergence" in the manual). Try to play with the damping parameters which might enhance the convergence.

-Finally, you can play with the SCF itself, especially with steps and even the convergence criteria both for the SCF as well as your geometry optimization. A problem that often occurs is that if the starting structure deviates too heavily from a stable state the SCF convergence also gets worse. If your structure has at least once converged with lax criteria it is often much simpler to proceed with stronger criteria. If you follow this strategy: always enforce the SCF and the geo-opt convergence criteria simultaneously, otherwise you might end up with a structure that seems less stable despite being actually better.

Dear Weippert

Thank you for your helpful suggestion. I got it and will try it.