Speaking only to the use of a Dounce homogenizer, these typically come with two glass pestles of different widths. One pestle fits tightly within the shaft of the dounce for maximum friction and cell disruption, while the other pestle has a looser fit and is ideal for generating a homogenous sample. If the protocol you are following doesn't stipulate which one, then you'll have to determine that.
What degree of lysis are you looking for? If you want the nuclear envelope to stay intact, then using Dounce is too harsh of a condition; we use Dounce homogenizer only when working with tissues; when working with cells, gentle pipetting with a precut tip would suffice. The detergents in your lysis buffer would also contribute to sufficient lysis.
May I just ask, when you used the dounce homogeniser how did you remove the lysate from the homogeniser? Did you decant it or did you use a needle and syringe to do so?