With a mini drill (Dremel or other brand - near-to-perfect ), a drill press and cemented carbide drills 0.5 mm should be achievable.
It is not exactly "with hand" (due to the requirement for a small drill press) but it is still low-cost.
In addition I'd recommend to use collets to achieve the smallest drill eccentricity possible. Otherwise you risk to break the drill which is sensible vs. radial forces.
Any basic cheap drill stand can use 0.5mm drill bits for vias. You can do it with a hand (by that I mean rechargeable/cordless drill), but you have to be steady. They can snap very easily if you don't go in straight. Something like 1mm is fine with a hand drill, but that might be too big for you.