First of all, you need to use a die with internal walls polished as better as possible.
Second, if you can chose powder particles shape, prefer a spherical one. For better compaction use of various populations (by sizes) of particles.
Third, try to built your rigid die optimally, i.e. too high height-to-diameter ration will be harmful from the point of view of high powder-to walls friction. Always try to make you die more "shallow".