The electrophoresis conditions seem right, as shown with the molecular weight markers, only the lanes corresponding to the samples appear poorly defined. What kind of sample did you loaded? It could be the sample may contain traces of lipids?
Purified bacteriophage dissolved in DW was loaded.
sample preparation: mix 10ul of sample + 2.5 ul 4X protein loading buffer(contain b-ME) and then boil the sample at 90℃ for 10 min to denaturing proteins.
I see two distinct but very faint bands in your samples. The 'weed' at the bottom of the gel has the distortion fairly typical of the dye/salt front of the sample. Not knowing the MW of the lowest MW standard you are using, it is hard to say for certain, but I suspect that is what is going on. Also, you said you are running the gel for 5 hours total on ice? I am not aware of how that is done, and I would expect that could be a serious problem. Passing a current through a solution heats it, which would make maintaining 0oC difficult.
Since you used a tricine gel, I expect your protein/polypeptide was very small?
What was the MW of the protein? If it corresponds to the bands at the bottom of the lanes, don't worry about the ragged edge - it worked and it's quite clean (>90% by rough estimate)
The protein ladder markers seem to be ok. You have much diluted the 4x protein loading buffer. Decrease the sample volume to keep the concentration of the sample buffer 1x in solution. The bottom of the ladder should be 1 to 2 kDa roughly, thus do not worry to see dye front and abnormal view since the sample has completed its migration under the lowest MW band. I guess you could not see the band of interest, what was the expected MW of the peptide/protein, and did you perform fixation after electrophoresis? In some cases, it is needed to fixate your protein using glutaraldehyde fixation before staining and imaging. Keeping the temperature relatively cold is an important consideration of course but your ladders gave symmetrical bands and there is no smiling effect which is an apparent symptom of the increasing temperature. By the way, I am also struggling with a similar problem and I am wondering if the sample buffer composition of the ladder and sample buffer composition of your protein is similar?
That is a very small protein to run on an SDS gel. You might want to increase your acrylamide concentration. What concentration of acrylamide are you using? It's been so long since I have run gels, I don't remember the correlation between acrylamide concentration and size separation, but it definitely does have an effect.