Need not be. Please note that the value of frequency factor also plays an important role in deciding the value of activation energy. However many times higher temperature peaks has higher E value in some materials. You can also see results of CGCD performed by various researchers where you will find all possibility.
Shape of the TL Peaks of phosphor depends up on the activation energy and frequency factor, both are characteristics of the phosphor in addition to the rate of heating. But important parameter is the activation energy. So different peaks with different shape are being seen at different temperature of phosphor when heated at constant rate of heating. You can ref many books on Thermoluminescence of soild. One of the ref book is Cambridge Soild sate Science Series - Thermo luminescence of Solids by S.W.S Mckeever for further information.
for most of the cases : if heating rate and frequency factor is consider to be constant than activation energy for low temperature peak should be small as compare to high temperature peaks.