Generally Barrier less reactions happens in bimolecular reactions , not in unimolecular reactions. which one you are trying ? Reaction is barrier less or not , you have to calculate activation energy from reactant to transition state. Negative activation energy is corresponding to barrier less reaction. We can not determine it from imaginary frequency. Imaginary frequency in TS just indicate that displacement vector which is connecting reactant with product. 

The TS is not a barrier (it can be related to a potential barrier, but it is not necessary). The TS is a dividing surface in the phase space of the reaction which separates the reactants and the products. In a simple elementary reaction the TS is positioned at the top of the potential barrier, but in the barrierless reactions there are only a centrifugal (dynamical) barrier, not a real potential barrier. The position of the transition state will move together with this dynamical barrier at different temperatures. Such kind of TS is called loose TS. The description of the kinetics of the barrierless reactions with statistical rate theories is not trivial due to loose TS. You can use the following methods for the treatment of barrierless reactions: RRKM, variational TST, Phase Space Theory (PST) or  Statistical Adiabatic Channel Model (SACM) methods.

and your original question "how can we exactly know if a reaction is barrier-less?"

We need to discover the potential energy surface of the reaction. You can make sure, If you can not find barriers and you did 1-2D scans on PES or IRC calculations.

I really appreciate you for explaining it very nicely. So by loose TS you mean TS which has very low imaginary frequency (below -100 cm-1) ?

According to your answer a barrierless reaction might have dynamical barrier which means that the products are not necessarily lower in energy compared to the reactants. is that right?

No, there is no "only one" imaginary freq which could be related to a barrier. There is no barrier, the PES is flat and attractive. It means you must find a complex on the PES between the reactants and products. The energy of the product is lower than than the energy of the reactants. Due to the complex you can treat these reactions with RRKM (unimolecular version of TST).  You should observe a negative temperature dependence of the reaction.

There are much worse cases than the simple barrierless reactions. A more tricky case when you find a submerges barrier (barrier with "negative energy", with smaller energy than the reactants)

The best what you can do if you do 1D or 2D scans in more orientations.

Check this review (mainly from page 31, where you can find the topic of barrierless reactions)