It depends on which version of TRNSYS you use. If you use the version 17 or higher, so its OK and you can use GenOpt.
Also, you can use EnergyPlus software, which is opensource, and it has a great potential to use the MATLAB optimization tool. In this case, you need a script to link the MATLAB and EnergyPlus.