The results in the transmission monitor can be greater than 1 or less than 0 because of the following reasons.
Simulation auto-shutoff level is not small (i.e.>1e-5)
Simulation is terminating or finishing before reaching the auto-shutoff level (increase simulation time and making it large enough to make sure that the simulation runs until auto-shutoff level).
The nanostructures have high Q resonances which need small auto-shutoff levels (i.e. 1e-7 or smaller).
The source is too close to the nanostructure and the reflection from the nanostructures is interfering with the source that can result in the abnormal outcome in the transmission monitor. (Keep the source far away from the nanostructure. i.e. 1.5-2.5 um away from the nanostructure surface).
Use of standard stretched coordinate PML with maximum PML layers to avoid reflections from the PML layers. This works well for normal incidence but if you are simulating oblique angles then using steep angle PML instead of standard one.
Sometimes using too fine mesh sizing can also be problematic.
Use of broadband simulations
From my personal experience, 1D simulations are easier to work and troubleshoot as they don't take much time.
I hope this helps but if you have some specific problems or questions. You can mention in the question thread.