As far as I know, there is no open source that has capabilities equivalent to Optiwave OptiSystem, Rsoft OptSim, or VPIphotonics VPItransmissionMaker. You can use GNU Octave or MATLAB to build simulation program where many sub-programs are available on the internet.