MODBUS-RTU is a serial protocol very common in the industrial world, the protocol is not secure and doesn't have any build-in authentication mechanism, I'm looking for research projects that try to address this problem.
The original standard for serial line encryption started out as AGA 12 and I believe it later became the foundation for IEEE 1711. Read those standards first before going any further.