You're asking a question that is a profession in itself for many people. It's like asking automobile mechanics how they fix cars.
First, look at the state machines for the protocol. Look for ways to put the protocol in strange states. Observe precisely how the code gets in and out of each state and look for ways to screw up that state information.
Second, Fuzz the protocol. People write books about fuzzing techniques alone. So I'm not going to get in to details here.
Third, if applicable, analyze the cryptography. Look for weak parts such as poorly randomized nonce, weak hash methods, and other such things.
Fourth, analyze the parser of the protocol and confirm that it is stable in all conditions. If the parser crashes, whatever authentication you have won't matter much.
Those are things I would do. Note that none of these efforts are trivial tasks.
Also please note that I have experience from having been on the DNP3 SCADA protocol committee for over ten years, during which we reviewed the authentication features. From personal experience, I can tell you this is not a simple task.
This depends on what you want to analyze and give priority first as security parameter. Although, you can have a look at my paper for the same Article IoT-Chain: Security of things for Pervasive, Sustainable and...
Security testing has recently received much attention in the research and especially in industrial areas. Its aim is to increase the user’s confidence in the system by ensuring its conformance to security requirements.
Two main goals for security testing are considered:
Functional security testing, which ensures that the security properties and functions are implemented correctly
Vulnerability testing, which aims to identify and discover potential vulnerabilities based on risk and threat analysis.
In order to satisfy the authentication model to secure IoT, namely, mutual authentication, perfect forward secrecy, anonymity, and un-traceability, the authentication protocols use both cryptosystems and non-cryptosystems countermeasures. Formal security verification techniques used in authentication protocols for the IoT.
Dear Dr. Mohammed El-hajj Thanks a lot for your detailed answer i really appreciate it and considered, i will try to apply it and use this technique in the testing result.
i wish if we can communicate to take advantage of your experience in the field IoT Security. Thanks in advance.