Hi Pieter. That is the question "million-dollar", right? The most common method I've seen used is by examination (test) but always cover the theoretical part of it. On the other hand, I've also been using experiential workshops where you put them in situations you want to identify (attacks)

On the side of the Help Desk, you can help make comparative studies of the values of certain metrics before and after training. What metric? For example:

Number of incidents reported before and after awarness program.

Greeting

Dear Pieter,

Take a read at the following sources for more ideas and tools:

https://www.researchgate.net/profile/Bilal_Khan13/publication/267805604_Effectiveness_of_information_security_awareness_methods_based_on_psychological_theories/links/00b7d535f5e11a9872000000.pdf

https://www.researchgate.net/profile/Hennie_Kruger/publication/222422461_A_prototype_for_assessing_information_security_awareness/links/00b4951840cd905812000000.pdf

http://icsa.cs.up.ac.za/issa/2005/Proceedings/Full/018_Article.pdf

https://su-plus.strathmore.edu/bitstream/handle/11071/3537/A%20Model%20to%20measure%20information%20security%20awareness%20level%20in%20an%20organization.pdf?sequence=1&isAllowed=y

Good luck,

Simon Cleveland

Article Effectiveness of information security awareness methods base...

Article A prototype for assessing information security awareness

I think you should get ethical hackers to setup a trap and see if the respondents fall to the bait.

If briefly - questionaries, tests, periodical certification, skills and abilities excercises 

while security effectiveness can be an elusive object to measure, there are highly effective ways of determining it, short of recording incidents of catastrophic failure. The key dimensions of information security effectiveness:

Uptime: The ability to withstand cyber attacks and avoid costly business disruption.

Compliance: The ability to achieve compliance with all applicable regulations and laws.

Threat containment: The ability to prevent or quickly detect external security threats such as cybercrime, social engineering or malicious attacks.

Cost efficiency: The ability to manage investments in information security and data protection in a competent (non-wasteful) manner.

Data breach prevention: The ability to prevent or quickly detect internal security threats such as the negligent or incompetent insider.

Policy enforcement: The ability to monitor and strictly enforce compliance with internal policies, procedures and other security requirements.

http://www.csoonline.com/article/2134334/metrics-budgets/measuring-the-effectiveness-of-your-security-awareness-program.html

Awareness might be an interesting measurement, however it only takes one point of exposure to undermine the security for a system and/or compromise risk across an organization. Consider a broader framework that focuses on Universal Threat Management (UTM), where Awareness and Education are small pieces to a much larger capability to minimize exposure. 

One question I typically ask my clients to answer is how well they can measure risk and how well do they understand their current state of exposure.  Most can't answer this with any level of clarity and that is the point where the awareness can be most helpful.

If it's awareness of the staff, then email a sample phishing attack and see how many bite. This will give you an idea as to how aware they are. If it's awareness of the ICT staff, then fake attacks will also work but you can review process and procedure to ensure security is covered appropriately. For example, if a new system is being sent live, then does require a mandatory pen test? if there is no mention in your process, then you probably have a problem!

I think that would depend on context.  Is your requirement a practical and on-going one?  I have just reviewed a product line from Cisco Meraki that is (surprisingly?) capable of providing a great deal of information about devices attaching to the monitored network, as well as how users behave on that network.  Their products can even track devices like mobile phones that have not necessarily connected as they enter and leave the WiFi coverage area, giving quite a bit of intelligence on various activities.  Powerful if applied with imagination!  I suspect this reply is not quite on topic, but I hope it helps?  I expect to trial their systems in the "wild" soon.

why some people appear to think that *security awareness* is causally related to *risk threat* is baffling to me. As far as I know - people may be *aware* but it might not matter if they *do not care*. As in any risk-taking activity. Also even if people *do care* (what does this mean anyway?) - there is something called *priority*. How about when people can choose to either *follow the rules* OR *doing a good job*. Does that matter? Are you measuring awareness OR behaviour? They are not synonymous!

I agree, Peter. Awareness and behaviour are two quite different things.  I knew I could be slightly off-topic with my reply above, but from a practical and pragmatic point of view, monitoring behaviour can be the start of understanding the risk that users (and certain ones in particular) may pose to a system. That done, you have the ability to respond within the configuration of the affected systems and / or modify behaviour through training and further monitoring. Anyway, simply monitoring the behaviour of users can provide a great deal of intelligence on how any system is used, with varying benefits.

https://www.researchgate.net/publication/305815882_Challenges_for_Atomix_System_Ltd_The_ITES_industry_in_Bangladesh

Article Challenges for Atomix System Ltd.: The ITES industry in Bangladesh

Dear Pieter,

my recent publication on Information security awareness programs deals with banks measures and methods. Please have a look at:

https://www.researchgate.net/publication/308918183_From_Information_Security_Awareness_to_Reasoned_Compliant_Action_Analyzing_Information_Security_Policy_Compliance_in_a_Large_Banking_Organization

Soon we will publish also qualitative research on information security awareness programs, in which we discuss measuring information security awareness. The range of methods reaches from surveys (quizzes) to fake phone calls.

I keep you updated!

Stefan

Article From Information Security Awareness to Reasoned Compliant Ac...

Guys please check out my latest article on this topic Article Prevention is better than cure! Designing information securi...

I did a systematic literature research and ran some interviews with small and medium-sized companies to retrieve the state of the art about measuring information security awareness.

You can have a look at my recently published paper:

https://www.researchgate.net/publication/338630876_About_the_Measuring_of_Information_Security_Awareness_A_Systematic_Literature_Review