In my master thesis, I proposed a set of Human-Computer Interaction (HCI) guidelines for inclusive design focused on users with autism. A challenging aspect of the research was the evaluation of the guidelines' effectiveness, since I coundn't find a well stablished method, tool, technique or framework to perform this task. I decided to use a pilot evaluation through a qualitative survey and then I performed a second qualitative evaluation adapting the methods Level of Evidence and Strength of Recommendations (or Strength of Evidence) applied in Healthcare papers [1-5].
Is there some robust and well stablished method in HCI to evaluate a proposal of guidelines? How to ensure the effectiveness of new recommendations?
References:
[1] BRODERICK, J. P. et al. Guidelines for the management of spontaneous intracerebral hemorrhage: a statement for healthcare professionals from a special writing group of the Stroke Council. American Heart Association. Stroke. v. 30, p. 905-915, 2005.
[2] GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ: British Medical Journal, v. 328, n. 7454, p. 1490, 2004.
[3] LOBIONDO-WOOD, G. P.; HABER, J. Nursing research: Methods and critical appraisal for evidence-based practice. St. Louis, MO: Mosby Elsevier. 2010. 7ª Ed.
[4] NKF, National Kidney Foundation. KDOQI Clinical Practice Guidelines and Clinical Practice Recommendations for Diabetes and Chronic Kidney Disease. 2007. https://www2.kidney.org/professionals/KDOQI/guideline_diabetes/appendix2.htm
[5] SIMON, S. Special guidelines for overviews and meta-analyses. 2010. http://www.pmean.com/12a/journal/meta-analysis.asp
Article Criteria For Evaluating Usability Evaluation Methods
https://www.researchgate.net/topic/Human-Computer-Interaction2
https://pdfs.semanticscholar.org/23e8/33a500352b2d8fcabb8de20c17582d7ea7a2.pdf
http://eprints.nottingham.ac.uk/35000/1/Establishing%20usability%20heuristics%20for%20heuristics%20evaluation.pdf
https://www.nngroup.com/articles/ten-usability-heuristics/
http://www.idemployee.id.tue.nl/m.m.bekker/hartson.pdf
http://www.scielo.br/pdf/jistm/v13n3/1807-1775-jistm-13-03-0363.pdf
Plz refer to our article on Heuristic Evaluation of medical interface -
https://www.researchgate.net/publication/225800519_Usability_Heuristics_and_Qualitative_Indicators_for_the_Usability_Evaluation_of_Touch_Screen_Ventilator_Systems
I believe your approach has been quite appropriate and extensive, considering how this is usually done.
The ultimate evidence of the effectiveness of guidelines is their successful application by practitioners. However this application is often ill defined. Who do you envision to be using your guidelines, and how do they go about it in practice. If you know this, you can set up an experiment simulating this practice and check whether your guidelines lead to an improvement. This is a lot easier if the intended use is heuristic evaluation than if the intended use is design. So many things influence design that is merely impossible to establish the positive impact of a certain set of design guidelines.
An alternative, as you suggest, is to compare your guidelines to generic criteria for guidelines such as understandability, comprehensiveness etc. I would say guidelines need to be decisive (thus help to distinguish good from bad alternatives), specific (it should be very clear what they apply to and what not), practical (it should be easy to apply the guidelines) and robust (they should be based on the best evidence available). These are properties that can be evaluated by experts from practice or academia, and thus form the basis for a heuristic evaluation of your heuristics.
For each of these approaches, I think the most important thing is to have a clear idea of your users of the guidelines whether they are designers or evaluators and a model of their practices.
- Badges
- Science topic
- Similar topics
- Cell Biology
- Methods