Hello Zheng,

I think that the right questionnaire selection depends on your goals and the amount of effort that you want to invest, and of course the system that you're trying to assess.

There are several available tools:

• System usability scale - http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
• Post-Study Usability Questionnaire - http://michaelyeap.blogspot.com/2009/10/oct-9-post-study-system-usability.html
• Questionnaire for User Interaction Satisfaction
• The Usability Metric for User Experience - http://www.sciencedirect.com/science/article/pii/S095354381000038X

If you want to publish your results, I suggest that you review papers from the conference or journal to see which one is used the most.

On other topic, I agree with Ather Nawaz, if you have recordings of transcripts of your evaluations, you could categorize the comments' polarity and establish your metrics based on that review.

Best of luck.

Hi Zheng, 

It will be difficult to use a quantitate method for post-study system usability. If you have qualitative data, for pre-post study, perhaps you can do the sentiment analysis. 

Hello Zheng,

I think that the right questionnaire selection depends on your goals and the amount of effort that you want to invest, and of course the system that you're trying to assess.

There are several available tools:

• System usability scale - http://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html
• Post-Study Usability Questionnaire - http://michaelyeap.blogspot.com/2009/10/oct-9-post-study-system-usability.html
• Questionnaire for User Interaction Satisfaction
• The Usability Metric for User Experience - http://www.sciencedirect.com/science/article/pii/S095354381000038X

If you want to publish your results, I suggest that you review papers from the conference or journal to see which one is used the most.

On other topic, I agree with Ather Nawaz, if you have recordings of transcripts of your evaluations, you could categorize the comments' polarity and establish your metrics based on that review.

Best of luck.

For the comparison part of your question, usually there are no standards to compare evaluations because the protocol used is not published. You could find differences in applications of the same questionnaire and they would probably be due to the application protocol or users.

Each system have different usability requirements and you should compare the results with such requirements (i.e., justify the requirements and assess that the system complies)

Dear Zheng, throughout this semester, I have done this same question to myself. About the aspect of what post-study questionnaire is really good? the answers I found were the PSSUQ, UMUX and the SUS, but the important point of this discussion, I think that is to identify if it provides one appropriate metric and reliability of data, and if there is a psychometric approach.

I suggest you assess whether the questionnaire is appropriate for your study, for this you can use some questions addressed by McNamara (2013).

McNamara, N., 2013. The psychometric approach to user satisfaction measurement. Interacting with Computers, 25(4), p.294-298.

I used to make usability evaluations based primarily on task taxonomy and with that base i would design a task by task questioonaire. To me the major lesson is to make sure to use some convergent data such as, for example, task recall a week later, etc. The idea is to have alternative ways to test the same usability aspect and then comparing one against another.  The main disadvantage of this approach is you have to build it for each device or application considered.

Using a satisfaction questionnaire to estimate usability is all very well, but beware of the well-known dissociation of satisfaction results from performance outcomes (speed, accuracy).  Participants tend to give higher satisfaction ratings than would be consistent with their performance.  Usability is not just a matter of user satisfaction.  You will be likely to miss usability issues if you do not collect data on participant performance.

I agree with Elizabeth. It is important to analyze usage data (performance, interactions, etc) along with usability survey data in order to get more reliable results.

From my own experience, usability questionnaires are more useful if you are testing an application or technological artifact that is already developed; before you should develop a scenarios guide for the most critical tasks. You can use use eye tracking systems (if they are available) to help you identify where people look at most and if it corresponds to a rapid identification of visual affordances for tasks on the text, or if they are looking for some fancy images or graphics that lower they performance. The questionnaire (of course all the cited examples are valid for general questions about usability) should address the most critical tasks you have selected before and put in the scenarios guide. From observation and passing the questionnaire just after the test, you will see that satisfaction is strongly connected to your observation data. If interaction is good and participants go easy on doing tasks the satisfaction is high. The opposite is also true.

We occasionally use http://attrakdiff.de in our studies.

Based on: Hassenzahl, M. (2006).

Hedonic, emotional and experiential perspectives on product quality

In: Ghaoui, C. (Hrsg.), Encyclopedia of Human Computer Interaction, S. 266-272

and some other publications. It also offers various measurements based on many questions.

The interesting thing is that you can do a comparison between two applications within the system they provide, increasing the comparability.

As every system and every study is different a comparison with existing software within a study is essential though. That way you can provide the same setup for a fair comparison.

1. How to count PSSUQ ?

2. What a formula ?

3. Does pssuq have the same formula as SUS?