I'm not sure if it's suitable in this case, but what you can do is a heuristic evaluation, checking against known usability criteria. But if you do this type of testing you should take the role of a usability expert rather than the user. You can read more about e.g. Nielsen's criteria here: https://www.usability.gov/how-to-and-tools/methods/heuristic-evaluation.html

If this work is part of e.g. a thesis project and/or if it is going to be published as a scientific article, I would strongly suggest that you instead perform an evaluation where you let other users solve different tasks using the framework.

 Do you really mean usability in the sense of human factors, or in the sense of usable for your tool? 

A usability study with only one person, being also the developer of the study, is scientifically seen not very convincing (and as a reviewer, I would probably reject a paper with such a study).

As Jonas suggested, you should instead perform an evaluation with multiple users, to gather statistically significant conclusions. Evaluating a framework isn't an easy thing, as most usability studies are on HMI, and not on API's. Nevertheless, see attached link for a paper on Usability criteria for API's that may help in setting up your own evaluation. 

If you want to know if eclipse is the right tool for your purpose, I would start by collecting requirements that a framework has to provide, and then check if eclipse fulfils these requirements. From my experience, eclipse is quite a big framework, with a high learning curve, especially for novices. If you are not sure, if eclipse provides a certain function, you should post a specific question to one of the many eclipse forums. The experts there can usually tell you which plug-in/framework you need to solve that. 

Article What Makes APIs Difficult to Use?

Thank you very much for your responses! 

To clarify, I will be using the relatively new Sirius Modeling Framework (a plug-in of Eclipse) to create a UML-notation EER modeling tool (Extended entity relationship). The tool would be mainly used for educational purposes, to teach UML/EER to Business Engineering students.

Sirius declares to be the easiest way to make your own modeling tool. So, we would like to see whether this is actually true. That is why I was wondering if there is an acceptable measure of testing usability on one user. I would be the user testing the usability of Sirius.

An extension of the research could be that I test the new tool that I have developed in Sirius against the existing tool that is being used by the university right now. But that would not allow us to asses if Sirius actually is the easiest way to get your own modeling tool.

I hope I have made myself clear now. I am rather new to doing academic research, so I'm sorry if I am a bit unstructured. 

Thank you.

Hello Sonny,

In your research context, the promise of Sirius should be evaluated in comparison with how the existing tool has been created without Sirius.

Without Sirius, there are two main approaches :

1) Create a new tool (with the help of other frameworks)

An article about this kind of comparison exists: http://erk.fe.uni-lj.si/2014/vujovic(comparative_analysis)p.pdf

2) Adapt an existing tool (for example, by creating an UML profile in your case)

Selecting this solution depends on hwo far you are from the metamodel of the tool to extend:

- how many new concepts do you have to add ?

- how many useless UML concepts will remain in your tool (and provide confusion for the end-user)

The more your concepts are far from the UML concepts, the more Sirius will be helpful (and vice versa). 

For you study, I can also invite you to follow the basic Sirius tutorial (https://wiki.eclipse.org/Sirius/Tutorials/StarterTutorial),

And then to try to create the same modeling tool with another approach.

Best regards

Frédéric