for example, participants selected user-interface design preferences for a particular system.
Knowing that the conditions of the experiment were the same for all participants.
Selecting preferences for an interface is really just a survey - so not an experiment.
However, if you get the participants to all do an activity using the interface to come up with an answer - or complete a task - this is an experiment. For example, we wanted to see if the interface we designed enabled taxpayers to answer key questions about lodgement of their Business Activity Statement (BAS) and then see if they could successfully complete the lodgement without assistance. We then wanted to see how generalisable this was to the entire population.
The design itself had gone through a UCD process and prototyping. So this experiment was to establish a baseline and then see when changes were made to the interface in the future, whether they improved or diminished the effectiveness of the interface. We used a group of about 50 people. This was conducted in controlled conditions.
We tracked the time, the steps, where people got lost and whether they successfully located the necessary information. At the end participants were asked to answer comprehension questions (which determined understanding) and then also satisfaction questions. From this we could determine a "lostness" score (ie how lost did people get?) and a usability rating. This same task (as well as others ) are repeated every 6 months - and the usability of the tool is tracked.
This technique is quantitative usability testing. If you are interested in this type of thing - see Measuring the user experience, Tom Tullis and Bill Albert or Quantifying the user experience by Jeff Sauro and James Lewis.
Hope that helps!
If you compare persons in the same population/group to each other across conditions, it is called within subject design: A within-subjects design is a type of experimental design in which all participants are exposed to every treatment or condition. The term treatment is used to describe the different levels of the independent variable. In other words, all of the subjects in the study are treated with the critical variable in question."
NB be aware of carryover effect here - try to minimize through experiment design e.g. alternating which treatment is given first and second etc.
The alternative to within subject is between-subject design, were one group is given one treatment and another group another, and then compare. For ex some get new medicine versus existing medicine versus placebo. Good luck 😊
A "within subjects" design has one group of subjects that participates in all the conditions. The data analysis compares the data of each subject to within his/her own responses. In effect, each subject serves as their own control. See "The Design of Experiments" by Roland Fisher.