Georgios,

I'll do my best. Your notation is, I think, making your question harder to understand than it needs to be.

Q01 and Q02: Unless you have a very large sample size, many of your 13 difference-values will be unpopulated, especially toward the extremes. If I understand correctly, your interest is in whether the distributions are different between pre- and post-, in which case I think a Mann-Whitney U test would do the job. This also addresses Q03 and Q04.

Q05: A U test would also address whether responses are higher on one item than on another.

Q06: This one gets much trickier. What might work is a categorical-data SEM. The pre- factor is defined as indicated by frequency and quality as is the post-. In order for this to be identified, you'd need the intercepts and thresholds for each indicator to be equated between pre- and post-, and that would not be testable. Then your question is the difference between factor means.

Q07: There is literature on "retrospective pre-testing." It's certainly a limitation in an intervention study, but there are cases where it can have advantages.

Q08: I think you're asking about an interaction or effect modification from the demographic factors on the pre/post relation. I'm not sufficiently expert in ordinal-data methods to have a suggestion off-hand. There are extensions of chi-square that do it, but that would treat the responses as nominal and lose the advantage of ordinality.

Q09: R can almost certainly handle these things, but there's a learning curve if you're not familiar with object-oriented programming. Others may have more concrete suggestions.

Pat

Dear Dr Patrick S Malone

I wish to thank you off hand for your generous" Boxing Day" response to my "Christmas Day" wish!

As you correctly state, my notation does indeed make my questions harder to understand.

Nevertheless, I maintained this notation in case more "refined" observations/comments would eventually have to be made.

Furthermore, having only some rudiments of statistical knowledge, especially as concerns surveys, I apologize once again for the lack of any specialised terminology in the description of my project.

The study in question could be qualified as a

RETROSPECTIVE pre-test survey,

or as a

RETROSPECTIVE

INTERVENTIONAL (Experimental) and

Sequential (Self-controlled) study.

Your comments, Patrick, are most useful, and I would really like to analyze a few, if I may, to make sure that I, in turn, have well understood your thought process.

Concerning SAMPLE SIZE:

I am hoping for a large sample size (N = 300), since this will be an internet survey. This may perhaps allow for

ORDINAL => NOMINAL transformation,

but, for the time being, I'd like to think this out using only ORDINAL values.

Q01 to Q04:

PSM:

If "interest is in whether the distributions are different between pre- and post-".... "I think a Mann-Whitney U test would do the job".

GKM

Yes, my analysis will study ISOLATED ITEMS

and their INDIVIDUAL pre vs post EVALUATION

Using the wikipedia definitions

(http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)

I seem to be in difficulty understanding how to use this test adequately, and thought out two "methods".

Taking "Item ONE" as an example, for "N" candidates:

METHOD 1

if my understanding is correct, the rank-order test will require that I

group ALL PRE (Sum of Rank ordered PRE) ="sample 1"

and then ALL POST (Sum of Rank ordered POST) =sample 2"

so as to obtain "U"

Problem: this method does not differentiate INDIVIDUAL candidates, and their "evolution" (either positive, neutral or even negative), but rather, seems to "pool" all results.

Method 2

Can one use ORDINAL values

to FIRST obtain a POST vs PRE DIFFERENCE

for EACH candidate

and only then do a

Mann-Whitney U test?

PSM:

Q06: ....What might work is a categorical-data SEM.

GKM

I knew nothing of the method, found the following article

Skrondal, A. and Rabe-Hesketh S. "Structural Equation Modeling: Categorical Variables"

(http://www.gllamm.org/SEMcat.pdf)

and will definitely try using your advice.

PSM:

Q07: There is literature on "retrospective pre-testing."

GKM

The key words you provide were most ENLIGHTENING!

Indeed, a quick search lead to articles that more accurately express my reasons for choosing such a study method.

Q08

Concerning encoding various demographic categories (factors) and determining any interaction on the pre/post relation (of each item).

You suggest a possible "ordinal to nominal" conversion and "extensions of chi-square" with inevitable loss in the advantage of ordinality.

That will have to be looked into.

Q09:

You suggest "R " or welcome other suggestions.

OK. That will have to be looked into, as well.

Final comment/question:

All in all, can one consider my survey methodology as something that can be "realised" or should this project be considered "difficult, statistically unsound" (in a few words, "to be rejected")?

Once again, Patrick, I am most obliged for your rapid, generous and very enlightening response.

Georgios

motto

Πράξιν ἀγαθήν, εὖ ποιείτω ταύτην.

It suffices not to do good (to others), but to do it well

Il ne suffit pas de faire le bien, mais le bien faire.

Georgios,

I am hoping for a large sample size (N = 300), since this will be an internet survey. This may perhaps allow for

ORDINAL => NOMINAL transformation,

but, for the time being, I'd like to think this out using only ORDINAL values.

PSM: I can virtually guarantee will not be large enough to populate the 13 possible values of your subtraction transformation, especially for large declines.

Q01 to Q04:

Taking "Item ONE" as an example, for "N" candidates:

METHOD 1

if my understanding is correct, the rank-order test will require that I

group ALL PRE (Sum of Rank ordered PRE) ="sample 1"

and then ALL POST (Sum of Rank ordered POST) =sample 2"

so as to obtain "U"

Problem: this method does not differentiate INDIVIDUAL candidates, and their "evolution" (either positive, neutral or even negative), but rather, seems to "pool" all results.

PSM Yes. The U test measures whether the distribution of responses (across respondents) changes between pre and post. If you're interested in each respondent's change, then there is no summary statistic or test that can help you -- you'd have a very large qualitative study.

All in all, can one consider my survey methodology as something that can be "realised" or should this project be considered "difficult, statistically unsound" (in a few words, "to be rejected")?

PSM: It depends on whether I've understood the goal correctly -- to evaluate the effect of the intervention, I would group across respondents (with the caveat that you have no control group). If you're asking about examining change individually for the 300 respondents, I can only say that I'm not a qualitative researcher and my answers won't help much.

Pat

Dear Patrick,

Your answers are clear, and really help me to better understanding what I caneventually  do.

++++

GKM : Sample size = 300

PSM: .... " virtually guarantee will not be large enough to populate the 13 possible values of your subtraction transformation, especially for large declines"

GKM: All right.  Thank you for pointing this out: I'll see what can be done.

++++

Distribution of responses (across respondents) using the  Mann-Whitney U test

vs.

a very large qualitative study "per respondent"  (no summary statistic)

GKM: Thank you for clearly differentiating and for clarifying, as well,  the terminology and methodology concerning this issue. 

++++++++++++

PSM

.... to evaluate the effect of the intervention, I would group across respondents (with the caveat that you have no control group).

GKM

You have indeed understood correctly, and have once again answered clearly.

PROJECTION:

My feeling is that I should wait, as well, for some contribution from "QUALITATIVE" study specialists.

According to my humble understanding of statistics, I'd say that

-- if the sample size should be large enough (would N= 1000 do?), I could eventually "group across repsondents"

whereas,

-- if the sample size should turn out  to be small,

perhaps I can then create some

"post minus pre" value per item per respondent,

and subsequently  categorize all respondents into histograms per item

so as to content myself with a descriptive analysis.

I forgot to mention that all contributions will be officially acknowledged in my final presentation, at least in terms of contributor names, should the ideas not be used..

Once again, Patrick, I am very much obliged, and extend to you my warmest wishes for this Season's festivities.

Georgios

You might have better luck finding qualitative expertise with a much shorter question in groups targeted to qualitative researchers.

Good luck!

Once again, thank you very much Dr. MALONE.

The following message was posted in hope of getting some additional advice, to complement yours.

(https://www.researchgate.net/post/QUALITATIVE_STATISTICS-RETROSPECTIVE_pre-test_survey_suggestions_for_qualitative_comparisons_of_results)

QUALITATIVE STATISTICS - RETROSPECTIVE pre-test survey: suggestions for qualitative comparisons of results?

QUALITATIVE STATISTICS: In a RETROSPECTIVE pre-test survey,used to evaluate the effect of an intervention (an information pamphlet), is there some standard, qualitative method one may use to present, analyse or even correlate the various pre/post changes to other demographic variables as per

sampled candidate? In other words, can one avoid POOLING the PRE vs POST results (and thereby avoid using quantitative statistics)?

Season's greetings, Dr Yarnold.

Thank you very much for your response!

I browsed through a number of your articles and also found the first chapter of the 2005 book you co-authored on OPTIMAL DATA ANALYSIS (ODA).

Looking through the ressources and bibliography pages of your website, I discovered that you have worked extensively on a "novel" statistical approach called the ODA Paradigm, for which you have provided examples of at least one of the programs you provide.

Apparently, you provide various statistical software for this novel approach, of which the

UniODA®, MegaODA®, CTA®, as well as the

(2004? DOS?) software provided in the 2005 book you co-authored

Optimal Data Analysis: A Guidebook With Software for Windows

as well as an even newer one

Data Informeter®!.

Thanks the generous contributions of various readers, especially in the present forum, I can eventually state that my project ideas and possible methods involve the following survey particularities:

Part I - DEMOGRAPHICS (multivariable, categorial)

Part II - EMPATHY ASSESSMENT using the DAVIS' Interpersonnal Reactivity Index (IRI) questionnaire) (multivariable, categorial [four subgroup score])

Part III - "Retrospective pre-testing" of my ACRONYM PROTOCOL ("for developping empathic communication")

ie, a retrospective survey type Likert, involving

== MANY items whereby

== each item is evaluated according to

====== THREE ordinal categories, the where the first two (frequency and quality) will be used to compare "PRE/POST" autoevaluation scores

The main questions will involve

1) search for an interventional effect

POST-PRE difference EACH ITEM Likert

according to at least two variables (frequency and quality)

==

2) search for correlations among variables

Mixed == (categorial, ordinal and interval) DEMOGRAPHICS (Part I)

or

Nominal (EMPATHY ASSESSMENT scores) (Part II)

vs.

Ordinal: (frequency and quality) POST-PRE difference of EACH ITEM Likert (Part III)

Concerning question 1

I have also understood that descriptive statistics will turn out to be quite cumbersome, and that I would be better off using non parametric statistics,

after "pooling" the PRE an POST effects on each ITEM,

while treating each dependent variable (Frequency and QUALITY evaluation) SEPARATELY

Suggestion 1a: Mann-Whitney U

Should I wish, however, to "combine" the two variables, some sort of other method has been suggested

Suggestion 1b: to "connect" variables "Frequency" and "Quality", use "categorical-data SEM (" Structural Equation Modeling [of Categorical Variables] ").

Concerning question 2

Suggestion 2a: use "extensions" of the statistical type ordinary least squares (OLS)

Suggestion 2b: use novel method, "optimal data analysis" (or ODA)

Now, in the first chapter of the book you co-authored, one finds that

= statistical methods used until now consist of

either

== the ordinary least squares (OLS) paradigm, which maximizes some variance ratio

or

== the maximum likelihood (ML) paradigm, which maximizes the value of the likelihood function for a given sample;

and that a novel approach method

== OPTIMAL DATA ANALYSIS (ODA), maximizes the accuracy of a given model.

Unfortunately, there are no simple examples/diagrams etc. to make all this paradigm comparison "easy" for statistics "dummies" such as myself

(in spite of the fact that I have passed all my university "basic statistics" courses without any serious problem).

Therefore, I'd like to ask a direct question:

If I buy the book

and

if I buy the CTA® software ($ 249 for students)

1) Will I manage to get the job done?

2) Will I finally understand what OPTIMAL DATA ANALYSIS (ODA) does,

in comparison to ordinary least squares (OLS) and maximum likelihood (ML) paradigms?

and

3) Will I be able to eventually prove that it was worth using for my project?

In other words

- Will a financial investment (circa $300 [book and software], accompanied by a one month learning phase, allow me to climb the "learning curve" sufficiently, so as to finish my work within 2 months after I've collected my data?

Once again, I wish to thank you for providing generous literature to the novel statistical paradigm you work on, and for trying to make it easier to understand for those who are not specialist.

Sincerely,

Georgios

Dear  Paul,

I wish to thank you for your very informative response, and for the reminder that any "human plans make God (or Fate, for others) laugh"!

I also sincerely wish you good health, as should do all those you generously help out, so that you may continue persevering in what seems to be an important breakthrough, of which I may eventually be able to understand some basics through your book.

With all my gratitude,

Georgios

Dear Paul,

I am very much humbled not only by the various hardships and suffering you have gone through lately, but by the fact that your human qualities allow you to publicly speak both, your piece and peace, while actively contributing with trying to add your "little" stone ("megalithic", as fas as I'm concerned) to humanity and a "better" world.

I quickly read through the Fibromyalgia paper ("Ascertaining an Individual Patient’s Symptom Dominance Hierarchy) as soon as you had sent the link, and found that ODA does a lot of data "clarifying" and "connecting" (and, from what I gather, I'll understand all this after reading the book you co-authored), that habitual statistics do not perform (at least, as "easily").

A great part of medicine is evidence-based knowledge and ability to use it ... yet some consider that,  in spite of progress, an even greater part of it  is still "flair" ("scent" or "instinct").

Overall, I honestly feel that you are proposing something serious, and decided to try it out, not only because of the CONCEPT, but because of its devouted promoter.

There are no words to comfort one for losses such as yours in such a short interval, but may the memories of your beloved mother, brother and three dear friends remain eternal (mathematical point of view), and may their souls rest in peace (theo-metaphysical point of view, for those who do not reject it).

Once again, "thumb" s up for your help, as painful as it may be... if I were to be asked, I'd vote straight A's for your final exams....

With gratitude,

Georgios

Dear Paul

I wish to thank you publicly for keeping your word and living up to your promises!

I also with to thank RESEARCHGATE and all those who provided ideas (eg. Patrick S Malone): you will be acknowledged in my final work.

Paul, you were kind enough to order the book from your publisher and send it out to me,here in France,  along with the various programs of ODA, all for 110 Euros.

I installed the ODA program, which works perfectly well on Windows 8.1 (it was originally designed for Win 98 to XP)

and read through your book

(http://odajournal.com/)

while working through a few initial exercises.

The fundamentals of the method are "revolutionary" and  ingenious!

One has to work through all the exercises, which, unfortunately, I have not yet had the time to do.

Yet, the book is full of examples, applications, results and interpretations,  and demonstrates how classical statistics (ordinary least squares paradigm) can eventually be "left behind" when it comes down to comparing many variables.

Basically, and please correct me if I'm wrong, the ODA method takes the data, applies a number of permutations, and determines the probability that such data originated "by chance", for any type of distribution (continuous or discreet).

It "crashes out" a "Classification Performance Index", which is a list of "parameters" with % values for each "item", which allow one to discuss results.

Since most of the examples provided use the same data to obtain results from classical statistics (ordinary least squares paradigm) as well, one can compare them to those of ODA, and easily observe some extended possibilities that ODA can provide.

After speaking with you over the telephone, I got a clearer idea of how ODA works, and put my project down on paper.

One part will use the same data for "classical" and "ODA" statistics: this will eventually provide "just another example" in your book!

The other part will use ODA alone, to "decant" those variables that may  somehow be "connected" (i.e. determine if there isn't some sort of "conditional probability" involved), which I find hard if not impossible to obtain using classical statistics methods.

In spite of having studied "Statistics for mathematicians", "Biostatistics for biologists", "Medical statistics" and the list can go on, statistics remain quite an "obnubilous" territory in my mind.

Even if I probably will not understand many of the mysteries of  "classical statistics" and "ODA", the  examples and discussions provided in your book helped me pick out EXACTLY the type of ODA analysis that can be used.

I wish to thank you once again for all the work you, Paul R. YARNOLD and your collaborator, Robert C. SOLTYSIK have done to provide us all with this precious tool, and you in particular, Paul, for having given me ideas, as well as having offered to continue helping me so that my project will become reality: yes, empathic communication can be taught (in France just as anywhere else), just as much as ODA can!

Respectfully and most obliged,

Georgios

PS:

Hopefully you'll someday post some sort of  "ABCs for ODA" on youtube

- I'm sure it will get many views!