Your data is binary form ,so analysis aplicable for binary data is to be performed.

The analysis of binary data involves goodness-of-fit tests of a sample of binary variables to a theoretical distribution.

You may also use eg. SAS software, where binary data analysis is possible Regards,

Since you only have 16 possible results for each responder the first thing I would do is make a bar chart to illustrate each of the 16 outcomes. You might be able to make some valuable conclusions by looking at the chart - especially if some of the 16 possible results are empty.

You can analysyze it once in a k x 2 table or do it pair-wise comparison in 2x2 table. Both follows chi square test for testing of significance.

Set up k x 2 table

YES | NO

x1 |

x2 |

x3 |

x4 |

For calculation, see this link: https://link.springer.com/chapter/10.1007%2F978-1-4612-5246-7_9

In the alternative, you can do 2 x 2 table for two pairs comparison. Possible combinations are: (n(n+1) / 2) = 4(3)/2 = 6 possible pairs.

(x1, x2)

(x1, x3)

(x1, x4)

(x2, x3)

(x2, x4)

(x3, x4)

The 2 x 2 table set up would look like:

-------------------------------------------

Yes | No

-------------------------------------------

x1 a b

x2 c d

... (repeat pairs)

... (repeat pairs)

----------------------------------------------

Yes | No

x3 a b

x4 c d

---------------------------------------------

Then set up 2 x 2 table and test the significance under chi square distribution:

X2 = [(n - 1) (ad - bc)2 ] / sqrt[(a+b) (a + c) (b + d) (c + d) ]

This approach may be more informative than the k x 2 approach since you are examining pair-by-pair.