Thank you for asking this and bringing to discussion the validity and robustness of the DEA models when comparing with respect to outputs only. I and Gurdal Ertek faced with same challenge and were asked similar questions by reviewers in our recent project which can be accessed at: "Mind the perception gap: An integrative performance management framework for service supply chains", International Journal of Physical Distribution & Logistics Management, 2018. https://doi.org/10.1108/IJPDLM-09-2017-0302

To address your question, one need to first ensure that the selected DEA model is robust and valid.

1) Robustness: Robustness of DEA has been subject of a number of studies. Seiford and Zhu (1998) decisively report that the convex BCC model family of DEA is robust, and especially more robust than the earlier CCR model family. An important decision is the selection of the most appropriate DEA model from among many alternatives. From among a large collection of DEA models reported in the literature, in our paper, we preferred the BCC model family because of its robustness (Seiford and Zhu, 1998). In our data the values for all the outputs were non-negative, and the BCC allows for variable returns to scale. Since the input values (all equal to one) cannot be changed, the goal of a DMU is to increase the values of its outputs (the values given by respondents), hence we could justify the use of the output-oriented BCC model (BCC-O) in our study. For required calculations, we used an interactive DEA modelling software, SmartDEA Solver, which enabled us to produce DEA results in tabular format and facilitated subsequent data analysis (Akçay et al., 2012).

2) Validity: In our paper, the number of respondents evaluating each supplier was also the number of DMUs in the DEA model constructed for suppliers. A well-accepted modeling rule in DEA research is that the number of DMUs should be at least two times the number of input and outputs (Golan and Roll, 1989). In our paper, for each DEA model, the total number of input and outputs was nine (one input and eights outputs) and thus there should have been at least 2*9=18 DMUs in each DEA model. The number of DMUs was the total number of respondents evaluating each supplier (24+17=41 for Supplier 1, 7+18=25 for Supplier 2, and so on), giving (41, 25, 26, 38, 60, 30, 8, 24) DMUs for each of our models. Except for Supplier 7, where there were only eight respondents, we had the number of DMUs for each of our DEA models being at least twice the number of DMUs (18), satisfying the mentioned rule. Thus, the sample size was proven to be sufficient for the DEA models.

Another rule with respect to the minimum suggested number of DMUs is by Dyson et al. (2001). The authors recommend a total of two times the product of the number of input and outputs. In our model, we had one input and eight outputs, so according to this rule, the number of DMUs in each model should be at least 2*(1*8) = 16. The sample size in each of our models (except for Supplier 7) was sufficient according to this rule, as well.

References:

• Dawei Lu, Sobhan Asian, Gurdal Ertek, Mete Sevinc, (2018) "Mind the perception gap: An integrative performance management framework for service supply chains", International Journal of Physical Distribution & Logistics Management, https://doi.org/10.1108/IJPDLM-09-2017-0302

• Dyson, R.G., Allen, R., Camanho, A.S., Podinovski, V.V., Sarrico, C.S., and Shale, E.A., (2001), “Pitfalls and Protocols in DEA”, European Journal of Operational Research, Vol. 132, pp. 245-259.

• Golany, B. and Roll, Y. (1989), “An Application Procedure for DEA”, Omega, Vol. 17, pp. 237-250.

• Ertek, G., Akyurt, N., and Tillem, G. (2012) “DEA-based benchmarking models in supply chain management: an application-oriented literature review”, X. International Logistics and Supply Chain Congress 2012, November 8-9, Istanbul, Turkey.

• Seiford, L. M., and Zhu, J. (1998), “Sensitivity analysis of DEA models for simultaneous changes in all the data”, Journal of the Operational Research Society, Vol. 49 No. 10, pp. 1060-1071.

Weighted additive models can overcome the issue of negative values, as they are characterized by the translation invariant property. You can also get an output-oriented version of the weighted additive DEA models by setting the weights on inputs as zero.

Thanks Saeed, Sobhan, and Samah,

I am not a DEA expert but trying to help a colleague in ranking DMUs. Now, from Sohban answer, I am having 10 DMUs and 10 output variables, it seems that I need to remove some of the features to maintain robustness. Also from Samah and Sobhan, I have two suggestions to go forward: trying output-oriented weighted additive model or BCC-O!