You need first to well define the selfishness metric and study its impact on your new algorithm or protocol. The selfishness behavior may vary from full to partial (See attached) and it could be a probabilistic metric.
Please refer to these attached research papers. Hope they help to answer your question.