I did a conference paper that may be relevant to your question.  It defines criteria for evaluation of a Computer Mediated Communication platform, and then evaluates whether certain social media platforms can meet the criteria.  

It may not be perfectly focused to your need, but there may be some thing you c an use.

Conference Paper Selecting CMC Systems for EFL Instruction: Can Standard Soci...

Dear Sizwe,

In addition to Michael's recommendation, try the following two sources for other ideas:

http://sighci.org/Research/ijhcs_03/Yi.pdf

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.231.8887&rep=rep1&type=pdf

Good luck,

Simon Cleveland

Thanks for your contribution. it certainly helped a lot.

Hi Sizwe,

I know you specified "IS theory", but as LMSs are in the area of e-learning, I would say the "effectiveness" should take under consideration the learning aspect. Naturally, from that point of view, some established literature in the education and e-learning domain is useful. Here's one book I've found useful:

Phillips et al. (2012) Evaluating e-Learning: Guiding Research and Practice

https://www.routledge.com/products/9780415881944

Thanks Marko

Sizwe

I'm also like Marco thinking that the evaluation of an LMS should maybe consider the learner side as opposed to the user (IS) side.  Attached is a book chapter I co-authored

Moore, J. L., Dickson-Deane, C., & Liu, M. (2014). Designing CMS courses from a pedagogial usability perspective. In Perspectives in Instructional Technology and Distance Education : Research on Course Management Systems in Higher Education. (p. 169). Information Age Publishing.

Thanks a lot Camille.

Hi everyone. This is an interesting and valid question. Can we use ideas from the world of IS and IT, to evaluate a learning management system? I have taught information systems, and systems analysis and design for a number of years. I have also used a variety of different learning systems. There are some general frameworks or tools that you can use to evaluate any system in general, including a learning management system. First, I remember an article from 2012 by Aggarwal et. al (from Maryland, USA) that you may find interesting. They applied system development life cycle and requirements analysis (page 288) as a way of comparing different candidate systems (the link is below). Next, the PIECES framework is used for analyzing problems within the current system; you may be able to use that kind of framework for evaluation of learning management systems (one link is below). There is also a nice article by Palmius (from Sweden) on evaluating information systems in general, which you can get some ideas from (page 4 in particular). The link for that one is also at the bottom of this message. Good luck with your research, 

Dr. E 

http://www.jite.org/documents/Vol11/JITEv11IIPp283-300Aggarwal1108.pdf

http://www.cs.toronto.edu/~sme/CSC340F/readings/PIECES.html

http://www.palmius.com/joel/text/IRIS-30-final.pdf

Thanks Stefan

Embedding design support within the online environment of a learning management system 

Implementation Process

• Initial analysis

• Design-development-evaluation cycle

• Final analysis

http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1093&context=edupapers

Thank you