I don't have much knowledge on Muslim settlements but chaos and fractal theories can be a way of understanding. Look into Cities in Disequilibrium by Michael Batty and other books by Michael Batty. There is a lot on the internet. I like Batty, M., Longley, P. (1994), Fractal Cities.
I am happy this helped you! I also think that the inquiries on borders and the middle landscape (topics which I have worked recently) is interesting to look at - I only have the text in Spanish and it is here on Research Gate on my page. I think that Batty addresses these topics in a significantly manner from the geographical view. I also think that from the urban design, architectural design there is a different approach. Books on these can be: Kevin Lynch; David Grahame Shane- Recombinant urbanism!!!! and David Gouverneur "Planning and Design for Future Informal Settlements" - Shaping the Self-Constructed City, Routledge, 2014.
Chaos Theory or complexity science in general might be able to explains the complexity of urban fabric or traditional Muslim settlements, but it is limited to explanation or understanding, and it hardly contributes to making or re-making. However, Christopher Alexander's complexity science is, from the very beginning, for making or re-making the complexity of urban fabric. This making or re-making makes Alexander's work unique:
Presentation A Mathematical Model of Beauty for Sustainable Urban Design
Built on the legacy of Alexander and his life's work pursuing beauty, a new master program on architecture http://buildingbeauty.net/ has been established. The beauty is defined mathematically, exists in space and matter physically, and reflects in our minds and cognition psychologically. The program aims for wholeness-oriented design, beginning with construction rather than paper based design as most architecture schools do. It intends to create buildings or cities with a high degree of wholeness, instead of slick buildings as most modernist architects do.