I suggest you to read the following books, there you will find answers to your questions. Geotechnical Enfgineering: Unsaturated and Saturated soils by Jean - Louis Briaud (2013), published by John Wiley & Sons. Another book; Soil Mechanics and Foundations by Muni Budhu (2011), 3rd edition Published by John Wiley & Sons. Read these two text books, you will find very interesting understand about slope stability problems. I hope you will find e-book also from the web site.   

For sandy soil, the slip surface me be parallel to the slope. For c-fi soil, the slip surface becomes non-linear. Note that the circular slip surface is rather idealized failure condition of slope. For multi-layer soil with a soft layer, the slip surface becomes complex.

The determination of the critical sliding surface (if it is not specified a priori) in the methods of limiting equilibrium is multistep. At the first step, a probable PS is determined and the slope stability coefficient is calculated. This procedure is repeated for a number of possible sliding surfaces. As a result, the slip surface with the lowest coefficient of stability is defined as critical. However, in the case of inhomogeneous slopes, the real slip surfaces can differ cOOstantially from simple ones. Often the optimal slip plane is controlled by stratigraphic boundaries or confined to individual geological bodies. Recent studies have revealed the possibility of incremental changes in parts of the sliding surface (Greco VR., "Efficient Monte Carlo Technique for Locating Critical Slip Surface," Journal of Geotechincal Engineering, Vol. 7, No. 122, 1996. pp. 517-525; Malkawi AIH , Hassan WF, Sarma SK, "An efficient search method for finding the critical circular slip surface using the Monte Carlo technique," Canadian Geotechnical Journal, Vol. 38, No. ISSN: 0008-3674, 2001. pp. 1081-1089) , as a result of which there is a decrease in the coefficient of stability of the slope, and the PS itself becomes different from the simple one (circular-cylindrical). This technique of minimizing the coefficient of stability due to incremental changes in the parts of the calculated simple sliding surface is called the optimization of the slip surface. The main methods for optimizing the slip surface are: Greco Method, Annealing Simulation Algorithm (SA), and the cuckoo method.

The main thing to remember is that the position of the slip surface with the minimum coefficient of stock is determined by the geological structure of the slope, the properties of the rocks and the hydrogeological data. Very good, when the position of the calculated slip surface is consistent with geological information.

The type of soil, and number of layers of your slope, play role in the position and the type of slip surface. you can do analysis assessment of slope by finite element method.

First, you should develop a conceptual model to understand what kind of materials and what kind of controls are acting on your slope instability.

1) you should define the type of landslide you expect: shallow landslide within the soil cover? large bedrock slide? Relatively shallow earthslide? For each typology, the shape and the location of slip surface can be different (for instance, for shallow landslides within soil cover we usually assume the failure parallel to the slope, located at the soil-bedrock contact).

2) you should make a hypothesis (that you'll demonstrate or not) on the geological control. Is the slide occurring on fractured bedrock? In this case, the slip surface is controlled by the joints. Is the slide occurring on soil or extremely fractured rocks (according to the scale of the landslide)? In this case, the slip surface can be circular, sub-circular or logarithmic spiral. Is the slide occurring on layered materials? In this case, different layers can control the slip surface.

3) you should finally evaluate other possible controls, such as groundwater level, slope morphology, the presence of tension cracks on the top, water seepage, geological structures (faults, major discontinuities), etc.

Once you have developed your conceptual model, you can run numerical simulations with either a stress-strain approach or a limit equilibrium approach, as Kreuzer commented above. However, if you don't have a clear conceptual model of teh landslide and a good geological model, you would spare time with millions of useless attempts.

Regarding the literature, it's huge, but you should direct your search to the appropriate type of landslide and type of materials.

There are many publication dealing with complex polygonal slip surfaces which normally occur in complex ground conditions. Many limit equilibrium programs can model and calculate this situation easily. Moreover, using this software you could define one and check the appropriate safety factor. Another possibility is to perform FEM analysis using phi-c reduction technique.