Have a look at Article Optimization of endwall contouring in axial compressor S-shaped ducts

You can study end wall profiling in either of the two rigs you suggest. And as you say the annular one looks rather more like an engine. The use of incompressible flow is fine unless the "real" case you are simulating happens to be supersonic where shockwaves will cause steps in pressure that you cannot simulate at low Mach number. The important rule is that the design of your endwalls is carried out at the same Mach number as your experiment.

Back to your question about annular or otherwise. An important feature of annular cascades is that the spinning of the fluid at row exit is associated with a radial pressure gradient (as happens in engine). This pressure gradient transforms the nature of the secondary flow. High loss fluid will migrate radially inwards. This makes the suction side boundary layer and the loss cores fall towards the hub suction side corner. If your design focus is to modify the end walls only then you cannot address these radial flow effects satisfactorily. This means that the proportional reduction of secondary loss is reduced. How ever the general rule is that the stronger the secondary flow in the initial geometry the stronger the potential benefit of endwall optimisation. This suggests that having a cascade of low aspect ratio will make the secondary flow worse, but the improvement better.

Good luck either way

Martin Rose