Which cooling fluid is recommended, and is it advisable to write Matlab code to find the analytical solution for the different geometrical and thermal properties?
Dr. Vitalii has given some very useful references. All I want to add is the following:
What is the application you are talking about? For example, you have power plant condensers and regenerative feedwater preheaters (among others). Then there are condensers for refrigerant vapor. If you are interested in the shell-and-tube type of condensers, then in all these cases water is the coolant, it being boiler feed water in regenerative feedheaters. If your choice for coolant is between air and water, please be advised that air (or gas) is inadvisable for shell-and-tube types. Air-cooled condensers are very different in construction and Google images or a good book on Power Plant Engineering will give you a good idea of its geometry. Water in unbeatable as a coolant except in those unusual circumstances outlined in the second reference given by Vitalii.
In CHP (Combined Heat and Power) plants, the coolant could be a process medium. Therefore, as you might see, the coolant depends on the application.
In all these applications, the vapor is always on the shellside. Except for some designs of feedwater preheaters (where they have segmented baffles on the shellside and zigzag flow of the vapor), the shell is generally a TEMA X-type (i.e., simple crossflow over the tube bank). The number of tubes in any row decreases steeply as you go towards lower rows and this must be taken into cognizance in more serious commercial designs. If your interest is an academic exercise, then a single mean heat transfer coefficient for condensation would do. Kern's prescription is for a power of 1/4 for N rows of tubes while current practice uses an index of 1/6.
The design approach is similar in all cases. In steam condensation, the condensation heat transfer coefficients will be many times higher than for water in the tubes (fully developed turbulent flow). Therefore plain tubes (smooth surface, not augmented surface) are generally used.
As several variables like tube diameter or shell diameter are not continuous (unlike velocity for example), mathematical methods of optimization do not work here. To select the optimal design, you need to consider different designs and compare them. And yes, you could code it in MATLAB. My favorite approach was Excel as I understand it better!