I am currently studying on flat heat exchangers with various fins shapes for my research. The objective is to identify the best fin shape. Colburn factor and friction factor are to be calculated.
Better to refer to Heat Transfer Data book for specific correlations. Else one can use the regular closed form solutions available in terms of Bessel's function for unconventional geometries to estimate the fin efficiency. I would also suggest a CFD analysis. Your task can be completed both numerically or analytically.