I want to use a photo directly from Matlab in my paper. Should I use reference for the Matlab software in my work? and if yes, would you please tell me how I should refer it?
I think you do not need to refer to Matlab. This concept applies to any other software that produces graph/figure. For example, you never refer to Excel or origin to show the reference.
I don't think citing Matlab in this case would be required. However, if you prefer, you may add a footnote mentioning that the image was drawn/obtained/plotted using Matlab version xyz. Hopefully, you are not trying to use the Matlab logo or other similar images that may lead to copyright violation.