How many residues are not satisfying the Rama plot?

It is frequently hard to bring every residue into the allowed region of RPlot, so if your residues are few, then it is fine.

Generally, disorder regions in proteins do not "properly" follow Rplot, so you also need to look for that.

Multi-template modeling can be used in the Modeller program to build an improved homology model.

Finally, you can build the outlier region independently and copy/paste it onto the model.

You can refine the model by GalaxyRefine webserver , it will generate 10 models and usually all refined models meet the criteria but occasionally model 1 is best fit.

Look at the templates used by swissmodel - what is their quality? If multiple well-resolved structures show the residue with the same poor phi/psi angles, it may be real.

e.g. for antibody variable domains:

"Immunoglobulin variable domains contain a number of positions with conserved positive Phi angles. Most, but not all of these positions show a strong preferences for Gly. Other residue types may have a destabilizing effect in those positions. VL kappa domains usually also contain two cis-prolines. cis-Pro L8 forms the "kink" in framework I. S.Spada et al. have shown that replacement of this cis-Pro by other residue types has a destabilizing effect. A similar cis-Pro is present in the equivalent position of many T-cell receptor variable domains. VL lambda and heavy chain variable domains have a one-residue deletion compared to VL kappa and do not contain conserved cis-prolines in the FR1 kink. A second conserved cis-Pro in Position L136 is responsible for the omega-loop conformation of the CDR 3 of most of the kappa domains"

https://plueckthun.bioc.uzh.ch/antibody/Structures/Torsions/index.html

If you want to know the quality of the model you get, then you can use the web server https://saves.mbi.ucla.edu/