Perhaps observing temperature may not be sufficient in this case. Depending on your discretization scheme, it may help identifying the required CFL condition [https://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition] and validate that it is satisfied throughout the time evolution. You may need to identify and observe some additional parameters e.g. Courant Number, time-step size (and grid size if its adaptive). Often plotting such parameters separately and observing their derivatives helps to identify non-physical sources/sinks.

Hint: Note the peculiar point at Flow-time = ~470s on the second plot. At this point, first derivative of temperature dosen't exist. Observing relevant parameters around this time-point would perhaps provide better insights.

Absent someone being able to quickly identify the cause, you're going to have to step back to a simple case and slowly add complexity to your model. Can you correctly model a steady state condition? Do the temperatures make sense with the heat sources and sinks present? Can you do a simple non-udf case? Then can you do a simple transient? Back to steady state, can you independently test the radiative and convective heating modes? Then again independently a simple transient for each. Can you make a UDF that just emulates the simple cases? If all those independently look fine, start combining slowly.

It's just like bisecting code to identify a programming bug. Simple pieces to help identify the error. Could be as simple as a UDF flux or boundary condition being defined negative to the rest of the model pulling heat against the temperature gradient. Start small and tease out the error.

Try to do an Heat balance instead of just looking at the temperatures of wall and fluid near the wall

Heat flux (W/m2 )getting in from the wall goes in to the bulk fluid. So total Q into fluid equal W/m2 * wall area m2, which equal Qfluid ( m *cp*dt).