I don't know which algorithm they actually chose for ASPEN but here is a nice discussion of some algorithms that could be used for the task. that might be of interest.
I think code from the original ASPEN project (MIT late 70's) are publicly available but they have probably changed quite a bit in the past 40 years. FWIW most of the convergence algorithms I have seen or developed from KBC/Aspen/Hyprotech typically use a simple Newton convergence technique with adjustable step sizes. For most problems I usually keep track of max that is too low and min that is too high and move to interval halving when the next guess is outside those bounds.
If you want to know which equations use the radfrac model, you'd watch the Inside out method for solving distillation columns, These algorithm could be useful to understand how ASPEN plus solving the Temperature, composition, and pressure profiles!
If you want to investigate more deep, I recommed you a book (You can find free online) called "
Distillation Design And Control Using Aspen Simulation - Luyben (2013)