It is very broad question as there are many different types of photodetectors which can be used in different ways.
The most widespread case is photodiode being limited by (R+r)C where R- effective impedance of readout circuitry, r - intrinsic resistance, C-capacitance.
Again, in some cases (e.g. very thick structures) you have to consider charge transit/collection time too.
On another hand, photoresistive detectors might be defined by combined play of such parameters as recombination time, charge sweep-out or collection time (for both electrons and holes), dielectric relaxation time (e.g. if injection is involved) and various charge trapping mechanisms.
Following the above comments, the area of a photodetector is vital to determine the rising and falling time of a photodetector, which is related to the capacitance of a device. The nature of materials made from photodetector for sure is a key for the response of a photodetector. The interface between the layers within a device is the factor as you mentioned.
First, need to define what "rise" and "decay" times mean. I assume you refer to the scenario that a square pulsed signal light shines onto a photodetector and the sloped leading edge and trailing edge durations are the rise and decay times. I'm sure they are not the right terms, but I don't remember the correct one either. Probably something like response time and relaxation time.
Anyhow, as above said, the circuit RC characteristics typically dominate. The rest differ among the types of detectors. Photomultiplier tubes and APDs have higher relaxation times because the electrons or holes need time to stop multiplying. Semiconductor photodiodes don't have multiplication and have good relaxation times.
I fully agree with all previous comments, it is tricky to answer.
However for a particular case (UV MSM photodetector), we could significantly reduiced the photodetector response time (rise and decay) by using a better geometry of the top metal contact (finger structure) and by improving also the metal-semiconductor contacts (we tryed to "passivate" the surface defects). This improvement was observed using the same bulk semiconductor (diamond) material. Note that UV detectors are very sensitive to surface defects.
The main factor is the exponential trap distribution in the bulk or at the surface. Most probably surface defects and recombination centers formed by vacancies control the growth and decay processes. cases where trap centers are converted to sensitizing centers are associated with instability of photocurrent at short time scales.
Please read
photoelectronic Properties of Semiconductors, Richard Bube