Point-to-Point satellite communications involves several factors: 1) they operate a low power levels so high-gain antennas are needed, usually at both ends, 2) because of the narrow beam width of high-gain antennas, antenna steering is needed but that is difficult to acheive with tumbling satellites so satellite stabization is needed but this requires an energy system (steering jets) that is expensive to put into satellites, 3) the communication is line-of-site so when the earth is between the satellites, they cannot directly communicate and relay satellites must be used. Suggest you read the original papers on the Motorola Irridium system that had to face these same challenges. Good luck.
Totally performance of any communication system can be evaluated based on the Bit Error Rate (BER) that mainly controlled by controlling interference. Thus, improving interference and BER could be mention as the top priority of satellite communication, however it is investigated too much in the researches. You may find many many paper that deal with these topics.
Inter satellite communications involves several criticality: orbit and trajectory of satellites; Size and type of antenna arrays.Total Link Budget; Path Loss; EIRP; Figure of merit of receiver; Carrier-to-Noise Ratio (CNR) at Receiver input; Bandwidth; Frequency allocation; Frequency reuse Concept; Propagation time; ...
One needs to understand the end requirements first. Some of the obvious questions are (i) whether this ISL requirement between the satellites have to be DIRECT or INDIRECT ( i.e., via Earth Station) ? (ii) whether the requirement is for connecting LEO satellites to GEO, and that calls for Store & Forward requirements, or (iii) is it for REAL TIME connectivity between LEO satellites themselves?. Then you will start getting different scenarios with their technical/technological challenges.