Is there a signaling pathway or a particular signal induced by arrest between S phase and mitosis (cells don't divide after synthesizing new DNA) that initiates cell death? Is this death necessarily mediated by caspases?
I am sure that in the living cell there will always be a new mechanism in order to avoid fatal off process between S phase and mitosis. Alternative routes include, for example, in cancer cells. If such alternative paths were not included, we would be able to defeat the cancer. Please, try to develop your idea. It is splendid way in science.
S-phase checkpoint deals with this. If DNA synthesis is not completed by the cells (which usually will have naked DNA ends due to topoisomerase cuts: unwinding the tangles) DNA damage response kicks-in to activate cells death. This needs a threshold to convert the DNA damage repair signal to cell death signal, involving p53, BRCA etc.
G2-M arrest also induced apoptosis or mitotic catastrophe depending upon how long the cells stay in G2-M stage (example: prolonged nocodazole induced G2-M arrest leads to apoptosis).
Caspases are involved in such cell deaths but I guess there could be alternate cell death (caspase independent) depending upon the mode of mitotic catastrophe involved.
There could be various pathways related to this majorly can be p53 or caspase pathways. But dowstream pathways can alternate. Best to know all is to do micro-array.
DNA damage signalling can also lead to cell senescence (at least partly through CHK2-p53-p21) rather than cell death. It is not entirely clear how cells choose one or the other, but probably with lower levels of damage signalling you would get more senescence (permanent arrest) and less death. So cells that fail to complete S phase or mitosis can end up in a kind of G1 phase with tetraploid or subtetraploid DNA, as several groups have shown.