I am looking on drug discovery manuals describing aspects like therapeutic windows in cancer, how to choose or rank drugs according to their effect on target, but also on toxicity comparing as example normal vs cancer cells.
"The Practice of Medicinal Chemistry" is a pretty good textbook on such topics.
That being said, there's no hard and fast answer. If you go to conferences, you'll see that there's still a lot of effort going into determining when to keep going on a discovery project, and when to quit. Some more classical efforts are to look at e.g. Lipinski Rule-of-5 criteria, but the field has been finding more and more good drugs to violate those criteria - and all they ever did, after all, was describe the characteristics of drugs that existed right at the time. Similarly, the PAINS concept has gotten a lot of attention, but more recent studies have suggested that the pain was in the assays conducted, and not in the compounds used.
In terms of whether to proceed, a lot tends to depend upon your goal. In academia/basic research, you might proceed with a molecule with poor selectivity and no therapeutic window to speak of because you're more interested in validating a target, or just developing a novel scaffold. In industry/applied research, it's more 'kill quick, kill cheap', so failure on any of a host of different criteria may end a project.