I have tried the following methods unsuccessfully:
1- To a stirring solution of beta carboline derivatives ( harmaline, harmine , tryptoline or harmane) in DMSO, i added powdered NaOH and allowed to react for an hour @ rt; i added the dibromoalkane linker in excess (1:5 ratio) and allowed the reaction to run overnight. Reaction monitors by TLC. The spot i suspect to be my desire compound above my starting material on the TLC is not the one after i isolated it from CC (confirm by Proton NMR)
2- I carried on the same reaction in DMF as a solvent in place of DMSO.
3- The mixture of beta carboline, powdered KOH and CH3CN was irradiated in sealed tube in microwave: 100o C, 150W, 250 PSI, power max on, high stirring, 30 sec Ramping time for an hour.
Are trying to link two same beta carboline derivatives by a linker? If yes, I would suggest to use a different method. In case of tryptoline you might have to protect (e.g. by Boc) one of the NHs before you add the linker.
My suggested method: Take NaH (1.2 equiv.) in dry THF (3 mL/mmol of your substrate) under argon atmosphere in ice cold condition. Add your sample (dissolved in dry THF) slowly using needle and syringe. Bring the reaction mixture slowly to room temperature and monitor the the reaction by TLC. If you are a skilled organic chemist having knowledge in handling moisture sensitive reactions, it will be very easy to follow my instruction. If not, you can request any expert in your institute to set up the experiment. I believe this will help you.
The method you are using (the first one) is usually used for alkylating indole NH and as betacarbolines are indole based compounds hence the method is presumed to be valid for alkylation of betacarbolines as well. Another way to perform this reaction could be using NaH as a base in the solvent DMF.
However if you are looking for preparing the dimer, then I suggest you to carry out the reaction at elevated temperatures such as heating in acetone in presence of K2CO3 or any other mild base in polar aprotic solvents.
KOH is a poor choice, as hydroxide is a powerful nucleophile - and your substrate gives it two chances to interfere. NaH, as suggested by others, is a better choice, as would be DBU (an excellent base, although costly on a large scale.) Best solvents are likely to be DMF and DMSO. THF may need extended reflux, but it's worth trying because it's so much more convenient.
My recommendation:
dioxane as solvent, molar excess (1.5-2 eq) of equimolar mixture of K2CO3/KOH good pulverized, Cetrimide as PTC, substoichiometric (cca 5 mol%) NaI, temp. 50-60 °C, ultrasound action and your reagents in molar ratio 1:1.