Hi Nariswari, have you used an appropriate base? This is a Claisen-Schmidt type reaction and this link may assist you

http://www.synarchive.com/syn/13

Also a procedure that may help

A mixture of 2-(4-formyl-2-methoxyphenoxy)acetic acid 1 (0.01 mol, 2.10 g) and appropriate acetophenone 2 (0.01 mol) is reacted as per Claisen–Schmidt condensation, in absolute methanol were stirred at room temperature in the presence of base (alcoholic solution of potassium hydroxide 30 %; 5 ml) for 30 min. The reaction mixture was allowed to stand overnight and then poured into ice-cold water followed by neutralization with HCl. The separated solid product was filtered, dried, and crystallized from ethanol. The purity of the chalcone was checked by column chromatography, viz: toluene: ethyl acetate: formic acid (5:4:1) and petroleum ether: toluene: acetic acid (5:4:1) and elemental analysis using Perkin Elmer analyzer.

Source:http://rd.springer.com/article/10.1007/s00044-012-0086-y

Hi Nariswari, have you used an appropriate base? This is a Claisen-Schmidt type reaction and this link may assist you

http://www.synarchive.com/syn/13

Also a procedure that may help

A mixture of 2-(4-formyl-2-methoxyphenoxy)acetic acid 1 (0.01 mol, 2.10 g) and appropriate acetophenone 2 (0.01 mol) is reacted as per Claisen–Schmidt condensation, in absolute methanol were stirred at room temperature in the presence of base (alcoholic solution of potassium hydroxide 30 %; 5 ml) for 30 min. The reaction mixture was allowed to stand overnight and then poured into ice-cold water followed by neutralization with HCl. The separated solid product was filtered, dried, and crystallized from ethanol. The purity of the chalcone was checked by column chromatography, viz: toluene: ethyl acetate: formic acid (5:4:1) and petroleum ether: toluene: acetic acid (5:4:1) and elemental analysis using Perkin Elmer analyzer.

Source:http://rd.springer.com/article/10.1007/s00044-012-0086-y

I agree with Andrew Jordan, you need either base or acid present for an aldol condensation. Also, some heat to favor the dehydration, or else it wont be productive.

1.May be the activity of vanillin is  not enought, compared with the other  aromatic aldehydes;

2.You might have got the target compounds and they could not separate from reaction as precipitation since the mixtures you got are too motley. so you need to do like this: first, you need to confirm you have got your compounds by TLC or LC or other test methods(I greatly believe that you have got them); second, get your compounds by Column Chromatography (I also believe the yield might be very low)

3. You have to synthesize chalcone by other methods.

Maybe it is necessary to protect the hydroxy of the vanillin first.

0.5 or 1 eq. of NH4OAc in 5-10 ml of absolute ethanol and followed by the addition of your Carbonyl compounds vanillin (1.1 eq.). vanillin and NH4OAc will give the imine or aldmine... this u reflux for 15-20 min. followed by the addition of other carbonyl compound such as 4-chloroacetophenone (1 eq))..... and the reflux was continued to 24 h.

finally  you can preform the Column chromatography... 

or

Otherwise without column also u can isolate the product... but yield is bit lower.... when compare to column isolation.. the procedure:  remove the solvent from the crude reaction mixture... after that U can add methanol or ethanol to the ppt gently warm up with 40-50oC to remove the unreacted starting material and some of the volatile NH4OAc......    

I think this procedure will work... please give me your suggestion.... after you preformed the reaction.

Try mild bses e.g. Na2CO3 or NH4 acetate at room temperature for a long time [TLC follow] ... the chalcone product is probably with low M.p. .... If it is formed, extract it and perform TLC for purification

I send you a reference, Solvent-Free Synthesis of Chalcones.

www.JCE.DivCHED.org • Vol. 81 No. 9 September 2004 • Journal of Chemical Education 1345

Regards

Jorge Cárdenas

I just want to ask, Why are you dissolving your base in water?

Try to avoid water in the first steps, dissolve it in MeOH, or ethanol the solvent which you will be using in your reaction.