My suspicion is that 1) the small radius rotor would have to rotate at an unacceptably high rate to counter the torque by the main rotor, 2) the resulting turbulence would likely affect the performance of the main rotor, and 3) it would be a very wasteful design, as the small rotor would not contribute at all to lift, so all the energy spent on making it rotate is wasted.

The reason why the standard helicopter design works is because the small rotor is at the end of a large moment arm. The reason why the coaxial design works is because the two counterrotating rotors both contribute to lift.

This design cannot work. Suppose the helicopter body is stabilized by applying two opposite torques to it: one from the atmosphere through the main rotor engine, and another from the "reaction propeller". Then this "reaction propeller" should also experience a torque from the helicopter body. If nothing compensates for this torque, the "reaction propeller" should increase its rate to infinity, not to any constant  value. 

The contrarotative rotor has to be as big as the first rotor and outside the duct. (werewolf)

To balance to main torque, a old concept was to move principal rotor with high speed air ejector at end of blade (no torque went through rotor axis) (remicopter, Djinn).

The tail rotor is sometimes replace by air thrust nozzle, to create the equilibrium torque.

http://www.google.fr/imgres?imgurl=http://1.bp.blogspot.com/_Hkorffi-QAY/TUOxosq5hkI/AAAAAAAAAJ4/VpdOM0zt9Lo/s400/notar.gif&imgrefurl=http://helicopter-world.blogspot.com/2011/01/this-article-is-about-helicopter-main.html&h=257&w=400&tbnid=eXuAivdwXWRS5M:&zoom=1&tbnh=90&tbnw=140&usg=__j7aPr4J1sztyGzGdRVqbQ13d_1M=&docid=UA3w9jbF_lJpbM

http://www.pistonheads.com/gassing/topic.asp?t=1069465