Aside from authenticity in instrumentation, I think a good starting point for rules, is the modes that these music were usually written in. Particularly in Celtic tunes that are still performed or sung. There is also conjecture that there are traces of Arabic influences in their music, which shef D more light into the nature of this kinds of music.

http://journalofmusic.com/focus/arab-influence-irish-traditional-music

You should base your research on the abilities of the musical instruments of this culture. For example, Trossingen Lyres were widely used in vikiing music. These are usually 6 stringed instruments for playing by "block'n'strum' technique. These lyres allowed to play four triads, Am, C, Em and G. But such tuning is modern adaptation. We can judge about the tuning of these instruments from surviving flutes, because flutes serves their tuning as long as they live. Reed instruments can be used as tuning ethalon too, but they are less reliable, because their tuning depends on the state of the reed. They can  be used to research the intervals, but not ablosute tone.

Listen to Ben Bagby.  You might start with this:

http://www.amazon.com/Songs-Rhineland-Harper-BAGBY-SEQUENTIA/dp/B0002XEEBQ

You write: "I can´t find any reliable music notation". Well, you didn't look very far...

Gregorian chant has been written in readable notation since the 11th century and any modern chant book of the Catholic church reproduce similar notations of the same pieces; these books are available on Internet (e.g. on http://imslp.org/).

For medieval compositions in reliable notation, see for instance http://imslp.org/wiki/Category:People_from_the_Medieval_era.

But composing such music today is quite another matter. Musicologists and theorists devote their time to understanding these works and figuring out how they were composed. The better the understanding, the better one could write "in the style". It takes several lifes to master...

Modes might indeed be a possible approach. The problem is that one cannot be sure of the extent to which medieval music in the West was modal, nor whether modern Arabic modes (maqam) have much in common with former counterparts. This has occupied generations of musicologists and will occupy them for as many generations to come.