Are your X and Y scales continuous or the variables take only integer values?

You asked  Does this make sense? 

The answer is No.

First of all, we use transformation such as square root of X if the variable is skewed and needs transformation.

Second, we use polynomial or exponential regression if there is a theoretical reasoning for it.

Third , we use dichotomous dummy variable to test for a structural shift (two eqs having two different intercepts)

Fourth, the interaction between X and dummy is used to test the change in slope.

I believe you should do some more review of literature to be able to form the right model

I believe you are on the right track. Suggest you look at chapt. 3 in the book below. This may help. best wishes

http://www.amazon.com/Analysis-Experiments-Chapman-Statistical-Science/dp/1439868131/ref=sr_1_7?s=books&ie=UTF8&qid=1455429638&sr=1-7&keywords=John+Lawson

Thank you for your responses!

Kaloyan:

Thank you for your comment.

X is is the amount of money people receive for a certain task. However there are only 7 different possible values but the distance between those is equal (1$, 2$, 3$, etc.)

Y is an aggregate of 3 likert scales, so yes, continuous.

Abdulrazzak:

Thank you for taking the time and your candid assessment. However, I am not sure I understand all of your points.

1. That is certainly a way to use this transformation. However, if the relationship between X and Y is not a linear relationship, I assume it makes sense to model it accordingly.

2. Of course. I have my theoretical reasons to assume this relationship. To keep this already quite long post as short as possible I did not elaborate on that. I hypothesize that the relationship should like like I described. But I am unsure if that is the right way to test it statistically.

4. Exactly! I am interested in different lopes in the two groups. So I would assume that there are differences in those slopes reflecting in interactions with the linear and with the square root term. The biggest biggest question for me is if I can include both terms (linear and square root). If we, for example test for quadratic functions, we would have to include the linear term and the quadratic term in the function. Is it possible for a X1/2 function as well?

From a theoretical point I think that it is the right model. I only want to find out how to test it properly.

David: Thank you very much for the book suggestion. I will look into it.

Thanks for the clarification, Arne. Now, to the main point. You cannot obtain the two curves shown on your graph using a square root of your X variable. No matter what your regression coefficients are, for one of the groups the graph will always lie above (below) the other graph. To obtain a result similar to your graph, you need to have two different power functions (see the attached chart where as an example I used 0.5 and 0.3 as powers of X). In your example it seems that those powers need to be estimated in the two cases. See also my attached short pdf.

The parametric model you used might suffice, but often this is not the case. There are more flexible methods that are impossible to explain quickly. Suggesting looking at the methods in Wilcox (2012, Ch 11, particularly section 11.11) The are easily applied using R

Wilcox (2012) Introduction to Robust Estimation and Hypothesis Testing. Elsevier

The Wilcox book is very good in my opinion. I would still look at the Lawson book to get a good understanding of interaction prior to looking at the Wilcox book.

Kaloyan, thank you very much for suggesting to test for different power functions. that looks like a promising approach. And thank you so much that you took the time to write down the functions.

I think the reason that my graphs cut is that I had a linear and a square root term in the equation. And in one group the ß of the square root was bigger while in the other group the ß of the linear term was bigger.

Rand, thank you very much for the literature suggestion. I will take a look into the ANCOVA chapter!

It's always possible to move away from testing your hypothesis of moderation with a statistical interaction term: Given you have a dichotomous moderator, you could carry out a two group model (which would also test for moderation of the mean structure, e.g. intercept).  Perhaps work up a series of comparisons testing all the parts of your moderation?  i.e. That X on Y differs across for the x2 Z groups: mean structure, shape of linear regression, then form of regression.  It's an interesting delve into how far hypotheses of moderation can take us.  Thanks for sharing.