An indicator function is a function that takes two values, depending on whether the conditions stipulated are fulfilled or not. The indicator function typically takes the value 0 for all points (in whatever space we consider) that fulfill the condition. But for points that do NOT fulfill the conditions stipulated, sometimes the value 1 is used - making it a  0/1 indicator -, and in others the value corresponding to non-fullfillment can be + infinity (that is, a 0/infinity indicator function). This latter case is a classic form of ultimate penalty function for being outside of a specified region of interest, defined perhaps by a set of constraints. Penalty methods for optimization, devised in the 1960s already, can work as follows: initially the penalty value for violating a constraint is small, so one tends to yield infeasible points when minimizing the some of the original objective function and the penalty term. One then utilizes this solution as a starting point for the next iteration, in which the penalty value has increased a lot - let's say that you multiply it by 10 in each iteration. Eventually the penalty term will become more important to keep low than the original objective, so you will notice that the points obtained will come closer and closer to being feasible. Under nice conditions one can in fact prove that the sequence of such iteration points converge to an optimal solution to the original problem.

Of course, I do not know about indicator function. But we refer to this kind of function as characteristic function. We denote it by greek letter "Ki". The characteristic function of a set A is defined as Ki(x)=1 if x belongs to A and 0 otherwise. Your definition of indicator function is identical with characteristic function. Characteristic function is a very standard function. It requires no reference.

 Your  example is so called characteristic function. More precisely to name it as set indicator.As for the reference see e.g. Kolmogorov&Fomin Elements of the Theory of Functions and Functional Analysis (Trnslation from Russian)

Thanks for your help, I will it a characteristic function, not an indicator function