The impulse response of an ideal filter used in OTFS transmission is a sinc function. This is because the sinc function is bi-orthogonal, meaning it is symmetrically orthogonal in both the time and frequency domains. This property is necessary for OTFS transmission, as it ensures that the transmitted symbols are orthogonal to each other and thus have minimal interference.