I designed a rectangular waveguide filtering antenna but finding it difficult to control the bandwidth and transmission zeros. Your professional advice will be highly appreciated
The antenna needs to be designed as a filter, using the appropriate filter couplings, for maximally flat or Tchebychev filter characteristics for example, to give the desired bandwidth and roll-off, perhaps with cross-coupling between non-adjacent elements to give zeros. I think you may find this under "elliptic filters".
You may have done this already, in which case I have no further advice!
Matthaie Young and Jones is on the internet as a free pdf and probably covers most of what you need. https://www.microwaves101.com/uploads/MYJ-part-1.pdf