Each of the steps on that page tells you which file to modify. The instructions can't tell you where the files are, because that depends on how you installed Gromacs. The force field files are in subdirectories of /your/path/to/gromacs/share/gromacs/top in the relevant directory of the force field you are modifying. Modifications are done with your favorite plain text editor.

I think i need to elaborate my question.

Suppose i need to make a peptide ALA-ALA-XXX-ALA-ALA and simulate it using gromacs.

As I need partial charges and parameters for my non-natural amino acid (XXX), I used swissparam webserver (http://www.swissparam.ch/) for getting .itp file compatible with charmm27 force field. If i were to do simulation of a protein with ligand XXX then it is no problem because just i have to include this xxx.itp file obtained from swissparam webserver into the topol.top file of protein (generated by pdb2gmx). But i want this XXX to be incorporated into the peptide.

Now i am not able to understand what to do because from gromacs manual what i understand is i have to modify aminoacids.rtp, aminoacids.hdb, atomtypes.atp etc files and I only have xxx.itp and some other files like .par, .psf, .rtf etc and i don't understand how to include these to make it work.

The primary file you will have to modify is the .rtp file. The .rtp is a topology, just in a slightly different format. Chapter 5 of the Gromacs manual covers this. You should study existing examples of the other, standard amino acids to see how it works. The .hdb entry is the same story - read and understand the manual section regarding its format and see how those elements are applied to a known system. If your residue introduces new atom types, add them to the .atp and ffnonbonded.itp files.

If you need more detailed guidance, I would suggest posting specific information to the gmx-users mailing list, which is a very active community of Gromacs experts. Given that you didn't tag "Gromacs" in your post, the most qualified people may not even be seeing your question here. The mailing list will also allow you to easily share files and post the necessary information.

You need to create your own force field basically.

You can either write out .top file entirely, or create a .rtp file and then use pdb2gmx to generate your .top.

It is easier to create .rtp file first, since you only need partial charge and bond information. The dihedral and angle will be automatically generated. Then you run grompp to check for any non-standard bond/angle/dihedral, correct for them (modify the ffbonded.itp).

However, in terms of the file system, I have another question.

Can you create _appendix file for the standard .rtp and ffbonded.itp and include them using #include "DIRECTORY/_appendix" in .rtp/.itp respectively?

I tried but Gromacs seem not be aware of the inclusion. I can only copy the appendix.rtp to the source folder, and include ffbonded_appendix.itp in the forcefield.itp.

I there anything wrong with my inclusion?

I doubt #include statements within .rtp files work.  Using the #include mechanism within other files that are #included is very dangerous; it's easy to lose track of nested parameters!  There is really no reason to try to #include anything with respect to an .rtp file; pdb2gmx reads everything that has a file type of .rtp when generating the list of available residues.

Hey Justin, thanks for the discussion.

The main reason I want to use include is not to mess with the gromacs folders. Since I work on a local computer and a cluster, it is easy to get things disordered. So to avoid malfunctionalize my Gromacs, I tend to add appendix files to gromacs forcefield that are in a separate work folder, and include them in the .top or other files.

But you are right, the .rtp does not support include in itself.