I have got the FM and AFM1 configurations and able to optimize by giving MAGMOM. But how to get AFM 2 and AFM 3 ? and how the MAGMOMS are to be provided? I need step by step procedure. Thank you.
The three types of antiferromagnetism are likely those referred to as G-type, A-type, and C-type. The simplest type, G-type, puts a spin on each Ni which is opposite to the spin on its nearest neighbors. A-type has 2D planes of atoms with the same spin, but alternating planes are opposite. C-type has parallel spins along 1D chains, I remember this one as Column-type. Figure 1 of this paper shows what it looks like. Article The structural, electronic, and magnetic properties of SrFeO...
To get the correct MAGMOMS, I recommend that you open the POSCAR in the free VESTA software to visualize it. You can then click on each Ni atom in the structure and VESTA will tell you, for example, that the selected Ni atom is the third Ni in the the structure. This way, you can visually identify which atoms must be spin-up and spin-down for each type of antiferromagnetism, and obtain your list of MAGMOM by hand.
Please note that you may need to make a larger supercell of NiO to build some types of antiferromagnetism. For example, if you want A-type AFM with the parallel spins on the Ni in the AB plane, you must have an even number of atomic planes along the C-axis of your supercell, to balance the number of spin-up planes and spin-down planes. Remember that the supercell has a periodic boundary condition, so if the supercell is three atomic planes high, it will be like this
Steven Hartman has already provided a very good answer to this question; but there's one further comment worth making for a beginner.
NiO is *the* classic example of what is termed a "strongly correlated material". For most 3d transition metals (as well as the 4f block of the periodic table) it is worth adding in a correction for this. Within VASP the simplest way to do this is the Hubbard-U correction (see the Vasp wiki for details)