I'd recommend using the Xilinx boards, as you will typically find more resources/datasheets/forum posts to get you started. Startng from the '6' series, Xilinx FPGAs have an integrated PCIe endpoint and many boards feature a dedicated PCIe connector on the edge -- see e.g.:
Check out Pico Computing: http://picocomputing.com/
They make PCIe FPGA modules and provide the communication systems for you. This should let you emulate HPC communication across multiple FPGAs seamlessly.