I am using a U-Net architecture with Xception net as encoder . The visual area of the segmentation mask is very small and after training, it is giving a lot of false positives. I am thinking of changing the kernel size of 3 by 3 to 5 by 5. What precautions/care shall I take while creating the model? I am getting the issue of dimensionality mismatch if just change 3 by 3 to 5 by 5 kernel size.
Also,while concatenating the skip connections, the dimensions don't match