Duplications in the genome. The intrinsic complexity of the genome. If you get a repeated segment from your sequencing, where you are going to place it in your assembly. That is hard.
Of course, repeats can be exact and inexact. Will the minor differences in your raw sequence matter? It may be simply sequencing errors.
I guess if you have very long reads, with many unique regions that can anchor it in the genome, then the assembly task is not that hard. That said, where are you going to get the extremely long, unfragmented sequences. I guess PacBio is one option, Oxford Nanopore is the other. Oxford Nanopore reads are not good quality reads, so it may be hard to use for now. Long term, I hope Oxford Nanopore can provide high-quality reads. If the read length is over 10K, then assembly of genome is trivial. I repeat, it is trivial!
A major problem is also that limited variation is represented by the homozygous human reference genomes up untill Hg37. The human reference genome is based on a consensus of a few individuals whose ethnic backgrounds are a strictly held secret.
Thus parts of the genome that vary quite a bit from the reference genome (such as in populations not well represented by the reference) either map to wrong regions or not at all.
The latest Hg38 alleviates this problem some because unlike its predecessors it has alternate contigs. So assuming an alternate contig aware aligner is used, the resulting assembled sequence should be more accurate.