No you do not need the empty vector. You can assemble a plasmid as long as your fragments contain an ori and a marker hence you will end up with a circular self-replicating plasmid (it is important that the final product is circular though).

The vector in Gibson assembly is no more than just a fragment. A fragment could be from PCR amplification, synthetic double-strand DNA or single-strand DNA or restriction enzyme digested product. A restriction enzyme digested vector (AKA empty vector) is preferred since it is straightforward and has less risk of carrying mutations in comparison to PCR amplified fragments. However, it is not always possible nor necessary to use such backbones.