The first question to ask yourself is if you want to evaluate the overhead analytically or empirically. If you want to do an analytical evaluation then just look at the specification for Java RMI and RPC over DDS -- BTW notice that RTI-DDS is a DDS implementation not the Standard, you may also consider some more widely used and Open Source alternatives as OpenSplice (http://www.prismtech.com/dds-community).
If you are looking to do an empirical analysis then you can somehow estimate the temporal overhead performing some basic tests for small sample. The space overhead can be estimated indirectly by looking at the throughout delivered to the application vs. the bandwidth used on the network. In that case recall to take into account he protocol headers for level 4 and level 3, e.g. UDP + IP or TCP + IP.