Along the depth you have a change in the hydrostatic pressure and you can have some density variation due to difference in salinity. However, the drag is mainly due to the viscous stress and is mainly dependendent on the velocity of the body in water as well as surface roughness and body shape.
If we consider that the resistance is the main opposition to the high velocity, we can consider the Reynolds number as main parameter, higher in water that in air that has incidence on the viscous stresses. Think about the biomimetic shark skin developed after the study of the turbulence over riblets. However, the interface air-water can be involved by the creation of waves that creates resistence to the motion as well as vortical structures near the interface are different than those in depth.
A significant amount of energy of a swimmer goes to a surface wave when swimming at the surface. The speed of the (deep-water) wave depends on the body length. Taller swimmers therefore have an advantage (when swimming at Olympic level). To avoid wave drag you need to get about half-wave-length below the surface. Further increase of depth does not change drag. Perhaps different effects will be at the bottom due to the boundary layer but I suspect that they will be small.
Yakov's answer appeals to me. I would add that swimming at intermediate depth in a fjord, where there can be a strong stratification can be slowed by the drag of generating internal waves.
As a weak swimmer I can add some empirical experiences: I always use much more energy when swimming on the surface compared to snorkeling, even if the depth one is immersed to while snorkeling is not so large. Yes, the explanation could be you avoid producing free surface (or interfacial) waves. Counter-argument: one tends to take a much more streamlined position while snorkeling or diving compared to the free-surface swimming.
For humans, as noted by Dan and alluded to by Yakov, energy is spent on surface waves when free swimming, but swimmers stay below the surface off of walls where they use the force of pushing off the wall. As stated by Jacek, less energy is spent swimming sub surface, however this does not account for the speed difference. When snorkeling or free swimming at depth swimmers are going to have much lower horizontal speeds than at the surface (assuming wave action is not sufficient to greatly hinder progress. Swimmers generally swim at the surface because to propel them selves they pull/push against the water and recover in air to continue the motion.
A Submarine uses a propeller and as such movement at the surface is hindered by the lack of thrust when the propeller interacts with air. Along with avoiding wave action, submarine speed would then benefit from being subsurface. After getting below a depth where surface waves effects are negligible additional depth is unlikely to provide additional benefit.
However, if moving from the pool to the ocean the answer is going to be geographically dependent due to changes in current with depth. Consider the two layer flow model of an estuary.