One key factor is whether you can trust the operators of the particular cloud. Are legal agreements with the cloud operator enough? In many cases it is not, which is why some organizations go with private clouds, assuming that they can trust their own employees more than another organizations.
Another factor is the ability to audit a system. A cloud user might have the requirement to be able to physical audit and even tap the network connections to a cloud resource. That kind of requirement could easily dictate the public/private cloud decision.
A final factor (there are probably more, but these three are the ones that immediately come to mind) is exposure to other organizations on the same hardware. Some people might be suspicious of sharing VMs on the same hardware with others (and leaks in hypervisors have happened), which is why some cloud providers offer dedicated hardware for customers, where no one else but a single customer will be using that hardware. Of course, that means you need to trust the cloud provider!
I would think that some of these considerations will change when things like Intel's Software Guard Extensions (SGX) are widely deployed. SGX makes it possible for applications to run safely without interference, modification, and eavesdropping from other applications and even the operating system.