In short, the tool used is reason--our brains. There are several standard techniques (differential/linear cryptanalysis for the algorithm, etc.) and it's standard to consider a number of known/common attacks (such as known plaintext, chosen plaintext, adaptive attacks, etc.) But at the end, each security/crypto system is unique, and the system as a whole must be considered. In most cases, the crypto primitive (i.e. encryption algorithm or whatever) is not the point of failure, but rather bad key generation/scheduling, bad protocols, bad management, or some other aspect of how the system is used. It's also important to have some security goals you want to measure--authentication, secrecy, guarantee of service, etc. What's important to this use of the system?

For bluetooth specifically, one must analyze all the information that an attacker can gather--is the device discoverable, what is required to authenticate a connection, how many transmissions can they collect, etc. The protocols and settings must be analyzed to see if there's a way to violate any of the defined security goals given information that can be obtained by the attacker.

For bluetooth, you might look at the security suggestions from NIST: http://csrc.nist.gov/publications/nistpubs/800-121-rev1/sp800-121_rev1.pdf. NSA has a shorter brief w/ good references at http://www.nsa.gov/ia/_files/factsheets/I732-016R-07.pdf.

Thank you very much