"Breaking a cipher simply means finding a weakness in the cipher that can be exploited with a complexity less than brute-force." from Bruce Schneier in A self-study course in block-cipher cryptanalysis
But i guess there is not a "formal" or globally accepted definition.
When we say a particular cipher text which is encrypted with a particular key of any cipher system is broken, it means that the key which is used to encrypt the plain text is recovered and decryption algorithm is identified through which the original plain text can be recovered.
When we speak about any cipher system, like RSA-512 or DES is broken, we mean that at least one ciphertext of the cipher system is broken with normally available computational power and it is no more safe to use that particular system. Second case speaks about the security level of the particular cipher system.