I think having your brain sound-occupied helps your mind to concentrate. When I write, I like to have movies going on, in another language than the one I wrote. A kind of music. I don't follow the movie, I just appreciate the sound presence. I think it's a way to joke your brain like you joke a baby with a tickling sound.
It will depend on who is listening, on what kind of main activity is being done, and on the type of music. Two extremes come to mind: (a) a naturally active listener (e.g. a professional musician) in a highly investigative acitivity (reading, writing paper, or composing—performing or practicing is a completely different matter); (b) a completely passive listener (e.g. a musically non-literate person) in a menial activity (repetitive, or extremely routine task). The active listener (case a) tends (as a natural routine effort) to listen to compositional structures in detail, as well as the performance inflections and skills that are present (inexorably) in the performance. It seems unlikely that the passive listener (case b) will spend any effort in recognizing structures, inflections or skills, but may simply hear the information in much the same way as background noise (one has to consider points as those in the answers by prof. Aparna Sathya Murthy and Fred Romano). If the piece of music is complex (e.g. contemporary, acousmatic music), the active-listener-by-trade not only will find it hard to concentrate in anything else, but will probably not even want to deviate the listening toward anything else. If the piece of music is fairly well-known and simple (e.g. a particular monophonic music that happens to be, in itself, repetitive and predictable), that active-listener will not spend to much effort in listening and immediately figuring out its musical characteristics, and may even value-judge or relegate it to the status of background noise.
Naturally, there are many studies in Music Therapy that would point the benefits and harms of listening to music while performing other activities. I (as musician) tend to see that those studies are applied mainly (if not only) to passive listeners, disregarding the specialized audition. Some of those studies even try to compare different types of music in order to assess their effects on cognition, but there must extra care in consulting them, for there are those without scientific methodology and no unbiased basis for comparison (e.g. the flawed studies that claim the benefits of, or that have constructed the so-called “Mozart Effect”).