Speaking of someone who, with Latham, has developed one of the most successful theory building projects in the history of psychology, I will say that it is a gradual, inductive process-feelings come and go like rain drops but they do not lead to a validated theory without extensive cognitive work-

Theory building needs both a creative mind and a systematic approach. One of the main strengths of grounded theory is to have been able to marry those two opposite attitudes. Remind Pierre Bourdieu's saying "la sociologie est un sport de combat". Martial arts require both system and creativity. Under this regard The Art and Science of Portraiture by Sara Lawrence-Lightfoot and Jessica Hoffmann Davis released in 1997 has been an inspiring book for me. Creativity helps to produce ideas, orientations, options to check, systematic work helps to select. Without ideas you will not move, without selecting you will get lost. Moreover theory building requires reflective work in order to be aware of your position on the shoulder of giants. Working with the classic texts must be done in a systematic way.

I recall one of great contemporary mathematicians, Prof. Petr Vopenka (http://en.wikipedia.org/wiki/Petr_Vopěnka), describing his feelings when he was working on a major treatise on non-Euclidean geometry: he was saying that after the months of immersion in the work, he could actually "see" parallel lines crossing each other, and all other "strange" things of non-Euclidean universe. This allowed him to pursue various theorems and their proofs much more creatively and with ease, and he was much more able to tell the promising ideas from the dead ends. The theories and formalisms are then what we are left with after such a creative, very subjective and impartable enlightenment fades away; they are casts of a kind, and potentially anyone can use them to direct his own creativity to immerse back and carry on the hard exploratory road.

So my opinion: it's fifty-fifty; without creativity, there's no progress, and without formal and precise methodology, there's just dreaming.

Speaking of someone who, with Latham, has developed one of the most successful theory building projects in the history of psychology, I will say that it is a gradual, inductive process-feelings come and go like rain drops but they do not lead to a validated theory without extensive cognitive work-

Briefly, I would offer that Initially, science is a process for establishing relatively objective and observable facts about some aspect of

experience that is subject to direct or indirect observation and measurement. Once a sufficient body of related facts are established a scientific (i.e.; testable) theory or theories are proposed to account for those facts. In other words, a tentative explanation or explanations are proposed that the proposers think best account for the related facts. I think there are both steps and creativity involved in producing a theory.

I would take issue with one aspect of David's comment: the objective facts are about reality not about experience--I am not a Kantian--

Illuminating comments. The tentative conclusion that I have reached so far reading the comments is this: Creativity provides the potential but like any other potential it should be realized by means of a systematic procedure and hardwork. We need both to build strong theories.  

Well, there is facticity and correspondence with what appears to be reality. But is a theory always completely and rationally driven by this causation? How does one explain competing theories? I am not sure I am consistently Kantian, but I like the challenge of trying to distinguish between the role my mind plays in my understanding and the actual event I am trying to understand. I can be puzzled by wondering if I have unwittingly relied upon an inference or if the imaginary structure I develop that "fits the facts" is useful.

Then of course, there are some facts about language we ignore when we believe that we can turn our perceptions into language and not lose something in the conversion. We might try mightily to limit our terms to specific meanings, to create that which baffles synonymy, but then we have certified correspondence between our terms and things and defeated all attempts to make meaning of the language. Marklar marklar marklar marklar marklar...

I am thinking of Christopher Michael Langan's website "The theory of Theories." He ends up where a lot of us do, at the paradox in mathematics between language and result in the set-of-all-sets-that-do-not-contain-themselves. We are not dealing with empirical science here alone, but particularly with the logos. In language we can state a condition under which there must be conflicting answers. But then one wonders if the question of quantum entanglement isn't simply the means by which language describes nature, not as it is but as it must grammatically be.

I think theory gets bogged down way too much is rationalism: concepts not tied to reality--so that theory building becomes a word game detached from the world--we can thank Plato for this unfortunate mess-

But Plato was a philosopher!!  Not a scientist or theorist  And an early one at that-- I happen to enjoy /learn a lot from Plato.

Kantians, platonians ... philosophy from former times is attractive, especially to understand how human thinking evolved, but it will not solve epistemological problems of contemporary social science. I for my part like the following quote from H.S. Becker: "[...] my favorite way of developing concepts is in a continuous dialogue with empirical data. Since concepts are ways of summarizing data, it's important that they be adapted to the data you're going to summarize." To maintain this constant dialogue, we should not loosen the ties to reality (as Edwin said).

Dear Charles, I am no ..ian of any kind and enjoy philosophy PERIOD. 

Without, " philosophy from former times," we'd miss out  on a lot of knowledge.  Again, philosophy is philosophy and it does not aim to or have the purpose  to, "solve epistemological problems of contemporary social science."  One of  my favorite quotes, from my dear RIP mother, is "a place for everything and everything in its place.

Shall I conclude that you are a logical positivist?

P.S.  I was merely responding to the put down of Plato that he is responsible for muddled thinking/writing.  Sorry for digressing from the question.

But is there theory without rationality, Edwin? Theory predicts something, an event or repetition because of the meaning made of observation. Some great theories are pretty counter intuitive and cannot be simply and directly inducted from observation. I have a theory that I will feel better when I stop hitting myself in the head with this hammer (or voting for fiscal austerity). I am right, but not because of the data.

Some answers to several people: 1. Why I am opposed to Plato: because he divorced concepts from the senses(reality)--he did not believe the senses were valid--this leaves concepts floating in space and also makes them arbitrary- I believe Aristotle had the right approach though he was not perfect; 2. How does philosophy help social science? one way is by providing a valid theory of concepts. 3. Re Scott: you can form hypotheses about anything out of the blue but an hypothesis is not a validated theory--to validate a theory you need facts.

To this extent?

“There are in science three fundamentally different kinds of reasoning… “

“These three kinds of reasoning are Abduction, Induction, and Deduction. [—] All the ideas of science come to it by the way of Abduction. Abduction consists in studying facts and devising a theory to explain them. Its only justification is that if we are ever to understand things at all, it must be in that way. “

“Now, in an inquiry concerning a hypothesis in general, three distinct stages have to be recognized, these stages being governed by entirely different logical principles. The first stage consists in the invention, selection, and entertainment of the hypothesis. This I call the abduction. “

~ http://www.commens.org/dictionary/term/abduction

“When theorists build theory, they design, conduct, and interpret imaginary (imagined, JR) experiments.  In doing so, their activities resemble the three processes of evolution: variation, selection, and retention (reproduction, a continuation, JR). 

Because the theorist rather than nature intentionally guides the evolutionary process, theorizing is more like artificial selection than natural selection, and theorizing becomes more like natural selection the more the process is dominated by validation and empiricism.

The close parallel between theory building and evolutionary processes can be illustrated with the example of…(phi spiral abduction?  JR)”

~Karl E. Weick, “Theory Construction as Disciplined Imagination”

"Theory predicts something, an event or repetition because of the meaning made of observation.."  Observation(s) can lead / be used to develop theory.  If theory predicted anything, there would be no need for research.

"Every inquiry whatsoever takes its rise in the observation…of some surprising phenomenon, some experience which either disappoints an expectation, or breaks in upon some habit of expectation of the inquisiturus…The inquiry begins with pondering these phenomena in all their aspects… At length a conjecture arises that furnishes a possible Explanation, by which I mean a syllogism exhibiting the surprising fact as necessarily consequent upon the circumstances of its occurrence together with the truth of the credible conjecture, as premisses. On account of this Explanation, the inquirer is led to regard his conjecture, or hypothesis, with favour. As I phrase it, he provisionally holds it to be "Plausible";...

The whole series of mental performances between the notice of the wonderful phenomenon and the acceptance of the hypothesis,…the search for pertinent circumstances and the laying hold of them,… the scrutiny of them, the dark labouring, the bursting out of the startling conjecture, the remarking of its smooth fitting to the anomaly, as it is turned back and forth like a key in a lock, and the final estimation of its Plausibility, I reckon as composing the First Stage of Inquiry. Its characteristic formula of reasoning I term Retroduction, i.e. reasoning from consequent to antecedent."

~Peirce, A Neglected Argument for the Reality of God

Compare with phi spiral abduction:

Observation→  hypothesis →, experiment/research→ results = hypothesis accepted or rejected.  This is science not religious faith in god/no god.

oh the irony...

:)

Really, has no one heard of Peircean abduction, abductive inference, the First Stage of Inquiry, A Neglected Argument?

https://www.researchgate.net/post/What_is_the_best_example_of_Peircean_abduction

“

‘The Scientific imagination dreams of explanations and laws.’  CP 1.48

“Charles Sanders Peirce was a restless and passionate man.  His greatest passion was philosophy, especially that branch known as logic. 

A person of immense intellectual power, Peirce devoted much of his thinking life to the analysis of reason itself.  He was not content to use reason to dissect narrow sub-problems in philosophy, so he turned to his own mind in order to understand how reason itself functions; and his contributions in this area place him among the great thinkers in the history of philosophy. 

The origin of new ideas and scientific hypotheses was always of particular interest to him, and he seems to have had a life-long fascination with the question of genius and creativity.  The analysis of genius and creativity was not to be left to the psychologists, however…Producing a harmony of creativity and logic was one of Peirce’s underlying philosophic tasks-and in order to do so he had to reformulate logic itself in a radical way...”

~Kathleen Hull, “Why Hanker After Logic?  Mathematical Imagination, Creativity and Perception in Peirce’s Systematic Philosophy”

"Though neglected for much of the twentieth century, his thought is enjoying a renaissance…Of the myriad concepts that make up the constellation of Peirce’s thought, there is one that is of special interest to many researchers in cognitive science and artificial intelligence, namely, his notion of hypothesis formation or what he called abduction. 

As the available literature attests, the AI community is interested in abduction in hopes that the process can be formalized and implemented in a wide array of intelligent or expert systems (….).  Although the idea is intriguing and merits research, I have found that most discussions of abduction in the AI context betray little or no understanding of Peirce’s thought as a whole.  The abductive process is considered in isolation from the general architecture of Peirce’s philosophical vision…

The best way to understand Peirce’s philosophy is by first understanding how its diverse components are structured and interrelated…."

~Costa, “Can Creativity be Formalized? Peircean Reflections on the Role of Abduction in Human Intelligence”

In "Creativity: Surprise and abductive reasoning", Gonzalez & Haselager explore directly this conflict between "mechanical and non-mechanical" ingredients of creativity by building a more precise notion of creativity, and going so far as to place importance of order parameters as discriminators for goodness/completeness. 

"We have selected the experience of surprise as a possible candidate to help us address the polemic ‘mechanistic versus non- mechanistic’ debate on creativity...

Thus, it seems that, according to Peirce, not only the experience of surprise... constitutes the first step of abductive reasoning underlying creative thinking, but also that its absence should be a characteristic of a good (completed) abduction."  ~Gonzalez & Haselager 

There's lots of good stuff out there on Peirce's contributions to formalizing creativity, which makes spreading the intended philosophy coherent.