I want to monitor the status of the China Mainland lottery industry, and wonder if it is possible to develop a based-on-big-data system that shows what is happening in the field automatically. My so-called big data means information from public websites. Who can give me some idea or help?
Look up any resource on "web scraping," such as:
RCurl for R: http://tinyurl.com/kkvtdr8 or www.omegahat.org/RCurl/
SCrapy for Python: http://tinyurl.com/76ts4u
It's a bit difficult, given that your code may be broken when the websites change their own code! I hope this helps.
I believe this is a very difficult task that you have given yourself - good luck!
thank Mr. Kelly very much for your information. It seems you opened a new window for me. I don't understand R, but I would try to get rough picture of it.
thank Ms Issa! I know what you mean. you are so kind. I have a small team, but all of us are freshers in the field. we are learning!
You might also look at web crawlers/spiders. eg http://sourceforge.net/projects/openwebspider/
I'd be interested in what you end up doing Albert. I plan a study looking at postings on Tumblr. You have given me some ideas about automating it using something like this.
- Badges
- Science method
- Similar topics
- Methodology
- Research Methodology
- Qualitative Research