You can use 

REST API https://dev.twitter.com/rest/public and Streaming API https://dev.twitter.com/streaming/overview

e.g. twitteR, RTwitterAPI

See also blog posts on Twitter data on R-bloggers: http://www.r-bloggers.com/search/twitter

http://www.r-bloggers.com/search/twitter

have you tried NodeXL? http://nodexl.codeplex.com/

You can download your own file going to your Settings -> "Request your archive" button. And you'll get to your first tweet :-)

@Chee Siang Ang: I didn't try.  I am using R packages.

@ Francesc grau: Thank you for your reply. I want to download some other tweets not my own tweets.

You can't access to file of others by this way :-(

@ Francesc Grau: Can you tell me, is there a way to do it?

You can try https://www.exporttweet.com or http://www.tweetdownload.net

I used them long time ago and both were well. Nowadays, I don't know. Sorry

@ Francesc Grau: Thanks  a lot.

You can use 

REST API https://dev.twitter.com/rest/public and Streaming API https://dev.twitter.com/streaming/overview

e.g. twitteR, RTwitterAPI

See also blog posts on Twitter data on R-bloggers: http://www.r-bloggers.com/search/twitter

http://www.r-bloggers.com/search/twitter

You probably want to connect to the public streams. There are libraries for many languages: https://dev.twitter.com/overview/api/twitter-libraries

So far I've used the Perl Net::Twitter successfully.

Depending on your setup (i.e. account/keywords you pass to the API to look for) you may not get many tweets at the beginning, and so you will need to be connected to the stream for quite some time,

Depending on what you are after, NodeXL can be a good package. Does not require programming skills, it's simply a plugin for Excel. And the guys behind it are helpful if you get stuck.

you can use putty and hive to download tweets if you are using twitter Api...That was the apps I used to extract the tweets

You can have a look at this link.

Mathematica allows you to download and manage a lot of Social Media data, including twitter.

http://reference.wolfram.com/language/ref/SocialMediaData.html

Hope it is useful.

Mariano.

Thank you all for your kind reply.

@ Marius: Is  NODE XL useful for doing sentiment analysis?.  I tried NODE XL, it is useful for doing network analysis. I want to use tweets for doing sentimental analysis.

@ Udeh: Is puuty and hive are mobile apps? 

@ Mariano:  I will try this. 

you can install an oracle VM as the server and then use putty to connect to it. You need some queries to extract the tweets  in hive and then add some json file for transforming the data before using it for sentiment analysis.

@ Udeh: Thanks a lot. If you have any material related to putty, can you please share with me. 

Try TWICTION, which is connected to a Twitter Account and realized (what I know) via a Twitter/streaming API.

For further information have a look at: www.econob.com

@ Gunther Fliedl: Thank you for your reply. Could you please explain how it helps to my study?

@venkata I'm not sure if NodeXL is the best tool for sentiment analysis. I've only used it for SNA.

@ Marius: Thank you.

Some good suggestions above, but you may also want to take a look at Chorus Tweetcatcher (TCD) for a simple GUI data collection tool. You can run queries and download user-specific timelines. It can also be used in conjunction with a prototype visual analytics tool called Tweetvis which provides summary statistics and various visualizations to help you explore your datasets.

Chorus is free to download and use: http://chorusanalytics.co.uk

where can i download the movie reviews where users have given opinions using rating. I tried using rotten tomatoes. But its not working. I want it for Sentiment Analysis. Please any body can help me. 

Also suggest me if there are other product sites that allow users to download opinions of rating on movies or any other product.

i suggest you use R package TwitteR (for backward search) or StreamR (for forward search). I found this package is easy to use and we can crawl up to 1500 on one go. i attach the appearance of this TwitteR package 

http://cran.r-project.org/web/packages/twitteR/index.html

@ Andry: Thank you for your reply. I have used twitteR package in R software. But i am able to download only 3 tweets. I want minimum 1000 tweets.

@Vankata if you dont specify anything by default twitter search will get 25 last tweet from your search. This is the default :

searchTwitter(searchString, n=25, lang=NULL, since=NULL, until=NULL, locale=NULL, geocode=NULL, sinceID=NULL, retryOnRateLimit=120, ...)

the more variable you put above makes the search is more specific and better accuracy. i am able to get more accurate result by putting for example n=1000, since(date) or sinceID (twitter ID) or geocode ..

@Andy: Thank you for your reply. I am specifying n=1000 still it is giving only 3 tweets.

@Venkata i forgot one thing the API can not retrieve tweet older than 7 days. other than that i never had any problem to crawl twitter data. by the way you can give me one of your keyword and i will check from my end if i experience the same problem.

@Andry: Ya i agree. API will not provide more than 7days tweets. My keyword is stayzilla. Please tell me how many you are able to download.

Do you know programming in R? I will soon publish a project on data visualization in R. So you can use the analysis to download tweets

@ Udeh: ya i know R programming.

ok so you can also download tweets using R, then save it and use it for your project

@udeh: I am using twitteR package to downlaod tweets.  I am able to download only 5 tweets , i want at least 1000 tweets. Do you know how to download? 

> library(twitteR)

> library(wordcloud)

> library(RColorBrewer)

> library(plyr)

> library(ggplot2)

> library(sentiment)

# Find OAuth settings for twitter:

> library(httr)

> oauth_endpoints(“twitter”)

##

## request: https://api.twitter.com/oauth/request_token

## authorize: https://api.twitter.com/oauth/authenticate

## access: https://api.twitter.com/oauth/access_token

# Register an application (API) at https://apps.twitter.com/

# Once done registering, look at the values of api key, secret and token

# Insert these values below:

> api_key api_secret access_token access_token_secret setup_twitter_oauth(api_key,api_secret,access_token,access_token_secret)

## [1] “Using direct authentication”

# Now let us collect some tweets (2000 in our example) containing the term '#UDEH_BIGDATA' from twitter (language = English,

# if you wish you can set other languages to fetch tweets in those languages in your analytics)

>  tweets = searchTwitter('#UDEH_BIGDATA', n=2000, lang=”en”)

You should specify the programming language you are using. Anyway, a list of libraries to interact with the Twitter API is available at https://dev.twitter.com/overview/api/twitter-libraries.

Depending on your needs, you could use:

• the REST API (https://dev.twitter.com/rest/public), if you want to execute searches to retrieve tweets matching some specified criteria, or
• the STREAMING API (https://dev.twitter.com/streaming/overview), if you need to monitor some keywords or account (https://dev.twitter.com/streaming/reference/post/statuses/filter) or if you want to sample the whole twitter stream (https://dev.twitter.com/streaming/reference/get/statuses/sample).

If you are using Java as programming language, consider that Twitter4J (http://twitter4j.org) is, probably, the best library to interact with Twitter. It is also well-documented.  

@Venkata sorry for late reply over the weekend ..

your keyword appearance just to few in the twitter, i am only able to gather 184 data when specify n = 1000, i will try on another computer today ..

@  Udeh: Sorry for the late reply. I am using the similar code in R. Still i can download only 6 tweets.

@ Marco: Thank you very much for your reply. I am using R Software. I will try.

@ Andry: Thank you very much. It has 2176 tweets. Let me know how many you are able to download.

Do you have to code for that?

I mean: there are ready made software for this. You have to check Deen Freelon list : https://docs.google.com/document/d/1UaERzROI986HqcwrBDLaqGG8X_lYwctj6ek6ryqDOiQ/edit

This list is curated by Deen but collected in contribution of a lot of internet researchers' chat through the AOIR list ([email protected]). "Proudly" I have contributed too in this :)

I also suggest you begin following this list for there are many helpful guys there !

Good luck!

@Noha: Thank you for your reply. Definitely i will look into this.

You have to!

Good luck :)

thank you all for your support. Not able to thank all specifically, sorry for that.

really it helped me. 

Hi, I'm in a similar situation. I need historical twitter data for social network analysis. What would be the advantages/disadvantages of scraping tweets from Twitter's advanced search and downloading data from the Internet Archive respectively? Both in terms of coverage/completeness of data set and in terms of the tools and/or programming skills I would need. Thanks.

@Lone Sorensen

It seems to me that to obtain full historical data from Twitter is not possible. Please see this paper http://www.sciencedirect.com/science/article/pii/S0378873314000057 for a discussion of biases of different methods of data collection.

You can learn the procedures by following the steps in this project https://www.researchgate.net/publication/275947996_BIG_DATA_STATISTICS_WITH_R?ev=prf_pub

Conference Paper BIG DATA STATISTICS WITH R

@Dominik - thanks for the article! I gather that I can't use the API as it won't provide me with historical data (I need to go further back than one week). Hence I'm wondering what the bias is of using Twitter's advanced search, compared to the firehose and/or API. I've seen studies comparing the API and the firehose like the one you posted but nothing about the web search itself. Have you heard of any such studies? And am I right in guessing that the Internet Archive is based on the streaming API?

@Udeh - thanks, this will be really useful as a next step!

@ Udeh: thank you very much for you R code. That function perfectly!.  mayble if someone doesn´t know how to get the api and tocken codes could get it in https://apps.twitter.com .

The Software MAXQDA 12 provides an Interface for selective downloads.

Hi,

You can get twitter data from Podargos Data. All you have to do is to tell them your needs.

Podargos can provide twitter historical and realtime data. In additional, it's able to help you retrieve other mobile App data of social networks and e-commerce, and supports for nearly 100 languages. I hope this information can help you.

Podargos' website is http://www.podargos.com

Here is the data sample: 

http://goo.gl/dRxyow

I would also suggest that you check: twitteR

Hi venkat....I am also facing the same problem. But i have a solution to download data from facebook using netvizz app. But Now a days i can't download data from FB.

Any one can help me how to download fb comments using netvizz or some other easiest way and also i need .gdf file to import in Gephi.

Please guide me.

Hi, You can scrap data using "Search Twitter" operator in RapidMiner. With this operator you can specify query and get Twitter statuses containing this query.

with R Programming

Hi, 

I'm a researcher from University of Melbourne in Australia. The group I work in is scraping Twitter data to analyse aspects of emotion control. After many months we are now collecting tweets and user timelines in a mongoDB database using docker, pymongo and tweepy. Currently have a dataset of 20 million tweets from 8000 individual users and it's growing daily.

Happy to share information as I think helping other researchers will help us.

[email protected]

If you are looking for information older than a week, a current limitation of the Twitter API, you can crawl the advanced search with the script in the link below.

http://tomkdickinson.co.uk/2016/12/extracting-a-larger-twitter-dataset/

Hi, you can get twitter data from Podargos Data. It can provide twitter historical and realtime data. In additional, it's able to help you retrieve other mobile App data of social networks and e-commerce, and supports for nearly 100 languages. I hope this information can help you.

Pordargos' website is http://www.podargos.com.

Here is the data sample: 

http://goo.gl/dRxyow

Realtime data collection using the Twitter Search API and the Gnip PowerTrack 2.0 is available using a point and click (no programming) interface at https://discovertext.com. Here are the top 10 reasons to use DiscoverText:

https://vimeo.com/170157685

For three free historical Twitter data estimates per day, create a free account on Sifter (https://sifter.texifter.com).

For more information about "Gnip" PowerTrack data, please refer to:

https://discovertext.com/gnip-enabled-access-for-discovertext-users/

I used this tool a month ago to download twitter data, am not sure if the free trial would let you export 30,000 rows at a time,. Feel free to check it out though https://www.exporttweet.com/

I used KNIME (https://www.knime.com/) to monitor and download Twitter data daily.

If you try Twitter API, you'll face 7 day limitation. Now Twitter offers a Premium API for bulk requests. Still I don't understand why they don't offer any option for educational researches. But still you can still try another options like trimming data from Twitter search services. There's a good option here, even if you don't know Python, this tool will help you to get bulk tweets and also you can get different time spans: https://github.com/Jefferson-Henrique/GetOldTweets-python

And also follow our project SentiTweet: https://www.sentitweet.com

streaming twitter API using python language the package of twepy

Have you considered using a social media monitoring tool?

There are many you could test, and I'd advise using a paid tool. Free ones usually lack features that you might actually find the most valuable, whether it's analytics or data visualisation.

As far as my experience goes, the tool I've spent quite a big chunk of my time with is Brand24 (https://brand24.com). It's a media monitoring tool that allows for monitoring both social and traditional web and then exporting the data to .xls files, infographics or .pdf reports.

Coming back to crawling Twitter data, it's very effective due to the access to Twitter's API. However, media monitoring tools are able to collect only those mentions that are publicly available and don't have access restrictions assigned to them. On top of that, historical data is limited, so the tool is the most accurate from the moment of creating the monitoring project onwards.

As far as other tools are concerned, you might also want to have a look at Brandwatch(https://brandwatch.com) or Sysomos (https://sysomos.com)

Hope you'll find this answer helpful!

Thank you for your contribution. I read with great interest. I am now able to download the tweets as described in some links. Now, I would like to build a database of tweets to be continuously enriched so as to overcome the constraint of the 18,000 tweets and dey 7 days free of the free API. Do you have any advice for me?

You can use this:

https://github.com/Jefferson-Henrique/GetOldTweets-python

Hi, if you want to download historical tweets related to any search term then you can fetch it from Twitter API for past 7 days.

As per the description provided by you, it looks like that you had already tried that method and received just 3 tweets in past 7 days.

Hence, if you want to go further than 7 days then you will either need to purchase the data directly from Twitter to use its Premium API or purchase it from third party data providers like TrackMyHashtag.

Here is the link to submit your query at TrackMyHashtag: https://www.trackmyhashtag.com/historical-twitter-data

Mr. Venkata Prasad Palakiti

At the end, which website or software did you use to download Twitter data?

Python and R. Between the two I am more familiar with R but I am moving towards Python: for my uses of artificial intelligence it is more powerful.

Good Answer Abdullah Önden

You can use Twint. It is an advanced Twitter scraping tool written in Python that allows for scraping Tweets from Twitter profiles without using Twitter's API.

Some of the benefits of using Twint vs Twitter API:

• Can fetch almost all Tweets (Twitter API limits to last 3200 Tweets only);
• Fast initial setup;
• Can be used anonymously and without Twitter sign up;
• No rate limitations.

Reference: https://github.com/twintproject/twint

You ca use scrapy.