Not Retrieving All of a User's Timeline When Under 3200 Tweets

[pravin-santhanam27]

Hello,

I am running the following loop to scrape around 60 user's timelines and expecting to get 3200 tweets for each user if they have more than 3200, or the maximum number of tweets they have if its less than 3200. The code for the loop is below:

print("Querying All Player Tweets")
tweetQuery <- vector("list",n)

for (i in seq_along(tweetQuery)) {
tweetQuery[[i]] <- get_timeline(twitterHandles[i,]$Handle, n = 3200)
## assuming full rate limit at start, wait for fresh reset every 52 users
if (i %% 52L == 0L) {
rl <- rate_limit("get_timeline")
Sys.sleep(as.numeric(rl$reset, "secs"))
}
## print update message
cat(i, " ")
}

tweetQuery <- do_call_rbind(tweetQuery)

However, for some guys that have less than 3200 tweets, it does not pull all of their tweets. For example, an account that has 352 tweets, the loop above for that user returned only 328. Is there a reason for this? And does it have to do with looping through many users and the rate limit being exceeded mid-loop?

Thanks for your help.

[pravin-santhanam27]

I think the reason for the discrepency is that the twitter user_timeline function does not get replies, but only tweets and retweets. On the twitter website, the tweet count includes replies, so there can be many more than what is returned from the rtweet get_timeline function. Can anyone confirm this? I want to make sure I am getting as many tweets as possible under the 3200 limit.

[sadettindemirel]

I came across with a similar problem. Normally I set the get_timeline() function with the n = 3200 but I can only get fewer number of tweets. In the past I mean like 5 months ago I can reach at least over 3000 tweets. This might happen because of a change in Twitter API Policy or limits.

[hadley]

Now tracking in #510