Importing bookmarked tweets

[esanfar]

I looked for a function to import own account bookmarks and I couldn't find anything. Bookmarking tweets was not universally available, but it is now, and it would be great to be able to import tweets saved in it.

[Arf9999]

There doesn't seem to be a Twitter API endpoint for bookmarks (yet?) https://developer.twitter.com/en/docs/api-reference-index

[llrs]

Still not available. Apparently it is something that twitter don't want to expose. Closing the issue, if they make it available open an issue again please

[tchakravarty]

Hi @llrs

This seems to now be available: https://twittercommunity.com/t/build-with-bookmarks-on-the-twitter-api-v2/168804.

It would be great if rtweet exposed a get_bookmarks function based on this.

Thanks.

[llrs]

Thanks for the head up.
As it is with API v2 it will need to wait till release rtweet > 1.0.0 as next release is to prepare for API v2.

[rbcavanaugh]

Any chance of some movement on this? Would love to be able to pull all of my bookmarks before twitter completely implodes. Thanks!

[Arf9999]

If you pull your archive from Twitter via the web site, you should be able to parse the bookmarks and lookup the statuses via rtweet. Not a single step, but should deliver the results that you need.

…

________________________________ From: Rob Cavanaugh ***@***.***> Sent: Friday, November 11, 2022 2:51:43 PM To: ropensci/rtweet ***@***.***> Cc: Andrew Fraser ***@***.***>; Comment ***@***.***> Subject: Re: [ropensci/rtweet] Importing bookmarked tweets (#344) Any chance of some movement on this? Would love to be able to pull all of my bookmarks before twitter completely implodes. Thanks! — Reply to this email directly, view it on GitHub<#344 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIY5MIK5DUBFYW7CATN6UETWHZMP7ANCNFSM4IFNPLWA>. You are receiving this because you commented.Message ID: ***@***.***>

[llrs]

@rbcavanaugh Yes, I am working on supporting API v2 but I had started by the streaming API which I thought would stop working last week.

The initial support for API v2 is already present in the devel branch but it is still not yet fully functional to be released (it doesn't have support for extensions and fields yet). If you want you can start a PR and I can help with it (you are not the first person to ask for it).

[AmeliaMN]

I got my twitter archive, but it doesn't seem to contain bookmarked tweets. I poked through the directory structure looking for anything that looked related, but couldn't find anything. If someone has figured out what file they are in, I'd love to know!

[llrs]

Mmh, yes I checked my own archive and it didn't even contain a reference to saved or bookmarked tweets. Sorry, I'll up this in the list of endpoints to provide support soon. However, the API requires a different authentication method for this endpoint, so existing code for API v2 doesn't work (I hoped it would).

[llrs]

I add here an app to retrieve all bookmarked tweets https://twitter-bookmarks-exporter.herokuapp.com/. It provides with a webpage with links to all the bookmarked tweets (and contents referenced there). Note it might take a long time to retrieve all the content.
Thanks @adriaaula for mentioning it.

[Lextuga007]

I've just used the app to get 289 links and it was really fast! Thanks for building and sharing @llrs

[deborahapthorp]

Thanks, but I couldn't get that app to work at all - is there a newer version?

[mpettis]

I get an Application Error on the herokuapp above, to confirm what I think @deborahapthorp is saying as well.

[llrs]

Good news! I think I figured out the problem with the authentication method required for this (and others endpoints). It needs more testing and I need to understand more about it: if it will work with the current default token provided by rtweet (it doesn't seem to), and why it fails after some time using it (I think this is Twitter's fault as other tokens that did work also stopped working).
But in the devel branch (version 1.1.0.9002) there is a function to retrieve them! Hopefully next release (If I'm lucky next month) you will be able to download them.

However, the endpoint only returns the "Allows you to get an authenticated user's 800 most recent bookmarked Tweets." So if we'll get all or not (while testing it seemed at one point like it could continue to the next page above the 800 tweets).

[llrs]

Given that Twitter won't be free to download bookmarks next week (See announcement: https://twitter.com/TwitterDev/status/1621026986784337922) I recommend to extract the downloads with the code as is:

devtools::install_github("ropensci/rtweet@devel")
library("rtweet")
client <- rtweet:::rtweet_client() # You'll need to provide the id and secret of your app.
client_as(client)
token_oa2 <- rtweet:::rtweet_oauth2() 
ub <- rtweet:::user_bookmarks("123456789", parse = FALSE, n = Inf, token = token_oa2)
saveRDS(ub, "my_bookmarks.RDS")

The authentication mechanism might not work, if it does, it will be for only 2 hours.
I'm reconsidering the future of the package and my investment in it, but I'll keep monitoring and post more as the changes develop.

[adriaaula]

Thanks a ton for the code snippet Lluis!
It's disappointing to say the least, and it adds up more pressure to go somewhere else, let's hope I have enough pressure to go now haha

[Lextuga007]

Yes, echo that, thanks for the tip off that it's going to charge soon. Much appreciated!

[bassamsdata]

The authentication mechanism might not work, if it does, it will be for only 2 hours.
I'm reconsidering the future of the package and my investment in it, but I'll keep monitoring and post more as the changes develop.

Thank you so much for all the support and effort you put in this package.

unfortunately, The message always appears when I try to do anything, the error message appears when token_oa2 <- rtweet:::rtweet_oauth2() even though I provided the id and secret.

is there a way to export the bookmarks 🫣. note: I used to authenticate using the default in the browser.

[mpettis]

@llrs I completely understand your question to further support of this package. I appreciate your work to this point, and will just say my use case is to just extract meaningful history from Twitter before I stop using it altogether. Thanks for the work I've used!

[bassamsdata]

unfortunately, The message always appears when I try to do anything, the error message appears when token_oa2 <- rtweet:::rtweet_oauth2() even though I provided the id and secret.

I just want to update on my comment that after removing the s from https://127.0.0.1:1410/ I wan able to authenticate yet another problem appeared when doing ub <- rtweet:::user_bookmarks("123456789", parse = FALSE, n = Inf, token = token_oa2)

I have the last verion of httr2 and the last version of rtweet@devel.

Thanks a lot

[llrs]

@bassamsdata Did you replace 123456789 by your own Twitter id?

[bassamsdata]

@bassamsdata Did you replace 123456789 by your own Twitter id?

yea absolutely :)

[llrs]

I'll need more information to try to help you. What is the token_oa2 output? What is the client output? What did you try?

[bassamsdata]

Thanks for you help, I'm just trying to download my book marks. they are around 500 to 600.

token_oa2 output:

rtweet:::rtweet_client() output with client id and client secret

no output for this rtweet:::client_as(client)

output for this token_oa2 <- rtweet:::rtweet_oauth2()

so I added client manually token_oa2 <- rtweet:::rtweet_oauth2(client = client)
and the output is to open the browser and authorize. I did authorize,

and checked the app inside the twitter settings.

then when I try to download my bookmarks, it shows this error:

just to note: if i did something like this useRs <- search_users("#tidytuesday", n = 100) it works.

[llrs]

@bassamsdata Please next time post text instead of images, it is easier to search and copy to the terminal.
It would be helpful to know the id of your client (and without the secret I cannot do anything). It would help me verify if you used the default client or not. If you used the default client it won't work.
The search_users used a different token for the requests. It wasn't used to make the requests, you should have used search_users("#tidytuesday", n = 100, token = token_oa2). The OAuth2 token is only valid in some endpoints of the the API v2 not in the v1.

[bassamsdata]

Please next time post text instead of images, it is easier to search and copy to the terminal.

My bad, sorry.

It would be helpful to know the id of your client

client <- rtweet:::rtweet_client(client_id = "dWJSZXljQ2NSQkNlNFFlc3lNZm86MTpjaQ",
                                 client_secret = )
<httr2_oauth_client>
name: rtweet
id: dWJSZXljQ2NSQkNlNFFlc3lNZm86MTpjaQ
secret: <REDACTED>
token_url: https://api.twitter.com/2/oauth2/token
auth: oauth_client_req_auth_header

If you used the default client it won't work.

I used the twitter provided client.

The search_users used a different token for the requests

> search_users("#tidytuesday", n = 100, token = token_oa2)
Error in `check_token()`:
! `token` is not a valid access token
Run `rlang::last_error()` to see where the error occurred.

the output of client and token_oa2

> client
<httr2_oauth_client>
name: rtweet
id: dWJSZXljQ2NSQkNlNFFlc3lNZm86MTpjaQ
secret: <REDACTED>
token_url: https://api.twitter.com/2/oauth2/token
auth: oauth_client_req_auth_header

> token_oa2
<httr2_token>
token_type: bearer
access_token: <REDACTED>
expires_at: 2023-02-04 12:59:07
refresh_token: <REDACTED>
scope: mute.write tweet.moderate.write block.read follows.read offline.access list.write bookmark.read list.read tweet.write space.read
block.write like.write like.read users.read tweet.read bookmark.write mute.read follows.write

here is the output of the user_bookmarks

> ub <- rtweet:::user_bookmarks("bassamstreet", parse = FALSE, n = Inf, token = token_oa2)
Error in `resp_abort()`:
! HTTP 401 Unauthorized.
Run `rlang::last_error()` to see where the error occurred.

[llrs]

I haven't heard complains from any other users (after last update of the sample code), I don't know for sure what might be the cause. It could be that your app is not authorized for API v2 and you might need to refresh the authorization or select a different type of APP in the developer.twitter.com menu, but using http://127.0.0.1:1410/ for confirming the app is fixed in rtweet code.

[bassamsdata]

I haven't heard complains from any other users (after last update of the sample code), I don't know for sure what might be the cause. It could be that your app is not authorized for API v2 and you might need to refresh the authorization or select a different type of APP in the developer.twitter.com menu, but using http://127.0.0.1:1410/ for confirming the app is fixed in rtweet code.

Thank you so much for your efforts to help me here, I really appreciated. It's probably an issue with my API. Thanks again

[llrs]

@bassamsdata Rechecking your code, I realize now that you were using you twitter name, not your id with numbers. I've added a check to detect those kind of errors.

I'm closing the issue now, as I've made user_bookmarks to be exported (for next release).

devtools::install_github("ropensci/rtweet@devel")
library("rtweet")
client <- rtweet_client() # You'll need to provide the id and secret of your app.
client_as(client)
token_oa2 <- rtweet_oauth2() 
ub <- user_bookmarks("123456789", 
                     expansion = NA, fields = NA, # To export everything: alt text, images, urls, ...
                     parse = FALSE, n = Inf, token = token_oa2)
saveRDS(ub, "my_bookmarks.RDS")