Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submission: rtweet #302

Closed
15 of 25 tasks
mkearney opened this issue May 17, 2019 · 27 comments
Closed
15 of 25 tasks

Submission: rtweet #302

mkearney opened this issue May 17, 2019 · 27 comments

Comments

@mkearney
Copy link

mkearney commented May 17, 2019

Submitting Author: Michael W. Kearney (@mkearney)
Repository: https://github.com/mkearney/rtweet
Version submitted: v0.6.9
Editor: @sckott
Reviewer 1: @andrewheiss
Reviewer 2: @briatte
Archive: TBD
Version accepted: TBD


  • Paste the full DESCRIPTION file inside a code block below:
Package: rtweet
Type: Package
Version: 0.6.9
Title: Collecting Twitter Data
Authors@R: c(
    person("Michael W.", "Kearney", ,
    email = "kearneymw@missouri.edu", role = c("aut", "cre"),
    comment = c(ORCID = "0000-0002-0730-4694"))
    ## add contributor template (middle name/initial optional)
    #person("First Middle", "Last", ,
    #email = "email@address.com", role = c("ctb"))
    )
Description: An implementation of calls designed to collect and
    organize Twitter data via Twitter's REST and stream Application
    Program Interfaces (API), which can be found at the following URL:
    <https://developer.twitter.com/en/docs>.
Depends:
    R (>= 3.1.0)
Imports:
    httr (>= 1.3.0),
    jsonlite (>= 0.9.22),
    magrittr (>= 1.5.0),
    tibble (>= 1.3.4),
    utils,
    progress,
    Rcpp,
    httpuv
License: MIT + file LICENSE
URL: https://CRAN.R-project.org/package=rtweet
BugReports: https://github.com/mkearney/rtweet/issues
Encoding: UTF-8
Suggests:
    ggplot2,
    knitr,
    magick,
    openssl,
    readr,
    rmarkdown,
    testthat (>= 2.1.0),
    webshot,
    covr,
    igraph
VignetteBuilder: knitr
LazyData: yes
RoxygenNote: 6.1.1
LinkingTo: 
    Rcpp

Scope

  • Please indicate which category or categories from our package fit policies this package falls under: (Please check an appropriate box below. If you are unsure, we suggest you make a pre-submission inquiry.):

    • data retrieval
    • data extraction
    • database access
    • data munging
    • data deposition
    • reproducibility
    • geospatial data
    • text analysis
  • Explain how and why the package falls under these categories (briefly, 1-2 sentences):

Data retrieval because the package allows users to easily request and import data from Twitter's REST and stream APIs.

Data munging because significant work is done to convert JSON objects returned by Twitter's APIs into tabular data frames.

  • Who is the target audience and what are scientific applications of this package?

The target audience is researchers. The scientific applications span a range of topics. Here's is the relevant excerpt from paper.md

Although rtweet provides some coverage to user context-behaviors (e.g.,
posting statuses, liking tweets, following users, etc.), the primary audience
for the package to date has been researchers. Accordingly, rtweet has been
featured in numerous popular press [e.g.,
@bajak2019democrats; @machlis2019r; @riley2019twitter] and academic publications
[e.g.,
@bossetta2018simulated; @bradley2019major; @buscema2018media;
@erlandsen2018twitter; @gitto2019brand; @kearney2019analyzing;
@kearney2018analyzing; @li2018sentiment; @lutkenhaus2019tailoring;
@lutkenhaus2019mapping; @molyneux2018media; @tsoi2018can; @unsihuay2018topic;
@valls2017urban; @wu2018finding].

To date no other package interfaces with both REST and stream APIs. The twitteR package is most similar, but it has entered a stage of deprecation (I've agreed to carry the torch, so to speak). So, not only does twitteR not reflect some recent changes to Twitter's API (most notably the introduction of 'extended tweet' mod–the new 280 character limit), but it lacks active maintenance thanks–in part–to rtweet filling the void.

  • If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

#193

Technical checks

Confirm each of the following by checking the box. This package:

Publication options

JOSS Options
  • The package has an obvious research application according to JOSS's definition.
    • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
    • The package is deposited in a long-term repository with the DOI:
    • (Do not submit your package separately to JOSS)
MEE Options
  • The package is novel and will be of interest to the broad readership of the journal.
  • The manuscript describing the package is no longer than 3000 words.
  • You intend to archive the code for the package in a long-term repository which meets the requirements of the journal (see MEE's Policy on Publishing Code)
  • (Scope: Do consider MEE's Aims and Scope for your manuscript. We make no guarantee that your manuscript will be within MEE scope.)
  • (Although not required, we strongly recommend having a full manuscript prepared when you submit here.)
  • (Please do not submit your package separately to Methods in Ecology and Evolution)

Code of conduct

@briatte
Copy link

briatte commented May 18, 2019

As I said in openjournals/joss-reviews#1454 -- I'll happily review the package here. I've reviewed for JOSS (alongside @andrewheiss, in fact), and would gladly review for @ropensci :-)

@mkearney
Copy link
Author

mkearney commented May 21, 2019

I tentatively listed @andrewheiss and @briatte as reviewers because they initially expressed their willingness to review on a [now closed] JOSS thread. I don't want to presume they are/will be the official reviewers–I'm not completely familiar with the ropensci process, but I'd imagine the selection of reviewers isn't entirely up to me–but I figured reviewer suggestions/volunteers were worth knowing!

@sckott
Copy link
Contributor

sckott commented May 28, 2019

thanks for the reviewer suggestions, much appreciated.

@sckott
Copy link
Contributor

sckott commented May 28, 2019

Editor checks:

  • Fit: The package meets criteria for fit and overlap
  • Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
  • License: The package has a CRAN or OSI accepted license
  • Repository: The repository link resolves correctly
  • Archive (JOSS only, may be post-review): The repository DOI resolves correctly
  • Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

Editor comments

Thanks for your submission @mkearney !

Here's the output from goodpractice. If you haven't used goodpractice it's an R package that checks a number of things with another package - most of which we agree with and want authors to follow. You don't need to address these now, it's info for reviewers to use to get started.

  • Regarding tests: all tests fail, because I don't have the auth set up. How do you suggest one gets set up to run your test suite? I clearly need a twitter_tokens file. How do I get that file? and where does it go? Seems like a good thing to go in a CONTRIBUTING.md file or so - but maybe there's a good reason not to add that?
── GP rtweet ─────

It is good practice toavoid long code lines, it is bad for readability. Also, many people prefer editor windows that are about 80 characters wide. Try make your lines
    shorter than 80 characters

    R/bearer_token.R:56:1
    R/bearer_token.R:80:1
    R/coords.R:119:1
    R/coords.R:131:1
    R/coords.R:132:1
    ... and 37 more linesavoid sapply(), it is not type safe. It might return a vector, or a list, depending on the input data. Consider using vapply() instead.

    R/favorites.R:75:45
    R/post.R:326:12
    R/post.R:336:12
    R/stream.R:338:13
    R/utils.R:327:40
    ... and 1 more linesavoid the library() and require() functions, they change the global search path. If you need to use other packages, import them. If you need to
    load them explicitly, then consider loadNamespace() instead, or as a last resort, declare them as 'Depends' dependencies.

    R/utils.R:485:5fix this R CMD check NOTE: Note: found 113868 marked UTF-8 stringschecking tests ...  SEE QUESTION ABOVE

Seeking reviewers now 🕐


Reviewers:

@briatte
Copy link

briatte commented May 29, 2019

Here's version 1 version 2 of my review, with comments shown as list sub-items. I'll add the functionality tests soon.

Package Review (version 2, final)

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

  • As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

  • A statement of need clearly stating problems the software is designed to solve and its target audience in README
  • Installation instructions: for the development version of package and any non-standard dependencies in README
  • Vignette(s) demonstrating major functionality that runs successfully locally
    • This step used to require Twitter API credentials, but does not anymore.
  • Function Documentation: for all exported functions in R help
  • Examples for all exported functions in R Help that run successfully locally
  • Community guidelines including contribution guidelines in the README or CONTRIBUTING, and DESCRIPTION with URL, BugReports and Maintainer (which may be autogenerated via Authors@R).
    • The package does not have a CONTRIBUTING file, but it does have the URL and BugReports fields in its DESCRIPTION. This seems sufficient to me.
For packages co-submitting to JOSS

The package contains a paper.md matching JOSS's requirements with:

  • A short summary describing the high-level functionality of the software
  • Authors: A list of authors with their affiliations
  • A statement of need clearly stating problems the software is designed to solve and its target audience.
  • References: with DOIs for all those that have one (e.g. papers, datasets, software).
  • The .bib file is clean enough, but could be cleaned further (some DOI fields come as URLs, other as just DOIs).

Functionality

  • Installation: Installation succeeds as documented.
  • Functionality: Any functional claims of the software been confirmed.
  • Performance: Any performance claims of the software been confirmed.
  • Automated tests: Unit tests cover essential functions of the package
    and a reasonable range of inputs and conditions. All tests pass on the local machine.
  • Packaging guidelines: The package conforms to the rOpenSci packaging guidelines

Final approval (post-review)

  • The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: 0.5 (draft version 1)

  • Should the author(s) deem it appropriate, I agree to be acknowledged as a package reviewer ("rev" role) in the package DESCRIPTION file.

Review Comments

Reviewed using R version 3.6.0 on x86_64-apple-darwin15.6.0 (64-bit).

Installed with

devtools::install_github("mkearney/rtweet", dependencies = TRUE)

Tested with both the "out of the box" access to the Twitter API and personal OAuth credentials via create_token -- both worked as expected, although I ran into exactly the same list of issues as @andrewheiss when testing the package with testthat::auto_test_package().

Typo at https://rtweet.info/articles/auth.html -- "autheticate via web browser".

I have only one request beyond what Andrew is suggesting in terms of better documenting how to fix the tests (excellent table suggestion, by the way, Andrew!):

Could the thread about how rwteet compares to other packages also document how the package compares to RTwitterAPI?

I have used both twitteR and RTwitterAPI in the past: in this project, we collected a large amount of Twitter followers, and RTwitterAPI was, back in the days and in our experience, quicker and less error-prone than twitteR. I hope rtweet compares favourably re: RTwitterAPI in that respect.

Last, something not related to the package itself: since Twitter has sunset apps.twitter.com, old tokens created there do not seem to work properly -- for instance, get_followers("famous_person", n = 5000, retryonratelimit = TRUE) will not go beyond 5000 followers. Users need to go through the new vetting process at developer.twitter.com.

@andrewheiss
Copy link

andrewheiss commented Jun 5, 2019

Package Review

Please check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide

  • As the reviewer I confirm that there are no conflicts of interest for me to review this work (If you are unsure whether you are in conflict, please speak to your editor before starting your review).

Documentation

The package includes all the following forms of documentation:

  • A statement of need clearly stating problems the software is designed to solve and its target audience in README
  • Installation instructions: for the development version of package and any non-standard dependencies in README
  • Vignette(s) demonstrating major functionality that runs successfully locally
  • Function Documentation: for all exported functions in R help
  • Examples for all exported functions in R Help that run successfully locally
  • Community guidelines including contribution guidelines in the README or CONTRIBUTING, and DESCRIPTION with URL, BugReports and Maintainer (which may be autogenerated via Authors@R).
For packages co-submitting to JOSS

The package contains a paper.md matching JOSS's requirements with:

  • A short summary describing the high-level functionality of the software
  • Authors: A list of authors with their affiliations
  • A statement of need clearly stating problems the software is designed to solve and its target audience.
  • References: with DOIs for all those that have one (e.g. papers, datasets, software).

Functionality

  • Installation: Installation succeeds as documented.
  • Functionality: Any functional claims of the software been confirmed.
  • Performance: Any performance claims of the software been confirmed.
  • Automated tests: Unit tests cover essential functions of the package
    and a reasonable range of inputs and conditions. All tests pass on the local machine.
  • Packaging guidelines: The package conforms to the rOpenSci packaging guidelines

Final approval (post-review)

  • The author has responded to my review and made changes to my satisfaction. I recommend approving this package.

Estimated hours spent reviewing: ≈5 hours

  • Should the author(s) deem it appropriate, I agree to be acknowledged as a package reviewer ("rev" role) in the package DESCRIPTION file.

Review Comments

Documentation and vignettes

All vignettes worked great and were a perfect introduction to the package. It might be helpful to move up the vignette listing near the beginning of the README so that users can find them more easily.

Each function is generally very well documented. There are some sparsely documented functions like bearer_token() and invalidate_bearer() that seem to only be used internally—it might be useful to either add @keywords internal to the roxygen block or add additional documentation explaining what a bearer token is and including some examples.

It's really helpful that the US and the world are baked into lookup_coords() (e.g. lookup_coords("usa")) so users don't need an API key for that. This should probably be documented, though. It caught me by surprise that "usa" was working just fine and all other locations triggered a message—I only figured out why because I looked at the source code for lookup_coords(), and that has a huge all caps note about needing a Google Maps API key. (A future expansion could maybe include hardcoded boundaries for a bunch of other countries (perhaps make an internal lookup table of lat/long coordinates?) so the package is less US-centric and less dependent on another API.)

The pkgdown site is great. It might be helpful (and recommended by rOpenSci) to add @family tags to roxygen documentation so that there are automatic groups of functions in the documentation and in the reference website.

Installation and testing

Installation worked fine with devtools::install_github("mkearney/rtweet", dependencies = TRUE)

Testing doesn't work out of the box and it took me a while to figure out how to generate the /tests/testthat/twitter_tokens RDS file. When logged in through the embedded rstats2twitter app, running this created an auth token that the test suite can use:

saveRDS(get_token(), "tests/testthat/twitter_tokens")

It might be useful to explain that process somewhere in the documentation—not in the README though, since it's not something a typical user would do.

190 tests passed (super impressive coverage!!), and 5 tests failed:

  • direct_messages: This is because the rstats2twitter app doesn't provide read-write DM access. It worked when I used my own auth token

  • get_my_timeline: One test failed because rtweet:::home_user() not equal to "kearneymw". Mike's account is hardcoded in the tests and this test can't pass if someone is using a different account. I don't know what else could be done to test it though.

    test_get_my_timeline.R:6: failure: get_my_timeline
    rtweet:::home_user() not equal to "kearneymw".
    1/1 mismatches
    x[1]: "andrewheiss"
    y[1]: "kearneymw"
    
  • lookup_coords: This returns an error that seems unrelated to token issues and I don't know while it's failing

    test_lookup_coords.R:36: error: lookup_coords returns coords data
    all arguments must be named
    1: do.call(Sys.setenv, ng) at /Users/andrew/Downloads/rtweet-master/tests/testthat/test_lookup_coords.R:36
    2: (function (...) 
       {
           x <- list(...)
           nm <- names(x)
           if (is.null(nm) || "" %in% nm) 
               stop("all arguments must be named")
           .Internal(Sys.setenv(nm, as.character(unlist(x))))
       })()
    3: stop("all arguments must be named")
    ──────────────────────────────────────────
    
  • tweet_shot: This failed because I didn't have PhantomJS installed. The test worked after installing it. Running tweet_shot() interactively gives a note that it should be installed with webshot::install_phantomjs(), but this message is invisible when testing on a fresh system. PhantomJS isn't installed as a dependency when the package in installed, but that's because it's not an R package. idk what the best way to handle this is, though, since it's 16+ MB and not handled automatically with R ¯\_(ツ)_/¯

  • test-test_username-r: Like get_my_timeline, this failed because Mike's account is hardcoded in. Again, I don't know what the best way to handle this is.

    test_username.R:7: failure: test authenticating user name
    `sn` not equal to "kearneymw".
    1/1 mismatches
    x[1]: "andrewheiss"
    y[1]: "kearneymw"
    

Token creation and authentication

It's so great that people can use this without creating their own Twitter apps, since that process has become long and arduous.

It might be helpful to more clearly explain the benefits of creating an app with tokens vs. just using the interactive browser-based token. Right now, the README says "You may still choose to do this [create an app] (gives you more stability and permissions)", and the auth vignette explains how to do it, but there isn't any explanation of why users might want to do it. Later in the README, under "Post actions," it explains that users would need their own app to post tweets and access DMs, but it could be helpful to have that information up above when describing authentication in general.

It might even be helpful to have a table of sorts showing pros/cons/features available when using a private app vs. the embedded rstats2twitter app, like this maybe?

Task rstats2twitter application Personal application
Work interactively
Create reproducible research
Search tweets
Stream tweets
Get friends
Get timelines
Post tweets
Run package tests

The page for creating apps looks slightly different from the screenshots, and the direction to go to apps.twitter.com is out of date—according to Twitter, that aspect of developer accounts is getting sunset. Instead, the URL is now https://developer.twitter.com/en/apps .

Community

It would be helpful to have some community guidelines in the package too, such as a CONTRIBUTING.md file (perhaps modeled after something like this) and a CONDUCT.md file (like this)

Creating an RStudio project .Rproj file could be helpful for encouraging package contributions, since it would enable RStudio's package building interface.

rOpenSci-related issues

JOSS-related issues

There are a few issues with paper.bib:

  • "Twitter as a tool of para-disploomacy" is misspelled
  • Several of the cited articles that have actual DOIs are missing DOIs (like "How can we better use Twitter to find a person who got lost due to dementia?" is https://dx.doi.org/10.1038/s41746-018-0017-5)

Kicking the tires with some use cases

Look at tweets

library(tidyverse)
library(rtweet)

# Check out my tweets
my_tweets <- get_timeline("andrewheiss", n = 500)

head(my_tweets)
#> # A tibble: 6 x 90
#>   user_id status_id created_at          screen_name text  source display_text_wi… reply_to_status…
#>   <chr>   <chr>     <dttm>              <chr>       <chr> <chr>             <dbl> <chr>           
#> 1 167410… 11360038… 2019-06-04 20:16:30 andrewheiss @dva… Twitt…                0 113600361493217…
#> 2 167410… 11359881… 2019-06-04 19:14:09 andrewheiss I ME… Twitt…                6 113598739853218…
#> 3 167410… 11359873… 2019-06-04 19:11:16 andrewheiss Revi… Twitt…               94 NA              
#> 4 167410… 11359848… 2019-06-04 19:01:06 andrewheiss "boo… Twitt…              171 113424196604718…
#> 5 167410… 11359781… 2019-06-04 18:34:42 andrewheiss oh m… Twitt…              140 NA              
#> 6 167410… 11359750… 2019-06-04 18:22:13 andrewheiss As a… Twitt…              140 NA              

ts_plot(my_tweets)

image

Geocoding with sf

library(sf)
library(rnaturalearth)

searched_tweets <- search_tweets(
  q = "lang:en",
  geocode = lookup_coords("usa"), n = 10000
) %>% 
  lat_lng() %>% 
  drop_na(lng) %>% 
  sf::st_as_sf(coords = c("lng", "lat"), crs = 4326) 

us <- ne_states(country = "united states of america", returnclass = "sf") %>% 
  filter(!(postal %in% c("HI", "AK", "PR")))

ggplot() +
  geom_sf(data = us) +
  geom_sf(data = searched_tweets) + 
  coord_sf(crs = 102003, datum = NA) +  # Albers
  theme_void()

image


Magical. This is a great package—fantastic work!


Tested and reviewed using R 3.6.0 on x86_64-apple-darwin15.6.0 (64-bit).

@sckott
Copy link
Contributor

sckott commented Jun 5, 2019

thanks for your review @andrewheiss - @briatte let me know when you're done reviewing

@briatte
Copy link

briatte commented Jun 25, 2019

Hi @sckott

I have updated my draft review, which can be considered final. My new comments are at the bottom of the review. In a nutshell, I hit the same issues as @andrewheiss did with the tests, and am asking for additional details to be included about how the package compares to RTwitterAPI.

Generally speaking, the package is awesome, thanks a lot for your work @mkearney!

@sckott
Copy link
Contributor

sckott commented Jun 25, 2019

thanks @briatte

@mkearney follow up here with responses to reviewers

@briatte
Copy link

briatte commented Jun 26, 2019

FYI, I've just posted an issue to rtweet that has to do with getting followers for accounts with over 5,000 followers: https://github.com/mkearney/rtweet/issues/340 (update: issue now closed, thanks!)

Perhaps this issue could lead to better documentation re: how to collect large amounts of followers/friends? As the issue says, I am unsure that I have understood how get_followers(retryonratelimit = TRUE) works.

Last and still related to followers, I subscribe to the suggestion made here to harmonize the output of get_followers and get_friends: https://github.com/mkearney/rtweet/issues/308

@sckott sckott added the package label Aug 29, 2019
@mkearney
Copy link
Author

mkearney commented Sep 5, 2019

@sckott:

First, thanks to @briatte and @andrewheiss for taking the time to review rtweet and providing such great feedback!

Second, please see my detailed reply to each reviewer below. This revision process has so far resulted in a lot of changes to rtweet–many of which were quite difficult😅–but I am quite certain the package is much better for it.

Thank,

-Mike

@briatte

Should the author(s) deem it appropriate, I agree to be acknowledged as a package reviewer ("rev" role) in the package DESCRIPTION file.

  • Added @briatte as reviewer role to DESCRIPTION

Tested with both the "out of the box" access to the Twitter API and personal OAuth credentials via create_token -- both worked as expected, although I ran into exactly the same list of issues as @andrewheiss when testing the package with testthat::auto_test_package().

  • This is addressed in my reply to @andrewwheis below

Typo at https://rtweet.info/articles/auth.html -- "autheticate via web browser".

  • Fixed typo (autheticate to authetincate) in auth vignette.

Could the thread about how rwteet compares to other packages also document how the package compares to RTwitterAPI?

  • Adding a package comparison table to README. Note: The RTwitterAPI package is outdated and very bare bones, but while it does technically offer a way to hit most endpoints it doesn't do much to automate that process, parse the response, or document endpoints besides a 'get friends' example. And, thus, I've used the red question mark on the relevant GET API endpoints for RTwitterAPI.
Task rtweet twitteR twitteR RTwitterAPI
Available on CRAN
Updated since 2016
Non-'developer' access
Extended tweets (280 chars)
Parses JSON data
Converts to data frames
Automated pagination
Search tweets
Search users
Stream sample
Stream keywords
Stream users
Get friends
Get timelines
Get mentions
Get favorites
Get trends
Get list members
Get list memberships
Get list statuses
Get list subscribers
Get list subscriptions
Get list users
Lookup collections
Lookup friendships
Lookup statuses
Lookup users
Get retweeters
Get retweets
Post tweets
Post favorite
Post follow
Post messsage
Post mute
Premium 30 day
Premium full archive
Premium 30 day
Run package tests

@andrewwheiss

Documentation and vignettes

Should the author(s) deem it appropriate, I agree to be acknowledged as a package reviewer ("rev" role) in the package DESCRIPTION file.

  • Added @andrewwheis as reviewer role to DESCRIPTION

All vignettes worked great and were a perfect introduction to the package. It might be helpful to move up the vignette listing near the beginning of the README so that users can find them more easily.

  • Moved vignettes section up to above key functions (and just after install/API authorization)

Each function is generally very well documented. There are some sparsely documented functions like bearer_token() and invalidate_bearer() that seem to only be used internally—it might be useful to either add @Keywords internal to the roxygen block or add additional documentation explaining what a bearer token is and including some examples.

  • Added @ keywords internal to invalidate_bearer()
  • Improved name/description, added more information about requirements and usage, and several examples to bearer_token()

It's really helpful that the US and the world are baked into lookup_coords() (e.g. lookup_coords("usa")) so users don't need an API key for that. This should probably be documented, though. It caught me by surprise that "usa" was working just fine and all other locations triggered a message—I only figured out why because I looked at the source code for lookup_coords(), and that has a huge all caps note about needing a Google Maps API key. (A future expansion could maybe include hardcoded boundaries for a bunch of other countries (perhaps make an internal lookup table of lat/long coordinates?) so the package is less US-centric and less dependent on another API.)

  • Document US/world coordinates in lookup_coords(). This can be found in the details section of lookup_coords()
  • Add other (less US-centric) defaults to lookup_coords(). Latitude and longitudes for major world cities are now hardwired in. This is documented in the details section of lookup_coords(). And I've added new automated tests for these as well.

The pkgdown site is great. It might be helpful (and recommended by rOpenSci) to add @family tags to roxygen documentation so that there are automatic groups of functions in the documentation and in the reference website.

  • Add @family tags for reference/function documentation grouping

Installation and testing

Testing doesn't work out of the box and it took me a while to figure out how to generate the /tests/testthat/twitter_tokens RDS file. When logged in through the embedded rstats2twitter app, running this created an auth token that the test suite can use...It might be useful to explain that process somewhere in the documentation—not in the README though, since it's not something a typical user would do.

  • Added description of configuration/setup of test token. I've created a 'Github action' for this–so forking {rtweet} should trigger a message about automated testing and working with encrypted keys on travis-ci

(FAILED) direct_messages: This is because the rstats2twitter app doesn't provide read-write DM access. It worked when I used my own auth token

  • Testing now checks for appropriate permissions.
  • As for the limited permissions, this was a concsious decision on my part. I limited write and DM-related permissions in tokens generated via rstats2twitter because I don't want to encourage or be associated with automated/spam/TOS-violating behaviors.

(FAILED) get_my_timeline: One test failed because rtweet:::home_user() not equal to "kearneymw". Mike's account is hardcoded in the tests and this test can't pass if someone is using a different account. I don't know what else could be done to test it though.

  • Agreed. This may have been possible back when I had a dedicated rtweet Twitter account, but that got shut down by Twitter (I learned you can't use multiple accounts/tokens to do the same thing the hard way), so I'm hesitant/careful not to further hurt my standing with them.

(FAILED) lookup_coords: This returns an error that seems unrelated to token issues and I don't know while it's failing

  • Edited test to check if an environment variable named GOOGLE exists and optionally replaces with empty for following test.

(FAILED) tweet_shot: This failed because I didn't have PhantomJS installed. The test worked after installing it. Running tweet_shot() interactively gives a note that it should be installed with webshot::install_phantomjs(), but this message is invisible when testing on a fresh system. PhantomJS isn't installed as a dependency when the package in installed, but that's because it's not an R package. idk what the best way to handle this is, though, since it's 16+ MB and not handled automatically with R ¯_(ツ)_/¯

  • The webshot dependencies are especially difficult to configure in travis integrations. Since webshot is recommended and not required, I've changed the test to first check whether the webshot package is installed.

(FAILED) test-test_username-r: Like get_my_timeline, this failed because Mike's account is hardcoded in. Again, I don't know what the best way to handle this is.

  • This may have been possible back when I had a dedicated rtweet Twitter account, but that got shut down by Twitter (I learned you can't use multiple accounts/tokens to do the same thing the hard way), so I'm hesitant/careful not to further hurt my standing with them.

Token creation and authentication

It might be helpful to more clearly explain the benefits of creating an app with tokens vs. just using the interactive browser-based token. Right now, the README says "You may still choose to do this [create an app] (gives you more stability and permissions)", and the auth vignette explains how to do it, but there isn't any explanation of why users might want to do it. Later in the README, under "Post actions," it explains that users would need their own app to post tweets and access DMs, but it could be helpful to have that information up above when describing authentication in general.
It might even be helpful to have a table of sorts showing pros/cons/features available when using a private app vs. the embedded rstats2twitter app, like this maybe?

Task rstats2twitter user-app
Work interactively
Search/lookup tweets/users
Get friends/followers
Get timelines/favorites
Get lists/collections
Post tweets
Run package tests
Use Bearer token
Read/Write Direct Messages
  • Futher developed comparison table for embedded app versus user-created apps (now included in README) and added more description of pros/cons of creating a token in documentation.

The page for creating apps looks slightly different from the screenshots, and the direction to go to apps.twitter.com is out of date—according to Twitter, that aspect of developer accounts is getting sunset. Instead, the URL is now https://developer.twitter.com/en/apps .

  • I've updated the auth and FAQ vignettes to reflect the new developer portal. I've also updated the steps, labels, and screen shots in the auth vignette.

Community

It would be helpful to have some community guidelines in the package too, such as a CONTRIBUTING.md file (perhaps modeled after something like this) and a CONDUCT.md file (like this)

Creating an RStudio project .Rproj file could be helpful for encouraging package contributions, since it would enable RStudio's package building interface.

rOpenSci-related issues

* [Following rOpenSci's package guidelines](https://ropensci.github.io/dev_guide/building.html#creating-metadata-for-your-package), it might be helpful to use the **codemetar** package to generate a `codemeta.json` file.
* Consider adding a repostatus.org badge too ([following rOpenSci's guidelines](https://ropensci.github.io/dev_guide/building.html#readme))
  • Added repostatus badge!
* Also add the rOpenSci peer review badge
  • Added rOpenSci badge!

JOSS-related issues

There are a few issues with paper.bib:

* "Twitter as a tool of para-disploomacy" is misspelled
  • Fixed to 'para-diplomacy'
* Several of the cited articles that have actual DOIs are missing DOIs (like "How can we better use Twitter to find a person who got lost due to dementia?" is https://dx.doi.org/10.1038/s41746-018-0017-5)
  • Added all the DOIs I could find (including the one for the dementia paper)

@sckott
Copy link
Contributor

sckott commented Sep 6, 2019

thx for your responses to reviews @mkearney

any feedback/thoughts on the comments reviewers ? (@briatte @andrewheiss )

@andrewheiss
Copy link

Other than a duplicate "Premium 30 day" row in the big comparison table, this addresses all my concerns and I think it's good to go ✅

@stefaniebutland
Copy link
Member

@mkearney since you're keen on contributing a blog post once the review process is complete 🙂 ...
This tag gets you (diverse) examples of blog posts by authors of peer-reviewed packages: https://ropensci.org/tags/software-peer-review/.

Here are some technical and editorial guidelines: https://github.com/ropensci/roweb2#contributing-a-blog-post. Publication date is flexible. I like to get a draft via pull request a week before the planned publication date so I can review.

Happy to answer any questions. I'm thrilled that you've put rtweet through review.

@sckott
Copy link
Contributor

sckott commented Oct 9, 2019

@briatte Are you happy with the changes made?

@mkearney
Copy link
Author

Any update on this?

@sckott
Copy link
Contributor

sckott commented Oct 16, 2019

@mkearney let's move on. I'll take a final look and get back to you ASAP

@sckott
Copy link
Contributor

sckott commented Oct 17, 2019

looks good, just a single comment: ^codemeta\.json$ should be in .Rbuildignore

Approved! Thanks @mkearney for submitting and @briatte @andrewheiss for your reviews!

To-dos:

  • Transfer the repo to rOpenSci's "ropensci" GitHub organization under "Settings" in your repo. I have invited you to a team that should allow you to do so. You'll be made admin once you do.
  • Add the rOpenSci footer to the bottom of your README
[![ropensci_footer](https://ropensci.org/public_images/ropensci_footer.png)](https://ropensci.org)
  • Fix all links to the GitHub repo to point to the repo under the ropensci organization.
  • for the pkgdown site, we usually have people point to https://docs.ropensci.org/package_name but it looks like you have your own domain name https://rtweet.info/ , and I imagine you'd prefer it live there
  • Add a mention of the review in DESCRIPTION via rodev::add_ro_desc().
  • Fix any links in badges for CI and coverage to point to the ropensci URL. We no longer transfer Appveyor projects to ropensci Appveyor account so after transfer of your repo to rOpenSci's "ropensci" GitHub organization the badge should be [![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/github/ropensci/pkgname?branch=master&svg=true)](https://ci.appveyor.com/project/individualaccount/pkgname).
  • We're starting to roll out software metadata files to all ropensci packages via the Codemeta initiative, see https://github.com/ropensci/codemetar/#codemetar for how to include it in your package, after installing the package - should be easy as running codemetar::write_codemeta() in the root of your package.

For JOSS:

  • Activate Zenodo watching the repo
  • Tag and create a release so as to create a Zenodo version and DOI
  • Submit to JOSS at https://joss.theoj.org/papers/new, using the rOpenSci GitHub repo URL. When a JOSS "PRE REVIEW" issue is generated for your paper, add the comment: This package has been reviewed by rOpenSci: https://LINK.TO/THE/REVIEW/ISSUE, @ropensci/editors

We've put together an online book with our best practice and tips, this chapter starts the 3d section that's about guidance for after onboarding. Please tell us what could be improved, the corresponding repo is here.

@mkearney
Copy link
Author

@sckott I’m not currently seeing an invite (and I wasn’t authorized to transfer to ropensci). Where would I find this?

@briatte
Copy link

briatte commented Oct 20, 2019

@briatte Are you happy with the changes made?

Yes! With apologies for not signalling it earlier.

@mkearney Thanks again for a brilliant package. I can report it's currently in use in various research projects in Paris and Lille, France :-)

@sckott
Copy link
Contributor

sckott commented Oct 21, 2019

@mkearney sorry, now you should have gotten an invite ... let me know if you don't see it soon

@mkearney
Copy link
Author

Got it! Thanks!

@mkearney
Copy link
Author

mkearney commented Oct 21, 2019

TO DO for rOpenSci:

  • Transfer the repo to rOpenSci's "ropensci" GitHub organization under "Settings" in your repo. I have invited you to a team that should allow you to do so. You'll be made admin once you do.
  • Add the rOpenSci footer to the bottom of your README
  • Fix all links to the GitHub repo to point to the repo under the ropensci organization.
  • for the pkgdown site, we usually have people point to https://docs.ropensci.org/package_name but it looks like you have your own domain name https://rtweet.info/ , and I imagine you'd prefer it live there
    • Yes, I'd like to keep the rtweet.info URL, but I updated the pkgdown site
  • Add a mention of the review in DESCRIPTION via rodev::add_ro_desc().
  • Fix any links in badges for CI and coverage to point to the ropensci URL. We no longer transfer Appveyor projects to ropensci Appveyor account so after transfer of your repo to rOpenSci's "ropensci" GitHub organization the badge should be [![AppVeyor Build Status](https://ci.appveyor.com/api/projects/status/github/ropensci/pkgname?branch=master&svg=true)](https://ci.appveyor.com/project/individualaccount/pkgname).
  • We're starting to roll out software metadata files to all ropensci packages via the Codemeta initiative, see https://github.com/ropensci/codemetar/#codemetar for how to include it in your package, after installing the package - should be easy as running codemetar::write_codemeta() in the root of your package.

TODO for JOSS:

  • Activate Zenodo watching the repo
  • Tag and create a release so as to create a Zenodo version and DOI
  • Submit to JOSS at https://joss.theoj.org/papers/new, using the rOpenSci GitHub repo URL.
  • When a JOSS "PRE REVIEW" issue is generated for your paper, add the comment: This package has been reviewed by rOpenSci: https://LINK.TO/THE/REVIEW/ISSUE, @ropensci/editors

We've put together an online book with our best practice and tips, this chapter starts the 3d section that's about guidance for after onboarding. Please tell us what could be improved, the corresponding repo is here.

@stefaniebutland
Copy link
Member

@mkearney When would you like to publish the post? Tues Nov 12 is earliest date available, and that would mean submitting a draft via pull request by Tues Nov 5. Later dates are also open.

Details #302 (comment)

@jeroen
Copy link
Member

jeroen commented Oct 23, 2019

Docs are now live on https://docs.ropensci.org/rtweet/

@mkearney
Copy link
Author

@stefaniebutland Nov 5th/12th works for me!

@stefaniebutland
Copy link
Member

@mkearney Will you submit a draft post soon?
Technical and editorial guidelines: https://github.com/ropensci/roweb2#contributing-a-blog-post

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants