Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AlchemiscalClient async+bulk for results, other methods; add request, response compression for large objects #150

Merged
merged 50 commits into from
Jun 29, 2023
Merged

AlchemiscalClient async+bulk for results, other methods; add request, response compression for large objects #150

merged 50 commits into from
Jun 29, 2023

Commits on Jun 17, 2023

  1. Small optimization to Neo4jStore.get_task_status

    @dotsdl
    dotsdl committed Jun 17, 2023
    Configuration menu
    Copy the full SHA
    6d98952 View commit details
    Browse the repository at this point in the history

  2. AlchemiscaleClient get_tasks_status and set_tasks_status single… 

    … request
    @dotsdl
    dotsdl committed Jun 17, 2023
    Configuration menu
    Copy the full SHA
    703dd71 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2023

  1. Working async/await implementation of get_tasks_status 

    Need to implement the `set_tasks_status` async/await version next
    @dotsdl
    dotsdl committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    1ce1de9 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2023

  1. Merge branch 'main' into issue-126-task-status-async

    @dotsdl
    dotsdl authored Jun 22, 2023
    Configuration menu
    Copy the full SHA
    1686ca3 View commit details
    Browse the repository at this point in the history

  2. Black!

    @dotsdl
    dotsdl committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    e8f859f View commit details
    Browse the repository at this point in the history

  3. May have optimized get_task_status for bulk usage

    @dotsdl
    dotsdl committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    443470a View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into issue-126-task-status-single

    @dotsdl
    dotsdl committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    19b8bd0 View commit details
    Browse the repository at this point in the history

  5. Faster implementation, single query for get_task_status

    @dotsdl
    dotsdl committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    66cbfa8 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2023

  1. Think I've got token refreshes working with async 

    Using/abusing an async lock for this.
    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    e535729 View commit details
    Browse the repository at this point in the history

  2. Black!

    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    287326b View commit details
    Browse the repository at this point in the history

  3. Merge branch 'issue-126-task-status-single' into user-async-batch

    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    2bf4971 View commit details
    Browse the repository at this point in the history

  4. Hybrid approach complete for get_tasks_status 

    We now hit the bulk API endpoint with async using batches.
Get 45s locally for 10,000 tasks.
    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    491d2a4 View commit details
    Browse the repository at this point in the history

  5. Small edit

    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    d80dfce View commit details
    Browse the repository at this point in the history

  6. Black!

    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    c3f8d90 View commit details
    Browse the repository at this point in the history

  7. Switched to using requests for synchronous HTTP via clients 

    We're seeing what looks like weird performance issues using `httpx` for
synchronous requests vs. `requests`. Sticking with `requests` for
synchronous, `httpx` for async for now.
    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    86b8417 View commit details
    Browse the repository at this point in the history

  8. Allow string form of Task scoped keys in get_tasks_status

    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    31675bb View commit details
    Browse the repository at this point in the history

  9. Async implementation for result retrieval via AlchemiscaleClient.

    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    2ed8c0a View commit details
    Browse the repository at this point in the history

  10. Black!

    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    e2c0307 View commit details
    Browse the repository at this point in the history

  11. Synchronous request token back to using requests 

    For consistency.
    @dotsdl
    dotsdl committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    d9e580b View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2023

  1. Only want solvent transformations computed for speed of test suite

    @dotsdl
    dotsdl committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    97a962d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into user-async-batch

    @dotsdl
    dotsdl authored Jun 24, 2023
    Configuration menu
    Copy the full SHA
    d9b4278 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into user-async-batch

    @dotsdl
    dotsdl authored Jun 24, 2023
    Configuration menu
    Copy the full SHA
    7860c27 View commit details
    Browse the repository at this point in the history

  4. Nondeterministic test fixes

    @dotsdl
    dotsdl committed Jun 24, 2023
    Configuration menu
    Copy the full SHA
    39732e9 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into user-async-batch

    @dotsdl
    dotsdl authored Jun 24, 2023
    Configuration menu
    Copy the full SHA
    5d99f26 View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into user-async-batch

    @dotsdl
    dotsdl authored Jun 24, 2023
    Configuration menu
    Copy the full SHA
    6180b03 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. Added verify kwarg to AlchemiscaleBaseClient. 

    Allows use of client with self-signed certs on API services.
    @dotsdl
    dotsdl committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    48e7d06 View commit details
    Browse the repository at this point in the history

  2. Adding compression of AlchemicalNetworks on `AlchemiscaleClient.creat… 

    …e_network`
    @dotsdl
    dotsdl committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    0332b80 View commit details
    Browse the repository at this point in the history

  3. Black!

    @dotsdl
    dotsdl committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    8ea236d View commit details
    Browse the repository at this point in the history

  4. Believe we now have gzip compression working correctly

    @dotsdl
    dotsdl committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    153d8ea View commit details
    Browse the repository at this point in the history

  5. Black!

    @dotsdl
    dotsdl committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    c0dede2 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2023

  1. Now create_network resilient against really short expiry on tokens 

    Short expiry like we have in the test suite, not real life :P
    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    4471046 View commit details
    Browse the repository at this point in the history

  2. Black!

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    d414ff3 View commit details
    Browse the repository at this point in the history

  3. Added ability to set compression for AlchemiscaleClient.create_network

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    7e33c47 View commit details
    Browse the repository at this point in the history

  4. Black!

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    e012ad4 View commit details
    Browse the repository at this point in the history

  5. Added compression of responses for certain AlchemiscaleClient.get c… 

    …alls

Now by default compress retrievals of AlchemicalNetwork, Transformation,
    and ChemicalSystem.

Also compress retrieval of ProtocolDAGResults.
    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    8786f09 View commit details
    Browse the repository at this point in the history

  6. Black!

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    2f83850 View commit details
    Browse the repository at this point in the history

  7. Added compress control to all result/failure AlchemiscaleClient methods 

    Also, added `rich`-based progress bar to result retrieval.
    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    fae60ff View commit details
    Browse the repository at this point in the history

  8. Black!

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    9ca8356 View commit details
    Browse the repository at this point in the history

  9. Added more progress output to various methods on AlchemiscaleClient 

    Also made set_tasks_status work as async/batch, same as get_tasks_status
    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    3864393 View commit details
    Browse the repository at this point in the history

  10. Black!

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    23f499b View commit details
    Browse the repository at this point in the history

  11. Added exposure of verify kwarg to compute service

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    3b85619 View commit details
    Browse the repository at this point in the history

  12. Black

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    6c51997 View commit details
    Browse the repository at this point in the history

  13. Close #133

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    c1d9e9a View commit details
    Browse the repository at this point in the history

  14. Black

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    685c411 View commit details
    Browse the repository at this point in the history

  15. Added finer grained control over compression for create_network

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    1a49f2d View commit details
    Browse the repository at this point in the history

  16. Fixes from @hmacdope review

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    9169b73 View commit details
    Browse the repository at this point in the history

  17. Add nest_asyncio for AlchemiscaleClient cases run in envs like Jupyter 

    Also, add scope-based ordering to query outputs.
    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    42c1a23 View commit details
    Browse the repository at this point in the history

  18. Black

    @dotsdl
    dotsdl committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    41b2239 View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2023

  1. set_tasks_status methods now run in single query for many Tasks 

    We use the same patterns we applied for `get_tasks_status`.
    @dotsdl
    dotsdl committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    5d6a09f View commit details
    Browse the repository at this point in the history

  2. Refinements to terminal set_task_status methods

    @dotsdl
    dotsdl committed Jun 29, 2023
    Configuration menu
    Copy the full SHA
    f70a6c9 View commit details
    Browse the repository at this point in the history