Anthropic prompt caching support #25625
mrdrprofuroboros
started this conversation in
Ideas
Replies: 2 comments
-
implemented in #27087 |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Checked
Feature request
Anthropic has recently added prompt caching with its own special pricing and cache write/hit token counts
https://www.anthropic.com/news/prompt-caching
Apparently this feature will start popping up in other providers. Spinning up this thread to ask if there are any plans on integrating it into langchain standard API
Motivation
I'm using langfuse for token/cost monitoring and right now I have to make quite ugly workarounds to use claud's prompt cache: https://github.com/orgs/langfuse/discussions/2987
I think there are a lot of devs that would want to use anthropic prompt caching within langchain infra
Proposal (If applicable)
Right now anthropic requires marking one of the messages as a prefix end to cache tools-system-messages until this flag. not sure if openai has any work in that direction. this whole thing is exposed through anthropic beta api (https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) A simple starting step might be inheriting
AnthropicBetaClient
and substituting the underlying anthropic calls + exposing cache writes / hits to the response objectWould love to hear your thoughts on that
Beta Was this translation helpful? Give feedback.
All reactions