Skip to content

feat(js/plugins/vertexai): add support to anthropic prompt caching #2886

New issue

Have a question about this project? Sign up for a free account to open an issue and contact its maintainers and the community.

By clicking “Sign up for ”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on ? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

amondnet
Copy link

@amondnet amondnet commented May 8, 2025

Description here... Help the reviewer by:

When using Anthropic models, you can set up cache control for system messages. You can enable this feature by adding a cacheControl property to the custom field of a message. The only currently supported cache type is ephemeral.

For example, to use cache control for a specific system message, you would set it up like this:

const llmResponse = await ai.generate({
  model: claude3Sonnet, // or another Anthropic model
  messages: [
    {
      role: 'system',
      content: [
        {
          text: 'This is an important instruction that can be cached.',
          custom: {
            cacheControl: {
              type: 'ephemeral',
            },
          },
        },
      ],
    },
    {
      role: 'user',
      content: [{ text: 'What should I do when I visit Melbourne?' }],
    },
  ],
});

https://.com/anthropics/anthropic-sdk-typescript/releases/tag/sdk-v0.33.0
anthropics/anthropic-sdk-typescript#631

Checklist (if applicable):

@github-actions-actions bot added docsImprovements or additions to documentationjs config labels May 8, 2025
@amondnetamondnet force-pushed the vertexai-anthropic-context-caching branch from 72b29ea to 20881ee Compare May 8, 2025 12:59
Sign up for free to join this conversation on . Already have an account? Sign in to comment
Labels
config docsImprovements or additions to documentationjs
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

[JS] Add support to anthropic prompt caching
1 participant
@amondnet