Skip to content

LanguageModelChat reasoning tokens #246993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
roblourens opened this issue Apr 20, 2025 · 0 comments
Open

LanguageModelChat reasoning tokens #246993

roblourens opened this issue Apr 20, 2025 · 0 comments
Assignees
Milestone

Comments

@roblourens
Copy link
Member

We should add a LanguageModelThinkingPart to handle thinking tokens. This would stream out of LanguageModelChat, and then stream out of chat participants, show up in the chat response history, and the LanguageModelChat request. They could be displayed in the UI, so they get their own part, rather than building on LanguageModelExtraDataPart.

  • Anthropic: in "extended thinking with tool use", thinking/redacted tokens are required to be captured and returned in the next request, along with a signature https://docs.anthropic.com/en/docs/build-with-claude/extended-thinking#understanding-thinking-blocks. But otherwise, thinking tokens can be discarded. Thinking tokens can also be 'redacted'
  • OpenAI: reasoning tokens are not returned
  • DeepSeek/other models (?): reasoning tokens are streamed in a simple format, and do not need to be included in the next request
// Redacted thinking tokens in Anthropic could use LanguageModelExtraDataPart, since they won't be displayed.
// "thinking" or "reasoning"? Different models use one or the other.
export class LanguageModelThinkingPart {
	thinking: string;
	// for Anthropic, to track signature
	metadata?: any;
	constructor(thinking: string, metadata?: any);
}
@roblourens roblourens added this to the April 2025 milestone Apr 20, 2025
@roblourens roblourens self-assigned this Apr 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant