Skip to content

Commit a52e33e

Browse files
authored
[Rust] Bump HuggingFace tokenizer version (#36)
This PR bumps the HuggingFace tokenizers package version to work with some of the latest models (e.g., Mistral v0.3).
1 parent 2db668a commit a52e33e

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

rust/Cargo.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,6 @@ crate-type = ["staticlib"]
88

99
[dependencies]
1010

11-
tokenizers = { version = "0.13.4", default-features = false, features = ["onig"] }
11+
tokenizers = { version = "0.19.1", default-features = false, features = ["onig"] }
1212
serde = { version = "1.0", features = [ "derive" ] }
1313
serde_json = "1.0"

0 commit comments

Comments
 (0)