CMake Configuration Issues on MacOS

[Darce-One]

Description

The current configuration of the CMakeLists.txt file does not support successful project builds on MacOS. This was verified using the build_and_run.sh script located in the example folder.

Workaround

I have implemented a workaround by bypassing the tokenizers_cpp_objs target and building the tokenizers_cpp target directly, which resolved the issue. Additionally, I included a find_executable command to locate the cargo executable at compile time and replaced the hardcoded cargo path with a variable. This change enhances the build process, particularly when working within Xcode.

Pull Request

I am preparing a Pull Request #65 to incorporate these modifications into the main branch.

[springkim]

@Darce-One

How about trying out a recently developed tokenizer library?
It's written in C++, supports all major operating systems, and provides extremely fast tokenization performance.
You can check it out here: https://github.com/NLPOptimize/flash-tokenizer

[Darce-One]

@Darce-One

How about trying out a recently developed tokenizer library? It's written in C++, supports all major operating systems, and provides extremely fast tokenization performance. You can check it out here: https://github.com/NLPOptimize/flash-tokenizer

@springkim Thanks for the recommendation. It looks good, I'll try it out. Do you plan to keep supporting it in the future?