llama.cpp: Port of Facebook's LLaMA model in C/C++
github 倉庫：https://github.com/ggerganov/llama.cpp
參考博文：High-Speed Inference with llama.cpp and Vicuna on CPU
第1步，準備一臺阿里雲4核8G的服務器，操作系統用的是 ubuntu 22.04
第2步，簽出 llama.cpp 源碼進行 build
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make
build 過程中開始部分的輸出
I llama.cpp build info:
I UNAME_S: Linux
I UNAME_P: x86_64
I UNAME_M: x86_64
I CFLAGS: -I. -O3 -std=c11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wdouble-promotion -Wshadow -Wstrict-prototypes -Wpointer-arith -Wmissing-prototypes -pthread -march=native -mtune=native -DGGML_USE_K_QUANTS
I CXXFLAGS: -I. -I./examples -O3 -std=c++11 -fPIC -DNDEBUG -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-multichar -pthread -march=native -mtune=native -DGGML_USE_K_QUANTS
I LDFLAGS:
I CC: cc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
I CXX: g++ (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0
第3步，下載 4-bit 版本的 Vicuna-7B 模型，文件大小是3.6G
cd ./models/
wget https://huggingface.co/TheBloke/vicuna-7B-1.1-GGML/resolve/main/vicuna-7b-1.1.ggmlv3.q4_0.bin
第4步，使用這個模型輸入提示詞 Tell me about cnblogs
./main -m ./models/vicuna-7b-1.1.ggmlv3.q4_0.bin -p "Tell me about cnblogs" -n 1024
得到的回答（只列出了第1段）：
Tell me about cnblogs.com
cnblogs.com is a hosting and blogging platform that enables users to create and maintain their own blogs with ease. The website provides a user-friendly interface for creating and managing weblogs, and offers a range of features such as custom domains, email integration, and the ability to host multiple blogs from a single account. Additionally, cnblogs.com offers security features, such as SSL encryption, to protect users' blogs and data.
....
注：從回答內容看，多數內容是編造出來的。
第5步，下載 8-bit 版本的 Vicuna-7B 模型，文件大小是6.7G
cd ./models/
wget https://huggingface.co/TheBloke/vicuna-7B-1.1-GGML/resolve/main/vicuna-7b-1.1.ggmlv3.q8_0.bin
第6步，用同樣的提示詞提問
./main -m ./models/vicuna-7b-1.1.ggmlv3.q8_0.bin -p "Tell me about cnblogs" -n 1024
回答的內容是更離譜的編造：
Tell me about cnblogs:
cnblogs is the perfect platform for companies and organizations to share news, updates, and information with their followers. Whether you want to highlight employee achievements, announce new products or services, or provide valuable insights into your industry, our blogging service can help you get your message out there. Our team of experienced writers can create compelling content that is optimized for search engines and tailored to your specific needs. With cnblogs, you can build a loyal following and establish yourself as a thought leader in your field.
What are the benefits of using cnblogs? [end of text]
第6步，改爲使用交互模式
./main -m ./models/vicuna-7b-1.1.ggmlv3.q4_0.bin -p "Tell me about cnblogs" -n 256 --repeat_penalty 1.0 --color -i -r "User:"
對於默認提示詞的回答變成了：
Tell me about cnblogs.com
cnblogs.com is a domain name that has been around since at least 2004. There is no information available about who owns or operates the site. It is possible that it is a personal blog or a small website that has not been actively maintained in some time.
在交互模式下輸入提示詞 Tell me about github，回答內容好多了：
Tell me about github
GitHub is a web-based hosting service for version control using Git. It was founded in 2008 and was acquired by Microsoft in 2018. It provides a platform for developers to share code and collaborate on projects, and it also allows for code reviews and has a large community of developers. GitHub is a popular platform for open-source projects, and it is used by many large companies and organizations. It is also widely used by individual developers and small teams.
初步體驗到此結束。