1. Plugins 是什麼
1.1 Plugins 的工作原理
1.2 Plugin開發
可能是史上最容易開發的 plugin。只需要定義兩個文件：
1. `yourdomain.com/.well-known/ai-plugin.json`，描述插件的基本信息
2. `openai.yaml`，描述插件的 API（Swagger生成的文檔）
而 OpenAI 那邊，更簡單，沒有任何人和你對接。是 AI 和你對接！AI 閱讀上面兩個文件，就知道該怎麼調用你了。
下面是官方的例子
ai-plugin.json
{ "schema_version": "v1", //配置文件版本 "name_for_human": "Sport Stats", //插件名字，給用戶看的名字 "name_for_model": "sportStats", //插件名字，給ChatGPT模型看的名字，需要唯一 "description_for_human": "Get current and historical stats for sport players and games.", //描述插件的功能，這個字段是在插件市場展示給用戶看的 "description_for_model": "Get current and historical stats for sport players and games. Always display results using markdown tables.", //描述插件的功能，ChatGPT會分析這個字段，確定什麼時候調用你的插件 "auth": { "type": "none" //這個是API認證方式，none 代表不需要認證 }, "api": { "type": "openapi", "url": "PLUGIN_HOSTNAME/openapi.yaml" //這個是Swagger API文檔地址，ChatGPT通過這個地址訪問我們的api文檔 }, "logo_url": "PLUGIN_HOSTNAME/logo.png", //插件logo地址 "contact_email": "[email protected]", //插件官方聯繫郵件 "legal_info_url": "https://example.com/legal" //與該插件相關的legal information }
openapi.yaml
openapi: 3.0.1 info: title: Sport Stats description: Get current and historical stats for sport players and games. version: "v1" servers: - url: PLUGIN_HOSTNAME paths: /players: get: operationId: getPlayers summary: Retrieves all players from all seasons whose names match the query string. parameters: - in: query name: query schema: type: string description: Used to filter players based on their name. For example, ?query=davis will return players that have 'davis' in their first or last name. responses: "200": description: OK /teams: get: operationId: getTeams summary: Retrieves all teams for the current season. responses: "200": description: OK /games: get: operationId: getGames summary: Retrieves all games that match the filters specified by the args. Display results using markdown tables. parameters: - in: query name: limit schema: type: string description: The max number of results to return. - in: query name: seasons schema: type: array items: type: string description: Filter by seasons. Seasons are represented by the year they began. For example, 2018 represents season 2018-2019. - in: query name: team_ids schema: type: array items: type: string description: Filter by team ids. Team ids can be determined using the getTeams function. - in: query name: start_date schema: type: string description: A single date in 'YYYY-MM-DD' format. This is used to select games that occur on or after this date. - in: query name: end_date schema: type: string description: A single date in 'YYYY-MM-DD' format. This is used to select games that occur on or before this date. responses: "200": description: OK /stats: get: operationId: getStats summary: Retrieves stats that match the filters specified by the args. Display results using markdown tables. parameters: - in: query name: limit schema: type: string description: The max number of results to return. - in: query name: player_ids schema: type: array items: type: string description: Filter by player ids. Player ids can be determined using the getPlayers function. - in: query name: game_ids schema: type: array items: type: string description: Filter by game ids. Game ids can be determined using the getGames function. - in: query name: start_date schema: type: string description: A single date in 'YYYY-MM-DD' format. This is used to select games that occur on or after this date. - in: query name: end_date schema: type: string description: A single date in 'YYYY-MM-DD' format. This is used to select games that occur on or before this date. responses: "200": description: OK /season_averages: get: operationId: getSeasonAverages summary: Retrieves regular season averages for the given players. Display results using markdown tables. parameters: - in: query name: season schema: type: string description: Defaults to the current season. A season is represented by the year it began. For example, 2018 represents season 2018-2019. - in: query name: player_ids schema: type: array items: type: string description: Filter by player ids. Player ids can be determined using the getPlayers function. responses: "200": description: OK
description的內容非常重要，決定了ChatGPT會不會調用你的插件，調用得是否正確。
1.3 Plugins的市場表現
1. 時間線： 1. 3月24日發佈， 提供11個插件，可以申請加入waitlist獲得使用權 2. 5月15日開始向Plus用戶全量開放插件和Browsing， 插件數70多個 3. 7月5日因安全原因，關閉Browsing（用戶可通過此功能訪問付費頁面） 4. 7月11日開始全量開放Code Interpreter。插件數已超400
2. 媒體將其類比爲App Store，獲得鼓吹
3. 6月7日（全面放開後三星期）一篇應OpenAI要求而[被刪除的帖子](https://humanloop.com/blog/openai-plans)中透露，Sam Altman 在一個閉門會中說：「插件的實際使用情況表明，除了Browsing以外，還沒有達到理想的產品市場契合點。他表示，很多人認爲他們希望自己的應用程序位於ChatGPT中，但他們真正想要的是應用程序中的ChatGPT。」
（被刪內容這裏可以看到：https://web.archive.org/web/20230531203946/https://humanloop.com/blog/openai-plans）
1.4 Plugins到目前還沒做起來的原因分析
它暫時歇菜了，主要原因：
1. 缺少強 Agent調度，只能手工選三個 plugin，使用成本太高。（解決此問題，相當於 App Store + Siri，可挑戰手機操作系統地位）
2. 不在場景中，不能提供端到端一攬子服務。（解決此問題，就是全能私人助理了，人類唯一需要的軟件）
3. 開銷大。（至少兩次 GPT-4 生成，和一次 Web API 調用）
這是我們做智能應用也要面對的問題。OpenAI 很快推出了大殺器 Function Calling 功能，來幫助我們開發更好的智能應用
2. Function Calling
2.1 Function Calling 的機制
Function Calling完整的官方接口文檔：https://platform.openai.com/docs/guides/gpt/function-calling
Function Calling詳細的參數說明: https://learn.microsoft.com/zh-cn/azure/ai-services/openai/how-to/function-calling
2.2 Function Calling 示例 1：加法計算器
需求：用戶輸入任意可以用加法解決的問題，都能得到計算結果。
# 加載環境變量 import openai import os import json from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) # 讀取本地 .env 文件，裏面定義了 OPENAI_API_KEY openai.api_key = os.getenv('OPENAI_API_KEY')
def get_completion(messages, model="gpt-3.5-turbo"): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=0, # 模型輸出的隨機性，0 表示隨機性最小 functions=[{ # 用 JSON 描述函數。可以定義多個，但是隻有一個會被調用，也可能都不會被調用 "name": "sum", "description": "計算一組數的求和", "parameters": { "type": "object", "properties": { "numbers": { "type": "array", "items": { "type": "number" } } } }, }], ) return response.choices[0].message
from math import * # prompt = "Tell me the sum of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10." prompt = "桌上有 2 個蘋果，四個桃子和 3 本書，一共有幾個水果？" # prompt = "1+2+3...+99+100" messages = [ {"role": "system", "content": "你是一個小學數學老師，你要教學生加法"}, {"role": "user", "content": prompt} ] response = get_completion(messages) messages.append(response) # 把大模型的回覆加入到對話中 print("=====GPT回覆=====") print(response) # 如果返回的是函數調用結果，則打印出來 if (response.get("function_call")): # 是否要調用 sum if (response["function_call"]["name"] == "sum"): args = json.loads(response["function_call"]["arguments"]) result = sum(args["numbers"]) print("=====自定義的函數返回=====") print(result) messages.append( {"role": "function", "name": "pythonRunner", "content": str(result)} # 數值result 必須轉成字符串 ) print("=====最終回覆=====") print(get_completion(messages).content)
運行結果：
2.3 Function Calling 示例2：計算數學表達式
def get_completion(messages, model="gpt-3.5-turbo"): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=0, # 模型輸出的隨機性，0 表示隨機性最小 functions=[{ # 用 JSON 描述函數。可以定義多個，但是隻有一個會被調用，也可能都不會被調用 "name": "calculate", "description": "計算一個數學表達式的值", "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "a mathematical expression in python grammar.", } } }, }], ) return response.choices[0].message
from math import * # prompt = "從1加到20" prompt = "3的平方根乘以2再開平方" messages = [ {"role": "system", "content": "你是一個數學家，你可以計算任何算式。"}, {"role": "user", "content": prompt} ] response = get_completion(messages) messages.append(response) # 把大模型的回覆加入到對話中 print("=====GPT回覆=====") print(response) # 如果返回的是函數調用結果，則打印出來 if (response.get("function_call")): if (response["function_call"]["name"] == "calculate"): args = json.loads(response["function_call"]["arguments"]) result = eval(args["expression"]) print("=====函數返回=====") print(result) messages.append( {"role": "function", "name": "calculate", "content": str(result)} # 數值result 必須轉成字符串 ) print("=====最終回覆=====") print(get_completion(messages).content)
運行結果：
Function Calling中的函數與參數的描述也是一種Prompt. 這種Prompt也需要調優，否則會影響函數的召回、參數的準確性，甚至讓GPT產生幻覺
2.3 Function Calling 示例3：計算數學表達式的一個反面教材
def get_completion(messages, model="gpt-3.5-turbo"): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=0, # 模型輸出的隨機性，0 表示隨機性最小 functions=[{ # 用 JSON 描述函數。可以定義多個，但是隻有一個會被調用，也可能都不會被調用 "name": "calculate", "description": "計算一個以Python形式表示的數學表達式的值", "parameters": { "type": "object", "properties": { "expression": { "type": "string", "description": "a mathematical expression in python format. it must be evaluatable by Python's eval()", } } }, }], ) return response.choices[0].message prompt = "從1加到20" messages = [ {"role": "system", "content": "你是一個數學家，你可以計算任何算式。"}, {"role": "user", "content": prompt} ] response = get_completion(messages) messages.append(response) # 把大模型的回覆加入到對話中 print("=====GPT回覆=====") print(response) # 如果返回的是函數調用結果，則打印出來 if (response.get("function_call")): if (response["function_call"]["name"] == "calculate"): args = json.loads(response["function_call"]["arguments"]) result = eval(args["expression"]) print("=====函數返回=====") print(result) messages.append( {"role": "function", "name": "calculate", "content": str(result)} # 數值result 必須轉成字符串 ) print("=====最終回覆=====") print(get_completion(messages).content)
運行結果:
上面的例子是做數學表達式的function call, 我的目標是計算數學表達，但是由於在function的description和parameters的description描述中過於強調python,導致GPT返回了錯誤的funcation name
2.4 Function Calling 示例4：多Function調用
def get_completion(messages, model="gpt-3.5-turbo"): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=0, # 模型輸出的隨機性，0 表示隨機性最小 function_call="auto", # 默認值，由系統自動決定，返回function call還是返回文字回覆 functions=[{ # 用 JSON 描述函數。可以定義多個，但是最多隻有一個會被調用，也可能不被調用 "name": "get_location_coordinate", "description": "根據POI名稱，獲得POI的經緯度座標", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "POI名稱，必須是中文", }, "city": { "type": "string", "description": "搜索城市名，必須是中文", } }, "required": ["location"], }, }, { "name": "search_nearby_pois", "description": "搜索給定座標附近的poi", "parameters": { "type": "object", "properties": { "longitude": { "type": "string", "description": "中心點的經度", }, "latitude": { "type": "string", "description": "中心點的緯度", }, "keyword": { "type": "string", "description": "目標poi的關鍵字", } }, "required": ["longitude","latitude","keyword"], }, }], ) return response.choices[0].message
import requests amap_key="baidu_map_key" def get_location_coordinate(location,city): url = f"https://restapi.amap.com/v5/place/text?key={amap_key}&keywords={location}®ion={city}" print(url) r = requests.get(url) result = r.json() if "pois" in result and result["pois"]: return result["pois"][0] return None def search_nearby_pois(longitude,latitude,keyword): url = f"https://restapi.amap.com/v5/place/around?key={amap_key}&keywords={keyword}&location={longitude},{latitude}" print(url) r = requests.get(url) result = r.json() ans = "" if "pois" in result and result["pois"]: for i in range(min(3,len(result["pois"]))): name = result["pois"][i]["name"] address = result["pois"][i]["address"] distance = result["pois"][i]["distance"] ans += f"{name}\n{address}\n距離：{distance}米\n\n" return ans
prompt = "惠州市大亞灣石化大道西39號附近的自助餐" messages = [ {"role": "system", "content": "你是一個地圖通，你可以找到任何地址。"}, {"role": "user", "content": prompt} ] response = get_completion(messages) messages.append(response) # 把大模型的回覆加入到對話中 print("=====GPT回覆=====") print(response) # 如果返回的是函數調用結果，則打印出來 while (response.get("function_call")): if (response["function_call"]["name"] == "get_location_coordinate"): args = json.loads(response["function_call"]["arguments"]) print("Call: get_location_coordinate") result = get_location_coordinate(**args) elif (response["function_call"]["name"] == "search_nearby_pois"): args = json.loads(response["function_call"]["arguments"]) print("Call: search_nearby_pois") result = search_nearby_pois(**args) print("=====函數返回=====") print(result) messages.append( {"role": "function", "name": response["function_call"]["name"], "content": str(result)} # 數值result 必須轉成字符串 ) response = get_completion(messages) print("=====最終回覆=====") print(get_completion(messages).content)
運行結果:
2.5 Function Calling 示例5：用Function Calling實現信息抽取
def get_completion(messages, model="gpt-3.5-turbo"): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=0, # 模型輸出的隨機性，0 表示隨機性最小 function_call="auto", functions=[{ "name": "add_contact", "description": "添加聯繫人", "parameters": { "type": "object", "properties": { "name": { "type": "string", "description": "聯繫人姓名" }, "address": { "type": "string", "description": "聯繫人地址" }, "tel": { "type": "string", "description": "聯繫人電話" }, } }, }], ) return response.choices[0].message prompt = "中秋禮品A，收貨人是Brian,收貨地址是深圳市寶安區西鄉街道，電話190xxxx123。" messages = [ {"role": "system", "content": "你是一個聯繫人錄入員。"}, {"role": "user", "content": prompt} ] response = get_completion(messages) print("====GPT回覆====") print(json.dumps(response,ensure_ascii=False,indent=2)) args = json.loads(response["function_call"]["arguments"]) print("====函數參數====") print(args)
運行結果:
如果只想要個JSON格式數據，那麼Prompt和Function Calling哪個更好？因爲Function Calling能力是特別fine-tune在模型內的，所以輸出更穩定，用來獲取JSON更可靠。搞個假函數聲明，就能拿到JSON了。
2.6 Function Calling 示例 6：通過Function Calling查詢數據庫
需求：從訂單表中查詢各種信息，比如某個用戶的訂單數量、某個商品的銷量、某個用戶的消費總額等等。
import openai import os import json from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) # 讀取本地 .env 文件，裏面定義了 OPENAI_API_KEY openai.api_key = os.getenv('OPENAI_API_KEY') def get_sql_completion(messages, model="gpt-3.5-turbo"): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=0, # 模型輸出的隨機性，0 表示隨機性最小 function_call="auto", functions=[{ # 摘自 OpenAI 官方示例 https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb "name": "ask_database", "description": "Use this function to answer user questions about business. \ Output should be a fully formed SQL query.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": f""" SQL query extracting info to answer the user's question. SQL should be written using this database schema: {database_schema_string} The query should be returned in plain text, not in JSON. The query should only contain grammars supported by SQLite. """, } }, "required": ["query"], }, }], ) return response.choices[0].message
import openai import os import json from dotenv import load_dotenv, find_dotenv _ = load_dotenv(find_dotenv()) # 讀取本地 .env 文件，裏面定義了 OPENAI_API_KEY openai.api_key = os.getenv('OPENAI_API_KEY') def get_sql_completion(messages, model="gpt-3.5-turbo"): response = openai.ChatCompletion.create( model=model, messages=messages, temperature=0, # 模型輸出的隨機性，0 表示隨機性最小 function_call="auto", functions=[{ # 摘自 OpenAI 官方示例 https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_with_chat_models.ipynb "name": "ask_database", "description": "Use this function to answer user questions about business. \ Output should be a fully formed SQL query.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": f""" SQL query extracting info to answer the user's question. SQL should be written using this database schema: {database_schema_string} The query should be returned in plain text, not in JSON. The query should only contain grammars supported by SQLite. """, } }, "required": ["query"], }, }], ) return response.choices[0].message
# 描述數據庫表結構 database_schema_string = """ CREATE TABLE orders ( id INT PRIMARY KEY NOT NULL, -- 主鍵，不允許爲空 customer_id INT NOT NULL, -- 客戶ID，不允許爲空 product_id STR NOT NULL, -- 產品ID，不允許爲空 price DECIMAL(10,2) NOT NULL, -- 價格，不允許爲空 status INT NOT NULL, -- 訂單狀態，整數類型，不允許爲空。0代表待支付，1代表已支付，2代表已退款 create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- 創建時間，默認爲當前時間 pay_time TIMESTAMP -- 支付時間，可以爲空 ); """
import sqlite3 # 創建數據庫連接 conn = sqlite3.connect(':memory:') cursor = conn.cursor() # 創建orders表 cursor.execute(database_schema_string) # 插入5條明確的模擬記錄 mock_data = [ (1, 1001, 'TSHIRT_1', 50.00, 0, '2023-08-12 10:00:00', None), (2, 1001, 'TSHIRT_2', 75.50, 1, '2023-08-16 11:00:00', '2023-08-16 12:00:00'), (3, 1002, 'SHOES_X2', 25.25, 2, '2023-08-17 12:30:00', '2023-08-17 13:00:00'), (4, 1003, 'HAT_Z112', 60.75, 1, '2023-08-20 14:00:00', '2023-08-20 15:00:00'), (5, 1002, 'WATCH_X001', 90.00, 0, '2023-08-28 16:00:00', None) ] for record in mock_data: cursor.execute(''' INSERT INTO orders (id, customer_id, product_id, price, status, create_time, pay_time) VALUES (?, ?, ?, ?, ?, ?, ?) ''', record) # 提交事務 conn.commit()
def ask_database(query): cursor.execute(query) records = cursor.fetchall() return records # prompt = "上個月的銷售額" # prompt = "統計每月每件商品的銷售額" prompt = "哪個用戶消費最高？消費多少？" messages = [ {"role": "system", "content": "基於 order 表回答用戶問題"}, {"role": "user", "content": prompt} ] response = get_sql_completion(messages) print("====Function Calling====") print(response) if "function_call" in response: if response["function_call"]["name"] == "ask_database": arguments = response["function_call"]["arguments"] args = json.loads(arguments) print("====SQL====") print(args["query"]) result = ask_database(args["query"]) print("====DB Records====") print(result) messages.append({ "role": "user", "content": f"用戶問：{prompt}\n系統通過以下SQL查詢後，返回:"+str(result)+"\n據此請回答：" }) response = get_sql_completion(messages) print("====最終回覆====") print(get_completion(messages).content)
運行結果:
2.7 Function Calling 示例 7：用Function Calling實現多表查詢
# 描述數據庫表結構 database_schema_string = """ CREATE TABLE customers ( id INT PRIMARY KEY NOT NULL, -- 主鍵，不允許爲空 customer_name VARCHAR(255) NOT NULL, -- 客戶名，不允許爲空 email VARCHAR(255) UNIQUE, -- 郵箱，唯一 register_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP -- 註冊時間，默認爲當前時間 ); CREATE TABLE products ( id INT PRIMARY KEY NOT NULL, -- 主鍵，不允許爲空 product_name VARCHAR(255) NOT NULL, -- 產品名稱，不允許爲空 price DECIMAL(10,2) NOT NULL -- 價格，不允許爲空 ); CREATE TABLE orders ( id INT PRIMARY KEY NOT NULL, -- 主鍵，不允許爲空 customer_id INT NOT NULL, -- 客戶ID，不允許爲空 product_id INT NOT NULL, -- 產品ID，不允許爲空 price DECIMAL(10,2) NOT NULL, -- 價格，不允許爲空 status INT NOT NULL, -- 訂單狀態，整數類型，不允許爲空。0代表待支付，1代表已支付，2代表已退款 create_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP, -- 創建時間，默認爲當前時間 pay_time TIMESTAMP -- 支付時間，可以爲空 ); """ #prompt = "統計每月每件商品的銷售額" prompt = "這星期消費最高的用戶是誰？他買了哪些商品？ 每件商品買了幾件？花費多少？" messages = [ {"role": "system", "content": "基於 order 表回答用戶問題"}, {"role": "user", "content": prompt} ] response = get_sql_completion(messages) print(response)
運行結果:
{ "role": "assistant", "content": null, "function_call": { "name": "ask_database", "arguments": "{\n \"query\": \"SELECT c.customer_name, p.product_name, COUNT(o.id) AS quantity, SUM(o.price) AS total_cost FROM customers c JOIN orders o ON c.id = o.customer_id JOIN products p ON o.product_id = p.id WHERE o.create_time >= DATE_SUB(CURDATE(), INTERVAL WEEKDAY(CURDATE()) DAY) AND o.create_time < DATE_ADD(CURDATE(), INTERVAL 1 DAY) GROUP BY c.customer_name, p.product_name ORDER BY total_cost DESC LIMIT 1\"\n}" } }
2.8 Function Calling 的注意事項
1. 截至到目前只有 `gpt-3.5-turbo-0613` 和 `gpt-4-0613` 可用。它倆針對Function Calling做了fine-tuning，以儘可能保證正確率。
2. 但不保證不出錯，包括不保證json格式正確。所以官方強烈建議（原文：strongly recommend）如果有寫操作，一定插入人工流程做確認。但比純靠 prompt控制，可靠性是大了很多的
3. 函數聲明是消耗token的。要在功能覆蓋、省錢、節約上下文窗口之間找到最佳平衡
4. 實戰經驗：把自己的函數調用結果用自然語言給到OpenAI，效果有時更好。調優時可以試試