1. BPF 和 eBPF
BPF (Berkeley Packet Filter)最早的網絡數據包捕獲。
eBPF (extended Berkeley Packet Filter) 新出了一個 BPF 替換了之前老的 BPF ,但是名稱有的時候也常 BPF ,也有叫 eBPF 的。功能得到加強,除了能網絡數據包捕獲外,也能用於 trace 內核函數,內核中自帶的一個 ftrace 功能就使用這個技術。
如下圖所示:分爲用戶層和內核層,用戶層通過編譯 bpf 的指令,可以通過 bcc 庫,最終經過 llvm 編譯爲 elf 格式的文件。由 bpf loader 加載到內核中通過 BFP 虛擬機檢查,如 :死循環、不安全的調用等。在通過 JIT 編譯提高運行速度,最終被添加到內核中。
readelf -S /system/etc/bpf/netd.o
There are 24 section headers, starting at offset 0x28a0:
Section Headers:
[Nr] Name Type Address Offset
Size EntSize Flags Link Info Align
[ 0] NULL 0000000000000000 00000000
0000000000000000 0000000000000000 0 0 0
[ 1] .strtab STRTAB 0000000000000000 00002402
000000000000049b 0000000000000000 0 0 1
[ 2] .text PROGBITS 0000000000000000 00000040
0000000000000000 0000000000000000 AX 0 0 4
[ 3] cgroupskb/in[...] PROGBITS 0000000000000000 00000040
00000000000009d8 0000000000000000 AX 0 0 8
[ 4] .relcgroupsk[...] REL 0000000000000000 000020e0
0000000000000140 0000000000000010 23 3 8
[ 5] cgroupskb/eg[...] PROGBITS 0000000000000000 00000a18
0000000000000950 0000000000000000 AX 0 0 8
[ 6] .relcgroupsk[...] REL 0000000000000000 00002220
0000000000000140 0000000000000010 23 5 8
[ 7] skfilter/egr[...] PROGBITS 0000000000000000 00001368
0000000000000110 0000000000000000 AX 0 0 8
[ 8] .relskfilter[...] REL 0000000000000000 00002360
0000000000000030 0000000000000010 23 7 8
[ 9] skfilter/ing[...] PROGBITS 0000000000000000 00001478
0000000000000110 0000000000000000 AX 0 0 8
[10] .relskfilter[...] REL 0000000000000000 00002390
0000000000000030 0000000000000010 23 9 8
[11] skfilter/whi[...] PROGBITS 0000000000000000 00001588
00000000000000d0 0000000000000000 AX 0 0 8
[12] .relskfilter[...] REL 0000000000000000 000023c0
0000000000000010 0000000000000010 23 11 8
[13] skfilter/bla[...] PROGBITS 0000000000000000 00001658
0000000000000070 0000000000000000 AX 0 0 8
[14] .relskfilter[...] REL 0000000000000000 000023d0
0000000000000010 0000000000000010 23 13 8
[15] cgroupsock/i[...] PROGBITS 0000000000000000 000016c8
00000000000000a0 0000000000000000 AX 0 0 8
[16] .relcgroupso[...] REL 0000000000000000 000023e0
0000000000000010 0000000000000010 23 15 8
[17] .rodata PROGBITS 0000000000000000 00001768
0000000000000021 0000000000000000 A 0 0 4
[18] maps PROGBITS 0000000000000000 0000178c
0000000000000118 0000000000000000 WA 0 0 4
[19] license PROGBITS 0000000000000000 000018a4
000000000000000b 0000000000000000 WA 0 0 1
[20] .BTF PROGBITS 0000000000000000 000018af
0000000000000019 0000000000000000 0 0 1
[21] .BTF.ext PROGBITS 0000000000000000 000018c8
0000000000000020 0000000000000000 0 0 1
[22] .llvm_addrsig LOOS+0xfff4c03 0000000000000000 000023f0
0000000000000012 0000000000000000 E 23 0 1
[23] .symtab SYMTAB 0000000000000000 000018e8
00000000000007f8 0000000000000018 1 58 8
bcc 庫,是一個簡化開發 bpf.o 的 c 語言庫,安卓在早期版本,如 11 版本時使用了 一小部分的 bcc 提供的功能,在 安卓 13 後自己整出來一個 libbpf 的東西,不用 bcc 庫了。
安卓中查詢 APP使用流量詳細的方法
NetworkStatsManager networkStatsManager = (NetworkStatsManager)getApplicationContext().getSystemService(Context.NETWORK_STATS_SERVICE);
//網絡類型 TYPE_WIFI 移動數據 TYPE_MOBILE
//訂閱者ID 爲空表示不區分 SIM 卡
long startTime = new Date().getTime() - 7*24*60*60*1000; //最近7天 (ms)
long endTime = new Date().getTime();
NetworkStats.Bucket bucket = networkStatsManager.querySummaryForDevice(ConnectivityManager.TYPE_WIFI, null, startTime, endTime);
Log.d(TAG, "bucket uid:" + bucket.getUid());
Log.d(TAG, "bucket getRxBytes:" + bucket.getRxBytes());
Log.d(TAG, "bucket getTxBytes:" + bucket.getTxBytes());
1,bpf loader 掛載 bpf 文件系統,掛載 cgroup2 啓動 bpfloader ,cgroup 由 /system/etc/cgroups.json 初始化,這裏是手動初始化
#bpf start
mount bpf bpf /sys/fs/bpf
mkdir /dev/cg2_bpf 0600 root root
chmod 0600 /dev/cg2_bpf
mount cgroup2 none /dev/cg2_bpf
on late-init
trigger load_bpf_programs
on load_bpf_programs
write /proc/sys/kernel/unprivileged_bpf_disabled 0
write /proc/sys/net/core/bpf_jit_enable 1
write /proc/sys/net/core/bpf_jit_kallsyms 1
start bpfloader
service bpfloader /system/bin/bpf-loader
user root
group root
disabled
oneshot
#bpf end
2,cgroup 配置
對於cgroupskb
類型的bpf程序, 還需要通過BPF_PROG_ATTACH
命令把固定到/sys/fs/bpf
的代碼附着到對應的cgroup上(這樣我們就可以監控特定cgroup上的進程的網絡狀態了):
mount bpf bpf /sys/fs/bpf
mkdir /dev/cg2_bpf 0600 root root
chmod 0600 /dev/cg2_bpf
mount cgroup2 none /dev/cg2_bpf
3,安卓流量收集記錄功能
TrafficController::start()
initMaps() # 初始化 map 文件
initPrograms() # attach to cgroup
addInterface() # 添加網卡配置
4,讀取流量統計記錄,並清理結果
BpfNetworkStats.cpp
parseBpfNetworkStatsDetail()
BpfMapRO<uint32_t, uint8_t> configurationMap(CONFIGURATION_MAP_PATH);
auto configuration = configurationMap.readValue(CURRENT_STATS_MAP_CONFIGURATION_KEY);
const char* statsMapPath = STATS_MAP_PATH[configuration.value()];
TrafficController.cpp
swapActiveStatsMap() #切換 mapA mapB
5,安卓統計記錄實現,供參考:
# android11 流量統計相關代碼
NetworkStatsService.java
# 創建4個收集文件
// create data recorders along with historical rotators
mDevRecorder = buildRecorder(PREFIX_DEV, mSettings.getDevConfig(), false);
mXtRecorder = buildRecorder(PREFIX_XT, mSettings.getXtConfig(), false);
mUidRecorder = buildRecorder(PREFIX_UID, mSettings.getUidConfig(), false);
mUidTagRecorder = buildRecorder(PREFIX_UID_TAG, mSettings.getUidTagConfig(), true);
# 創建 performPoll() 更新流量統計用的 handle
private final class NetworkStatsHandler extends Handler {
NetworkStatsHandler(@NonNull Looper looper) {
super(looper);
}
@Override
public void handleMessage(Message msg) {
switch (msg.what) {
case MSG_PERFORM_POLL: {
performPoll(FLAG_PERSIST_ALL);
break;
}
# 註冊 更新廣播
private BroadcastReceiver mPollReceiver = new BroadcastReceiver() {
@Override
public void onReceive(Context context, Intent intent) {
// on background handler thread, and verified UPDATE_DEVICE_STATS
// permission above.
performPoll(FLAG_PERSIST_ALL);
// verify that we're watching global alert
registerGlobalAlert();
}
};
# 收集 更新函數 添加同步保護 synchronized
private void performPoll(int flags) {
synchronized (mStatsLock) {
mWakeLock.acquire();
try {
performPollLocked(flags);
} finally {
mWakeLock.release();
}
}
}
# 更新4個收集文件
private void performPollLocked(int flags) {
try {
recordSnapshotLocked(currentTime);
} catch (IllegalStateException e) {
Log.wtf(TAG, "problem reading network stats", e);
return;
}
# 拉取更新記錄
private void recordSnapshotLocked(long currentTime) throws RemoteException {
// For xt/dev, we pass a null VPN array because usage is aggregated by UID, so VPN traffic
// can't be reattributed to responsible apps.
Trace.traceBegin(TRACE_TAG_NETWORK, "recordDev");
mDevRecorder.recordSnapshotLocked(devSnapshot, mActiveIfaces, currentTime);
Trace.traceEnd(TRACE_TAG_NETWORK);
Trace.traceBegin(TRACE_TAG_NETWORK, "recordXt");
mXtRecorder.recordSnapshotLocked(xtSnapshot, mActiveIfaces, currentTime);
Trace.traceEnd(TRACE_TAG_NETWORK);
// For per-UID stats, pass the VPN info so VPN traffic is reattributed to responsible apps.
Trace.traceBegin(TRACE_TAG_NETWORK, "recordUid");
mUidRecorder.recordSnapshotLocked(uidSnapshot, mActiveUidIfaces, currentTime);
Trace.traceEnd(TRACE_TAG_NETWORK);
Trace.traceBegin(TRACE_TAG_NETWORK, "recordUidTag");
mUidTagRecorder.recordSnapshotLocked(uidSnapshot, mActiveUidIfaces, currentTime);
Trace.traceEnd(TRACE_TAG_NETWORK);
final NetworkStats uidSnapshot = getNetworkStatsUidDetail(INTERFACES_ALL);
NetworkStats uidSnapshot = readNetworkStatsUidDetail(UID_ALL, ifaces, TAG_ALL);
mStatsFactory.readNetworkStatsDetail(uid, ifaces, tag);
public NetworkStats readNetworkStatsDetail(
int limitUid, String[] limitIfaces, int limitTag) throws IOException {
// In order to prevent deadlocks, anything protected by this lock MUST NOT call out to other
// code that will acquire other locks within the system server. See b/134244752.
synchronized (mPersistentDataLock) {
// Take a reference. If this gets swapped out, we still have the old reference.
final VpnInfo[] vpnArray = mVpnInfos;
// Take a defensive copy. mPersistSnapshot is mutated in some cases below
final NetworkStats prev = mPersistSnapshot.clone();
if (USE_NATIVE_PARSING) {
final NetworkStats stats =
new NetworkStats(SystemClock.elapsedRealtime(), 0 /* initialSize */);
# 讀取 bpf 信息
if (mUseBpfStats) {
try {
requestSwapActiveStatsMapLocked();
} catch (RemoteException e) {
throw new IOException(e);
}
// Stats are always read from the inactive map, so they must be read after the
// swap
if (nativeReadNetworkStatsDetail(stats, mStatsXtUid.getAbsolutePath(), UID_ALL,
INTERFACES_ALL, TAG_ALL, mUseBpfStats) != 0) {
throw new IOException("Failed to parse network stats");
}
// BPF stats are incremental; fold into mPersistSnapshot.
mPersistSnapshot.setElapsedRealtime(stats.getElapsedRealtime());
mPersistSnapshot.combineAllValues(stats);
}
# nativeReadNetworkStatsDetail() jni
base\services\core\jni\com_android_server_net_NetworkStatsFactory.cpp:
328
329 static const JNINativeMethod gMethods[] = {
330: { "nativeReadNetworkStatsDetail",
331 "(Landroid/net/NetworkStats;Ljava/lang/String;I[Ljava/lang/String;IZ)I",
332 (void*) readNetworkStatsDetail },
static int readNetworkStatsDetail(JNIEnv* env, jclass clazz, jobject stats, jstring path,
jint limitUid, jobjectArray limitIfacesObj, jint limitTag,
jboolean useBpfStats) {
std::vector<std::string> limitIfaces;
if (limitIfacesObj != NULL && env->GetArrayLength(limitIfacesObj) > 0) {
int num = env->GetArrayLength(limitIfacesObj);
for (int i = 0; i < num; i++) {
jstring string = (jstring)env->GetObjectArrayElement(limitIfacesObj, i);
ScopedUtfChars string8(env, string);
if (string8.c_str() != NULL) {
limitIfaces.push_back(std::string(string8.c_str()));
}
}
}
std::vector<stats_line> lines;
if (useBpfStats) {
if (parseBpfNetworkStatsDetail(&lines, limitIfaces, limitTag, limitUid) < 0)
return -1;
}
int parseBpfNetworkStatsDetail(std::vector<stats_line>* lines,
const std::vector<std::string>& limitIfaces, int limitTag,
int limitUid) {
BpfMapRO<uint32_t, IfaceValue> ifaceIndexNameMap(IFACE_INDEX_NAME_MAP_PATH);
if (!ifaceIndexNameMap.isValid()) {
int ret = -errno;
ALOGE("get ifaceIndexName map fd failed: %s", strerror(errno));
return ret;
}
BpfMapRO<uint32_t, uint8_t> configurationMap(CONFIGURATION_MAP_PATH);
if (!configurationMap.isValid()) {
int ret = -errno;
ALOGE("get configuration map fd failed: %s", strerror(errno));
return ret;
}
auto configuration = configurationMap.readValue(CURRENT_STATS_MAP_CONFIGURATION_KEY);
if (!configuration.ok()) {
ALOGE("Cannot read the old configuration from map: %s",
configuration.error().message().c_str());
return -configuration.error().code();
}
const char* statsMapPath = STATS_MAP_PATH[configuration.value()];
BpfMap<StatsKey, StatsValue> statsMap(statsMapPath);
if (!statsMap.isValid()) {
int ret = -errno;
ALOGE("get stats map fd failed: %s, path: %s", strerror(errno), statsMapPath);
return ret;
}
// It is safe to read and clear the old map now since the
// networkStatsFactory should call netd to swap the map in advance already.
int ret = parseBpfNetworkStatsDetailInternal(lines, limitIfaces, limitTag, limitUid, statsMap,
ifaceIndexNameMap);
if (ret) {
ALOGE("parse detail network stats failed: %s", strerror(errno));
return ret;
}
Result<void> res = statsMap.clear();
if (!res.ok()) {
ALOGE("Clean up current stats map failed: %s", strerror(res.error().code()));
return -res.error().code();
}
return 0;
}
# 讀取 bpf 收集的數據
int parseBpfNetworkStatsDetailInternal(std::vector<stats_line>* lines,
const std::vector<std::string>& limitIfaces, int limitTag,
int limitUid, const BpfMap<StatsKey, StatsValue>& statsMap,
const BpfMap<uint32_t, IfaceValue>& ifaceMap) {
int64_t unknownIfaceBytesTotal = 0;
const auto processDetailUidStats =
[lines, &limitIfaces, &limitTag, &limitUid, &unknownIfaceBytesTotal, &ifaceMap](
const StatsKey& key,
const BpfMap<StatsKey, StatsValue>& statsMap) -> Result<void> {
char ifname[IFNAMSIZ];
if (getIfaceNameFromMap(ifaceMap, statsMap, key.ifaceIndex, ifname, key,
&unknownIfaceBytesTotal)) {
return Result<void>();
}
std::string ifnameStr(ifname);
if (limitIfaces.size() > 0 &&
std::find(limitIfaces.begin(), limitIfaces.end(), ifnameStr) == limitIfaces.end()) {
// Nothing matched; skip this line.
return Result<void>();
}
if (limitTag != TAG_ALL && uint32_t(limitTag) != key.tag) {
return Result<void>();
}
if (limitUid != UID_ALL && uint32_t(limitUid) != key.uid) {
return Result<void>();
}
Result<StatsValue> statsEntry = statsMap.readValue(key);
if (!statsEntry.ok()) {
return base::ResultError(statsEntry.error().message(), statsEntry.error().code());
}
lines->push_back(populateStatsEntry(key, statsEntry.value(), ifname));
# 添加 bpf 收集的統計信息
ALOGD("network bpf ifname:%s", ifname);
ALOGD("network bpf key.tag:%d", key.tag);
ALOGD("network bpf key.uid:%d", key.uid);
ALOGD("network bpf rxBytes:%d", (int)(statsEntry.value().rxBytes));
ALOGD("network bpf txBytes:%d", (int)(statsEntry.value().txBytes));
return Result<void>();
};
Result<void> res = statsMap.iterate(processDetailUidStats);
if (!res.ok()) {
ALOGE("failed to iterate per uid Stats map for detail traffic stats: %s",
strerror(res.error().code()));
return -res.error().code();
}
// Since eBPF use hash map to record stats, network stats collected from
// eBPF will be out of order. And the performance of findIndexHinted in
// NetworkStats will also be impacted.
//
// Furthermore, since the StatsKey contains iface index, the network stats
// reported to framework would create items with the same iface, uid, tag
// and set, which causes NetworkStats maps wrong item to subtract.
//
// Thus, the stats needs to be properly sorted and grouped before reported.
groupNetworkStats(lines);
return 0;
}
使用 瀏覽器下載一個大文件,使用 logcat 觀察數據記錄過程
# 查看 瀏覽器 UID 爲 10125
/ # cat data/system/packages.list | grep browser
org.chromium.browser 10125 0 /data/user/0/org.chromium.browser default:targetSdkVersion=29 3002,3003,3001 0 398713230
# logcat 中的記錄 key.uid rxBytes txBytes
09-07 10:41:12.281 5020 5326 D BpfNetworkStats: network bpf key.tag:0
09-07 10:41:12.281 5020 5326 D BpfNetworkStats: network bpf key.uid:10125
09-07 10:41:12.281 5020 5326 D BpfNetworkStats: network bpf rxBytes:3900464
09-07 10:41:12.281 5020 5326 D BpfNetworkStats: network bpf txBytes:207692
09-07 10:41:14.449 5020 5326 D BpfNetworkStats: network bpf ifname:wlan0
參考:
eBPF 流量監控
https://source.android.google.cn/devices/tech/datausage/ebpf-traffic-monitor?hl=zh-cn&skip_cache=true
使用 eBPF 擴展內核
https://source.android.google.cn/docs/core/architecture/kernel/bpf?hl=zh-cn
理解android-ebpf
https://sniffer.site/2021/11/26/理解android-ebpf/