filecoin lotus benchmarks v26 GeForce GTX 2080 Ti
1，編譯bench
- 版本
# git show
commit d11e375f828526eab2a59fd6fd3db88e3c5cb9d3 (HEAD -> interopnet, tag: interop.5.13, origin/interopnet)
Merge: 7b8ee402 1d8c8b9a
Author: Travis Person <[email protected]>
Date: Wed May 13 05:44:28 2020 +0000
Merge remote-tracking branch 'origin/fix/check-gas-limits' into interopnet
- 設置環境變量
# export ALL_PROXY=socks5://127.0.0.1:1080
# git config --global http.proxy 'socks5://127.0.0.1:1080'
# export GO111MODULE=on
# export GOPROXY=https://goproxy.cn
env RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean deps bench
- 最後顯示，編譯成功，當前目錄可以看到
bench
go build -o bench ./cmd/lotus-bench
go run github.com/GeertJohan/go.rice/rice append --exec bench -i ./build
bench版本
# ./bench --version
2020-05-14T15:01:36.340+0800 INFO lotus-bench lotus-bench/main.go:74 Starting lotus-bench
lotus-bench version 0.3.0
2，基準測試
- 設置環境變量
RUST_LOGrust日誌FIL_PROOFS_MAXIMIZE_CACHING最大化內存參數
# export RUST_LOG=Trace && export FIL_PROOFS_MAXIMIZE_CACHING=1
- 開始測試
# ./bench sealing --storage-dir=/storage/benchtmp --sector-size=32GiB
- 或者後臺運行
# nohup ./bench sealing --storage-dir=/raid5/benchtmp --sector-size=32GiB >> /raid5/bench.log 2>&1 &
- 報錯
2020-05-14T23:16:04.644 INFO filecoin_proofs::api::seal > seal_commit_phase1:end
2020-05-14T23:16:04.711 INFO filcrypto::proofs::api > seal_commit_phase1: finish
2020-05-14T23:16:04.805+0800 INFO lotus-bench lotus-bench/main.go:499 Generating PoRep for sector (2)
2020-05-14T23:16:04.825 INFO filcrypto::proofs::api > seal_commit_phase2: start
2020-05-14T23:16:04.900 INFO filecoin_proofs::api::seal > seal_commit_phase2:start
2020-05-14T23:16:04.901 INFO storage_proofs_porep::stacked::vanilla::graph > using parents cache of unlimited size
2020-05-14T23:16:04.901 INFO storage_proofs_porep::stacked::vanilla::graph > using parent_cache[1073741824]
2020-05-14T23:16:04.902 INFO filecoin_proofs::caches > trying parameters memory cache for: STACKED[34359738368]
2020-05-14T23:16:04.902 INFO filecoin_proofs::caches > no params in memory cache for STACKED[34359738368]
2020-05-14T23:16:04.904 INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: layered_drgporep::PublicParams{ graph: stacked_graph::StackedGraph{expansion_degree: 8 base_graph: drgraph::BucketGraph{size: 1073741824; degree: 6; hasher: poseidon_hasher} }, challenges: LayerChallenges { layers: 11, max_count: 18 }, tree: merkletree-poseidon_hasher-8-8-0 }
2020-05-14T23:16:04.904 INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/storage/filecoin-proof-parameters-v26/parameters-v26/v26-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params" exist
2020-05-14T23:16:04.904 INFO storage_proofs_core::parameter_cache > checking cache_path: "/storage/filecoin-proof-parameters-v26/parameters-v26/v26-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params" for parameters
2020-05-14T23:16:07.290 INFO storage_proofs_core::parameter_cache > read parameters from cache "/storage/filecoin-proof-parameters-v26/parameters-v26/v26-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.params"
2020-05-14T23:16:07.290 INFO filecoin_proofs::api::seal > got groth params (34359738368) while sealing
2020-05-14T23:16:07.290 INFO storage_proofs_porep::stacked::vanilla::graph > using parents cache of unlimited size
2020-05-14T23:16:07.290 INFO storage_proofs_porep::stacked::vanilla::graph > using parent_cache[1073741824]
2020-05-14T23:16:07.290 INFO filecoin_proofs::api::seal > snark_proof:start
2020-05-14T23:16:07.298 INFO bellperson::groth16::prover > Bellperson 0.7.0 is being used!
2020-05-14T23:20:18.490 INFO bellperson::gpu::locks > GPU is available for FFT!
2020-05-14T23:20:18.490 DEBUG bellperson::gpu::locks > Acquiring GPU lock...
2020-05-14T23:20:18.496 DEBUG bellperson::gpu::locks > GPU lock acquired!
2020-05-14T23:20:18.732 INFO bellperson::gpu::fft > FFT: 1 working device(s) selected.
2020-05-14T23:20:18.732 INFO bellperson::gpu::fft > FFT: Device 0: GeForce RTX 2080 Ti
2020-05-14T23:20:18.732 INFO bellperson::domain > GPU FFT kernel instantiated!
2020-05-14T23:23:12.352 DEBUG bellperson::gpu::locks > GPU lock released!
2020-05-14T23:24:13.677 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2020-05-14T23:24:13.677 DEBUG bellperson::gpu::locks > Acquiring GPU lock...
2020-05-14T23:24:13.840 DEBUG bellperson::gpu::locks > GPU lock acquired!
2020-05-14T23:24:13.933 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
2020-05-14T23:24:13.933 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce RTX 2080 Ti (Chunk-size: 7488999)
2020-05-14T23:24:13.933 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2020-05-14T23:50:07.906 DEBUG bellperson::gpu::locks > GPU lock released!
2020-05-14T23:50:09.849 INFO filecoin_proofs::api::seal > snark_proof:finish
2020-05-14T23:50:09.849 INFO storage_proofs_porep::stacked::vanilla::graph > using parents cache of unlimited size
2020-05-14T23:50:09.849 INFO storage_proofs_porep::stacked::vanilla::graph > using parent_cache[1073741824]
2020-05-14T23:50:09.849 INFO storage_proofs_porep::stacked::vanilla::graph > using parents cache of unlimited size
2020-05-14T23:50:09.849 INFO storage_proofs_porep::stacked::vanilla::graph > using parent_cache[1073741824]
2020-05-14T23:50:09.849 INFO filecoin_proofs::caches > trying parameters memory cache for: STACKED[34359738368]-verifying-key
2020-05-14T23:50:09.849 INFO filecoin_proofs::caches > no params in memory cache for STACKED[34359738368]-verifying-key
2020-05-14T23:50:09.850 INFO storage_proofs_core::parameter_cache > parameter set identifier for cache: layered_drgporep::PublicParams{ graph: stacked_graph::StackedGraph{expansion_degree: 8 base_graph: drgraph::BucketGraph{size: 1073741824; degree: 6; hasher: poseidon_hasher} }, challenges: LayerChallenges { layers: 11, max_count: 18 }, tree: merkletree-poseidon_hasher-8-8-0 }
2020-05-14T23:50:09.850 INFO storage_proofs_core::parameter_cache > ensuring that all ancestor directories for: "/storage/filecoin-proof-parameters-v26/parameters-v26/v26-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk" exist
2020-05-14T23:50:09.850 INFO storage_proofs_core::parameter_cache > checking cache_path: "/storage/filecoin-proof-parameters-v26/parameters-v26/v26-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk" for verifying key
2020-05-14T23:50:09.946 INFO storage_proofs_core::parameter_cache > read verifying key from cache "/storage/filecoin-proof-parameters-v26/parameters-v26/v26-stacked-proof-of-replication-merkletree-poseidon_hasher-8-8-0-sha256_hasher-82a357d2f2ca81dc61bb45f4a762807aedee1b0a53fd6c4e77b46a01bfef7820.vk"
2020-05-14T23:50:09.946 INFO filecoin_proofs::api::seal > got verifying key (34359738368) while verifying seal
2020-05-14T23:50:09.971 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2020-05-14T23:50:09.971 DEBUG bellperson::gpu::locks > Acquiring GPU lock...
2020-05-14T23:50:10.023 DEBUG bellperson::gpu::locks > GPU lock acquired!
2020-05-14T23:50:10.123 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
2020-05-14T23:50:10.124 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce RTX 2080 Ti (Chunk-size: 7488999)
2020-05-14T23:50:10.124 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2020-05-14T23:50:10.219 DEBUG bellperson::gpu::locks > GPU lock released!
2020-05-14T23:50:10.219 INFO filecoin_proofs::api::seal > seal_commit_phase2:end
2020-05-14T23:50:10.219 INFO filcrypto::proofs::api > seal_commit_phase2: finish
2020-05-14T23:50:10.219 INFO filcrypto::proofs::api > verify_seal: start
2020-05-14T23:50:10.220 INFO storage_proofs_porep::stacked::vanilla::graph > using parents cache of unlimited size
2020-05-14T23:50:10.220 INFO storage_proofs_porep::stacked::vanilla::graph > using parent_cache[1073741824]
2020-05-14T23:50:10.220 INFO storage_proofs_porep::stacked::vanilla::graph > using parents cache of unlimited size
2020-05-14T23:50:10.220 INFO storage_proofs_porep::stacked::vanilla::graph > using parent_cache[1073741824]
2020-05-14T23:50:10.220 INFO filecoin_proofs::caches > trying parameters memory cache for: STACKED[34359738368]-verifying-key
2020-05-14T23:50:10.220 INFO filecoin_proofs::caches > found params in memory cache for STACKED[34359738368]-verifying-key
2020-05-14T23:50:10.220 INFO filecoin_proofs::api::seal > got verifying key (34359738368) while verifying seal
2020-05-14T23:50:10.237 INFO bellperson::gpu::locks > GPU is available for Multiexp!
2020-05-14T23:50:10.237 DEBUG bellperson::gpu::locks > Acquiring GPU lock...
2020-05-14T23:50:10.414 DEBUG bellperson::gpu::locks > GPU lock acquired!
2020-05-14T23:50:10.522 INFO bellperson::gpu::multiexp > Multiexp: 1 working device(s) selected. (CPU utilization: 0)
2020-05-14T23:50:10.522 INFO bellperson::gpu::multiexp > Multiexp: Device 0: GeForce RTX 2080 Ti (Chunk-size: 7488999)
2020-05-14T23:50:10.522 INFO bellperson::multiexp > GPU Multiexp kernel instantiated!
2020-05-14T23:50:10.612 DEBUG bellperson::gpu::locks > GPU lock released!
2020-05-14T23:50:10.612 INFO filcrypto::proofs::api > verify_seal: finish
2020-05-14T23:50:22.791+0800 WARN lotus-bench lotus-bench/main.go:90 failed to run seals:
main.glob..func3
/root/interopnet/lotus/cmd/lotus-bench/main.go:233
- porep proof for sector 1 was invalid:
main.runSeals
/root/interopnet/lotus/cmd/lotus-bench/main.go:550
3，GeForce GTX 2080 Ti 測試
- CPU
3950X
# lscpu
AMD Ryzen 9 3950X 16-Core Processor
- 顯卡
GeForce GTX 1080 Ti
# lspci | grep -i nvidia
2d:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
2d:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
- 內存，交換分區
# free -h
總計 已用 空閒 共享 緩衝/緩存 可用
內存： 125Gi 729Mi 104Gi 0.0Ki 20Gi 123Gi
交換： 105Gi 302Mi 105Gi
- bench 編譯版本
# git show
commit 54b3eafd6d541a68199965bbf13ec2e339c625f5 (HEAD -> master, origin/master, origin/HEAD)
Merge: 5c73602d f6589407
Author: Whyrusleeping <[email protected]>
Date: Fri May 15 13:05:27 2020 -0700
Merge pull request #1758 from filecoin-project/fix/lotus-bench-secp
register secp signatures for lotus bench usage
- bench編譯
env RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make clean deps bench
- bench 運行
# export RUST_LOG=Trace && export FIL_PROOFS_MAXIMIZE_CACHING=1
nohup ./bench sealing --storage-dir=/raid5/benchtmp --sector-size=32GiB >> /raid5/bench.log 2>&1 &
- bench 測試結果，跑完大概12個小時
results (v26) (34359738368)
seal: addPiece: 6m26.577337189s (84.8 MiB/s)
seal: preCommit phase 1: 4h17m35.603586994s (2.12 MiB/s)
seal: preCommit phase 2: 1h43m21.981600421s (5.28 MiB/s)
seal: commit phase 1: 1m4.253378137s (510 MiB/s)
seal: commit phase 2: 1h2m37.451786187s (8.72 MiB/s)
seal: verify: 37.58607ms
unseal: 4h17m51.34613723s (2.12 MiB/s)
generate candidates: 7.194132ms (4.34 TiB/s)
compute winnnig post proof (cold): 14.527132336s
compute winnnig post proof (hot): 6.148464516s
verify winnnig post proof (cold): 98.07689ms
verify winnnig post proof (hot): 14.109101ms
compute window post proof (cold): 15m57.884786147s
compute window post proof (hot): 16m16.748839396s
verify window post proof (cold): 5.504320785s
verify window post proof (hot): 44.248935ms
- ben運行日誌
- bench完整運行日誌
參考：