在上一節,我們安裝好了所有caffe可能用到的軟件,我這裏再列舉一下:
- NVIDIA驅動367.57
- CUDA8.0
- Cudnn v5.1
- MKL(這裏我提一句,最後BLAS可以不用MKL,用openblas或者atlas都可以)
- MATLAB2014a
- Anaconda(Python 2.7版)
- OpenCV 2.4.13
接下來,我們進入caffe的安裝與編譯環節。
在安裝caffe前,我們先來安裝一些依賴
sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler; sudo apt-get install -y libatlas-base-dev; sudo apt-get install -y --no-install-recommends libboost-all-dev; sudo apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev; sudo apt-get install -y python-pip python-dev python-numpy python-scipy; sudo apt-get install -y libopencv-dev;
安裝依賴後,我們再來修改Make.config文件,如下:
## Refer to http://caffe.berkeleyvision.org/installation.html # Contributions simplifying and improving our build system are welcome! # cuDNN acceleration switch (uncomment to build with cuDNN). USE_CUDNN := 1 # CPU-only switch (uncomment to build without GPU support). # CPU_ONLY := 1 # uncomment to disable IO dependencies and corresponding data layers USE_OPENCV := 0 USE_LEVELDB := 0 USE_LMDB := 0 # uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary) # You should not set this flag if you will be reading LMDBs with any # possibility of simultaneous read and write # ALLOW_LMDB_NOLOCK := 1 # Uncomment if you're using OpenCV 3 # OPENCV_VERSION := 3 # To customize your choice of compiler, uncomment and set the following. # N.B. the default for Linux is g++ and the default for OSX is clang++ # CUSTOM_CXX := g++ # CUDA directory contains bin/ and lib/ directories that we need. CUDA_DIR := /usr/local/cuda # On Ubuntu 14.04, if cuda tools are installed via # "sudo apt-get install nvidia-cuda-toolkit" then use this instead: # CUDA_DIR := /usr # CUDA architecture setting: going with all of them. # For CUDA < 6.0, comment the *_50 lines for compatibility. CUDA_ARCH := -gencode arch=compute_20,code=sm_20 \ -gencode arch=compute_20,code=sm_21 \ -gencode arch=compute_30,code=sm_30 \ -gencode arch=compute_35,code=sm_35 \ -gencode arch=compute_50,code=sm_50 \ -gencode arch=compute_50,code=compute_50 # BLAS choice: # atlas for ATLAS (default) # mkl for MKL # open for OpenBlas BLAS := atlas # Custom (MKL/ATLAS/OpenBLAS) include and lib directories. # Leave commented to accept the defaults for your choice of BLAS # (which should work)! #BLAS_INCLUDE :=/opt/intel/mkl/include #BLAS_LIB :=/opt/intel/mkl/lib/intel64 # Homebrew puts openblas in a directory that is not on the standard search path # BLAS_INCLUDE := $(shell brew --prefix openblas)/include # BLAS_LIB := $(shell brew --prefix openblas)/lib # This is required only if you will compile the matlab interface. # MATLAB directory should contain the mex binary in /bin. MATLAB_DIR := /usr/local/MATLAB/R2014a # MATLAB_DIR := /Applications/MATLAB_R2012b.app # NOTE: this is required only if you will compile the python interface. # We need to be able to find Python.h and numpy/arrayobject.h. #PYTHON_INCLUDE := /usr/include/python2.7 \ /usr/lib/python2.7/dist-packages/numpy/core/include # Anaconda Python distribution is quite popular. Include path: # Verify anaconda location, sometimes it's in root. ANACONDA_HOME := $(HOME)/anaconda2 PYTHON_INCLUDE := $(ANACONDA_HOME)/include \ $(ANACONDA_HOME)/include/python2.7 \ $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \ # Uncomment to use Python 3 (default is Python 2) # PYTHON_LIBRARIES := boost_python3 python3.5m # PYTHON_INCLUDE := /usr/include/python3.5m \ # /usr/lib/python3.5/dist-packages/numpy/core/include # We need to be able to find libpythonX.X.so or .dylib. #PYTHON_LIB := /usr/lib PYTHON_LIB := $(ANACONDA_HOME)/lib # Homebrew installs numpy in a non standard path (keg only) # PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include # PYTHON_LIB += $(shell brew --prefix numpy)/lib # Uncomment to support layers written in Python (will link against Python libs) WITH_PYTHON_LAYER := 1 # Whatever else you find you need goes here. INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial # If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies # INCLUDE_DIRS += $(shell brew --prefix)/include # LIBRARY_DIRS += $(shell brew --prefix)/lib # Uncomment to use `pkg-config` to specify OpenCV library paths. # (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.) # USE_PKG_CONFIG := 1 # N.B. both build and distribute dirs are cleared on `make clean` BUILD_DIR := build DISTRIBUTE_DIR := distribute # Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171 # DEBUG := 1 # The ID of the GPU that 'make runtest' will use to run unit tests. TEST_GPUID := 0 # enable pretty build (comment to see full commands) Q ?= @
最後我們來編譯caffe和python,matlab與caffe的接口,並測試。
編譯caffe:sudo make all -j16; sudo make test -j16; sudo make runtest -j16;
編譯pycaffe和matcaffe:
sudo make pycaffe -j16; sudo make matcaffe -j16; sudo make pytest -j16; sudo make mattest -j16;
如果以上命令全部成功執行,我們就完全安裝好了caffe。
如果安裝過程中出現了錯誤,那該怎麼處理呢?
我在這裏就列舉一下,當初我安裝caffe與編譯pycaffe和matcaffe時,出現的錯誤。
編譯caffe時出現的錯誤
boost錯誤
CXX src/caffe/layers/hinge_loss_layer.cpp CXX src/caffe/layers/threshold_layer.cpp In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/relu_layer.hpp:6, from src/caffe/layers/relu_layer.cpp:4: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/power_layer.hpp:6, from src/caffe/layers/power_layer.cpp:3: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/relu_layer.o] Error 1 make: *** Waiting for unfinished jobs.... make: *** [.build_release/src/caffe/layers/power_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/filler.hpp:10, from src/caffe/layers/bias_layer.cpp:3: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/cudnn_relu_layer.hpp:6, from src/caffe/layers/cudnn_relu_layer.cpp:4: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_relu_layer.o] Error 1 make: *** [.build_release/src/caffe/layers/bias_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/silence_layer.hpp:6, from src/caffe/layers/silence_layer.cpp:3: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/reshape_layer.hpp:6, from src/caffe/layers/reshape_layer.cpp:3: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/silence_layer.o] Error 1 make: *** [.build_release/src/caffe/layers/reshape_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/threshold_layer.hpp:6, from src/caffe/layers/threshold_layer.cpp:3: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/threshold_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/sigmoid_layer.hpp:6, from src/caffe/layers/sigmoid_layer.cpp:4: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/sigmoid_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/concat_layer.hpp:6, from src/caffe/layers/concat_layer.cpp:3: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/concat_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/multinomial_logistic_loss_layer.hpp:6, from src/caffe/layers/multinomial_logistic_loss_layer.cpp:5: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/hinge_loss_layer.hpp:6, from src/caffe/layers/hinge_loss_layer.cpp:4: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/multinomial_logistic_loss_layer.o] Error 1 make: *** [.build_release/src/caffe/layers/hinge_loss_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/cudnn_tanh_layer.hpp:6, from src/caffe/layers/cudnn_tanh_layer.cpp:4: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_tanh_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/flatten_layer.hpp:6, from src/caffe/layers/flatten_layer.cpp:3: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/flatten_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/cudnn_lcn_layer.hpp:6, from src/caffe/layers/cudnn_lcn_layer.cpp:4: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_lcn_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/cudnn_conv_layer.hpp:6, from src/caffe/layers/cudnn_conv_layer.cpp:5: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_conv_layer.o] Error 1 In file included from ./include/caffe/blob.hpp:8:0, from ./include/caffe/layers/softmax_loss_layer.hpp:6, from src/caffe/layers/softmax_loss_layer.cpp:5: ./include/caffe/common.hpp:4:32: fatal error: boost/shared_ptr.hpp: No such file or directory #include <boost/shared_ptr.hpp> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/softmax_loss_layer.o] Error 1
解決方法:
這是由於沒有安裝boost導致的錯誤。要想使用caffe,安裝的boost版本必須高於1.55。所以在命令行執行以下即可。
sudo apt-get install libboost1.55-*
cblas錯誤
In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/filler.hpp:13, from src/caffe/layers/bias_layer.cpp:3: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/bias_layer.o] Error 1 make: *** Waiting for unfinished jobs.... In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/cudnn_tanh_layer.hpp:7, from src/caffe/layers/cudnn_tanh_layer.cpp:4: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_tanh_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/concat_layer.hpp:7, from src/caffe/layers/concat_layer.cpp:3: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/concat_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/multinomial_logistic_loss_layer.hpp:7, from src/caffe/layers/multinomial_logistic_loss_layer.cpp:5: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/multinomial_logistic_loss_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/softmax_loss_layer.hpp:7, from src/caffe/layers/softmax_loss_layer.cpp:5: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/softmax_loss_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/cudnn_conv_layer.hpp:7, from src/caffe/layers/cudnn_conv_layer.cpp:5: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_conv_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/cudnn_relu_layer.hpp:7, from src/caffe/layers/cudnn_relu_layer.cpp:4: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/relu_layer.hpp:7, from src/caffe/layers/relu_layer.cpp:4: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_relu_layer.o] Error 1 make: *** [.build_release/src/caffe/layers/relu_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/reshape_layer.hpp:7, from src/caffe/layers/reshape_layer.cpp:3: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/reshape_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/power_layer.hpp:7, from src/caffe/layers/power_layer.cpp:3: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/power_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/silence_layer.hpp:7, from src/caffe/layers/silence_layer.cpp:3: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/silence_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/cudnn_lcn_layer.hpp:7, from src/caffe/layers/cudnn_lcn_layer.cpp:4: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/cudnn_lcn_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/threshold_layer.hpp:7, from src/caffe/layers/threshold_layer.cpp:3: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/threshold_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/sigmoid_layer.hpp:7, from src/caffe/layers/sigmoid_layer.cpp:4: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/sigmoid_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/flatten_layer.hpp:7, from src/caffe/layers/flatten_layer.cpp:3: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/flatten_layer.o] Error 1 In file included from ./include/caffe/util/math_functions.hpp:11:0, from ./include/caffe/layer.hpp:12, from ./include/caffe/layers/hinge_loss_layer.hpp:7, from src/caffe/layers/hinge_loss_layer.cpp:4: ./include/caffe/util/mkl_alternate.hpp:14:19: fatal error: cblas.h: No such file or directory #include <cblas.h> ^ compilation terminated. make: *** [.build_release/src/caffe/layers/hinge_loss_layer.o] Error 1
libcudart 錯誤
當編譯caffe時,出現有關libcudart錯誤時,往往是cuda的路徑沒有設置對。這個時候我們要修改Makefile.config文件和ld.conf.d文件夾下的cuda.conf文件(如果該文件夾下沒有該文件則要新建一個)# 修改makefile文件,添加/usr/local/cuda/include和/usr/local/cuda/lib64 INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/local/cuda/include LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/local/cuda/lib64 #修改目錄/etc/ld.so.conf.d下的cuda.conf文件,將/usr/local/cuda/include和/usr/local/cuda/lib64添加進去 cd /etc/ld.so.conf.d sudo vim cuda.conf #insert two paths above
至此caffe應該可以順利的編譯完畢。
接下來我們看看編譯pycaffe時會出現的問題
安裝一些python依賴
進入caffe目錄下的python文件夾然後執行下列操作:
for req in $(cat requirements.txt); do pip install $req; done
這樣就安裝了一些python的依賴skimage.io錯誤
如果出現該錯誤,說明有一些依賴沒有正確的安裝,執行以下程序:sudo apt-get install python-numpy python-scipy python-matplotlib python-sklearn python-skimage python-h5py python-protobuf python-leveldb python-networkx python-nose python-pandas python-gflags Cython ipython;#主要是看看python-skimage有沒有安裝
ImportError: No module named google.protobuf.internal
這是由於沒有安裝protobuf導致的。
pip install protobuf
就ok了import caffe時出現了問題:Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so
這個跟anaconda有關,一般情況下,我們使用Linux自帶的python就可以了,沒必要整anaconda。如果非要使用anaconda的話。使用一下辦法解決這個問題。
conda install nomkl numpy scipy scikit-learn numexprconda remove mkl mkl-service
安裝一些包,移除一些包就可以了。
最後就是編譯matcaffe出現的問題了。
我最近重裝了一次標準版的caffe,編譯matcaffe時只出現了一個問題。就是要用-std=c++11的gcc去編譯matcaffe。這個很好解決,只需要在Makefile文件中查找CXXFLAG,然後在一個什麼mp的表達式下面加上一句話就行了。
編譯完之後的路徑設置問題。
我們編譯好了caffe和pycaffe以及matcaffe後,需要將caffe和pycaffe的庫文件路徑添加到系統路徑中,這樣我們才能正常的使用caffe。
涉及到的文件有:- ~/.bashrc
- /etc/profile
- /etc/ld.conf.d 目錄下的 ~.conf文件
編譯完之後的測試問題
- mnist測試caffe命令行版本
以上是編譯普通caffe 的方法,這一套流程我已經走過很多遍了,應該沒有問題。
這裏需要注意的是,當你需要編譯ssd 版的caffe或者別的研究團隊改過的caffe時,最好使用cmake 的方法編譯,具體內容以後有時間我會補充上來的。