首先我是在python3的環境上面完成的。我保留了python2的版本,然後安裝python3的版本。然後在安裝scrapy的過程中出現的錯誤,以及切換python版本後出現的錯誤。
一、安裝python3
cd /usr/local/src/
wget https://www.python.org/ftp/python/3.4.2/Python-3.4.2.tgz
tar -xf Python-3.4.2.tgz
cd Python-3.4.2
./configure --prefix=/usr/local/python3
make && make install
##將python做一個備份,然後把python3的建立一個軟連接
mv /usr/bin/python /usr/bin/pythonbak
ln -fs /usr/local/python3/bin/python3 /usr/bin/python
再在終端進入python交互模式,出現的是python3了。
[root@lsf ~]# python
Python 3.4.2 (default, Mar 13 2018, 11:37:48)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux
Type "help", "copyright", "credits" or "license" for more information.
##python升級到3後,yum無法正常使用,此時需要改一個文件
[root@lsf ~]# cat /usr/bin/yum
#!/usr/bin/python2.7
將第一行改爲python2.7
二、安裝scrapy
##建立pip3的軟連接
ln -fs /usr/local/python3/bin/pip3 /usr/bin/pip3
##yum安裝依賴包,可以解決編譯過程中出現的許多錯誤
yum install -y gcc openssl-devel libxml2-devel libxslt-devel bzip2-devel bzip2* libffi-devel python-devel openssl-devel pycrypto
##用pip3安裝
pip3 install twisted --upgrade
pip3 install cryptography pycrypto
##安裝scrapy
pip3 install scrapy
##建立scrapy軟連接
ln -fs /usr/local/python3/bin/scrapy /usr/bin/scrapy
三、測試scrapy命令
[root@lsf ~]# scrapy
Scrapy 1.5.0 - no active project
Usage:
scrapy <command> [options] [args]
Available commands:
bench Run quick benchmark test
fetch Fetch a URL using the Scrapy downloader
genspider Generate new spider using pre-defined templates
runspider Run a self-contained spider (without creating a project)
settings Get settings values
shell Interactive scraping console
startproject Create new project
version Print Scrapy version
view Open URL in browser, as seen by Scrapy
[ more ] More commands available when run from project directory
Use "scrapy <command> -h" to see more info about a command
##創建一個scrapy項目
scrapy startproject myfirstpjt
##可能會報錯ImportError: cannot import name 'certificate_transparency'
pip3 install pip3 --upgrade
##再次安裝scrapy
pip3 install scrapy
##創建項目成功後,會出現下面的文件
[root@lsf test_scrapy]# cd myfirstpjt/
[root@lsf myfirstpjt]# ls
myfirstpjt scrapy.cfg
[root@lsf myfirstpjt]# cd myfirstpjt/
[root@lsf myfirstpjt]# ls
__init__.py items.py middlewares.py pipelines.py __pycache__ settings.py spiders
然後就可以直接使用scrapy命令進行項目管理了。