Linux升級wget/curl用於下載https文件的過程

因爲需要提升服務器的性能以及支持mysql更新版本的某些特性,因此決定升級mysql版本從5.1.308.0,目標確定下來就開始幹。

Mysql安裝方式選擇

Linux上安裝應用,一般有三種方式,優劣對比分別爲:
圖片描述

因此我們選擇二進制安裝,安裝簡單、方便,支持多個Mysql版本同時存在。

Linux上安裝二進制版本的應用,統一爲三步:

// 通過配置自動生成文件
./configure
// 編譯文件
make
// 檢查自測單元,看編譯是否通過,可以省略該步,不影響安裝
make check
// 安裝
make install

卸載通過二進制安裝的程序:

// 方式一: 在編譯目錄裏執行卸載
make uninstall
// 方式二:找到安裝目錄,然後刪除,如nettle程序
$find / -name nettle
/usr/include/nettle
rm -rf /usr/include/nettle

wget不支持https

我們可以在mysql官網下載最新版本的mysql8.0.13二進制文件,注意官網提供的下載鏈接是https協議的,當我們在服務器執行下載命令:

// 使用wget或者curl來下載文件
wget https://dev.mysql.com/downloads/file/?id=480751
curl -O https://dev.mysql.com/downloads/file/?id=480751

會報錯:

$wget https://dev.mysql.com/downloads/file/?id=480751
--2018-12-12 16:57:54--  https://dev.mysql.com/downloads/file/?id=480751
Resolving dev.mysql.com (dev.mysql.com)... 137.254.60.11
Connecting to dev.mysql.com (dev.mysql.com)|137.254.60.11|:443... connected.
GnuTLS: A TLS fatal alert has been received.
GnuTLS: received alert [40]: Handshake failed
Unable to establish SSL connection.

$curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error

根據網上查詢到的答案,原因均爲版本過低,當前的版本不支持https協議的下載:

So the error actually happens with www.coursera.org and the reason is missing support for SNI. You need to upgrade your version of wget.

當前的版本:

$wget --version
GNU Wget 1.16.3 built on linux-gnu.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie -psl +ssl/gnutls 

Wgetrc: 
    /usr/local/etc/wgetrc (system)
Locale: 
    /usr/local/share/locale 
Compile: 
    gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc" 
    -DLOCALEDIR="/usr/local/share/locale" -I. -I../lib -I../lib 
    -DHAVE_LIBGNUTLS -DNDEBUG 
Link: 
    gcc -DHAVE_LIBGNUTLS -DNDEBUG -lpcre -lnettle -lgnutls -lz -lidn 
    -lrt ftp-opie.o gnutls.o http-ntlm.o ../lib/libgnu.a 

Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <[email protected]>.
Please send bug reports and questions to <[email protected]>.

------------------

$curl.7.19.7 --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp 
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz 

wget升級

既然是版本過低,升級即可,直接安裝新版本的wget,然後卸載掉原有的wget文件即可。

下載完成1.20版本之後

// 獲得文件wget-1.20.tar.gz
wget http://mirror.sergal.org/gnu/wget/wget-1.20.tar.gz
// 解壓縮
tar -xzvf wget-1.20.tar.gz
// 進入解壓後的文件夾
cd wget-1.20
// 開始配置
./configure
// 然而報錯了:
...
checking for GNUTLS... no
configure: error: Package requirements (gnutls) were not met:

No package 'gnutls' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables GNUTLS_CFLAGS
and GNUTLS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

報錯信息顯示我們沒有安裝gnutls依賴,需要繼續進行安裝,更爲詳細的報錯信息,可以查看config.log:

// 查看詳細報錯信息
vim config.log

...
PKG_CONFIG='/usr/local/bin/pkg-config'
...
configure:44443: checking for GNUTLS
configure:44450: $PKG_CONFIG --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found
configure:44453: $? = 1
configure:44467: $PKG_CONFIG --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found
configure:44470: $? = 1
configure:44484: result: no
No package 'gnutls' found
configure:44500: error: Package requirements (gnutls) were not met:

No package 'gnutls' found

Consider adjusting the PKG_CONFIG_PATH environment variable if you                                                                            
installed software in a non-standard prefix.                                                                                                  
                                                                                                                                              
Alternatively, you may set the environment variables GNUTLS_CFLAGS                                                                            
and GNUTLS_LIBS to avoid the need to call pkg-config.                                                                                         
See the pkg-config man page for more details.

這裏可以看出配置腳本實際是執行了:

$ /usr/local/bin/pkg-config --exists --print-errors "gnutls"
Package gnutls was not found in the pkg-config search path.
Perhaps you should add the directory containing `gnutls.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gnutls' found

安裝gnutls

gnutls全稱 GNU Transport Layer Security Library,即基於GNU版權協議的傳輸層安全協議,是wget支持https中的ssl協議的基礎庫。

我們可以在官方提供的鏡像庫裏快速下載並安裝:

// 下載gnutls二進制文件
wget http://www.ring.gr.jp/pub/net/gnupg/gnutls/v3.6/gnutls-3.6.4.tar.xz
// 解壓xz文件
xz -d gnutls-3.6.4.tar.xz
tar -xvf gnutls-3.6.4.tar
cd gnutls-3.6.4
./configure

// 報錯:
...
checking for NETTLE... no
configure: error: 
  ***
  *** Libnettle 3.4 was not found.
// 如果覺得可能不安全,可以下載md5簽名文件驗證文件,但是這個文件驗證後發現簽名過期了,所以沒辦法驗證了
wget http://www.ring.gr.jp/pub/net...
gpg --verify gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.tar
gpg --recv-key F1679A65
gpg --verify --verbose gnutls-3.6.4.tar.xz.sig gnutls-3.6.4.tar

查看詳細報錯信息:

$ vim config.log

...
configure:10032: checking for NETTLE
configure:10039: $PKG_CONFIG --exists --print-errors "nettle >= 3.4"
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found
configure:10042: $? = 1
configure:10056: $PKG_CONFIG --exists --print-errors "nettle >= 3.4"
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found
configure:10059: $? = 1
configure:10073: result: no
No package 'nettle' found
configure:10090: error:
  ***
  *** Libnettle 3.4 was not found.

結果顯示我們需要3.4版本以上的Libnettle庫,繼續安裝。

安裝Libnettle

Nettle庫是用於跨平臺的底層密碼庫,包含加密和解密的不同算法。我們下載並安裝nettle庫

wget ftp://ftp.gnu.org/gnu/nettle/nettle-3.4.1.tar.gz
tar -xzvf nettle-3.4.1.tar.gz
cd nettle-3.4.1
./configure
// 安裝成功
...
configure: summary of build options:

  Version:           nettle 3.4.1
  Host type:         x86_64-unknown-linux-gnu
  ABI:               64
  Assembly files:    x86_64
  Install prefix:    /usr/local
  Library directory: ${exec_prefix}/lib64
  Compiler:          gcc
  Static libraries:  yes
  Shared libraries:  yes
  Public key crypto: no
  Using mini-gmp:    no
  Documentation:     yes

make
make install

根據官方文檔,我們安裝完成後應該會有兩個文件lib{hogweed,nettle}.so,然而我們只能發現其中一個:

make install &&
chmod -v 755 /usr/lib/lib{hogweed,nettle}.so &&
install -v -m755 -d /usr/share/doc/nettle-3.4.1 &&
install -v -m644 nettle.html /usr/share/doc/nettle-3.4.1
$ ll | grep '\.so'
-rwxr-xr-x 1 root root 3675341 Dec 12 19:15 libnettle.so
$ ll | grep weed
-rw-rw-r-- 1 work work     529 Dec 10 15:30 hogweed.pc
-rw-r--r-- 1 work work     590 Nov 19  2017 hogweed.pc.in
-rw-rw-r-- 1 work work     298 Dec 10 15:30 libhogweed.map
-rw-r--r-- 1 work work     338 Nov 19  2017 libhogweed.map.in

少了一個libhogweed.so文件,稍後我們編譯gnutls時會發現這個導致的問題。

繼續編譯gnutls

既然nettle安裝完成了,我們可以繼續安裝gnutls

./configure

...
configure: error: 
  ***
  *** Libnettle 3.4 was not found.

依然報錯缺失庫,但我們明明已經安裝了,爲什麼找不到呢?我們用包管理工具查找一下:

$ pkg-config --modversion nettle
Package nettle was not found in the pkg-config search path.
Perhaps you should add the directory containing `nettle.pc'
to the PKG_CONFIG_PATH environment variable
No package 'nettle' found

我們找下這個nettle.pc剛纔安裝到哪裏去了:

$ locate nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc
/home/work/lib/nettle-3.4.1/nettle.pc.in
/usr/lib64/pkgconfig/nettle.pc
/usr/local/lib64/pkgconfig/nettle.pc

而我們pkg-config默認的管理包檢索路徑爲/usr/lib/pkgconfig,所以無法正常找到,參考pkgconfig文檔,有兩種方案:

// 方案一:鏈接該文件到默認目錄中
ln -s /usr/local/lib64/pkgconfig/nettle.pc /usr/lib/pkgconfig/nettle.pc  
// 方案二:全局變量中更改包的檢索路徑(只在本次終端窗口生效,退出後恢復,所以只能臨時使用一下)
$ echo $PKG_CONFIG_PATH

$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib64/pkgconfig/
$ echo $PKG_CONFIG_PATH
:/usr/local/lib64/pkgconfig/

// 任一方案執行後結果
$ pkg-config --modversion nettle
3.4.1

此時,我們繼續安裝:

$ ./configure
...
checking for NETTLE... yes
checking for HOGWEED... no
configure: error: 
  ***
  *** Libhogweed (nettle's companion library) was not found. Note that you must compile nettle with gmp support.

可以看到,我們的nettle庫已經安裝成功,但是hogweed卻檢查沒有找到,提示中也寫明瞭libhogweed需要字gmp庫已經安裝完成的情況下重新編譯nettle纔可以被安裝。

有人提出過相關的問題,我們也可以從官網文檔上更詳細的知道這個Nettle對於libhogweed的依賴:

5 Linking
Nettle actually consists of two libraries, libnettle and libhogweed. The libhogweed library contains those functions of Nettle that uses bignum operations, and depends on the GMP library. With this division, linking works the same for both static and dynamic libraries.

If an application uses only the symmetric crypto algorithms of Nettle (i.e., block ciphers, hash functions, and the like), it’s sufficient to link with -lnettle. If an application also uses public-key algorithms, the recommended linker flags are -lhogweed -lnettle -lgmp. If the involved libraries are installed as dynamic libraries, it may be sufficient to link with just -lhogweed, and the loader will resolve the dependencies automatically.

總而言之,就是沒有libhogweed.so這個文件不行,而它只能由nettle進行安裝。根據nettle庫官方資料顯示,libhogweed.so應該在nettle安裝時被自動生成,然而我們在上面的安裝過程中並沒有生成。那是不是因爲我沒有安裝gmp導致的呢?

安裝gmp

我們下載gmp庫並安裝,可以在編譯Nettleconfig.log中查看有一條warning,指明瞭版本需求:

$ vim config.log

...
configure:6583: result: no
configure:6594: WARNING: GNU MP not found, or too old. GMP-6.0 or later is needed, see https://gmplib.org/.
    Support for public key algorithms will be unavailable.

所以我們需要下載6.0版本後的:

// 這裏我只找到了官網的https版本,沒辦法,只好本地下載,然後rz到服務器,因爲是二進制文件,要帶上-be參數
rz -be
// 然後正常編譯
$ ./configure & make & make install
...
Libraries have been installed in:
   /usr/local/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

這裏提醒我們需要將動態庫鏈接到緩存中,我們採用第四種方案,可以參考ldconfig命令

$ vim /etc/ld.so.conf

// 添上安裝的.so文件路徑
/usr/local/lib
:wq

$ ldconfig
$ ldconfig -v | grep gmp
        libgmp.so.10 -> libgmp.so.10.3.2
        libgmpxx.so.4 -> libgmpxx.so.4.1.0
        libgmp.so.3 -> libgmp.so.3.5.0

看到libgmp.so.10就是我們安裝的最新版本,現在OK了。

然後重新編譯安裝nettle,會生成libhogweed.so文件:

$ ll | grep weed
-rw-r--r-- 1 root root     541 Dec 12 22:12 hogweed.pc
-rw-r--r-- 1 work work     590 Nov 19  2017 hogweed.pc.in
-rw-r--r-- 1 root root 6154192 Dec 12 22:13 libhogweed.a
-rw-r--r-- 1 root root     298 Dec 12 22:12 libhogweed.map
-rw-r--r-- 1 work work     338 Nov 19  2017 libhogweed.map.in
-rwxr-xr-x 1 root root 5519996 Dec 12 22:13 libhogweed.so
-rw-r--r-- 1 root root       8 Dec 12 22:13 libhogweed.stamp
請注意如果安裝完成後,如果出現多個版本的gmp庫,請刪除老版本的。具體刪除哪一項請自行斟酌,我刪除了所有的,然後在編譯的過程中,會報錯:can't find libgmp.so.3,說明libgmp.so.3這個是基礎庫,請不要動!
等我刪除了老版本的,重新編譯nettle就OK。如果你安裝成功了新版本,依然編譯不成功,請參考這個。

恐怖的依賴地獄

用二進制來安裝的時候,總是會出現各種各樣的問題,缺少各種依賴的包,解決方法就是缺什麼就去安什麼,但是會非常恐怖。爲了解決nettle安裝的問題,除了上面的gmp,我還安裝了最新版本的各種庫:

同時,由於gnutls編譯不通過的問題,又升級了pkg-config,它依賴於Libtasn1

繼續安裝gnutls庫(失敗、暫時放棄更新)

./configure
// 此時沒有錯誤信息了,但是還有很多WARNING信息
*** autogen not found. Will not link against libopts.
*** You will not be able to create source packages with 'make dist'
  because gtk-doc >= 1.14 is not found.
*** LIBIDN2 was not found. You will not be able to use IDN2008 support
*** libunbound was not found. Libdane will not be built.
*** trousers was not found. TPM support will be disabled.
*** `guile-snarf' from Guile not found.  Guile bindings not built.

*** The DNSSEC root key file in /etc/unbound/root.key was not found.
*** This file is needed for the verification of DNSSEC responses.
*** Use the command: unbound-anchor -a "/etc/unbound/root.key"
*** to generate or update it.

// 繼續編譯,又報錯了
make
...
tlsproxy/buffer.c:40: error: redefinition of typedef 'buffer_t'
tlsproxy/buffer.h:31: note: previous declaration of 'buffer_t' was here

暫時放棄更新wget,過幾天繼續嘗試,解決各種問題太費時間了

如果想要減少warning信息,可以更新autogen等庫:

安裝autogen過程中需要依賴guile,然而安裝guile時又報錯:guile configure: error: Cannot find a type to use in place of socklen_t
放棄更新autogen。

嘗試curl更新,層層依賴,放棄

$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) SSL connect error

根據報錯原因和網上資料是由於版本過老,需要更新curl版本。

官方地址下載curl後安裝,再次用新版本的curl請求:

$ curl https://dev.mysql.com/downloads/file/?id=480751
curl: (35) error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

還是報錯,根據上面的資料,如果依然不能解決問題,需要更新NSSNSSOpenSSl類似,都屬於底層密碼學,由Mozilla維護,MDN文檔提供安裝說明,它跟前面的二進制文件略有不同,不提供configure自動配置,詳細的查看它的文檔。

安裝NSS庫比較麻煩,還需要再安裝GYP庫,想起來又是層層嵌套的依賴關係,放棄更新。

解決方案:本地下載,rz上傳

在耗費兩天的時間後,我及時的終止了無畏的嘗試,轉而使用本地下載mysql8.0文件,然後rz -be上傳到服務器,搞定。

結論:

Linux上層層依賴的二進制文件安裝簡直是地獄版的體驗,在給我們帶來高自由度的同時也有無盡的煩惱,然而yum安裝版本又過低,不能滿足需求。雖然最終還是沒有成功更新wget或者curl,但是在整個過程中,也學習到了很多的新東西,在這篇文章總結一下過程,希望也能幫助一些人在某一步驟遇到的問題。

解決問題整體思路和過程

圖片描述

參考資料

  1. mysql8.0官網下載地址:https://dev.mysql.com/downloa...
  2. Mysql三種安裝方式詳解:https://www.jianshu.com/p/a04...
  3. [StackOverFlow] wget ssl alert handshake failure:https://stackoverflow.com/que...
  4. 卸載二進制程序:http://www.blogjava.net/zhyiw...
  5. wget下載地址:http://mirror.sergal.org/gnu/...
  6. gnutls下載地址:http://www.ring.gr.jp/pub/net...
  7. 利用.sig文件驗證數據的完整性:https://blog.csdn.net/xiazhiy...
  8. 下載安裝nettle http://www.linuxfromscratch.o...
  9. nettle官方文檔:http://www.lysator.liu.se/~ni...
  10. gmp下載地址:https://gmplib.org/
  11. ldconfig命令:http://man.linuxde.net/ldconfig
  12. SecureCRT rz 上傳文件失敗問題:https://blog.csdn.net/heavend...
  13. 初識NSS,一文了解全貌:https://cloud.tencent.com/dev...
  14. MDN文檔 NSS:https://developer.mozilla.org...
  15. curl: (35) SSL connect error:https://stackoverflow.com/que...
  16. 簡述configure、pkg-config、pkg_config_path三者的關係:http://www.mike.org.cn/articl...
  17. How to compile GnuTLS: https://stackoverflow.com/que...
發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章