Zeppelin啓用https過程和Hack內核以滿足客戶需求的記錄。
原因是這客戶很有意思,該客戶中國分公司的人爲了驗證內網安全性,從國外找了一個***測試小組對Zeppelin和其他產品進行***測試,結果發現Zeppelin主要倆問題,一個是在內網沒用https,一個是zeppelin裏面可以執行shell命令和python語句。其實這不算大問題,zeppelin本來就是幹這個用的。但是***小組不瞭解zeppelin是做什麼的,認爲即使在內網裏,執行shell命令能查看操作系統的一些文件是大問題,然後發生的事就不說了,不是我們的問題了。
不過既然他們要求整改,我們也只好配合,雖然大家都覺得內網域名加https屬於脫了褲子放屁,然後不讓zeppelin幹他本來應該乾的事就更過分了,但鑑於客戶是甲方,也只好hack源碼了。
於是某個週末用了4個小時完成所有工作。
先記錄下zeppelin加https訪問,我們有自己的域名證書,所以直接用即可。如果沒有域名證書,需要自簽發,那麼可以看第二部分,雙向認證步驟。
https第一部分,已有域名添加jks:
openssl pkcs12 -export -in xxx.com.crt -inkey xxx.com.key -out xxx.com.pkcs12 keytool -importkeystore -srckeystore xxx.com.pkcs12 -destkeystore xxx.com.jks -srcstoretype pkcs12
https第二部分,自簽發證書雙向認證添加jks
# 生成root私鑰和證書文件。 openssl genrsa -out root.key(pem) 2048 # Generate root key file openssl req -x509 -new -key root.key(pem) -out root.crt # Generate root cert file # 創建客戶端私鑰和證書以及證書請求文件csr openssl genrsa -out client.key(pem) 2048 # Generate client key file openssl req -new -key client.key(pem) -out client.csr # Generate client cert request file openssl x509 -req -in client.csr -CA root.crt -CAkey root.key(pem) -CAcreateserial -days 3650 -out client.crt # Use root cert to generate client cert file # 生成服務器端私鑰,證書和證書請求文件csr openssl genrsa -out server.key(pem) 2048 # Generate server key file, use in Zeppelin openssl req -new -key server.key(pem) out server.csr @ Generate server cert request file openssl x509 -req -in server.csr -CA root.crt -CAkey root.key(pem) -CAcreateserial -days 3650 -out server.crt # Use root cert to generate server cert file # 生成客戶端端jks文件 openssl pkcs12 -export -in client.crt -inkey client.key(pem) -out client.pkcs12 # Package to pkcs12 format, must input a password, you should remember the password keytool -importkeystore -srckeystore client.pkcs12 -destkeystore client.jks -srcstoretype pkcs12 # The client password you just input at last step # 生成服務器端jks文件 openssl pkcs12 -export -in server.crt -inkey server.key(pem) -out server.pkcs12 @ Package to pkcs12 format, must input a password, you should remember the password keytool -importkeystore -srckeystore server.pkcs12 -destkeystore server.jks -srcstoretype pkcs12 # The server password you just input at last step
如果是不需要雙向認證,只要單向自簽發,不創建客戶端的各種就可以了。
然後找個地把這些文件放過去,再修改zeppelin配置即可。
mkdir -p /etc/zeppelin/conf/ssl cp server.crt server.jks /etc/zeppelin/conf/ssl
<property> <name>zeppelin.server.ssl.port</name> <value>8443</value> <description>Server ssl port. (used when ssl property is set to true)</description> </property> <property> <name>zeppelin.ssl</name> <value>true</value> <description>Should SSL be used by the servers?</description> </property> <property> <name>zeppelin.ssl.client.auth</name> <value>false</value> <description>Should client authentication be used for SSL connections?</description> </property> <property> <name>zeppelin.ssl.keystore.path</name> <value>/etc/zeppelin/conf/ssl/xxx.com.jks</value> <description>Path to keystore relative to Zeppelin configuration directory</description> </property> <property> <name>zeppelin.ssl.keystore.type</name> <value>JKS</value> <description>The format of the given keystore (e.g. JKS or PKCS12)</description> </property> <property> <name>zeppelin.ssl.keystore.password</name> <value>password which you input on generating server jks step</value> <description>Keystore password. Can be obfuscated by the Jetty Password tool</description> </property>
然後反代那裏也加上443的ssl證書以及443轉8443的upstream即可。
然後是hack zeppelin源碼加入關鍵字限制,這個確實找了一小會zeppelin發送執行源碼給interpreter的地方,zeppelin架構比較清晰,但是代碼挺複雜的,用到了很多小花活兒。比如thrift,interpreter腳本里建立nc監聽。然後各個解釋器插件用socket跟interpreter腳本通信,前端angular,後端jetty,還用shiro做驗證和授權。回頭可以單開好幾篇說說zeppelin安裝,使用和詳細配置,做這項目基本把zeppelin摸透了。
找到發送前端編寫內容給interpreter的java代碼,然後用很生硬的辦法限制執行命令。具體那個.java文件的名字我就不說了,有懸念有驚喜。我不寫java,只負責讀源碼找到代碼位置,hack的java是同事寫的。然後編譯,替換jar包,完成。後面改了改配置,後續的***測試順利通過。
static HashSet<String[]> blockedCodeString = new HashSet<>(); static { blockedCodeString.add(new String[]{"import", "os"}); blockedCodeString.add(new String[]{"import", "sys"}); blockedCodeString.add(new String[]{"import", "subprocess"}); blockedCodeString.add(new String[]{"import", "pty"}); blockedCodeString.add(new String[]{"import", "socket"}); blockedCodeString.add(new String[]{"import", "commands"}); blockedCodeString.add(new String[]{"import", "paramiko"}); blockedCodeString.add(new String[]{"import", "pexpect"}); blockedCodeString.add(new String[]{"import", "BaseHTTPServer"}); blockedCodeString.add(new String[]{"import", "ConfigParser"}); blockedCodeString.add(new String[]{"import", "platform"}); blockedCodeString.add(new String[]{"import", "popen2"}); blockedCodeString.add(new String[]{"import", "copy"}); blockedCodeString.add(new String[]{"import", "SocketServer"}); blockedCodeString.add(new String[]{"import", "sysconfig"}); blockedCodeString.add(new String[]{"import", "tty"}); blockedCodeString.add(new String[]{"import", "xmlrpmlib"}); blockedCodeString.add(new String[]{"etc"}); blockedCodeString.add(new String[]{"boot"}); blockedCodeString.add(new String[]{"dev"}); blockedCodeString.add(new String[]{"lib"}); blockedCodeString.add(new String[]{"lib64"}); blockedCodeString.add(new String[]{"lost+found"}); blockedCodeString.add(new String[]{"mnt"}); blockedCodeString.add(new String[]{"proc"}); blockedCodeString.add(new String[]{"root"}); blockedCodeString.add(new String[]{"sbin"}); blockedCodeString.add(new String[]{"selinux"}); blockedCodeString.add(new String[]{"usr"}); blockedCodeString.add(new String[]{"passwd"}); blockedCodeString.add(new String[]{"useradd"}); blockedCodeString.add(new String[]{"userdel"}); blockedCodeString.add(new String[]{"rm"}); blockedCodeString.add(new String[]{"akka "}); blockedCodeString.add(new String[]{"groupadd"}); blockedCodeString.add(new String[]{"groupdel"}); blockedCodeString.add(new String[]{"mkdir"}); blockedCodeString.add(new String[]{"rmdir"}); blockedCodeString.add(new String[]{"ping"}); blockedCodeString.add(new String[]{"nc"}); blockedCodeString.add(new String[]{"telnet"}); blockedCodeString.add(new String[]{"ftp"}); blockedCodeString.add(new String[]{"scp"}); blockedCodeString.add(new String[]{"ssh"}); blockedCodeString.add(new String[]{"ps"}); blockedCodeString.add(new String[]{"hostname"}); blockedCodeString.add(new String[]{"uname"}); blockedCodeString.add(new String[]{"vim"}); blockedCodeString.add(new String[]{"nano"}); blockedCodeString.add(new String[]{"top"}); blockedCodeString.add(new String[]{"cat"}); blockedCodeString.add(new String[]{"more"}); blockedCodeString.add(new String[]{"less"}); blockedCodeString.add(new String[]{"chkconfig"}); blockedCodeString.add(new String[]{"service"}); blockedCodeString.add(new String[]{"netstat"}); blockedCodeString.add(new String[]{"iptables"}); blockedCodeString.add(new String[]{"ip"}); blockedCodeString.add(new String[]{"route "}); blockedCodeString.add(new String[]{"curl"}); blockedCodeString.add(new String[]{"wget"}); blockedCodeString.add(new String[]{"sysctl"}); blockedCodeString.add(new String[]{"touch"}); blockedCodeString.add(new String[]{"scala.sys.process"}); blockedCodeString.add(new String[]{"0.0.0.0"}); blockedCodeString.add(new String[]{"git"}); blockedCodeString.add(new String[]{"svn"}); blockedCodeString.add(new String[]{"hg"}); blockedCodeString.add(new String[]{"cvs"}); blockedCodeString.add(new String[]{"exec"}); blockedCodeString.add(new String[]{"ln"}); blockedCodeString.add(new String[]{"kill"}); blockedCodeString.add(new String[]{"rsync"}); blockedCodeString.add(new String[]{"lsof"}); blockedCodeString.add(new String[]{"crontab"}); blockedCodeString.add(new String[]{"libtool"}); blockedCodeString.add(new String[]{"automake"}); blockedCodeString.add(new String[]{"autoconf"}); blockedCodeString.add(new String[]{"make"}); blockedCodeString.add(new String[]{"gcc"}); blockedCodeString.add(new String[]{"cc"}); } static boolean allMatch(String aim, String[] checker){ if(checker == null || checker.length < 1){ return false; }else { // by default, treat as match, every not match change it for (String i : checker) { if (!aim.matches(".*\\b" + i + "\\b.*")){ return false; } } return true; } } static String anyMatch(String aim, HashSet<String[]> all) throws Exception{ if(aim.contains("FUCK P&G")){ throw new Exception("How do you know this ????"); } else { for (String[] one : all) { if (allMatch(aim, one)) { StringBuilder sb = new StringBuilder(); for (String s : one) { sb.append(s + " "); } return sb.toString(); } } throw new Exception("No one match"); } } //......此處是個public類 try{ String matchesStrings = anyMatch(st, blockedCodeString); result = new InterpreterResult(Code.ERROR, "Contains dangerous code : " + matchesStrings); }catch (Exception me){ // no match any scheduler.submit(job); while (!job.isTerminated()) { synchronized (jobListener) { try { jobListener.wait(1000); } catch (InterruptedException e) { logger.info("Exception in RemoteInterpreterServer while interpret, jobListener.wait", e); } } } if (job.getStatus() == Status.ERROR) { result = new InterpreterResult(Code.ERROR, Job.getStack(job.getException())); } else { result = (InterpreterResult) job.getReturn(); // in case of job abort in PENDING status, result can be null if (result == null) { result = new InterpreterResult(Code.KEEP_PREVIOUS_RESULT); } } } //......直到該public類結束
因爲客戶有deadline限制,所以快速定位源碼位置的過程還是挺有意思的,比較緊張刺激,在這個以小時計算deadline壓力下,什麼intelliJ, Eclipse都不好使啊,就grep和vi最好用,從找到到改完,比客戶定的deadline提前了好幾個小時。