從瀏覽器輸入URL到頁面渲染的過程

零、背景

---

---

一個web安全工程師在學習web安全和web滲透時候，非常有必要了解整個WEB工作過程。

一、輸入URL

---

---

這裏是最基本的知識：URL是URI的一種實際應用，URI統一資源表示符，URL統一資源定位符。URL舉例： https://www.xxx.com:443/index.php 前面是協議 https 然後是主機www.xxx.com 然後是端口 443，最後是資源在該主機上針對web系統根目錄（通常爲www）的相對路徑。所以可以想象，根據WEB訪問就是獲取資源，已經知道端口、路徑、協議，唯獨缺乏對方的IP了，畢竟HTTP下層是TCP/IP嘛，所以這裏就用到從域名（或者是主機名）獲取其對應的IP地址。

二、解析IP地址

---

---

這裏爲啥不用DNS呢，是因爲在windows裏面不一定是用DNS解析的IP地址哦，還可能是netbios哈

DNS 解析順序

---

在linux中一般先解析/etc/hosts中的靜態配置的解析，如果不行再利用dns遞歸請求解析域名。在windows中一般是先dns cache（dns緩存），然後再hosts文件，最後是dns遞歸解析。

Netbios 解析順序

---

首先這個只有windows纔會有，一般只在局域網或域內生效，其解析順序域操作系統也就是windows的版本有關係。另外和配置的模式有關係。一般的NetBIOS cache （NetBIOS緩存），然後是WINS Server和廣播，最後是LMHOSTS。如果模式不一樣，WINS服務器解析和廣播的順序有調整。

三、發送HTTP請求

---

---

其實這裏未必是HTTP協議或者HTTPS協議，有可能是其他協議，但是HTTP（HTTPS）協議非常典型，這裏就是以此爲例進行講解。這裏發送HTTP請求，表明請求的資源或者提交的資源，在頭部字段裏會帶有相應的身份驗證信息、回話信息、登錄狀態信息和控制信息等。

四、服務器處理並返回響應報文

---

---

HTTP請求到大服務器後，服務器會進行處理，然後根據處理結果返回相應的返回包，包括返回狀態、返回內容，頭部字段中的對應的控制信息等。

五、瀏覽器渲染頁面

---

---

現代瀏覽器的渲染過程如下：

• 1.解析HTML構建DOM樹
• 2.構建渲染樹
• 3.佈局渲染樹
• 4.繪製渲染樹

最近在工作中還是遇到了一些攻擊者追查等等事情，結合自己過去做過一年的安全分析師的經驗，對幾款比較好的追蹤溯源網站的使用做一個簡介。

一、關聯神器passivetotal

1、網址：https://community.riskiq.com/

2、簡介：常用來搜索IP、Domain、URL、whois信息、ipwhois信息等，是一個存儲數量比較全面，信息量較大，而且可以和時間關聯分析。實現這些維度的互查反查是很重要的一個功能。需要註冊哦。

3、截圖：

二、樣本神器virustotal

1、網址：https://www.virustotal.com/#/home/upload

2、簡介：可以查詢樣本在多家殺毒引擎的掃描檢測結果，另外可以查詢IP、url、hash，這樣可以關聯下載downloadurl 、惡意域名、IP，以及惡意的樣本hash。（註冊之後可付費，在線樣本可下載）

3、截圖：

4、自己查看vt文檔寫的一個apilib庫

  1 #!/usr/bin/env python
  2 #-*-coding:UTF-8-*-
  3 #引入頭文件
  4 import os
  5 import sys
  6 import json
  7 import urllib
  8 import urllib2
  9 import hashlib
 10 import postfile
 11 import simplejson
 12 
 13 #全局變量
 14 #APIKEY = None
 15 APIKEY = '033e1a2ae071f8800d558a968717c2674f8eb728cfb6b470d4713dd744600fb7'
 16 MYVTOBJ = None
 17 CRONTOL_BIT = 0
 18 #創建VirusTotal類
 19 class VirusTotal:
 20     def __init__(self,apikey):
 21         self.virus_dict = {}
 22         self.apikey = apikey
 23         
 24     def get_virus_dict(self):
 25         return self.virus_dict
 26         
 27     def upload_one_check_file(self,check_file):
 28         check_host = "www.virustotal.com"
 29         selector = "https://www.virustotal.com/vtapi/v2/file/scan"
 30         fields = [("apikey",self.apikey)]
 31         file_name = os.path.basename(check_file)
 32         try:
 33             file_to_check = open(check_file,'rb').read()
 34         except Exception,ex:
 35             #log.log(ex,3,1)
 36             return (-1,ex)
 37         file_struct = [("file",file_name,file_to_check)]
 38         try:
 39             result_json = postfile.post_multipart(check_host,selector,fields,file_struct)
 40         except Exception,ex:
 41             #log.log(ex,3,2)
 42             return (-1,ex)
 43         return result_json
 44         
 45     def upload_check_files(self,files_list):
 46         result_json_list = []
 47         if len(files_list) <= 0:
 48             return (-1,'files_list error')
 49         for item in files_list:
 50             ret = self.quick_check(item)
 51             result_json_list.append(ret)
 52         return result_json_list
 53         
 54     def check_file_rescan(self,scanid):
 55         url = "https://www.virustotal.com/vtapi/v2/file/rescan"
 56         scanid_string = None
 57         if type(scanid) is list:
 58             for item in scanid:
 59                 scanid_string += str(item)+','
 60             scanid_string = scanid_string[0:-1]
 61             parameters = {"resource":scanid_string,"apikey":self.apikey}
 62             try:
 63                 data = urllib.urlencode(parameters)
 64                 check_request = urllib2.Request(url,data)  
 65                 response = urllib2.urlopen(check_request)  
 66                 result_json = response.read()
 67             except Exception,ex:
 68                 ##log.log(ex,3,2)
 69                 return (-1,ex)
 70             return result_json
 71         parameters = {"resource":scanid,"apikey":self.apikey}
 72         try:
 73             data = urllib.urlencode(parameters)  
 74             check_request = urllib2.Request(url,data)  
 75             response = urllib2.urlopen(check_request)  
 76             result_json = response.read()
 77         except Exception,ex:
 78             #log.log(ex,3,2)
 79             return (-1,ex)
 80         return result_json
 81     
 82     def quick_check(self,check_file):
 83         file_md5 = None
 84         print str(check_file)
 85         with open(str(check_file),'rb') as f:
 86             file_md5 = hashlib.md5()
 87             file_md5.update(f.read())
 88             file_md5 = file_md5.hexdigest()
 89         ret = self.upload_one_check_file(check_file)
 90         if ret.find('Error 400') >= 0:
 91             ret = self.check_file_rescan(file_md5)
 92         ret = simplejson.loads(ret)
 93         return ret["sha256"]
 94 
 95     def get_report(self,scanid):
 96         url = "https://www.virustotal.com/vtapi/v2/file/report"  
 97         parameters = {"resource":scanid,"apikey":self.apikey}
 98         try:
 99             data = urllib.urlencode(parameters)  
100             report_requset = urllib2.Request(url,data)  
101             response = urllib2.urlopen(report_requset)  
102             result_json = response.read()
103         except Exception,ex:
104             #log.log(ex,3,2)
105             return (-1,ex)
106         return result_json
107 
108     def get_report_from_designated_company(self,company,scanid):
109         url = "https://www.virustotal.com/vtapi/v2/file/report"  
110         parameters = {"resource":scanid,"apikey":self.apikey}
111         try:
112             data = urllib.urlencode(parameters)  
113             report_request = urllib2.Request(url,data)  
114             response = urllib2.urlopen(report_request)  
115             result_json = response.read()    
116             response_dict = simplejson.loads(result_json)
117         except Exception,ex:
118             #log.log(ex,3,2)
119             return (-1,ex)
120         return response_dict.get("scans",{}).get(company,{}).get("result")
121         
122     def url_check(self,check_url):
123         url = "https://www.virustotal.com/vtapi/v2/url/scan"
124         check_urls = None
125         if type(check_url) is list:
126             for item in check_url:
127                 check_urls += str(item)+'\n'
128             check_urls = check_urls[0:-1]
129             parameters = {"url":check_urls,"apikey":self.apikey} 
130             try:
131                 data = urllib.urlencode(parameters)  
132                 url_check_request = urllib2.Request(url,data)  
133                 response = urllib2.urlopen(url_check_request)  
134                 result_json = response.read() 
135             except Exception,ex:
136                 #log.log(ex,3,2)
137                 return (-1,ex)
138             return result_json
139         check_urls = str(check_url)
140         parameters = {"url":check_urls,"apikey":self.apikey}  
141         try:
142             data = urllib.urlencode(parameters)  
143             url_check_request = urllib2.Request(url,data)  
144             response = urllib2.urlopen(url_check_request)  
145             result_json = response.read()  
146         except Exception,ex:
147             #log.log(ex,3,2)
148             return (-1,ex)
149         return result_json  
150         
151     def get_url_report(self,check_url):
152         url = "https://www.virustotal.com/vtapi/v2/url/report"
153         check_urls = None
154         if type(check_url) is list:
155             for item in check_url:
156                 check_urls += str(item)+','
157             check_urls = check_urls[0:-1]
158             parameters = {"resource": "http://www.sohu.com","apikey":self.apikey,"scan":"1"}  
159             try:
160                 data = urllib.urlencode(parameters)  
161                 url_check_request = urllib2.Request(url,data)  
162                 response = urllib2.urlopen(url_check_request)  
163                 result_json = response.read()
164             except Exception,ex:
165                 #log.log(ex,3,2)
166                 return (-1,ex)
167             return result_json
168         check_urls = str(check_url)
169         parameters = {"resource":check_urls,"apikey":self.apikey}  
170         try:
171             data = urllib.urlencode(parameters)  
172             url_check_request = urllib2.Request(url,data)  
173             response = urllib2.urlopen(url_check_request)  
174             result_json = response.read()
175         except Exception,ex:
176             #log.log(ex,3,2)
177             return (-1,ex)
178         return result_json
179 
180     def get_url_report_from_designated_company(self,check_url,company):
181         url = "https://www.virustotal.com/vtapi/v2/url/report"
182         check_urls = str(check_url)
183         parameters = {"resource":check_urls,"apikey":self.apikey}  
184         try:
185             data = urllib.urlencode(parameters)  
186             url_check_request = urllib2.Request(url,data)  
187             response = urllib2.urlopen(url_check_request)  
188             result_json = response.read()
189         except Exception,ex:
190             #log.log(ex,3,2)
191             return (-1,ex)
192         response_dict = simplejson.loads(result_json)  
193         return response_dict.get('scans',{}).get(company).get('result')
194     
195     def ip_check(self,ip):
196         url = 'https://www.virustotal.com/vtapi/v2/ip-address/report'  
197         parameters = {'ip':ip,'apikey':self.apikey}  
198         try:
199             response = urllib.urlopen('%s?%s'%(url,urllib.urlencode(parameters))).read()  
200             response_dict = json.loads(response)
201         except Exception,ex:
202             #log.log(ex,3,2)
203             return (-1,ex)
204         return response_dict 
205         
206     def domain_check(self,domain):
207         url = 'https://www.virustotal.com/vtapi/v2/domain/report'  
208         parameters = {'domain':domain,'apikey':self.apikey}
209         try:
210             response = urllib.urlopen('%s?%s'%(url,urllib.urlencode(parameters))).read()  
211             response_dict = json.loads(response)  
212         except Exception,ex:
213             #log.log(ex,3,2)
214             return (-1,ex)
215         return  response_dict 
216 
217 
218 #快速使用函數
219 def vtocq():
220     global APIKEY
221     global MYVTOBJ
222     global CRONTOL_BIT
223     if CRONTOL_BIT != 0:
224         return ('-2','ERROR')
225     if APIKEY != None:
226         MYVTOBJ = VirusTotal(APIKEY)
227     else:
228         return ('-1','APIKEY information set error!')
229     CRONTOL_BIT = 1
230     return MYVTOBJ
231     
232 def fcq(file_check,hash_sha1):
233     global MYVTOBJ
234     ret = MYVTOBJ.quick_check(file_check)
235     ret = MYVTOBJ.get_report(hash_sha1) 
236     ret = simplejson.loads(ret)
237     vtnumber = ret['positives']
238     av_macfee = ret['scans']['McAfee']['result']
239     av_bkav = ret['scans']['Bkav']['result']
240     av_360 = ret['scans']['Qihoo-360']['result']
241     ret = { 
242         "vtnumber":vtnumber,
243         "av_360":av_360,
244         "av_bkav":av_bkav,
245         "av_macfee":av_macfee
246     }
247     return ret
248     
249 def vt_check(filecheck,hash_sha1):
250     vtocq()
251     ret = fcq(filecheck,hash_sha1)
252     print ret
253     return ret
254     
255 #vt_check('C://PuTTY//plink.exe')
256 MYVTOBJ = VirusTotal(APIKEY)
257 print MYVTOBJ.domain_check("lx.com")

三、其他工具：

1、站長之家--站長工具：做一些域名、IP、註冊郵箱等信息的關聯查詢。

2、www.ipip.net：做一些IP地理信息查詢。