在傳輸層上壓縮WebService的請求和響應

[size=x-large]在傳輸層上壓縮WebService的請求和響應[/size]

[size=medium]場景[/size]

場景是這樣的：客戶端.NET 3.5應用程序，WCF實現WebService調用，服務端Java，通過CXF提供WebService。有一個方法提供了有一個字符串類型的參數，實際生產環境裏會傳100k以上的字符串。在併發量比較大的情況下，帶寬佔用很嚴重。所以尋找一種可以把傳輸的SOAP消息在客戶端壓縮，服務端解壓縮的方法。

這裏提供的方式在是客戶端通過WCF的MessageEncoder機制對所有的SOAP請求消息壓縮，SOAP響應消息解壓縮，反過來在服務端通過一個Filter對所有的SOAP請求消息，對SOAP響應消息壓縮。

[b]請求的流程如下：[/b]
Client -> SOAP Request -> GzipMessageEncoder -> gzip binary -> GzipWebSericeFilter -> SOAP Request -> CXF

[b]響應的流程如下：[/b]
CXF -> SOAP Response -> GzipWebServiceFilter -> gzip binary -> GzipMessageEncoder -> SOAP Response -> Client

其中.NET的WCF的GzipMessageEncoder是參照[url=http://msdn.microsoft.com/en-us/library/ms751458.aspx]WCF的Samples[/url]，下載解壓後路徑WF_WCF_Samples\WCF\Extensibility\MessageEncoder\Compression

[size=medium]客戶端[/size]

下面先來看一下客戶端部分的代碼：

GZipMessageEncoderFactory.cs 這文件主要是提供GZipMessageEncoder，在裏面通過重寫ReadMessage和WriteMessage方法來實現壓縮和解壓縮。實際壓縮和解壓處理是使用GZipStream實現的。


namespace ConsoleApplication2
{
    //This class is used to create the custom encoder (GZipMessageEncoder)
    internal class GZipMessageEncoderFactory : MessageEncoderFactory
    {
        readonly MessageEncoder _encoder;

        //The GZip encoder wraps an inner encoder
        //We require a factory to be passed in that will create this inner encoder
        public GZipMessageEncoderFactory(MessageEncoderFactory messageEncoderFactory)
        {
            if (messageEncoderFactory == null)
                throw new ArgumentNullException("messageEncoderFactory", "A valid message encoder factory must be passed to the GZipEncoder");
            _encoder = new GZipMessageEncoder(messageEncoderFactory.Encoder);

        }

        //The service framework uses this property to obtain an encoder from this encoder factory
        public override MessageEncoder Encoder
        {
            get { return _encoder; }
        }

        public override MessageVersion MessageVersion
        {
            get { return _encoder.MessageVersion; }
        }

        //This is the actual GZip encoder
        class GZipMessageEncoder : MessageEncoder
        {
            private const string GZipMediaType = "application/x-gzip";
            private const string GZipContentType = GZipMediaType + "; charset=utf-8";

            //This implementation wraps an inner encoder that actually converts a WCF Message
            //into textual XML, binary XML or some other format. This implementation then compresses the results.
            //The opposite happens when reading messages.
            //This member stores this inner encoder.
            readonly MessageEncoder _innerEncoder;

            //We require an inner encoder to be supplied (see comment above)
            internal GZipMessageEncoder(MessageEncoder messageEncoder)
            {
                if (messageEncoder == null)
                    throw new ArgumentNullException("messageEncoder", "A valid message encoder must be passed to the GZipEncoder");
                _innerEncoder = messageEncoder;
            }

            public override string ContentType
            {
                get { return GZipContentType; }
            }

            public override string MediaType
            {
                get { return GZipMediaType; }
            }

            //SOAP version to use - we delegate to the inner encoder for this
            public override MessageVersion MessageVersion
            {
                get { return _innerEncoder.MessageVersion; }
            }

            public override bool IsContentTypeSupported(string contentType)
            {
                return contentType.StartsWith(GZipMediaType, StringComparison.OrdinalIgnoreCase) || contentType.StartsWith("text/xml", StringComparison.OrdinalIgnoreCase);
            }

            //Helper method to compress an array of bytes
            static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager, int messageOffset)
            {
                var memoryStream = new MemoryStream();
                memoryStream.Write(buffer.Array, 0, messageOffset);

                using (var gzStream = new GZipStream(memoryStream, CompressionMode.Compress, true))
                {
                    gzStream.Write(buffer.Array, messageOffset, buffer.Count);
                }


                var compressedBytes = memoryStream.ToArray();
                var bufferedBytes = bufferManager.TakeBuffer(compressedBytes.Length);

                Array.Copy(compressedBytes, 0, bufferedBytes, 0, compressedBytes.Length);

                bufferManager.ReturnBuffer(buffer.Array);
                var byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset, bufferedBytes.Length - messageOffset);

                return byteArray;
            }

            //Helper method to decompress an array of bytes
            static ArraySegment<byte> DecompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager)
            {

                var memoryStream = new MemoryStream(buffer.Array, buffer.Offset, buffer.Count - buffer.Offset);
                var decompressedStream = new MemoryStream();
                const int blockSize = 1024;
                byte[] tempBuffer = bufferManager.TakeBuffer(blockSize);
                using (var gzStream = new GZipStream(memoryStream, CompressionMode.Decompress))
                {
                    while (true)
                    {
                        var bytesRead = gzStream.Read(tempBuffer, 0, blockSize);
                        if (bytesRead == 0)
                            break;
                        decompressedStream.Write(tempBuffer, 0, bytesRead);
                    }
                }
                bufferManager.ReturnBuffer(tempBuffer);

                var decompressedBytes = decompressedStream.ToArray();
                var bufferManagerBuffer = bufferManager.TakeBuffer(decompressedBytes.Length + buffer.Offset);
                Array.Copy(buffer.Array, 0, bufferManagerBuffer, 0, buffer.Offset);
                Array.Copy(decompressedBytes, 0, bufferManagerBuffer, buffer.Offset, decompressedBytes.Length);

                var byteArray = new ArraySegment<byte>(bufferManagerBuffer, buffer.Offset, decompressedBytes.Length);
                bufferManager.ReturnBuffer(buffer.Array);

                return byteArray;
            }

            //One of the two main entry points into the encoder. Called by WCF to encode a Message into a buffered byte array.
            public override ArraySegment<byte> WriteMessage(Message message, int maxMessageSize, BufferManager bufferManager, int messageOffset)
            {
                //Use the inner encoder to encode a Message into a buffered byte array
                ArraySegment<byte> buffer = _innerEncoder.WriteMessage(message, maxMessageSize, bufferManager, messageOffset);
                //Compress the resulting byte array
                return CompressBuffer(buffer, bufferManager, messageOffset);
            }

            public override Message ReadMessage(Stream stream, int maxSizeOfHeaders, string contentType)
            {
                var gzStream = new GZipStream(stream, CompressionMode.Decompress, true);
                return _innerEncoder.ReadMessage(gzStream, maxSizeOfHeaders);
            }

            public override Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager, string contentType)
            {
                //Decompress the buffer
                ArraySegment<byte> decompressedBuffer = DecompressBuffer(buffer, bufferManager);
                //Use the inner encoder to decode the decompressed buffer
                Message returnMessage = _innerEncoder.ReadMessage(decompressedBuffer, bufferManager);
                returnMessage.Properties.Encoder = this;
                return returnMessage;
            }

            public override void WriteMessage(Message message, Stream stream)
            {
                using (var gzStream = new GZipStream(stream, CompressionMode.Compress, true))
                {
                    _innerEncoder.WriteMessage(message, gzStream);
                }

                // innerEncoder.WriteMessage(message, gzStream) depends on that it can flush data by flushing 
                // the stream passed in, but the implementation of GZipStream.Flush will not flush underlying
                // stream, so we need to flush here.
                stream.Flush();
            }
        }
    }
}

下面是GZipMessageEncodingBindingElement.cs 這裏的GZipMessageEncodingBindingElement類是爲了在app.config裏添加配置項。


namespace ConsoleApplication2
{
    //This is the binding element that, when plugged into a custom binding, will enable the GZip encoder
    public sealed class GZipMessageEncodingBindingElement 
                        : MessageEncodingBindingElement //BindingElement
    {

        //We will use an inner binding element to store information required for the inner encoder
        MessageEncodingBindingElement _innerBindingElement;

        //By default, use the default text encoder as the inner encoder
        public GZipMessageEncodingBindingElement()
            : this(new TextMessageEncodingBindingElement()) { }

        public GZipMessageEncodingBindingElement(MessageEncodingBindingElement messageEncoderBindingElement)
        {
            _innerBindingElement = messageEncoderBindingElement;
        }

        public MessageEncodingBindingElement InnerMessageEncodingBindingElement
        {
            get { return _innerBindingElement; }
            set { _innerBindingElement = value; }
        }

        //Main entry point into the encoder binding element. Called by WCF to get the factory that will create the
        //message encoder
        public override MessageEncoderFactory CreateMessageEncoderFactory()
        {
            return new GZipMessageEncoderFactory(_innerBindingElement.CreateMessageEncoderFactory());
        }

        public override MessageVersion MessageVersion
        {
            get { return _innerBindingElement.MessageVersion; }
            set { _innerBindingElement.MessageVersion = value; }
        }

        public override BindingElement Clone()
        {
            return new GZipMessageEncodingBindingElement(_innerBindingElement);
        }

        public override T GetProperty<T>(BindingContext context)
        {
            if (typeof(T) == typeof(XmlDictionaryReaderQuotas))
            {
                return _innerBindingElement.GetProperty<T>(context);
            }
            return base.GetProperty<T>(context);
        }

        public override IChannelFactory<TChannel> BuildChannelFactory<TChannel>(BindingContext context)
        {
            if (context == null)
                throw new ArgumentNullException("context");

            context.BindingParameters.Add(this);
            return context.BuildInnerChannelFactory<TChannel>();
        }

        public override IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
        {
            if (context == null)
                throw new ArgumentNullException("context");

            context.BindingParameters.Add(this);
            return context.BuildInnerChannelListener<TChannel>();
        }

        public override bool CanBuildChannelListener<TChannel>(BindingContext context)
        {
            if (context == null)
                throw new ArgumentNullException("context");

            context.BindingParameters.Add(this);
            return context.CanBuildInnerChannelListener<TChannel>();
        }
    }

    //This class is necessary to be able to plug in the GZip encoder binding element through
    //a configuration file
    public class GZipMessageEncodingElement : BindingElementExtensionElement
    {
        //Called by the WCF to discover the type of binding element this config section enables
        public override Type BindingElementType
        {
            get { return typeof(GZipMessageEncodingBindingElement); }
        }

        //The only property we need to configure for our binding element is the type of
        //inner encoder to use. Here, we support text and binary.
        [ConfigurationProperty("innerMessageEncoding", DefaultValue = "textMessageEncoding")]
        public string InnerMessageEncoding
        {
            get { return (string)base["innerMessageEncoding"]; }
            set { base["innerMessageEncoding"] = value; }
        }

        //The only property we need to configure for our binding element is the type of
        //inner encoder to use. Here, we support text and binary.
        [ConfigurationProperty("messageVersion", DefaultValue = "Soap12")]
        public string MessageVersion
        {
            get { return (string)base["messageVersion"]; }
            set { base["messageVersion"] = value; }
        }

        //Called by the WCF to apply the configuration settings (the property above) to the binding element
        public override void ApplyConfiguration(BindingElement bindingElement)
        {
            var binding = (GZipMessageEncodingBindingElement)bindingElement;
            PropertyInformationCollection propertyInfo = ElementInformation.Properties;
            var propertyInformation = propertyInfo["innerMessageEncoding"];
            if (propertyInformation == null || propertyInformation.ValueOrigin == PropertyValueOrigin.Default) return;

            var version = System.ServiceModel.Channels.MessageVersion.Soap12;
            if ("Soap11" == MessageVersion)
            {
                version = System.ServiceModel.Channels.MessageVersion.Soap11;
            }

            switch (InnerMessageEncoding)
            {
                case "textMessageEncoding":
                    binding.InnerMessageEncodingBindingElement = new TextMessageEncodingBindingElement() { MessageVersion = version };
                    break;
                case "binaryMessageEncoding":
                    binding.InnerMessageEncodingBindingElement = new BinaryMessageEncodingBindingElement();
                    break;
            }
        }

        //Called by the WCF to create the binding element
        protected override BindingElement CreateBindingElement()
        {
            var bindingElement = new GZipMessageEncodingBindingElement();
            ApplyConfiguration(bindingElement);
            return bindingElement;
        }
    }
}

然後我們就可以把這個GZipMessageEncodingElement配置到app.config裏了


<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <system.serviceModel>
    <extensions>
      <bindingElementExtensions>
        <add name="gzipMessageEncoding" type="ConsoleApplication2.GZipMessageEncodingElement,ConsoleApplication2" />
      </bindingElementExtensions>
    </extensions>
    <bindings>
      <customBinding>
        <binding name="countServiceSoapBinding">
          <gzipMessageEncoding innerMessageEncoding="textMessageEncoding" messageVersion="Soap11"/>
          <httpTransport manualAddressing="false"
                         authenticationScheme="Anonymous"
                         bypassProxyOnLocal="false"
                         hostNameComparisonMode="StrongWildcard"
                         proxyAuthenticationScheme="Anonymous"
                         realm=""
                         useDefaultWebProxy="true" />
        </binding>
      </customBinding>
    </bindings>
    <client>
      <endpoint address="http://192.168.2.3:8080/binder/services/countService"
          binding="customBinding" bindingConfiguration="countServiceSoapBinding"
          contract="ServiceReference1.HolidayService" name="HolidayServiceImplPort" />
    </client>
  </system.serviceModel>
</configuration>

客戶端最後的部分就是調用webservice，這裏的壓縮和解壓對於調用者和陪調用者是透明的。也就是同沒有壓縮和解壓之前的使用方法一樣。


namespace ConsoleApplication2
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                var service = new ServiceReference1.HolidayServiceClient();
                var text =File.ReadAllText("c:\\words");

                var len = service.countText(text);
                Console.WriteLine("lenght = {0}", len);

            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message);
                Console.WriteLine(e.StackTrace);
            }
            Console.Read();
        }
    }
}

[size=medium]服務端[/size]

服務端是一個Filter，和HttpServletRequest和HttpServletResponse的包裝類。

入口：GzipWebServiceFilter.java


/**
 * 把使用Gzip壓縮的SOAP消息解壓縮。
 * @author matianyi
 *
 */
public class GzipWebServiceFilter implements Filter {

	public static final String CONTENT_TYPE = "application/x-gzip";
	public static final String CONTENT_ENCODING = "utf-8";

	@Override
	public void init(FilterConfig filterConfig) throws ServletException {
		// TODO Auto-generated method stub

	}

	@SuppressWarnings("unchecked")
	@Override
	public void doFilter(ServletRequest request, ServletResponse response,
			FilterChain chain) throws IOException, ServletException {

		HttpServletRequest req = (HttpServletRequest) request;
		HttpServletResponse resp = (HttpServletResponse) response;

		if(req.getContentType() == null || !req.getContentType().startsWith(CONTENT_TYPE)){
			chain.doFilter(request, response);
		} else {
			chain.doFilter(new GzipHttpServletRequestWrapper(req), new GzipHttpServletResponseWrapper(resp));
		}
	}

	@Override
	public void destroy() {
		// TODO Auto-generated method stub

	}

}

這裏就是判斷contentType，如果是gzip的就用GzipHttpServletRequestWrapper和GzipHttpServletResponseWrapper包裝原始的Request和Response以實現壓縮和解壓縮。

GzipHttpServletRequestWrapper

public class GzipHttpServletRequestWrapper extends HttpServletRequestWrapper {

	public static final String CONTNET_TYPE_SOAP_1_2 = "application/soap+xml";
	public static final String CONTNET_TYPE_SOAP_1_1 = "text/xml";

	public GzipHttpServletRequestWrapper(HttpServletRequest request) {
		super(request);
	}

	@Override
	public ServletInputStream getInputStream() throws IOException {
		return new GzipServletInputStream(super.getInputStream());
	}

	@Override
	public String getContentType() {
		return CONTNET_TYPE_SOAP_1_2;
	}

	@Override
	public String getHeader(String name) {
		if ("content-type".equalsIgnoreCase(name)) {
			return getContentType();
		} else {
			return super.getHeader(name);
		}
	}

}

class GzipServletInputStream extends ServletInputStream {

	private GZIPInputStream delegate;

	public GzipServletInputStream(ServletInputStream servletInputStream)
			throws IOException {
		super();
		this.delegate = new GZIPInputStream(servletInputStream);
	}

	@Override
	public int read() throws IOException {
		return delegate.read();
	}

}

GzipHttpServletResponseWrapper

public class GzipHttpServletResponseWrapper extends HttpServletResponseWrapper {

	public GzipHttpServletResponseWrapper(HttpServletResponse response) {
		super(response);
	}

	@Override
	public ServletOutputStream getOutputStream() throws IOException {
		return new GzipServletOutputStream(super.getOutputStream());
	}

	@Override
	public void setCharacterEncoding(String charset) {
		super.setCharacterEncoding(GzipWebServiceFilter.CONTENT_ENCODING);
	}

	@Override
	public void setContentType(String type) {
		super.setContentType(GzipWebServiceFilter.CONTENT_TYPE + "; charset=" + GzipWebServiceFilter.CONTENT_ENCODING);
	}

}

class GzipServletOutputStream extends ServletOutputStream{
	private GZIPOutputStream delegate;

	public GzipServletOutputStream(ServletOutputStream servletOutputStream)
			throws IOException {
		super();
		this.delegate = new GZIPOutputStream(servletOutputStream);
	}


	@Override
	public void write(int b) throws IOException {
		System.out.print((char)b);
		delegate.write(b);
	}


	public void close() throws IOException {
		delegate.close();
	}


	public void flush() throws IOException {
		delegate.flush();
	}


	public void write(byte[] buf, int off, int len) throws IOException {
		delegate.write(buf, off, len);
	}


	public void write(byte[] b) throws IOException {
		delegate.write(b);
	}


}

這裏做的主要事情就是在Resquest的getInputStream和Response的getOutputStream是返回一個擁有GZip功能的Stream，來代替原始的Stream。通過原始的Stream仍然是最終的輸入和輸出源。

然後在web.xml中把這個Filter作用於原來的WebService的Servlet

web.xml


<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">

	<!-- The definition of the Root Spring Container shared by all Servlets and Filters -->
	<context-param>
		<param-name>contextConfigLocation</param-name>
		<param-value>/WEB-INF/spring/root-context.xml</param-value>
	</context-param>

	<!-- Creates the Spring Container shared by all Servlets and Filters -->
	<listener>
		<listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
	</listener>

    <filter>
        <filter-name>GzipWebServiceFilter</filter-name>
        <filter-class>com.cccis.ws.GzipWebServiceFilter</filter-class>
    </filter>

    <filter-mapping>
        <filter-name>GzipWebServiceFilter</filter-name>
        <url-pattern>/services/*</url-pattern>
    </filter-mapping>

    <servlet>
        <description>Apache CXF Endpoint</description>
        <servlet-name>cxf</servlet-name>
        <servlet-class>org.apache.cxf.transport.servlet.CXFServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>cxf</servlet-name>
        <url-pattern>/services/*</url-pattern>
    </servlet-mapping>

</web-app>

webservice的配置和cxf原來的一樣

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xmlns:context="http://www.springframework.org/schema/context"
    xmlns:jaxws="http://cxf.apache.org/jaxws"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd
       http://cxf.apache.org/jaxws http://cxf.apache.org/schemas/jaxws.xsd">

    <import resource="classpath:META-INF/cxf/cxf.xml" />
    <import resource="classpath:META-INF/cxf/cxf-servlet.xml" />

    <bean id="countServiceImpl" class="com.cccis.ws.HolidayServiceImpl" />

    <jaxws:endpoint 
      id="countService" 
      implementor="#countServiceImpl" 
      serviceName="countService"
      address="/countService" />

</beans>

如果你想看一下實際的HTTP請求和響應是什麼樣子的可以用Fiddler Web Debugger來查看

[img]http://dl2.iteye.com/upload/attachment/0085/9897/42e0a5ba-6f96-3ad4-82a7-d80ba7cfe719.png[/img]

本文的源代碼在附件中。

本文的方案沒有在最終的被用於生產環境，一個原因是比較複雜，另外一個是服務器在對大XML進行unmarshal的效率並不高。單本文的方案的好處就是不用對原有的webservice接口和實現進行修改。最後在實際場景用我們使用[url=http://en.wikipedia.org/wiki/Message_Transmission_Optimization_Mechanism]MTOM[/url]來解決問題的，後面我還會寫一篇文章來介紹這個方法。

iteye_21091

發佈了15 篇原創文章 · 獲贊 2 · 訪問量 1萬+

私信關注

在傳輸層上壓縮WebService的請求和響應

詐騙（殺豬盤）網站進行滲透測試

Python 潮流週刊#50：我最喜歡的 Python 3.13 新特性！

【Python】保存gym截圖

【譯】使用 GitHub Copilot 作爲你的編碼 GPS

Linux 服務器配置-安裝portainer-ce社區版

外行也能讀懂的網絡硬件設備功能原理速成

Groovy: 把Map作爲接口的實現來使用

Apache Mina 文檔翻譯 - 第十六章 - JMX支持

Groovy：運行期把方法mixin到對象實例

Apache Mina 文檔翻譯 - 第十二章 - 日誌Filter

Apache Mina 2 文檔翻譯 - 章節彙總

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結