Netty實現TCP私有協議棧的開發

爲什麼使用TCP通信？
因爲在進行對接機器C語言開發的直播主機)的時候，TCP是一種更安全更高效率的連接，它是一種會確認通信對方，保持通信狀態，並且能檢查報文完整性的連接，而HTTP則是在TCP服務層上面的應用層，當然TCP通信稍微要複雜一點。

爲什麼要使用Netty?
先說一下BIO,NIO,AIO的關係：

BIO：是一種同步阻塞的io，當io建立連接等待應答的時候，當前的線程就被掛起不能做其他的事情。
NIO：是一種同步非阻塞的io，當io建立連接的時候，當前線程不會被掛起，可以去做其他的事情，它會不斷去詢問內核處理完消息沒，當內核返回消息的時候他就可以拿着消息進行讀寫，其間線程並沒有北掛起。
AIO：是一種異步非阻塞的io，當io建立連接的時候，當前線程不會被掛起，可以去做其他的事情，然而它也不會去詢問消息是否出來完成，內核消息處理完成後會自動通知它，這才叫異步。

舉個栗子：小明去食堂排隊打飯的時候，他只能排在那裏不能幹其他的事情這就是BIO，當食堂實行叫號取餐的時候，小明這個時候完全不用一直等在那裏，今天先去外面買瓶可樂打一把王者，他只需要時不時去看看取餐到他沒有，這就是NIO，小明在寢室打喫雞，於是叫了個外賣，餐好了會有外賣小哥送到寢室，他都不用去食堂了也不需要時不時去看是否做好了餐食，這就是AIO。

阻塞與非阻塞是看線程是否被掛起，同步與異步是看是否需要主動去詢問消息已準備好

如果使用BIO這樣的方式，一個線程只能應答一個請求同時還會被阻塞，不過可以採用線程池的方式來使用BIO,但是這樣會造成線程的極大開銷，導致系統資源匱乏，請求量過大會引起系統宕機，這時候基於NIO的Netty出來了，它首先不會北阻塞掛起，其次Netty是使用的是IO多路複用的技術，通過selector這個輪詢器去輪詢當前的請求，只需要一個線程就可以處理多個連接，這樣能極大的提高效率

直接上需求把！！！

1，私有協議棧需求

私有協議爲40字節的消息頭+Json格式的消息體，如下：

2，消息封裝

NettyMessage.class用於消息頭+消息體的封裝

public final class NettyMessage {

    private Header header;

    private Object body;

    /**
     * @return the header
     */
    public final Header getHeader() {
	return header;
    }

    /**
     * @param header
     *            the header to set
     */
    public final void setHeader(Header header) {
	this.header = header;
    }

    /**
     * @return the body
     */
    public final Object getBody() {
	return body;
    }

    /**
     * @param body
     *            the body to set
     */
    public final void setBody(Object body) {
	this.body = body;
    }

    @Override
    public String toString() {
        return "NettyMessage{" +
                "header=" + header +
                ", body=" + body +
                '}';
    }
}

Header.class用於消息頭的封裝，因爲消息頭裏面裏面數據是佔固定字節的，可以使用Java的不同數據類型定義來佔用固定字節

public final class Header {

    /**
     * 版本號佔2個字節
     */
    private short sVersion;

    /**
     * 命令類型佔2個字節
     */
    private short sCommand;

    /**
     * 包體長度佔4個字節
     */
    private int nPacketLen;

    /**
     * 命令標識，建議獲取當前時間的毫秒時間+4位隨機數
     * 佔8個字節
     */
    private long uIdentity;

    /**
     * 具體命令，詳見具體命令章節
     * 佔2個字節
     */
    private short pCmd;

    /**
     * 賬戶名要求64位的數字
     * 佔8個字節
     */
    private long account;

    /**
     * 密碼要求64位的數字
     * 佔8個字節
     */
    private long password;

    /**
     * szSession佔六個字節
     */
    private byte[] szSession;


    /**
     * 本地命令爲0，集控命令爲1
     */
    private int flag;


    public short getsVersion() {
        return sVersion;
    }

    public void setsVersion(short sVersion) {
        this.sVersion = sVersion;
    }

    public short getsCommand() {
        return sCommand;
    }

    public void setsCommand(short sCommand) {
        this.sCommand = sCommand;
    }

    public int getnPacketLen() {
        return nPacketLen;
    }

    public void setnPacketLen(int nPacketLen) {
        this.nPacketLen = nPacketLen;
    }

    public long getuIdentity() {
        return uIdentity;
    }

    public void setuIdentity(long uIdentity) {
        this.uIdentity = uIdentity;
    }

    public short getpCmd() {
        return pCmd;
    }

    public void setpCmd(short pCmd) {
        this.pCmd = pCmd;
    }

    public long getAccount() {
        return account;
    }

    public void setAccount(long account) {
        this.account = account;
    }

    public long getPassword() {
        return password;
    }

    public void setPassword(long password) {
        this.password = password;
    }

    public byte[] getSzSession() {
        return szSession;
    }

    public void setSzSession(byte[] szSession) {
        this.szSession = szSession;
    }

    public int getFlag() {
        return flag;
    }

    public void setFlag(int flag) {
        this.flag = flag;
    }

    @Override
    public String toString() {
        return "Header{" +
                "sVersion=" + sVersion +
                ", sCommand=" + sCommand +
                ", nPacketLen=" + nPacketLen +
                ", uIdentity=" + uIdentity +
                ", pCmd=" + pCmd +
                ", account=" + account +
                ", password=" + password +
                ", szSession=" + Arrays.toString(szSession) +
                '}';
    }
}

3，序列化數據

Java默認提供的序列化機制，需要序列化的Java對象只需要實現 Serializable / Externalizable 接口並生成序列化ID，這個類就能夠通過 ObjectInput 和 ObjectOutput 序列化和反序列化，不過Java的序列化效率低，而且不支持跨語言比如C,C++,目前主流的可跨語言的序列化方式有谷歌的Protobuf和Jboss的Marshalling，這裏我們使用Marshalling

(1)MarshallingCodecFactory.class

public final class MarshallingCodecFactory {

    /**
     * 創建Jboss Marshaller
     * 
     * @return
     * @throws IOException
     */
    protected static Marshaller buildMarshalling() throws IOException {
	final MarshallerFactory marshallerFactory = Marshalling
		.getProvidedMarshallerFactory("serial");
	final MarshallingConfiguration configuration = new MarshallingConfiguration();
	configuration.setVersion(5);
	Marshaller marshaller = marshallerFactory
		.createMarshaller(configuration);
	return marshaller;
    }

    /**
     * 創建Jboss Unmarshaller
     * 
     * @return
     * @throws IOException
     */
    protected static Unmarshaller buildUnMarshalling() throws IOException {
	final MarshallerFactory marshallerFactory = Marshalling
		.getProvidedMarshallerFactory("serial");
	final MarshallingConfiguration configuration = new MarshallingConfiguration();
	configuration.setVersion(5);
	final Unmarshaller unmarshaller = marshallerFactory
		.createUnmarshaller(configuration);
	return unmarshaller;
    }
}

(2)ByteBuf的數據讀寫封裝類

class ChannelBufferByteInput implements ByteInput {

    private final ByteBuf buffer;

    public ChannelBufferByteInput(ByteBuf buffer) {
        this.buffer = buffer;
    }

    @Override
    public void close() throws IOException {
        // nothing to do
    }

    @Override
    public int available() throws IOException {
        return buffer.readableBytes();
    }

    @Override
    public int read() throws IOException {
        if (buffer.isReadable()) {
            return buffer.readByte() & 0xff;
        }
        return -1;
    }

    @Override
    public int read(byte[] array) throws IOException {
        return read(array, 0, array.length);
    }

    @Override
    public int read(byte[] dst, int dstIndex, int length) throws IOException {
        int available = available();
        if (available == 0) {
            return -1;
        }

        length = Math.min(available, length);
        buffer.readBytes(dst, dstIndex, length);
        return length;
    }

    @Override
    public long skip(long bytes) throws IOException {
        int readable = buffer.readableBytes();
        if (readable < bytes) {
            bytes = readable;
        }
        buffer.readerIndex((int) (buffer.readerIndex() + bytes));
        return bytes;
    }

}

class ChannelBufferByteOutput implements ByteOutput {

    private final ByteBuf buffer;

    /**
     * Create a new instance which use the given {@link ByteBuf}
     */
    public ChannelBufferByteOutput(ByteBuf buffer) {
        this.buffer = buffer;
    }

    @Override
    public void close() throws IOException {
        // Nothing to do
    }

    @Override
    public void flush() throws IOException {
        // nothing to do
    }

    @Override
    public void write(int b) throws IOException {
        buffer.writeByte(b);
    }

    @Override
    public void write(byte[] bytes) throws IOException {
        buffer.writeBytes(bytes);
    }

    @Override
    public void write(byte[] bytes, int srcIndex, int length) throws IOException {
        buffer.writeBytes(bytes, srcIndex, length);
    }

    /**
     * Return the {@link ByteBuf} which contains the written content
     *
     */
    ByteBuf getBuffer() {
        return buffer;
    }
}

(3)MarshallingEncoder.class序列化消息體

@Sharable
public class MarshallingEncoder {

    private static final byte[] LENGTH_PLACEHOLDER = new byte[4];
    Marshaller marshaller;

    public MarshallingEncoder() throws IOException {
	marshaller = MarshallingCodecFactory.buildMarshalling();
    }

    protected void encode(Object msg, ByteBuf out) throws Exception {
		System.out.println("-----------------編碼時進行序列化--------------------");
	try {
	    int lengthPos = out.writerIndex();
	    out.writeBytes(LENGTH_PLACEHOLDER);
	    ChannelBufferByteOutput output = new ChannelBufferByteOutput(out);
	    marshaller.start(output);
	    marshaller.writeObject(msg);
	    marshaller.finish();
	    out.setInt(lengthPos, out.writerIndex() - lengthPos - 4);
	} finally {
	    marshaller.close();
	}
    }
}

(4)MarshallingDncoder.class反序列化消息體

public class MarshallingDecoder {

    private final Unmarshaller unmarshaller;

    /**
     * Creates a new decoder whose maximum object size is {@code 1048576} bytes.
     * If the size of the received object is greater than {@code 1048576} bytes,
     * a {@link StreamCorruptedException} will be raised.
     * 
     * @throws IOException
     * 
     */
    public MarshallingDecoder() throws IOException {
	unmarshaller = MarshallingCodecFactory.buildUnMarshalling();
    }

    protected Object decode(ByteBuf in) throws Exception {
		System.out.println("-----------------解碼時進行反序列化--------------------");
	int objectSize = in.readInt();
	ByteBuf buf = in.slice(in.readerIndex(), objectSize);
	ByteInput input = new ChannelBufferByteInput(buf);
	try {
	    unmarshaller.start(input);
	    Object obj = unmarshaller.readObject();
	    unmarshaller.finish();
	    in.readerIndex(in.readerIndex() + objectSize);
	    return obj;
	} finally {
	    unmarshaller.close();
	}
    }
}

4，編碼解碼器

(1)NettyMessageEncoder.class消息編碼器，對消息頭每個參數固定字節，對消息體進行序列化

public final class NettyMessageEncoder extends MessageToByteEncoder<NettyMessage> {

    MarshallingEncoder marshallingEncoder;

    public NettyMessageEncoder() throws IOException {
	this.marshallingEncoder = new MarshallingEncoder();
    }

    @Override
    protected void encode(ChannelHandlerContext ctx, NettyMessage msg,ByteBuf sendBuf) throws Exception {
		System.out.println("-----------------開始進行編碼--------------------");
	if (msg == null || msg.getHeader() == null){
		throw new Exception("The encode message is null");
	}else {
		// 信息寫入到流裏面並且固定字節
		sendBuf.writeShort(msg.getHeader().getsVersion());
		sendBuf.writeShort(msg.getHeader().getsCommand());
		sendBuf.writeInt(NettyUtil.getByteNum(msg.getBody()));
		sendBuf.writeLong(msg.getHeader().getuIdentity());
		sendBuf.writeShort(msg.getHeader().getpCmd());
		if (msg.getHeader().getFlag()==0) {
			// 本地tcp方式
			msg.getHeader().setSzSession(new byte[22]) ;
			sendBuf.writeBytes(msg.getHeader().getSzSession());
		}else {
			// 集控tcp方式
			sendBuf.writeLong(msg.getHeader().getAccount());
			sendBuf.writeLong(msg.getHeader().getPassword());
			msg.getHeader().setSzSession(new byte[6]);
			sendBuf.writeBytes(msg.getHeader().getSzSession());
		}

	}

	if (msg.getBody() != null) {
		Object body = msg.getBody();
		String jsonString = JSON.toJSONString(body);
		marshallingEncoder.encode(jsonString, sendBuf);
	}else {
		sendBuf.writeInt(0);
	}

	}

(2)NettyMessageDecoder.class消息編碼器，對消息頭每個參數進行讀取，對消息體進行反序列化

public class NettyMessageDecoder extends LengthFieldBasedFrameDecoder {


    MarshallingDecoder marshallingDecoder;

    public NettyMessageDecoder(int maxFrameLength, int lengthFieldOffset,
	    int lengthFieldLength) throws IOException {
	super(maxFrameLength, lengthFieldOffset, lengthFieldLength);
	marshallingDecoder = new MarshallingDecoder();
    }

    @Override
    protected Object decode(ChannelHandlerContext ctx, ByteBuf frame) throws Exception {
	System.out.println("-----------------開始進行解碼--------------------");
	System.err.println("frame.readableBytes():"+frame.readableBytes());
	NettyMessage message = new NettyMessage();
	Header header = new Header();
	header.setsVersion(frame.readShort());
	header.setsCommand(frame.readShort());
	header.setnPacketLen(frame.readInt());
	header.setuIdentity(frame.readLong());
	header.setpCmd(frame.readShort());
	// 獲取22字節的szSession
	frame.readLong();
	frame.readLong();
	frame.readInt();
	frame.readShort();

	// 開始對消息包進行解碼
	if (frame.readableBytes() > 4) {
	    message.setBody(marshallingDecoder.decode(frame));
	}
	message.setHeader(header);
	return message;
    }
}

5，服務端代碼

NettyServer.class用於監聽端口，處理客戶端請求

public class NettyServer {

    public void bind(int port) throws Exception {
		//bossGroup就是parentGroup，是負責處理TCP/IP連接的
        EventLoopGroup bossGroup = new NioEventLoopGroup(); 
        //workerGroup就是childGroup,是負責處理Channel(通道)的I/O事件
        EventLoopGroup workerGroup = new NioEventLoopGroup(); 

        ServerBootstrap sb = new ServerBootstrap();
        sb.group(bossGroup, workerGroup)
                .channel(NioServerSocketChannel.class)
                 //初始化服務端可連接隊列,指定了隊列的大小128
                .option(ChannelOption.SO_BACKLOG, 128)
                //保持長連接
                .childOption(ChannelOption.SO_KEEPALIVE, true) 
                //channel通道的處理器
                .childHandler(new ServerChannelInitializer());
        //綁定監聽端口，調用sync同步阻塞方法等待綁定操作完
        ChannelFuture future = sb.bind(port).sync();

        if (future.isSuccess()) {
            System.out.println("服務端啓動成功");
        } else {
            System.out.println("服務端啓動失敗");
            future.cause().printStackTrace();
            bossGroup.shutdownGracefully(); //關閉線程組
            workerGroup.shutdownGracefully();
        }

        //成功綁定到端口之後,給channel增加一個 管道關閉的監聽器並同步阻塞,直到channel關閉,線程纔會往下執行,結束進程。
        future.channel().closeFuture().sync();

    }
}

ServerChannelInitializer.class用於初始化channel通道，包括編碼器，解碼器，處理器等等

/**
 * @author David
 * @className ServerChannelInitializer
 * @date 2020/3/23 21:36
 */
public class ServerChannelInitializer extends ChannelInitializer<SocketChannel> {
    @Override
    protected void initChannel(SocketChannel channel) throws Exception {
        // 傳輸數據的解碼器
        channel.pipeline().addLast(new NettyMessageDecoder(1024 * 1024, 4, 4));
        // 傳輸數據的編碼器
        channel.pipeline().addLast( new NettyMessageEncoder());
        // 服務端收據接收處理器
        channel.pipeline().addLast(new ServerHandler());

    }
}

ServerHandler.class用於接收處理客戶端的消息

public class ServerHandler extends ChannelInboundHandlerAdapter{

    /**
     * 接受client發送的消息
     * @param ctx
     * @param msg
     * @throws Exception
     */
    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        System.out.println("-----------------服務器開始輸出消息--------------------");
        NettyMessage nettyMessage = (NettyMessage) msg;
        System.out.println("接收到客戶端信息:" + nettyMessage.toString());
        //返回的數據結構
        JsonBody jsonBody = new JsonBody();
        jsonBody.setUuid(UUID.randomUUID().toString());
        jsonBody.setData("hello nettyclient，這是給你的回覆!");
        // nettymessage
        NettyMessage sendNettyMessage = new NettyMessage();
        Header header = new Header();
        header.setsVersion((short) 0x6635);
        header.setsCommand((short) 0x2802);;
        header.setuIdentity((long)1232221);
        header.setnPacketLen(0);
        header.setpCmd((short)0x2901);
        header.setSzSession(null);
        header.setAccount((long) 42343244);
        header.setPassword((long) 31231213);
        header.setFlag(1);
        sendNettyMessage.setHeader(header);
        sendNettyMessage.setBody(jsonBody);
        ctx.writeAndFlush(sendNettyMessage);
    }

    /**
     * 通知處理器最後的channelRead()是當前批處理中的最後一條消息時調用
     * @param ctx
     * @throws Exception
     */
    @Override
    public void channelReadComplete(ChannelHandlerContext ctx) throws Exception {
        System.out.println("服務端接收數據完畢..");
        ctx.flush();
    }

    /**
     * 讀操作時捕獲到異常時調用
     * @param ctx
     * @param cause
     */
    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
        ctx.close();
    }

    /**
     * 客戶端去和服務端連接成功時觸發
     * @param ctx
     * @throws Exception
     */
    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
//        ctx.writeAndFlush("hello client");
    }
}

6，客戶端代碼

NettyClient.class用於建立連接，發送數據，配置處理器

public class NettyClient {

    private final String host;
    private final int port;
    private Channel channel;

    //連接服務端的端口號地址和端口號
    public NettyClient(String host, int port) {
        this.host = host;
        this.port = port;
    }

    public void start() throws Exception {

        final EventLoopGroup group = new NioEventLoopGroup();
        Bootstrap b = new Bootstrap();
        // 使用NioSocketChannel來作爲連接用的channel類
        b.group(group).channel(NioSocketChannel.class)
                .handler(new ClientChannelInitializer());
        //發起異步連接請求，綁定連接端口和host信息
        final ChannelFuture future = b.connect(host, port).sync();

        future.addListener(new ChannelFutureListener() {

            @Override
            public void operationComplete(ChannelFuture arg0) throws Exception {
                if (future.isSuccess()) {
                    System.out.println("連接服務器成功");

                } else {
                    System.out.println("連接服務器失敗");
                    future.cause().printStackTrace();
                    group.shutdownGracefully(); //關閉線程組
                }
            }
        });

        this.channel = future.channel();
    }

    public Channel getChannel() {
        return channel;
    }

ClientChannelInitializer.class用於初始化channel通道，包括編碼器，解碼器，處理器等等

public class ClientChannelInitializer extends ChannelInitializer<SocketChannel> {

    @Override
    protected void initChannel(SocketChannel channel) throws Exception {
        // 解碼器
        channel.pipeline().addLast(new NettyMessageDecoder(1024 * 1024, 4, 4));
        // 編碼器
        channel.pipeline().addLast( new NettyMessageEncoder());
        // 處理器
        channel.pipeline().addLast(new ClientHandler());
}
}

ClientHandler .class用於客戶端消息處理，處理服務端返回的數據

/**
 * @author david
 * client消息處理類
 */
public class ClientHandler extends SimpleChannelInboundHandler<NettyMessage> {

    /**
     * 處理服務端返回的數據
     * @param ctx
     * @param msg
     * @throws Exception
     */
    @Override
    protected void channelRead0(ChannelHandlerContext ctx, NettyMessage msg) throws Exception {
        System.out.println("接受到server響應數據: " + msg.toString());
    }

    @Override
    public void channelActive(ChannelHandlerContext ctx) throws Exception {
        super.channelActive(ctx);
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        ctx.close();
    }



}

7，測試

啓動服務端

public class NettyServerStart {

    public static void main(String[] args) throws Exception {

        new NettyServer().bind(8080);
    }
}

啓動客戶端

public class NettyClientStart {

    public static void main(String[] args) throws Exception {

        NettyClient client = new NettyClient(NettyConstant.LOCAL_IP, NettyConstant.LOCAL_PORT);
        //啓動client服務
        client.start();
        Channel channel = client.getChannel();
        
        JsonBody jsonBody = new JsonBody();
        jsonBody.setUuid("qwertyuiopuyiyiyiyiyit滬深 三分毒");
        jsonBody.setData("hello nettyserver!");
        // nettymessage
        NettyMessage nettyMessage = new NettyMessage();
        Header header = new Header();
        header.setsVersion((short) 0x6635);
        header.setsCommand((short) 0x2802);;
        header.setuIdentity((long)1232221);
        header.setpCmd((short)0x2901);
        header.setSzSession(null);
        header.setAccount((long) 121212151);
        header.setPassword((long) 784514544);
        header.setFlag(1);
        nettyMessage.setHeader(header);
        nettyMessage.setBody(jsonBody);

        //channel對象可保存在map中，供其它地方發送消息
        channel.writeAndFlush(nettyMessage);
    }

}

服務端啓動成功接收到消息，並返回消息，40字節的消息頭沒有錯位

客戶端發送消息，並接收服務器消息成功

總結：如上結束基於Netty的私有協議棧開發，其中還有登錄模塊，心跳模塊，超時模塊會在後面給出

Netty實現TCP私有協議棧的開發

1，私有協議棧需求

2，消息封裝

3，序列化數據

4，編碼解碼器

5，服務端代碼

6，客戶端代碼

7，測試

DO轉DTO映射MapStruct生產環境使用

dubbo的調用原理及泛化調用

Netty實現TCP私有協議棧的開發

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結