QNX® Neutrino 進程間通信編程之Message-passing/Pules

介紹

Interprocess Communication（IPC，進程間通信）在QNX Neutrino從一個嵌入式實時系統向一個全面的POSIX系統轉變起着至關重要的作用。IPC是將在內核中提供各種服務的進程內聚在一起的粘合劑。在QNX中，消息傳遞是IPC的主要形式，也提供了其他的形式，除非有特殊的說明，否則這些形式也都是基於本地消息傳遞而實現的。

QNX Neutrino提供以下形式的IPC：

Service:	Implemented in:
Message-passing	Kernel
Pules	Kernel
Signals	Kernel
Event Delivery	External process
POSIX message queues	External process
Shared memory	Process manager
Pipes	External process
FIFOs	External process

本篇幅主要講解Message-passing消息傳遞與Pulses脈衝。

Message-passing

比較傳統的IPC方式是基於主從式構架（client-server），並且是雙向通信。

再仔細來看的話，就是每一個process裏面都有一個thread來負責通信。當一個線程在等待回信的時候，就會傻傻的等待，什麼都不做了。直到收到回覆信息。

客戶端線程

• 客戶端線程調用MsgSend()後，如果服務器線程還沒調用MsgReceive()，客戶端線程狀態則爲SEND blocked，一旦服務器線程調用了MsgReceive()，客戶端線程狀態變爲REPLY blocked，當服務器線程執行MsgReply()後，客戶端線程狀態就變成了READY；

• 如果客戶端線程調用MsgSend()後，而服務器線程正阻塞在MsgReceive()上， 則客戶端線程狀態直接跳過SEND blocked，直接變成REPLY blocked；

• 當服務器線程失敗、退出、或者消失了，客戶端線程狀態變成READY，此時MsgSend()會返回一個錯誤值。

服務器線程

• 服務器線程調用MsgReceive()時，當沒有線程給它發送消息，它的狀態爲RECEIVE blocked，當有線程發送時變爲READY；

• 服務器線程調用MsgReceive()時，當已經有其他線程給它發送過消息，MsgReceive()會立馬返回，而不會阻塞；

• 服務器線程調用MsgReply()時，不會阻塞；

客戶端線程和服務器線程在時間主線裏顯示如下：

下面列出兩種場景Receive before Send和Send before Receive

服務器線程MsgReceive發生在客戶線程MsgSend之前

客戶線程MsgSend發生在服務器線程MsgReceive之前

由上面兩個場景看客戶線程MsgSend和服務器線程MsgReceive直接影響Message-passing性能。

Servers收到信息在通道上，Clients通過connection連接上channel，來發送信息。

一個進程可以有多個connections連接到另一個進程的channel上，是個多對一的關係。

Message-passing編程流程如下

– Server:

• creates a channel (ChannelCreate())
• waits for a message (MsgReceive())
• performs processing
• sends reply (MsgReply())
• goes back for more -> waits for a message (MsgReceive())

– Client:

• attaches to channel (ConnectAttach())
• sends message (MsgSend())
• processes reply

函數原型

int ChannelCreate( unsigned flags );
int ChannelDestroy( int chid );
int name_open( const char * name, int flags );
int name_close( int coid );

int ConnectAttach( uint32_t nd, pid_t pid, int chid, unsigned index, int flags );
int ConnectDetach( int coid );
name_attach_t * name_attach( dispatch_t * dpp, const char * path, unsigned flags );
int name_detach( name_attach_t * attach, unsigned flags );

long MsgSend( int coid, const void* smsg, size_t sbytes, void* rmsg, size_t rbytes );
int MsgReceive( int chid, void * msg, size_t bytes, struct _msg_info * info );
int MsgReply( int rcvid, long status, const void* msg, size_t bytes );

ssize_t MsgWrite( int rcvid, const void* msg, size_t size, size_t offset );
ssize_t MsgRead( int rcvid,  void* msg, size_t bytes,  size_t offset );

詳細命令使用請看以下命令鏈接：

Message-passing API

Function	Description
ChannelCreate()	Create a channel to receive messages on.
ChannelDestroy()	Destroy a channel.
ConnectAttach()	Create a connection to send messages on.
ConnectDetach()	Detach a connection.
name_open()	Open a name to connect to a server
name_close()	Close a server connection that was opened by name_open()
name_attach()	Register a name in the pathname space and create a channel
name_detach()	Remove a name from the namespace and destroy the channel
MsgSend()	Send a message and block until reply.
MsgSendv()	Send a message to a channel
MsgReceive()	Wait for a message.
MsgReceivev()	Wait for a message or pulse on a channel
MsgReceivePulse()	Wait for a tiny, nonblocking message (pulse).
MsgReply()	Reply to a message.
MsgError()	Reply only with an error status. No message bytes are transferred.
MsgRead()	Read additional data from a received message.
MsgReadv()	Read data from a message
MsgWrite()	Write additional data to a reply message.
MsgWritev()	Write a reply message
MsgInfo()	Obtain info on a received message.
MsgSendPulse()	Send a tiny, nonblocking message (pulse).
MsgDeliverEvent()	Deliver an event to a client.
MsgKeyData()	Key a message to allow security checks.

服務器端僞代碼如下：

#include <sys/neutrino.h>
int chid; // channel ID
main ()
{
	int rcvid; // receive ID
	chid = ChannelCreate (0 /* or flags */);
	/* create client thread */
	...
	while (1) {
		rcvid = MsgReceive (chid, &recvmsg, rbytes, NULL);
		// process message from client here...
		MsgReply (rcvid, 0, &replymsg, rbytes);
	} 
}

客戶端僞代碼如下：

#include <sys/neutrino.h>
extern int chid;
int coid; // connection id
int status;
client_thread() {
	// create the connection, typically done only once
	coid = ConnectAttach (0, 0, chid, _NTO_SIDE_CHANNEL, 0);
	...
	// at some point later we decide we want to send a message
	status = MsgSend (coid, &sendmsg, sbytes, &replymsg, rbytes);
}

Massage之間的通信數據總是通過拷貝，而不是指針的傳遞。

那麼如何設計消息傳遞策略呢？

• 定義公共消息頭消息類型結構體
• 所有消息都是同一個消息類型
• 具有匹配每個消息類型的結構
• 如果消息相關或它們使用共同的結構，請考慮使用消息類型和子類型
• 定義匹配的回覆結構體。如果合適，避免不同類型服務器的消息類型重疊

下面的消息傳遞策略僞代碼幫助你理解：

while(1) {
		recvid = MsgReceive( chid, &msg, sizeof(msg), NULL );
		switch( msg.hdr.type ) {
			case MSG_TYPE_1:
				handle_msg_type_1(rcvid, &msg);
				break;
		case MSG_TYPE_2:
				… 
	} 
}

頻道（Channel）與連接（Connect）實例代碼

服務器：這個服務器，準備好頻道後，就從頻道上接收信息。如果信息是字符串”Hello“的話，這個服務器應答一個”World“字符串。如果收到的信處是字符串“Ni Hao", 那麼它會應答”Zhong Guo"，其它任何消息都用MsgError()回答一個錯誤。

// Simple server
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <sys/neutrino.h>

int main()
{
	int chid, rcvid, status;
	char buf[128];

	if ((chid = ChannelCreate(0)) == -1) {
		perror("ChannelCreate");
		return -1;
	}

	printf("Server is ready, pid = %d, chid = %d\n", getpid(), chid);

	for (;;) {
		if ((rcvid = MsgReceive(chid, buf, sizeof(buf), NULL)) == -1) {
			perror("MsgReceive");
			return -1;
		}
	printf("Server: Received '%s'\n", buf);

	/* Based on what we receive, return some message */
	if (strcmp(buf, "Hello") == 0) {
			MsgReply(rcvid, 0, "World", strlen("World") + 1);
		} else if (strcmp(buf, "Ni Hao") == 0) {
			MsgReply(rcvid, 0, "Zhong Guo", strlen("Zhong Guo") + 1);
		} else {
			MsgError(rcvid, EINVAL);
		}
	}

	ChannelDestroy(chid);
	return 0;
}

服務器：這個服務器，準備好頻道後，就從頻道上接收信息。如果信息是字符串”Hello“的話，這個服務器應答一個”World“字符串。如果收到的信處是字符串“Ni Hao", 那麼它會應答”Zhong Guo"，其它任何消息都用MsgError()回答一個錯誤。

//simple client
#include <stdio.h>
#include <string.h>
#include <sys/neutrino.h>

int main(int argc, char **argv)
{
	pid_t spid;
	int chid, coid, i;
	char buf[128];

	if (argc < 3) {
		fprintf(stderr, "Usage: simple_client <pid> <chid>\n");
		return -1;
	}

	spid = atoi(argv[1]);
	chid = atoi(argv[2]);

	if ((coid = ConnectAttach(0, spid, chid, 0, 0)) == -1) {
		perror("ConnectAttach");
		return -1;
	}
	/* sent 3 pairs of "Hello" and "Ni Hao" */
	for (i = 0; i < 3; i++) {
		sprintf(buf, "Hello");
		printf("client: sent '%s'\n", buf);
		if (MsgSend(coid, buf, strlen(buf) + 1, buf, sizeof(buf)) != 0) {
			perror("MsgSend");
			return -1;
		}
		printf("client: returned '%s'\n", buf);

		sprintf(buf, "Ni Hao");
		printf("client: sent '%s'\n", buf);
		if (MsgSend(coid, buf, strlen(buf) + 1, buf, sizeof(buf)) != 0) {
			perror("MsgSend");
			return -1;
		}
		printf("client: returned '%s'\n", buf);
	}
	/* sent a bad message, see if we get an error */
	sprintf(buf, "Unknown");
	printf("client: sent '%s'\n", buf);
	if (MsgSend(coid, buf, strlen(buf) + 1, buf, sizeof(buf)) != 0) {
		perror("MsgSend");
		return -1;
	}
	ConnectDetach(coid);
	return 0;
}

分別編譯後的執行結果如下：

服務器：

$ ./simple_server
Server is ready, pid = 36409378, chid = 2
Server: Received 'Hello'
Server: Received 'Ni Hao'
Server: Received 'Hello'
Server: Received 'Ni Hao'
Server: Received 'Hello'
Server: Received 'Ni Hao'
Server: Received 'Unknown'
Server: Received ''

客戶端：

$ ./simple_client 36409378 2
client: sent 'Hello'
client: returned 'World'
client: sent 'Ni Hao'
client: returned 'Zhong Guo'
client: sent 'Hello'
client: returned 'World'
client: sent 'Ni Hao'
client: returned 'Zhong Guo'
client: sent 'Hello'
client: returned 'World'
client: sent 'Ni Hao'
client: returned 'Zhong Guo'
client: sent 'Unknown'
MsgSend: Invalid argument

name_open與name_attach實例代碼

用name_open與MsgSend消息傳遞的客戶端線程

#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <sys/dispatch.h>

#define ATTACH_POINT "myname"

/* We specify the header as being at least a pulse */
typedef struct _pulse msg_header_t;

/* Our real data comes after the header */
typedef struct _my_data {
    msg_header_t hdr;
    int data;
} my_data_t;

/*** Client Side of the code ***/
int client() 
{
    my_data_t msg;
	int msg_reply;
    int server_coid;

    if ((server_coid = name_open(ATTACH_POINT, 0)) == -1) {
		printf("client name open failed\n");
        return EXIT_FAILURE;
    }

    /* We would have pre-defined data to stuff here */
    msg.hdr.type = 0x00;
    msg.hdr.subtype = 0x00;
	msg.data = 1;

    /* Do whatever work you wanted with server connection */
    printf("client name open success, Client sending msg %d \n", msg.data);
    if (MsgSend(server_coid, &msg, sizeof(msg), &msg_reply, sizeof(msg_reply)) == -1) {
		printf("client send msg 1 error\n");    
	}
	
    printf("client receive msg 1 reply: %d \n", msg_reply);
	
	msg.hdr.type = 0x00;
    msg.hdr.subtype = 0x01;
	msg.data = 2;
	printf("client name open success, Client sending msg %d \n", msg.data);
	if (MsgSend(server_coid, &msg, sizeof(msg), &msg_reply, sizeof(msg_reply)) == -1) {
		printf("client send msg 2 error\n");    
	}
	
    printf("client receive msg 2 reply: %d \n", msg_reply);
	
    /* Close the connection */
    name_close(server_coid);
    return EXIT_SUCCESS;
}

int main(int argc, char *argv[]) {
	int ret;

    if (argc < 2) {
        printf("Usage %s -s | -c \n", argv[0]);
        ret = EXIT_FAILURE;
    }
    else if (strcmp(argv[1], "-c") == 0) {
        printf("Running client ... \n");
        ret = client();
    }
	else {
        printf("Usage %s -s | -c \n", argv[0]);
        ret = EXIT_FAILURE;
    }
    return ret;
}

用name_attach與MsgReceive，MsgReply實現消息傳遞的服務器線程

#include <stdio.h>
#include <errno.h>
#include <stdlib.h>
#include <string.h>
#include <sys/dispatch.h>

#define ATTACH_POINT "myname"

/* We specify the header as being at least a pulse */
typedef struct _pulse msg_header_t;

/* Our real data comes after the header */
typedef struct _my_data {
    msg_header_t hdr;
    int data;
} my_data_t;

int msg_update_fail =3;
int msg_update_success =4;

/*** Server Side of the code ***/
int server() {
   name_attach_t *attach;
   my_data_t msg;

   
   my_data_t msg_reply;
   msg_reply.hdr.type = 0x00;
   msg_reply.hdr.subtype = 0x00;
   
  // my_data_t msg_replaydata;
   int rcvid;

   /* Create a local name (/dev/name/local/...) */
   if ((attach = name_attach(NULL, ATTACH_POINT, 0)) == NULL) {
   		printf("server name_attach error\n");
       return EXIT_FAILURE;
   }
   printf("server name_attach suceess,wait masg from client\n");

   /* Do your MsgReceive's here now with the chid */
   while (1) {
       rcvid = MsgReceive(attach->chid, &msg, sizeof(msg), NULL);

       if (rcvid == -1) {/* Error condition, exit */
           break;
       }
       /* A message (presumable ours) received, handle */
	   switch(msg.data){

			case 1:
				printf("Server receive msg data %d \n", msg.data);
				MsgReply(rcvid, EOK, &msg_update_fail, sizeof(msg_update_fail));
				//MsgReply(UpdateReceiveId, EOK, &msg_update_fail, 0);
				break;
			case 2:
				printf("Server receive msg data %d \n", msg.data);		
				MsgReply(rcvid, EOK, &msg_update_success, sizeof(msg_update_success));
				break;
			default:
				break;
	   }
	   
       MsgReply(rcvid, EOK, 0, 0);

   }

   /* Remove the name from the space */
   name_detach(attach, 0);

   return EXIT_SUCCESS;
}

int main(int argc, char **argv) {
    int ret;

    if (argc < 2) {
        printf("Usage %s -s | -c \n", argv[0]);
        ret = EXIT_FAILURE;
    }
    else if (strcmp(argv[1], "-s") == 0) {
        printf("Running Server ... \n");
        ret = server();
    }
	else {
        printf("Usage %s -s | -c \n", argv[0]);
        ret = EXIT_FAILURE;
    }
    return ret;
}

如果您想在一條消息中發送以下三個緩衝區怎麼辦？

你可能通過三次memcpy()處理一個大的buffer。但是有個更有效的方法通過MsgSendv()傳遞指針數組。

使用函數原型如下：

long MsgSendv( int coid,  const iov_t* siov, size_t sparts,  const iov_t* riov, size_t rparts );
int MsgReceivev( int chid, const iov_t * riov, size_t rparts, struct _msg_info * info );
ssize_t MsgReadv( int rcvid, const iov_t* riov, size_t rparts, size_t offset );
void SETIOV( iov_t *msg, void *addr, size_t len );

typedef struct {
    void *iov_base;
    size_t iov_len;
} iov_t;

iov_t iovs [3];

實際IOVs應用實例，客戶端需要將一個12KBytes的文件傳給服務器端。

客戶端僞代碼如下：

write (fd, buf, size);
effectively does:
	hdr.nbytes = size;
	SETIOV (&siov[0], &header, sizeof (header));
	SETIOV (&siov[1], buf, size);
	MsgSendv (fd, siov, 2, NULL, 0);

實際上獲得的是連續的字節流。

服務器端獲得接收內容是什麼？

// assume riov has been setup
MsgReceivev (chid, riov, 4, NULL);

實際上我們是不知道接收的數據是多少，直到我們看見數據流的頭。

rcvid = MsgReceive (chid, &header,sizeof (header), NULL);
//獲取到數據流的頭信息後
SETIOV (iov [0], &cbuf [6], 4096);
SETIOV (iov [1], &cbuf [2], 4096);
SETIOV (iov [2], &cbuf [5], 4096);
//用MsgReadv直接根據sizeof(header)偏移把剩下的數據獲取出來。
MsgReadv (rcvid, iov, 3, sizeof(header));

整個消息傳遞從客戶端到服務器，你可以理解爲下面兩個流程：

那怎麼理解從服務器端拷貝數據給客戶端呢？

ssize_t MsgWrite( int rcvid, const void* msg, size_t size, size_t offset );
ssize_t MsgWritev( int rcvid, const iov_t* iov, size_t parts, size_t offset );

MsgWrite回傳數據實例如下：

Pulses脈衝

脈衝其實更像一個短消息，也是在“連接Connection”上發送的。脈衝最大的特點是它是異步的。發送方不必要等接收方應答，直接可以繼續執行。

脈衝的通信方式很特別，就像喊命令，不需要回應，執行就好了。便宜還快速，也不會發生blocking的現象。但是，這種異步性也給脈衝帶來了限制。脈衝能攜帶的數據量有限，只有一個**8位的"code"域 (1byte)用來區分不同的脈衝，和一個32位的“value"域 (4字節)**來攜帶數據。脈衝最主要的用途就是用來進行“通知”(Notification)。不僅是用戶程序，內核也會生成發送特殊的“系統脈衝”到用戶程序，以通知某一特殊情況的發生。

int MsgSendPulse ( int coid, int priority, int code, int value );
                                                |         |
                                      8bits <---|         |
                                     32bits <-------------|

• code 通常用於表示“脈衝類型”的有效範圍是 _PULSE_CODE_MINAVAIL 到 _PULSE_CODE_MAXAVAIL。
• priority 就像發送線程的消息優先級一樣
• 接收線程以該優先級運行
• 發送順序基於優先級

要跨進程邊界發送脈衝，發送者必須與接收者具有相同的有效用戶 ID 或者是 root 用戶

脈衝的接收比較簡單，如果你知道頻道上不會有別的消息，只有脈衝的話，可以用MsgReceivePulse()來只接收脈衝； 如果頻道既可以接收消息，也可以接收脈衝時，就直接用MsgReceive()，只要確保接收緩衝(ReveiveBuf)至少可以容下一個脈衝（sizeof struct _pulse)就可以了。 在後一種情況下，如果MsgReceive()返回的rcvid是0，就代表接收到了一個脈衝，反之，則收到了一個消息。所以，一個既接收脈衝，又接收消息的服務器，Pulses脈衝實例僞代碼如下：

#include <sys/neutrino.h>

struct _pulse {
    uint16_t                    type;
    uint16_t                    subtype;
    int8_t                      code;   // <---- 8-bit code
    uint8_t                     zero[3];
    union sigval                value; // <--- 32-bit value
    int32_t                     scoid;
};

typedef union {
	struct _pulse pulse;
	// other message types you will receive
} myMessage_t;
…
myMessage_t msg;
while (1) {
	rcvid = MsgReceive (chid, &msg, sizeof(msg), NULL);
	if (rcvid == 0) {
		// it’s a pulse, look in msg.pulse… for data
        process_pulse(&msgs, &info);
		} else {
		// it’s a regular message
        process_message(&msgs, &info);
	}
}
…

展開process_pulse處理實現

...
	rcvid = MsgReceive (chid, &msg, sizeof(msg), NULL);
	if (rcvid == 0) {
		// it’s a pulse, look in msg.pulse… for data
		switch (msg.pulse.code) {
		case _PULSE_CODE_UNBLOCK:
			// a kernel unblock pulse
			...
			break;
		case MY_PULSE_CODE:
			// do what's needed
			...
			break;
    } else {
    	process_message(&msgs, &info);
    }  
...

脈衝的發送，最直接的就是MsgSendPulse()。不過，這個函數通常只在一個進程中，用在一個線程要通知另一個線程的情形。在跨進程的時候，通常不會用到這個函數，而是用到下面將要提到的 MsgDeliverEvent()。與消息傳遞相比，消息傳遞永遠是在進程間進行的。也就是說，不會有一個進程向內核發送數據的情形。而脈衝就不一樣，除了用戶進程間可以發脈衝以外，內核也會向用戶進程發送“系統脈衝”來通知某一事件的發生。

如果您有一個頻道，您可能會在該頻道上接收來自 MsgSend*() 調用和脈衝的消息，但在某個時間點只想接收脈衝，使用MsgReceivePulse()則很有用。

int MsgReceivePulse( int chid, void * pulse, size_t bytes, struct _msg_info * info );

如果進程正在接收消息和脈衝：

• 接收順序仍然基於優先級，使用脈衝的優先級
• 內核將以其接收到的脈衝的優先級運行接收線程
• 脈衝和消息可能會混合在一起

來自線程 1 的Send Pulse比在它Send message之前先到達服務器。因爲脈衝的優先級比線程本身的優先級要高。

Pulse API	send	receive
	MsgSendPulse(coid, priority, code, value)	MsgReceivePulse()
	MsgDeliverEvent()	MsgReceive()

• MsgSendPulse() 只在一個進程中的通知，用與同一個進程中一個線程要通知另一個線程的情形, 其中 code 8bits; value 32bits
• MsgDeliverEvent() 在跨進程的時候的通知
• MsgReceivePulse() 用於頻道上只有pulse的接收
• MsgReceive() 用於頻道上既接收message又接收pulse

參考文獻：

Programming with POSIX Threads
QNC IPC---msg send receive example
QNX trying to send struct through MsgSend(), MsgReply() IPC message passing
從API開始理解QNX - 知乎 (zhihu.com)
QNX IPC機制