MPI-3 近鄰集合通信

本文從本人簡書博客同步過來

在上一篇中我們介紹了 mpi4py 中的非阻塞集合通信方法，下面我們將介紹 mpi4py 中的近鄰集合通信方法，另一個 MPI-3 引進的新特性。

近鄰集合通信（neighborhood collective communication） 是定義在拓撲通信子之上的一類集合通信操作，不同於一般集合通信操作中通信子上的所有進程都會參與，近鄰集合通信只要求拓撲通信子上的某個進程和其直接鄰居進程之間進行集合通信，其它進程可以不用參與，因此近鄰集合通信具有非常好的可擴展性。

近鄰集合操作也有阻塞和非阻塞兩個版本。

方法接口

下面給出 mpi4py 中近鄰集合通信的方法接口。

MPI.Topocomm.neighbor_allgather(self, sendobj)

近鄰全收集操作，對應 MPI.Comm.allgather。以小寫字母開頭的方法，可以傳遞任意可被 pickle 的 Python 對象 sendobj，爲阻塞通信方法。

MPI.Topocomm.neighbor_alltoall(self, sendobj)

近鄰全發散操作，對應 MPI.Comm.alltoall。以小寫字母開頭的方法，可以傳遞任意可被 pickle 的 Python 對象 sendobj，爲阻塞通信方法。

MPI.Topocomm.Neighbor_allgather(self, sendbuf, recvbuf)

近鄰阻塞全收集操作，對應 MPI.Comm.Allgather，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Neighbor_allgatherv(self, sendbuf, recvbuf)

近鄰阻塞向量全收集操作，對應 MPI.Comm.Allgatherv，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Neighbor_alltoall(self, sendbuf, recvbuf)

近鄰阻塞全發散操作，對應 MPI.Comm.Alltoall，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Neighbor_alltoallv(self, sendbuf, recvbuf)

近鄰阻塞向量全發散操作，對應 MPI.Comm.Alltoallv，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Neighbor_alltoallw(self, sendbuf, recvbuf)

近鄰阻塞向量全發散操作，對應 MPI.Comm.Alltoallw，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Ineighbor_allgather(self, sendbuf, recvbuf)

近鄰非阻塞全收集操作，對應 MPI.Comm.Iallgather，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Ineighbor_allgatherv(self, sendbuf, recvbuf)

近鄰非阻塞向量全收集操作，對應 MPI.Comm.Iallgatherv，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Ineighbor_alltoall(self, sendbuf, recvbuf)

近鄰非阻塞全發散操作，對應 MPI.Comm.Ialltoall，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Ineighbor_alltoallv(self, sendbuf, recvbuf)

近鄰非阻塞向量全發散操作，對應 MPI.Comm.Ialltoallv，只適用於傳遞具有緩衝區接口的對象。

MPI.Topocomm.Ineighbor_alltoallw(self, sendbuf, recvbuf)

近鄰非阻塞向量全發散操作，對應 MPI.Comm.Ialltoallw，只適用於傳遞具有緩衝區接口的對象。

由以上方法接口可以看出，近鄰集合通信最基本最主要的操作只有兩個：MPI.Topocomm.Neighbor_allgather 和 MPI.Topocomm.Neighbor_alltoall，其它的要麼是其向量版本，要麼是對應的非阻塞版本。Neighbor_allgather 會發送一份相同的數據給其所有直接鄰居，並從這些鄰居處接收一份數據；Neighbor_alltoall 會給其每個直接鄰居發送一份不同的數據，並從這些鄰居處接收一份數據。以 3 × 3 週期性的笛卡爾拓撲爲例，下圖顯示了 Neighbor_allgather 的發送和接收關係。

例程

以上圖中的 3 × 3 笛卡爾拓撲爲例，下面給出近鄰集合操作的使用例程。

# neighbor.py

"""
Demonstrates the usage of neighborhood collective communication.

Run this with 9 processes like:
$ mpiexec -n 9 python neighbor.py
"""

import numpy as np
from mpi4py import MPI


comm = MPI.COMM_WORLD
rank = comm.Get_rank()

dims = [3, 3]

# -----------------------------------------------------------------
# neighbor_allgather with periodic boundary
periods = [True, True]
cart_comm = comm.Create_cart(dims, periods)
recv_obj = cart_comm.neighbor_allgather(rank)
print 'neighbor_allgather (periodic): rank %d has %s' % (rank, recv_obj)


# -----------------------------------------------------------------
# neighbor_allgather with non-periodic boundary
periods = [False, False]
cart_comm = comm.Create_cart(dims, periods)
recv_obj = cart_comm.neighbor_allgather(rank)
print 'neighbor_allgather (non-periodic): rank %d has %s' % (rank, recv_obj)


# -----------------------------------------------------------------
# neighbor_alltoall with periodic boundary
periods = [True, True]
cart_comm = comm.Create_cart(dims, periods)
recv_obj = cart_comm.neighbor_alltoall(['a', 'b', 'c', 'd'])
print 'neighbor_alltoall (periodic): rank %d has %s' % (rank, recv_obj)


# -----------------------------------------------------------------
# Neighbor_allgather with periodic boundary
periods = [True, True]
cart_comm = comm.Create_cart(dims, periods)
send_buf = np.array([rank], dtype='i')
recv_buf = np.full((4,), -1, dtype='i') # initialize with all -1
cart_comm.Neighbor_allgather(send_buf, recv_buf)
print 'Neighbor_allgather (periodic): rank %d has %s' % (rank, recv_buf)


# -----------------------------------------------------------------
# Ineighbor_allgather with non-periodic boundary
periods = [False, False]
cart_comm = comm.Create_cart(dims, periods)
send_buf = np.array([rank], dtype='i')
recv_buf = np.full((4,), -1, dtype='i')
req = cart_comm.Ineighbor_allgather(send_buf, recv_buf)
req.Wait()
print 'Ineighbor_allgather (non-periodic): rank %d has %s' % (rank, recv_buf)

運行結果如下：

$ mpiexec -n 9 python neighbor.py
neighbor_allgather (periodic): rank 7 has [4, 1, 6, 8]
neighbor_allgather (non-periodic): rank 7 has [4, None, 6, 8]
neighbor_alltoall (periodic): rank 7 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 7 has [4 1 6 8]
Ineighbor_allgather (non-periodic): rank 7 has [ 4 -1  6  8]
neighbor_allgather (periodic): rank 8 has [5, 2, 7, 6]
neighbor_allgather (non-periodic): rank 8 has [5, None, 7, None]
neighbor_alltoall (periodic): rank 8 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 8 has [5 2 7 6]
Ineighbor_allgather (non-periodic): rank 8 has [ 5 -1  7 -1]
neighbor_allgather (periodic): rank 0 has [6, 3, 2, 1]
neighbor_allgather (non-periodic): rank 0 has [None, 3, None, 1]
neighbor_alltoall (periodic): rank 0 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 0 has [6 3 2 1]
Ineighbor_allgather (non-periodic): rank 0 has [-1  3 -1  1]
neighbor_allgather (periodic): rank 1 has [7, 4, 0, 2]
neighbor_allgather (non-periodic): rank 1 has [None, 4, 0, 2]
neighbor_alltoall (periodic): rank 1 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 1 has [7 4 0 2]
Ineighbor_allgather (non-periodic): rank 1 has [-1  4  0  2]
neighbor_allgather (periodic): rank 2 has [8, 5, 1, 0]
neighbor_allgather (non-periodic): rank 2 has [None, 5, 1, None]
neighbor_alltoall (periodic): rank 2 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 2 has [8 5 1 0]
Ineighbor_allgather (non-periodic): rank 2 has [-1  5  1 -1]
neighbor_allgather (periodic): rank 3 has [0, 6, 5, 4]
neighbor_allgather (non-periodic): rank 3 has [0, 6, None, 4]
neighbor_alltoall (periodic): rank 3 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 3 has [0 6 5 4]
Ineighbor_allgather (non-periodic): rank 3 has [ 0  6 -1  4]
neighbor_allgather (periodic): rank 4 has [1, 7, 3, 5]
neighbor_allgather (non-periodic): rank 4 has [1, 7, 3, 5]
neighbor_alltoall (periodic): rank 4 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 4 has [1 7 3 5]
Ineighbor_allgather (non-periodic): rank 4 has [1 7 3 5]
neighbor_allgather (periodic): rank 5 has [2, 8, 4, 3]
neighbor_allgather (non-periodic): rank 5 has [2, 8, 4, None]
neighbor_alltoall (periodic): rank 5 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 5 has [2 8 4 3]
Ineighbor_allgather (non-periodic): rank 5 has [ 2  8  4 -1]
neighbor_allgather (periodic): rank 6 has [3, 0, 8, 7]
neighbor_allgather (non-periodic): rank 6 has [3, None, None, 7]
neighbor_alltoall (periodic): rank 6 has ['b', 'a', 'd', 'c']
Neighbor_allgather (periodic): rank 6 has [3 0 8 7]
Ineighbor_allgather (non-periodic): rank 6 has [ 3 -1 -1  7]

由以上的輸出結果可以看出，對笛卡爾拓撲，近鄰集合操作某進程從其直接鄰居處接收數據的順序爲：按照笛卡爾網格的維度順序從小到大接收（如果拓撲爲週期性的，會考慮其對應的週期性）。比如對進程 4，首先接收其上面行的數據 1， 然後其下面行的數據 7， 然後其左邊列的數據 3， 最後其右邊列的數據5。如果拓撲爲非週期性的，則不會向其不存在的邊界鄰居發送數據，也不會從其不存在的邊界鄰居接收數據。

以上介紹了 mpi4py 中的近鄰集合通信方法，在下一篇中我們將介紹 mpi4py 中的非阻塞通信子複製和組集合通信子創建方法。