NumPy

1. Buffer protocol

python支持非常通用和方便地與plugin libraries交換數據。types可以expose a buffer view [1], 提供 fast direct access to the raw internal data representation. 假定我們需要bind如下簡單地Matrix class。

class Matrix {
public:
    Matrix(size_t rows, size_t cols) : m_rows(rows), m_cols(cols) {
        m_data = new float[rows*cols];
    }
    float *data() { return m_data; }
    size_t rows() const { return m_rows; }
    size_t cols() const { return m_cols; }
private:
    size_t m_rows, m_cols;
    float *m_data;
};

如下代碼會將Matrix contents暴露成一個 buffer object, 使得能夠將Matrices轉換爲numpy array。甚至能夠完全不需要拷貝操作，使用python 表達式： np.array(matrix_instance, copy = False).

py::class_<Matrix>(m, "Matrix", py::buffer_protocol())
   .def_buffer([](Matrix &m) -> py::buffer_info {
        return py::buffer_info(
            m.data(),                               /* Pointer to buffer */
            sizeof(float),                          /* Size of one scalar */
            py::format_descriptor<float>::format(), /* Python struct-style format descriptor */
            2,                                      /* Number of dimensions */
            { m.rows(), m.cols() },                 /* Buffer dimensions */
            { sizeof(float) * m.cols(),             /* Strides (in bytes) for each index */
              sizeof(float) }
        );
    });

對一個新的type支持buffer protocal 需要在py::class_ constructor中添加tag： py::buffer_protocal(), 並調用 def_buffer()method 和 lamda method py::buffer_info創建一個關於matrix instance 的描述記錄。py::buffer_info的內容是python buffer protocal specification 的鏡像。

struct buffer_info {
    void *ptr;
    ssize_t itemsize;
    std::string format;
    ssize_t ndim;
    std::vector<ssize_t> shape;
    std::vector<ssize_t> strides;
};

使用py::buffer作爲C++函數的形參，就能使C++function將 python buffer object作爲形參。buffers可以有很多中配置，因而函數體(function body)中進行safty-checks非常有必要。如下例是一個基本的爲Eigen double precision Matrix (Eigen::MatrixXd) 創建custom constructor的例子，支持從兼容的 buffer object 進行初始化(e.g. Numpy Matrix)。

/* Bind MatrixXd (or some other Eigen type) to Python */
typedef Eigen::MatrixXd Matrix;

typedef Matrix::Scalar Scalar;
constexpr bool rowMajor = Matrix::Flags & Eigen::RowMajorBit;

py::class_<Matrix>(m, "Matrix", py::buffer_protocol())
    .def("__init__", [](Matrix &m, py::buffer b) {
        typedef Eigen::Stride<Eigen::Dynamic, Eigen::Dynamic> Strides;

        /* Request a buffer descriptor from Python */
        py::buffer_info info = b.request();

        /* Some sanity checks ... */
        if (info.format != py::format_descriptor<Scalar>::format())
            throw std::runtime_error("Incompatible format: expected a double array!");

        if (info.ndim != 2)
            throw std::runtime_error("Incompatible buffer dimension!");

        auto strides = Strides(
            info.strides[rowMajor ? 0 : 1] / (py::ssize_t)sizeof(Scalar),
            info.strides[rowMajor ? 1 : 0] / (py::ssize_t)sizeof(Scalar));

        auto map = Eigen::Map<Matrix, 0, Strides>(
            static_cast<Scalar *>(info.ptr), info.shape[0], info.shape[1], strides);

        new (&m) Matrix(map);
    });

For reference, the def_buffer() call for this Eigen data type should look as follows:

.def_buffer([](Matrix &m) -> py::buffer_info {
    return py::buffer_info(
        m.data(),                                /* Pointer to buffer */
        sizeof(Scalar),                          /* Size of one scalar */
        py::format_descriptor<Scalar>::format(), /* Python struct-style format descriptor */
        2,                                       /* Number of dimensions */
        { m.rows(), m.cols() },                  /* Buffer dimensions */
        { sizeof(Scalar) * (rowMajor ? m.cols() : 1),
          sizeof(Scalar) * (rowMajor ? 1 : m.rows()) }
                                                 /* Strides (in bytes) for each index */
    );
 })

更簡單的binding Eigen types 的方法（with some limitation ）參見Eigen章節

2. Arrays

在上面的片段中通過交換 py::buffer with py::array，可以限制function只接受numpy array作爲輸入，不接受其他滿足buffer protocol 的python object。

許多情況下，我們希望function只接受特定數據類型的numpy array。這個可以通過py::array_<T>模板來實現，如下例，要求np.array() 的dtype爲double雙精度類型。

void f(py::array_t<double> array);

當使用其他類型進行調用的時候（e.g. int），pybind11 會嘗試驚醒類型轉換，轉換爲expected type。這個特性需要include pybind11/numpy.h,.

numpy array 中的數據不保證packed in a dense manner; 並且entry可能 be separated by arbitrary column and row strides. 有時候function只接受 dense array ( C (row-major) or Fortran (column-major) ordering)非常有用。可以通過 a second template argument with values py::array::c_style or py::array::f_style來實現, e.g.:

void f(py::array_t<double, py::array::c_style | py::array::forcecast> array);

The py::array::forcecast 參數是模板中第二參數的默認值，確保不符合要求的參數能夠轉換爲符合要求的array，這樣避免了對function重載路徑的嘗試。

3. Structured types

爲使得py::array_t work with 結構體，首先需要register the memory layout of the type. 可以通過PYBIND11_NUMPY_DTYPE macro, 在binding模塊中被調用， which expects the type followed by field names:

struct A {
    int x;
    double y;
};

struct B {
    int z;
    A a;
};

// ...
PYBIND11_MODULE(test, m) {
    // ...

    PYBIND11_NUMPY_DTYPE(A, x, y);
    PYBIND11_NUMPY_DTYPE(B, z, a);
    /* now both A and B can be used as template arguments to py::array_t */
}

該結構體應該包含基本的算術類型，e.g. std::complex, previously registered substructures, and arrays of any of the above。支持C++ array 和Std::array. 儘管有靜態的assertion 來prevent 許多不支持的數據類型，開發者仍然有責任儘量使用能夠安全地作爲原始內存進行操作的數據結構，without violating invariants.

4. Vectorizing functions

假定我們想一如下的方式bind一個function，進而可以處理除了正常的參數以外任意 nunmpy array參數(vectors, matrices, general N-D arrays)

double my_func(int x, float y, double z);

including the pybind11/numpy.h:

m.def("vectorized_func", py::vectorize(my_func));

如下，調用vectorized_func()會產生4個對my_func()的調用，每個調用針對array的一個元素。這個向量化操作相對於np.vectorize()的優勢是整個循環完全在C++上運行，並且可以通過編譯器優化爲a tight, optimized loop。返回結果是np.array()類型，
dtype=np.dtype.float64

>>> x = np.array([[1, 3],[5, 7]])
>>> y = np.array([[2, 4],[6, 8]])
>>> z = 3
>>> result = vectorized_func(x, y, z)

標量參數Z被調用了4次。x和z自動轉換爲正確的形式，本身爲np.dtype.int64, 但是需要分別轉換爲 np.dtype.int32 和 np.dtype.float32。

Note

只有通過value和reference進行傳遞的算術類型，complex， POD 類型能夠被向量化，所有其他的參數類型不會被向量化。右值不能被向量化。

左值與右值在C/C++中表示位於賦值運算符兩側的兩個值，賦值號左邊的就叫左值（left-value），右邊的就叫右值(right-value)。 比如：(1) int b = 3;(2) int a = b;第(2)行代碼，a爲左值，b爲右值。不過左值與右值的含義有了更深層次的含義：
L-value中的L指的是Location，表示可尋址。a value (computer science)that has an address.
R-value中的R指的是Read，表示可讀。in computer science, a value that does not have an address in a computer language.

對於計算場景非常複雜以至於不能vectorize的情況，需要手動 創建和access buffer contents 。
如下小節包含了一個完整的例子(the code is somewhat contrived, since it could have been done more simply using vectorize).

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>

namespace py = pybind11;

py::array_t<double> add_arrays(py::array_t<double> input1, py::array_t<double> input2) {
    py::buffer_info buf1 = input1.request(), buf2 = input2.request();

    if (buf1.ndim != 1 || buf2.ndim != 1)
        throw std::runtime_error("Number of dimensions must be one");

    if (buf1.size != buf2.size)
        throw std::runtime_error("Input shapes must match");

    /* No pointer is passed, so NumPy will allocate the buffer */
    auto result = py::array_t<double>(buf1.size);

    py::buffer_info buf3 = result.request();

    double *ptr1 = (double *) buf1.ptr,
           *ptr2 = (double *) buf2.ptr,
           *ptr3 = (double *) buf3.ptr;

    for (size_t idx = 0; idx < buf1.shape[0]; idx++)
        ptr3[idx] = ptr1[idx] + ptr2[idx];

    return result;
}

PYBIND11_MODULE(test, m) {
    m.def("add_arrays", &add_arrays, "Add two NumPy arrays");
}

5. Direct access

考慮到性能因素，當處理非常大的array時，很多時候期望當已經知道索引有效時，直接獲取元素而不必校驗元素的維數和邊界。爲了避免類似的校驗，array class, array_t template class提供了一個unchecked proxy object來提供unchecked access，對應的method爲unchecked and mutable_unchecked methods， N爲對應array的dimensionality。

m.def("sum_3d", [](py::array_t<double> x) {
    auto r = x.unchecked<3>(); // x must have ndim = 3; can be non-writeable
    double sum = 0;
    for (ssize_t i = 0; i < r.shape(0); i++)
        for (ssize_t j = 0; j < r.shape(1); j++)
            for (ssize_t k = 0; k < r.shape(2); k++)
                sum += r(i, j, k);
    return sum;
});

m.def("increment_3d", [](py::array_t<double> x) {
    auto r = x.mutable_unchecked<3>(); // Will throw if ndim != 3 or flags.writeable is false
    for (ssize_t i = 0; i < r.shape(0); i++)
        for (ssize_t j = 0; j < r.shape(1); j++)
            for (ssize_t k = 0; k < r.shape(2); k++)
                r(i, j, k) += 1.0;
}, py::arg().noconvert());

To obtain the proxy from an array object, 必須同時向template指定 data type 和 dimension, 比如：auto r = myarray.mutable_unchecked<float, 2>(). 如果在編譯階段維度信息未知，你可以忽略 dimensions template parameter,(i.e. calling arr_t.unchecked() or arr.unchecked<T>()). 這樣會返回一個proxy object，以同樣的方式工作，但是對於代碼的優化會變少，會有一小部分效率上的損失。

注意，返回的proxy對象直接對array的data進行reference，在創建的時候只讀取array的shape，strides， writable flag 。返回的object還存在期間，必須確保被引用的array沒有reshaped or destroyed，typically by limiting the scope of the returned instance.

The returned proxy object supports some of the same methods as py::array so that it can be used as a drop-in replacement for some existing, index-checked uses of py::array:

r.ndim() returns the number of dimensions
r.data(1, 2, ...) and r.mutable_data(1, 2, ...) returns a pointer to the const T or T data, respectively, at the given indices. The latter is only available to proxies obtained via a.mutable_unchecked().
itemsize() returns the size of an item in bytes, i.e. sizeof(T).
ndim() returns the number of dimensions.
shape(n) returns the size of dimension n
size() returns the total number of elements (i.e. the product of the shapes).
nbytes() returns the number of bytes used by the referenced elements (i.e. itemsize() times size()).

6. Ellipsis

Python 3 provides a convenient … ellipsis notation that is often used to slice multidimensional arrays.
For instance, the following snippet extracts the middle dimensions of a tensor with the first and last index set to zero.

a = # a NumPy array
b = a[0, ..., 0]

The function py::ellipsis() function can be used to perform the same operation on the C++ side:

py::array a = /* A NumPy array */;
py::array b = a[py::make_tuple(0, py::ellipsis(), 0)];

pybind11使用教程筆記__5.2_python C++ interface__numpy

NumPy

1. Buffer protocol

See also

2. Arrays

3. Structured types

4. Vectorizing functions

Note

See also

5. Direct access

See also

6. Ellipsis

np.vectorize np.piecewise 用法小結

conda更換國內源

docker_note_2

git 常用命令_note

pybind11使用教程筆記__4_數據類型轉換(typeconversion)

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結