【介紹】GNES ——"Pythonic"的直觀方式快速構建神經搜索框架

來自騰訊AI Lab肖涵(肖涵博士，騰訊AI Lab GNES項目組負責人)大神的開源項目，非常贊:

GNES Flow。GNES Flow讓你通過“Pythonic”的直觀方式快速構建神經搜索框架，將GNES架構中的各個微服務搭積木一樣組織起來。它支持可視化，多進程/線程/DockerSwarm/K8s後臺。GNES
Flow和GNES的關係就如同Keras和Tensorflow一樣，它爲不熟悉雲原生的用戶提供一個快速在本地搭建、調試GNES的方案。在GNES Readme中(https://github.com/gnes-ai/gnes/)，我寫了如何使用GNES Flow API一分鐘內搭建一個圖片語義搜索/詩詞語義引擎。

github地址：
https://github.com/gnes-ai/gnes/
博客鏈接：
https://hanxiao.github.io/2019/07/29/Generic-Neural-Elastic-Search-From-bert-as-service-and-Go-Way-Beyond/
GNES.ai項目地址：
ttps://github.com/gnes-ai

---

集成了非常多的框架與內容：

---

三分鐘構造圖像搜索引擎

第一步:Define the indexing workflow

from gnes.flow import Flow
flow = (Flow(check_version=False)
        .add_preprocessor(name='prep', yaml_path='yaml/prep.yml')
        .add_encoder(yaml_path='yaml/incep.yml')
        .add_indexer(name='vec_idx', yaml_path='yaml/vec.yml')
        .add_indexer(name='doc_idx', yaml_path='yaml/doc.yml', recv_from='prep')
        .add_router(name='sync', yaml_path='BaseReduceRouter', num_part=2, recv_from=['vec_idx', 'doc_idx']))

Here, we use the inceptionV4 pretrained model as the encoder and the built-in indexers for storing vectors and documents.

第二步:Indexing flower image data
To index our flower data, we need an iterator that generates bytes strings and feed those bytes strings into the defined flow

def read_flowers(sample_rate=1.0):
    with tarfile.open('17flowers.tgz') as fp:
        for m in fp.getmembers():
            if m.name.endswith('.jpg') and random.random() <= sample_rate:
                yield fp.extractfile(m).read()

第三步:Querying similar flowers

num_q = 20
topk = 10
sample_rate = 0.05

# do the query
results = []
with flow.build(backend='process') as fl:
    for q, r in fl.query(bytes_gen=read_flowers(sample_rate)):
        q_img = q.search.query.raw_bytes
        r_imgs = [k.doc.raw_bytes for k in r.search.topk_results]
        r_scores = [k.score.value for k in r.search.topk_results]
        results.append((q_img, r_imgs, r_scores))
        if len(results) > num_q:
            break