MLSQL is a SQL-Based language and MLSQL Stack is a platform including many systems which provide you the ability to do OLAP, ETL, Stream, Machine Learning jobs. In MLSQL Console, you can access almost any data which MLSQL give you the column-level visiting control in most populate storages and process them in distribute. Also, you can integrate any API to make your computing more efficiency and intelligence. With the include and micro support, you can reuse your work in a more high level than normal language e.g. Python, Scala, Java.
We will try to build three systems which help people easier to resolve their problems when they work.
DataSource Repository.
Since we hope people can easy to connect any data storage(e.g. MySQL, ElasticSearch, HBase....) but we can not package all connector jars into MLSQL Engine prebuild distribution. To resolve this question, we provide a remote DataSource repository service, people can use the following command line to add Datasource connector at runtime.
!connector add mongodb;
API Market
You can use MLSQL to invoke any API with function:
select crawler_request_image(image_url) as imageBin,mdImage
from imageUrls
as imageBins;
function crawler_request_image will send an HTTP request to get bytes of the image from image_url.
Or you can load data from API then process them as a table:
load json.`http://api..../data` as newTable;
So we can give people an API market , and at the same time, a function / datasource to help people use the API will also be provided. For example, suppose we have API provide spark news, you can use it like this:
load market.`/category/news/spark` as sparkNews;
MLSQL Script Store
Suppose you have a hello world script like this:
set name="" where type="defaultParam";
select "hello ${name}" as echo as output;
You want to provide this script as a command, e.g echo, you can do like this:
set echo='''
select "hello {}" as echo as output
''';
Suppose you put the script in store/forfun/echo.mslql, then others can use it with include statement:
include store.`forfun/echo.mlsql`;
!echo "jack";
We hope when more and more people use MLSQL Stack, and they can share their genius script to benefit more people.