awesome-compose/wasmedge-kafka-mysql
Michael Yuan e6b1d2755f
Feat: add Docker+wasm examples (#309)
* Add a Docker+wasm sample application featuring a WasmEdge-based microservice, a MySQL database and an Nginx web server for frontend UI files.

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Add a logo to indicate Docker+wasm compatibility. Add project descriptions to README.

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Add the example for WasmEdge + Kafka / Redpanda + MySQL application to take messages from a queue and save into a database table.

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Add a SVG icon to indicate Docker + Wasm req

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update the docker compose files for the new Docker Desktop release

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Use the correct platform to be compatible with Docker Desktop 4.15

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Update README.md

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-kafka-mysql/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-kafka-mysql/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-kafka-mysql/etl/Dockerfile

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Update wasmedge-mysql-nginx/README.md

Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

* Change the Nginx port to the default non-privileged 8090

Signed-off-by: Michael Yuan <michael@secondstate.io>

* My apologies. Need to correct the syntax for the Nginx port 8090.

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Remove commented lines

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Change wasi/wasm32 to wasi/wasm to conform with the latest spec

Signed-off-by: Michael Yuan <michael@secondstate.io>

* Update README.md

Co-authored-by: Michael Irwin <mikesir87@gmail.com>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>

Signed-off-by: Michael Yuan <michael@secondstate.io>
Signed-off-by: Michael Yuan <michael@michaelyuan.com>
Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Co-authored-by: Michael Irwin <mikesir87@gmail.com>
2023-01-09 16:30:59 -05:00
..
.docker Feat: add Docker+wasm examples (#309) 2023-01-09 16:30:59 -05:00
db Feat: add Docker+wasm examples (#309) 2023-01-09 16:30:59 -05:00
etl Feat: add Docker+wasm examples (#309) 2023-01-09 16:30:59 -05:00
kafka Feat: add Docker+wasm examples (#309) 2023-01-09 16:30:59 -05:00
README.md Feat: add Docker+wasm examples (#309) 2023-01-09 16:30:59 -05:00
compose.yml Feat: add Docker+wasm examples (#309) 2023-01-09 16:30:59 -05:00

README.md

Compose sample application

Compatible with Docker+Wasm

This sample demonstrates a WebAssembly (Wasm) microservice written in Rust. It subscribes to a Kafka queue topic on a Redpanda server, and then transforms and saves each message into a MySQL (MariaDB) database table. The microservice is compiled into Wasm and runs in the WasmEdge runtime, which is a secure and lightweight alternative to natively compiled Rust apps in Linux containers.

Use with Docker Development Environments

You will need a version of Docker Desktop or Docker CLI with Wasm support.

WasmEdge server with Redpanda and MySQL database

Project structure:

.
+-- compose.yml
|-- etl
    |-- Dockerfile
    |-- Cargo.toml
    +-- src
        |-- main.rs
|-- kafka
    |-- order.json
|-- db
    |-- db-password.txt

The compose.yml is as follows.

services:
  redpanda:
    image: docker.redpanda.com/vectorized/redpanda:v22.2.2
    command:
      - redpanda start
      - --smp 1
      - --overprovisioned
      - --node-id 0
      - --kafka-addr PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      - --advertise-kafka-addr PLAINTEXT://redpanda:29092,OUTSIDE://redpanda:9092
      - --pandaproxy-addr 0.0.0.0:8082
      - --advertise-pandaproxy-addr localhost:8082
    ports:
      - 8081:8081
      - 8082:8082
      - 9092:9092
      - 9644:9644
      - 29092:29092
    volumes:
      - ./kafka:/app

  etl:
    image: etl-kafka
    build:
      context: etl
      platforms:
        - wasi/wasm32
    environment:
      DATABASE_URL: mysql://root:whalehello@db:3306/mysql
      KAFKA_URL: kafka://redpanda:9092/order
      RUST_BACKTRACE: full
      RUST_LOG: info
    restart: unless-stopped
    runtime: io.containerd.wasmedge.v1

  db:
    image: mariadb:10.9
    environment:
      MYSQL_ROOT_PASSWORD: whalehello

The compose file defines an application with three services redpanda, etl and db. The redpanda service is a Kafka-compatible messaging server that produces messages in a queue topic. The etl service, in the WasmEdge container that subscribes to the queue topic and receives incoming messages. Each incoming message is parsed and stored in the db MySQL (MariaDB) database server.

Deploy with docker compose

$ docker compose up -d
...
 ⠿ Network wasmedge-kafka-mysql_default       Created                      0.1s
 ⠿ Container wasmedge-kafka-mysql-redpanda-1  Created                      0.3s
 ⠿ Container wasmedge-kafka-mysql-etl-1       Created                      0.3s
 ⠿ Container wasmedge-kafka-mysql-db-1        Created                      0.3s

Expected result

$ docker compose ps
NAME                              COMMAND                  SERVICE             STATUS              PORTS
wasmedge-kafka-mysql-db-1         "docker-entrypoint.s…"   db                  running             3306/tcp
wasmedge-kafka-mysql-etl-1        "kafka.wasm"             etl                 running
wasmedge-kafka-mysql-redpanda-1   "/entrypoint.sh 'red…"   redpanda            running             0.0.0.0:8081-8082->8081-8082/tcp, :::8081-8082->8081-8082/tcp, 0.0.0.0:9092->9092/tcp, :::9092->9092/tcp, 0.0.0.0:9644->9644/tcp, :::9644->9644/tcp, 0.0.0.0:29092->29092/tcp, :::29092->29092/tcp

After the application starts, log into the Redpanda container and send a message to the queue topic order as follows.

$ docker compose exec redpanda /bin/bash
redpanda@1add2615774b:/$ cd /app
redpanda@1add2615774b:/app$ cat order.json | rpk topic produce order
Produced to partition 0 at offset 0 with timestamp 1667922788523.

To see the data in the database container, you can use the following commands.

$ docker compose exec db /bin/bash
root@c97c472db02e:/# mysql -u root -pwhalehello mysql
mysql> select * from orders;
... ...