Skip to content
/ export Public

Mainflux Export service that sends messages from one Mainflux cloud to another via MQTT

Notifications You must be signed in to change notification settings

absmach/export

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

35 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Export

ci build Go Report Card License

Mainflux Export service can send message from one Mainflux cloud to another via MQTT, or it can send messages from edge gateway to Mainflux Cloud. Export service is subscribed to local message bus and connected to MQTT broker in the cloud.
Messages collected on local message bus are redirected to the cloud. When connection is lost, messages from local bus are stored into Redis stream. Upon connection reestablishment Export service consumes messages from Redis stream and sends it to the Mainflux cloud.

Install

Get the code:

go get github.com/mainflux/export
cd $GOPATH/github.com/mainflux/export

Make:

make

Usage

cd build
./mainflux-export

Configuration

By default Export service looks for config file at ../configs/config.toml if no env vars are specified.

[exp]
  cache_pass = ""
  cache_url = "localhost:6379"
  cache_db = "0"
  log_level = "debug"
  nats = "localhost:4222"
  port = "8170"

[mqtt]
  username = "<thing_id>"
  password = "<thing_password>"
  ca = "ca.crt"
  cert = "thing.crt"
  mtls = "false"
  priv_key = "thing.key"
  retain = "false"
  skip_tls_ver = "false"
  url = "tcp://mainflux.com:1883"

[[routes]]
  mqtt_topic = "channel/<channel_id>/messages"
  subtopic = "subtopic"
  nats_topic = "export"
  type = "plain"
  workers = 10

Http port

  • port - HTTP port where status of Export service can be fetched.
curl -X GET http://localhost:8170/version
{"service":"export","version":"0.0.1"}%

Redis connection

To configure Redis connection settings cache_url, cache_pass, cache_db in config.toml are used.

MQTT connection

To establish connection to MQTT broker following settings are needed:

  • username - Mainflux <thing_id>
  • password - Mainflux <thing_key>
  • url - url of MQTT broker

Additionally, you will need MQTT client certificates if you enable mTLS. To obtain certificates ca.crt, thing.crt and key thing.key follow instructions here.

Routes

Routes are being used for specifying which subscriber's topic(subject) goes to which publishing topic. Currently only MQTT is supported for publishing. To match Mainflux requirements mqtt_topic must contain channel/<channel_id>/messages, additional subtopics can be appended.

  • mqtt_topic - channel/<channel_id>/messages/<custom_subtopic>
  • nats_topic - Export service will be subscribed to NATS subject <nats_topic>.>
  • subtopic - messages will be published to MQTT topic <mqtt_topic>/<subtopic>/<nats_subject>, where dots in nats_subject are replaced with '/'
  • workers control number of workers that will be used for message forwarding.
  • type - specifies message transformation, currently only plain is supported, meaning no transformation.

Before running Export service edit configs/config.toml and provide username, password and url

  • username - matches thing_id in Mainflux cloud instance
  • password - matches thing_key
  • channel - MQTT part of the topic where to publish MQTT data (channel/<channel_id>/messages is format of mainflux MQTT topic) and plays a part in authorization.

In order for Export service to listen on Mainflux NATS deployed on the same machine NATS port must be exposed. Run mainflux using make run.

Environment variables

Service will look for config.toml first and if not found it will be configured with env variables and new config file specified with MF_EXPORT_CONFIG_FILE will be saved with values populated from env vars.
The service is configured using the environment variables presented in the following table. Note that any unset variables will be replaced with their default values.

Variable Description Default
MF_NATS_URL Nats url localhost:4222
MF_EXPORT_MQTT_HOST Mqtt url where to export tcp://localhost:1883
MF_EXPORT_MQTT_USERNAME MQTT username, thing id in case of mainflux
MF_EXPORT_MQTT_PASSWORD MQTT password, thing key in case of mainflux
MF_EXPORT_MQTT_CHANNEL MQTT channel where to publish
MF_EXPORT_MQTT_SKIP_TLS Skip tls verification true
MF_EXPORT_MQTT_MTLS Use MTLS for authentication false
MF_EXPORT_MQTT_CA CA for tls ca.crt
MF_EXPORT_MQTT_CLIENT_CERT Client cert for authentication in case when MTLS = true thing.crt
MF_EXPORT_MQTT_CLIENT_PK Client key for authentication in case when MTLS = true thing.key
MF_EXPORT_MQTT_QOS MQTT QOS 0
MF_EXPORT_MQTT_RETAIN MQTT retain false
MF_EXPORT_CONFIG_FILE Configuration file config.toml

for values in environment variables to take effect make sure that there is no MF_EXPORT_CONF file.

If you run with environment variables you can create config file:

MF_EXPORT_PORT=8178 \
MF_EXPORT_LOG_LEVEL=debug \
MF_EXPORT_MQTT_HOST=tcp://localhost:1883 \
MF_EXPORT_MQTT_USERNAME=<thing_id> \
MF_EXPORT_MQTT_PASSWORD=<thing_key> \
MF_EXPORT_MQTT_CHANNEL=<channel_id> \
MF_EXPORT_MQTT_SKIP_TLS=true \
MF_EXPORT_MQTT_MTLS=false \
MF_EXPORT_MQTT_CA=ca.crt \
MF_EXPORT_MQTT_CLIENT_CERT=thing.crt \
MF_EXPORT_MQTT_CLIENT_PK=thing.key \
MF_EXPORT_CONFIG_FILE=export.toml \
../build/mainflux-export&

Service will be subscribed to NATS <nats_topic>.> subject and send messages to channels/<MF_EXPORT_MQTT_CHANNEL>/messages + / + <NatsSubject>. For example if you are running Mainflux on a gateway if you set nats_topic="channel" you can make export service forward messages to other Mainflux instances i.e. into to the Mainflux cloud. When message gets published to local Mainflux instance it will end on NATS as channels.<local_channel_id>.messages.subtopic, Export service will pick it up and forward it to <mqtt_topic> ending on <mqtt_topic>/channels/<local_channel_id>/messages/subtopic. Created export.toml you can edit to add different routes and use in next run.

How to save config via agent

Configuration file for Export service can be send over MQTT using Agent service. save, export,

mosquitto_pub -u <thing_id> -P <thing_key> -t channels/<control_ch_id>/messages/req -h localhost -p 18831  -m  "[{\"bn\":\"1:\", \"n\":\"config\", \"vs\":\"save, export, <config_file_path>, <file_content_base64>\"}]"

vs="config_file_path, file_content_base64" - vs determines where to save file and contains file content in base64 encoding payload:

b,_ := toml.Marshal(export.Config)
payload := base64.StdEncoding.EncodeToString(b)

About

Mainflux Export service that sends messages from one Mainflux cloud to another via MQTT

Resources

Code of conduct

Stars

Watchers

Forks

Packages

No packages published