Skip to content
/ netlog Public

A lightweight, HTTP-centric, log-based (Kafka-like) message queue.

License

Notifications You must be signed in to change notification settings

ninibe/netlog

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

54 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Build Status godoc reference

NetLog

A lightweight, HTTP-centric, log-based (Kafka-style) message queue.

Alpha software

This is still early software and potentially buggy. To peek at the internals start with BigLog.

Roadmap

  • low-level log management
  • HTTP transport
  • scanner based pub/sub
  • custom data retention policy
  • persistent scanners
  • batching
  • compression
  • good test coverage
  • proper documentation
  • streaming based pub/sub
  • async replication
  • kinesis-compatible transport
  • gRPC transport

Non-goals

  • Match Kafka's performance.
  • Distributed system.

Getting started

While posting and fetching single messages is very inefficient, it's the simplest way to get started using nothing but curl commands.

# compile server
go get github.com/ninibe/netlog/cmd/netlog

# run server
bin/netlog

# create new topic
curl -XPOST localhost:7200/demo

# post messages
curl -XPOST localhost:7200/demo/payload --data-binary "message number one"
curl -XPOST localhost:7200/demo/payload --data-binary "message number two"
curl -XPOST localhost:7200/demo/payload --data-binary "message number three"

# check topic info
curl localhost:7200/demo

# create scanner
curl -XPOST "localhost:7200/demo/scanner?from=0"

export SC="...UUID RETURNED..."

# start scanning...
curl -XGET "localhost:7200/demo/scan?id=$SC"
x times ...

# wait 5 seconds for new messages
curl -XGET "localhost:7200/demo/scan?id=$SC&wait=5s"

# wait 5 minutes
curl -XGET "localhost:7200/demo/scan?id=$SC&wait=5m"

# post more messages in another window
curl -XPOST localhost:7200/demo/payload --data-binary "message number four"
curl -XPOST localhost:7200/demo/payload --data-binary "message number five"

# new scanner since 1 minute ago
curl -XPOST "localhost:7200/demo/scanner?from=1m"

One-line-ish pub/sub

# create new topic
curl -XPOST localhost:7200/pubsubdemo

# get scanner ID with jq
export SCANNER=$(curl -s -XPOST "localhost:7200/pubsubdemo/scanner?from=0&persist=true" | jq -r .id)

# subscribe to the topic
while true; do; curl "localhost:7200/pubsubdemo/scan?id=$SCANNER&wait=1h" && echo; done

# IN ANOTHER WINDOW

# publish on the topic
while true; do; read data; curl localhost:7200/pubsubdemo/payload --data-binary $data; done
# write something and hit enter

Contributing

Contributions are more than welcome, check the contributing guidelines. To ask any questions you can write to the netlog-dev mailing list.

About

A lightweight, HTTP-centric, log-based (Kafka-like) message queue.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages