Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wayne worker run into Infinite loop if rabbitmq is down #245

Closed
silenceshell opened this issue Jan 25, 2019 · 0 comments · Fixed by #322
Closed

wayne worker run into Infinite loop if rabbitmq is down #245

silenceshell opened this issue Jan 25, 2019 · 0 comments · Fixed by #322
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@silenceshell
Copy link
Contributor

Describe the bug

If we shutdown the rabbitmq, the wayne worker will print quite a lot of logs such as:

2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:66] delivery not initialized
2019/01/25 14:57:35.718 [C] [workers.go:87] delivery not initialized

To Reproduce
Steps to reproduce the behavior:

  1. deploy wayne on k8s
  2. scale replicas of rabbitmq to zeor
  3. display logs of wayne worker, will get a lot of logs as above.

Expected behavior

As rabbitmq is down, wayne worker should retry to reconnect, and should break this loop:

func processMessage(d *amqp.Delivery, w *BaseMessageWorker) {
	defer func() {
		if r := recover(); r != nil {
			logs.Critical(r)
			d.Reject(false)
		}
	}()

	var m message.Message
	err := json.Unmarshal(d.Body, &m)
	if err != nil {
		ackOrDie(d, false)
		return
	}
	if err = w.Process(&m); err != nil {
		logs.Error(err)
		ackOrDie(d, false) // 认为再次重试也无法处理
	} else {
		ackOrDie(d, false)
	}
}

Screenshots

NA

Desktop (please complete the following information):

NA

Server Version

- Kubernetes version: 1.11.1
- Wayne version: wayne v1.3.1-36-gc5157ca0c 

Additional context

@wilhelmguo wilhelmguo added the kind/bug Categorizes issue or PR as related to a bug. label Jan 31, 2019
@wilhelmguo wilhelmguo assigned wilhelmguo and unassigned chengyumeng Feb 27, 2019
@wilhelmguo wilhelmguo added the status/working-in-progress Indicates that a PR is working in progress. label Feb 28, 2019
@70data 70data removed the status/working-in-progress Indicates that a PR is working in progress. label Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants