AMQP output with timeout state affecting the data sent from telegraf to influxdb

[ricardojfernandes]

Bug report

When using telegraf and AMQP output, when the AMQP request is timed out, none data from telegraf nor amqp are sent to influxdb.

Relevant telegraf.conf:

[[outputs.amqp]] configuration

System info:

package - telegraf 1.1.1-1
SO - xenserver 7.0 and debian wheezy and jessie

Steps to reproduce:

1. Create a firewall rule to drop the packet on AMQP server.

Expected behavior:

Send to influxdb server at least the telegraf data.

Actual behavior:

Telegraf send nothing to influxdb

Additional info:

• Logs from telegraf.log:
  E! Failed to connect to output amqp, retrying in 15s, error was 'dial tcp amqpserver:port: i/o timeout'
  E! dial tcp amqpserver:port: i/o timeout
  E! Failed to connect to output amqp, retrying in 15s, error was 'dial tcp amqpserver:port: i/o timeout'
  E! dial tcp amqpserver:port: i/o timeout
  E! Failed to connect to output amqp, retrying in 15s, error was 'dial tcp amqpserver:port: i/o timeout'
  E! dial tcp amqpserver:port: i/o timeout

[danielnelson]

So you have an influxdb output in addition to the amqp output?

[ricardojfernandes]

@danielnelson Yes! I send the same data to influxdb and an AMQP server.

[danielnelson]

Are you able to test with the latest Telegraf release?

[ricardojfernandes]

@danielnelson I have tried with telegraf 1.2.1-1 and the same problem occurs.

But now I saw a new situation...the process is terminated after two errors on syslog and telegraf daemon dies.

======

I don't see the usual startup log. I only see these two fails and the process dies.

2017-03-30T22:29:14Z E! Failed to connect to output amqp, retrying in 15s, error was 'dial tcp amqpserver:port: i/o timeout'
2017-03-30T22:29:59Z E! dial tcp amqpserver:port: i/o timeout
2017-03-30T22:30:35Z E! Failed to connect to output amqp, retrying in 15s, error was 'dial tcp amqpserver:port: i/o timeout'
2017-03-30T22:31:20Z E! dial tcp amqpserver:port: i/o timeout

---

dpkg -l telegraf
||/ Name Version Description
+++-============================================-============================================-========================================================================================================
ii telegraf 1.2.1-1 Plugin-driven server agent for reporting metrics into InfluxDB.

[danielnelson]

Are you blocking AMQP before you start Telegraf? If so, what happens if you block it only after Telegraf starts?

[danielnelson]

I wasn't able to reproduce the exact symptoms but both the amqp_consumer and amqp output are not using a timeout, which could block the output flusher.

[ricardojfernandes]

You can try to block using iptables on AMQP server, blocking all incoming traffic from the server with output_amqp. But my problem isn't the amqp output doesn't posting data...Actually I need the data of telegraf with or without amqp output response OK.