Unify dashboard exporter tools #9097

kvch · 2018-11-15T13:51:43Z

The existing export_dashboards.go and the command export dashboards are unified, using the same code and slightly different logic. Configuration of export_dashboards is done using the following command line options: -kibana, -space, -output, -quiet and -index-pattern. The last flag is an odd one out, because it is part of the script, but its value is never used. So far no one complained, so I did not port it. The default value of that flag is the same in case of both methods.

Configuration the Kibana client of export dashboard is read from the config of the Beat. Thus, Kibana-related flags are not part of its CLI.

By default export dashboard does not decode the exported dashboard. If flag -decode passed to the command, the dashboard is decoded.

Equivalent commands

export dashboards from a dashboards.yml

$ ./filebeat export dashboard -yml path/to/dashboards.yml -decode
$ go run dev-tools/cmd/dashboards/export_dashboards.go -yml path/to/dashboards.yml

export a dashboard with an id and print to stdout

$ ./filebeat export dashboard -id {uuid} -decode
$ go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {uuid}

The interface of export dashboard is different than in the previous PR, because I wanted to follow existing methods and documentation.

TODO

add automated tests
add new export dashboard option to dev docs

libbeat/dashboards/export.go

libbeat/kibana/client.go

ruflin · 2018-11-19T08:22:24Z

docs/devguide/newdashboards.asciidoc

@@ -213,6 +213,8 @@ https://github.com/elastic/beats/tree/master/dev-tools/cmd/dashboards[dev-tools]
 for exporting Kibana 5.x dashboards. See the dev-tools
 https://github.com/elastic/beats/tree/master/dev-tools/README.md[readme] for more info.

+Alternatively, if the scripts above are not available, you can use your Beat binary to export Kibana 6.0 dashboards or later.


Will this PR be backported? If not, we could remove everything related to 5.x

Yes, it is planned to as the label suggests.

ph

good work @kvch and thanks for adding integration tests. I have added a few minors things.

I see we test for happy path in the integration tests could we add tests for non-happy-path like when dashboard don't exist?

dev-tools/cmd/dashboards/export_dashboards.go

libbeat/cmd/export/dashboard.go

libbeat/dashboards/export.go

libbeat/kibana/client.go

libbeat/dashboards/export.go

kvch · 2018-11-21T12:42:51Z

I added more E2E tests which cover requesting unknown dashboards and invalid files.

ph

Added a note about my mistake in parsing URL :(

ph · 2018-11-22T14:30:27Z

dev-tools/cmd/dashboards/export_dashboards.go

-		TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, // ignore expired SSL certificates
+	u, err := url.Parse(*kibanaURL)
+	if err != nil {
+		log.Fatalf("Error parsing Kibana URL: %v", err)


I have a mistake, url.Parse or url.ParseRequestURL is well really bad see for yourself:

https://play.golang.org/p/9NAU0-sasfT

I think we have to come up with something better, maybe we already have in other outputs.

In elasticsearch output url.Parse is used. Looking at the codebase, everytime we connect to something url.Parse is used. I think it is good enough.

ruflin · 2018-11-23T07:41:53Z

If I remember correctly the beat export dashboard command did export the dashboards without decoding to make sure they can be directly imported again. The dev export command also did the decoding to make it better for versioning. I skimmed through the code and it seems now in both cases the decoding happens? Didn't test locally.

kvch · 2018-11-23T14:12:37Z

@ruflin That's exactly how it worked. I added the option -decode to export dashboard, so users can decide if they want a decoded version or not. I left the functionality of export_dashboard.go as is.

ruflin · 2018-11-23T18:56:26Z

@kvch This is great. Didn't get to review the code itself yet but functionality is exactly what I hoped for. So if @ph is 👍 I'm good too :-D

ph

👍 LGTM, I've looked at the CI this is a metricbeat/kafka docker issue.. nothing that this PR introduce.

kvch · 2018-11-26T14:55:59Z

Added forgotten changelog entry.

The existing `export_dashboards.go` and the command `export dashboards` are unified, using the same code and slightly different logic. Configuration of `export_dashboards` is done using the following command line options: `-kibana`, `-space`, `-output`, `-quiet` and `-index-pattern`. The last flag is an odd one out, because it is part of the script, but its value is never used. So far no one complained, so I did not port it. The default value of that flag is the same in case of both methods. Configuration the Kibana client of `export dashboard` is read from the config of the Beat. Thus, Kibana-related flags are not part of its CLI. By default `export dashboard` does not decode the exported dashboard. If flag `-decode` passed to the command, the dashboard is decoded. export dashboards from a dashboards.yml ``` $ ./filebeat export dashboard -yml path/to/dashboards.yml -decode $ go run dev-tools/cmd/dashboards/export_dashboards.go -yml path/to/dashboards.yml ``` export a dashboard with an id and print to stdout ``` $ ./filebeat export dashboard -id {uuid} -decode $ go run dev-tools/cmd/dashboards/export_dashboards.go -dashboard {uuid} ``` (cherry picked from commit 1411852)

kvch added in progress Pull request is currently in progress. refactoring libbeat labels Nov 15, 2018

kvch requested review from ph and urso November 15, 2018 13:51

houndci-bot reviewed Nov 15, 2018

View reviewed changes

libbeat/dashboards/export.go Outdated Show resolved Hide resolved

libbeat/dashboards/export.go Show resolved Hide resolved

libbeat/dashboards/export.go Outdated Show resolved Hide resolved

libbeat/dashboards/export.go Outdated Show resolved Hide resolved

houndci-bot reviewed Nov 15, 2018

View reviewed changes

libbeat/dashboards/export.go Outdated Show resolved Hide resolved

kvch added review needs_backport PR is waiting to be backported to other branches. labels Nov 15, 2018

kvch mentioned this pull request Nov 16, 2018

Simplify exporting dashboards for Devs #7730

Merged

kvch removed the in progress Pull request is currently in progress. label Nov 16, 2018

kvch commented Nov 16, 2018

View reviewed changes

libbeat/kibana/client.go Outdated Show resolved Hide resolved

ruflin reviewed Nov 19, 2018

View reviewed changes

kvch mentioned this pull request Nov 19, 2018

Move generator packages of Filebeat from scripts/generator to generator #9147

Merged

kvch force-pushed the refactoring-libbeat-unify-export-dashboards branch from b2f0a30 to 52b1fac Compare November 20, 2018 08:44

ph suggested changes Nov 20, 2018

View reviewed changes

houndci-bot reviewed Nov 21, 2018

View reviewed changes

libbeat/kibana/client.go Outdated Show resolved Hide resolved

libbeat/dashboards/export.go Show resolved Hide resolved

ph suggested changes Nov 22, 2018

View reviewed changes

ph approved these changes Nov 26, 2018

View reviewed changes

kvch added 7 commits November 26, 2018 15:54

unify existing export_dashboard dev tool and export cmd

4b5e4c7

feed the hound && rename Yml to YML

98d013e

comment fix

9e1a9eb

add missing fixes

5a5c711

add more system tests

23ea1c1

add documentation

470e320

add missing test file

41e2375

kvch added 8 commits November 26, 2018 15:54

make fmt

94e05da

rm unnecessary docs

d465d14

fix rebase

0b0b5f2

add missing configuration

a41532e

address review notes

820990d

feed hound && rm weird line

c218c4a

add -decode to export dashboard && add E2E test

d20325e

add changelog

ee19147

kvch force-pushed the refactoring-libbeat-unify-export-dashboards branch from 6b096de to ee19147 Compare November 26, 2018 14:55

kvch merged commit 1411852 into elastic:master Nov 27, 2018

kvch mentioned this pull request Nov 27, 2018

Cherry-pick #9097 to 6.x: Unify dashboard exporter tools #9250

Merged

2 tasks

kvch added v6.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Nov 27, 2018

monicasarbu added the v7.0.0-alpha2 label Dec 19, 2018

jsoriano mentioned this pull request Aug 14, 2019

Custom beat: Cannot export and re-import dashboard. Missing steps? #8200

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unify dashboard exporter tools #9097

Unify dashboard exporter tools #9097

kvch commented Nov 15, 2018 •

edited

Loading

ruflin Nov 19, 2018

kvch Nov 19, 2018

ph left a comment

kvch commented Nov 21, 2018 •

edited

Loading

ph left a comment

ph Nov 22, 2018

kvch Nov 23, 2018

ruflin commented Nov 23, 2018

kvch commented Nov 23, 2018

ruflin commented Nov 23, 2018

ph left a comment

kvch commented Nov 26, 2018

Unify dashboard exporter tools #9097

Unify dashboard exporter tools #9097

Conversation

kvch commented Nov 15, 2018 • edited Loading

Equivalent commands

TODO

ruflin Nov 19, 2018

Choose a reason for hiding this comment

kvch Nov 19, 2018

Choose a reason for hiding this comment

ph left a comment

Choose a reason for hiding this comment

kvch commented Nov 21, 2018 • edited Loading

ph left a comment

Choose a reason for hiding this comment

ph Nov 22, 2018

Choose a reason for hiding this comment

kvch Nov 23, 2018

Choose a reason for hiding this comment

ruflin commented Nov 23, 2018

kvch commented Nov 23, 2018

ruflin commented Nov 23, 2018

ph left a comment

Choose a reason for hiding this comment

kvch commented Nov 26, 2018

kvch commented Nov 15, 2018 •

edited

Loading

kvch commented Nov 21, 2018 •

edited

Loading