Filebeat X-Pack Module Packaging #8615
Conversation
andrewkroh
force-pushed
the
feature-suricata-pkg
branch
from
5dce0ae
to
ed8ffd7
Compare
There was a problem hiding this comment.
@adriansr Can you please take over here for me since I'm on PTO this week? Thanks 💯x 💯
I have given you permissions in GH to push write to my repo if you want to go that route or you can close this and work from your own fork (any means is good with me).
filebeat/magefile.go
Outdated
| @@ -158,7 +169,17 @@ func customizePackaging() { | |||
| if args.Spec.License == "Elastic License" { | |||
| mods = moduleXPack | |||
| modsD = modulesDXPack | |||
|
|
|||
| // TODO: Replace files with X-Pack versions for: | |||
There was a problem hiding this comment.
This is the one TODO that needs to be resolved.
adriansr
force-pushed
the
feature-suricata-pkg
branch
from
fb68bd1
to
2473073
Compare
This PR adds an intermediate solution for packaging X-Pack modules with Filebeat. In this PR the dashboards, configuration files, and fields are generated in x-pack/filebeat. Packaging is still done entirely from the OSS filebeat directory by making the build run `mage update` in x-pack/filebeat then customizing the packaging configuration to point to different dashboards, config, and fields.yml for the Elastic licensed packages. Long term we will build, test, and package the OSS and Elastic licensed Beats from their respective directories, but this gives us a smaller step in order to be able to release the X-Pack content before the build system is fully transitioned.
X-pack and OSS binaries need a different include/assets.go to account for fields on non-oss modules.
adriansr
force-pushed
the
feature-suricata-pkg
branch
from
2fc6b2d
to
562ed8d
Compare
x-pack/filebeat/include/list.go imported suricata module to have it register its fields, but this is unnecessary in filebeat as it uses one big fields.go with all modules fields.
It was not properly replacing the beat binary with the x-pack version.
x-pack/filebeat/magefile.go
Outdated
| @@ -39,7 +39,7 @@ func CrossBuild() error { | |||
|
|
|||
| // Fields generates a fields.yml and fields.go for each module. | |||
| func Fields() { | |||
| mg.Deps(fieldsYML, mage.GenerateModuleFieldsGo) | |||
| mg.Deps(fieldsYML, mage.GenerateAllInOneFieldsGo) | |||
There was a problem hiding this comment.
The reason I did it this way was because I assumed that all the field content from OSS filebeat was going to be registered through its “all in one” fields.go so then non-OSS filebeat only needed to register its modules’ fields content. This is similar to what metricbeat is doing.
I just wanted to give some context. So if this is working for you then go your route.
There was a problem hiding this comment.
Thanks for the explanation.
What happened is that during my testing, the suricata fields were missing from the x-pack binary. That lead me to make those changes to the fields generation.
In the end this was caused by a mistake in the darwin/tar.gz package description fixed in 620b3778
I'm reverting all the changes to the fields generation.
There was a problem hiding this comment.
That issue you found is #8549. So we can mark that issue as fixed too once this merges.
Problems with x-pack fields missing were caused by a mistake in the packaging templates that caused the oss binary to be used for darwin/tar.gz package. In order to fix this issue a few unnecessary changes to the fields.yml generation were carried out.
There was a problem hiding this comment.
LGTM.
I started a packaging build at https://beats-ci.elastic.co/job/elastic+beats+pull-request+package/5/console.
This PR adds an intermediate solution for packaging X-Pack modules with Filebeat. In this PR the dashboards, configuration files, and fields are generated in x-pack/filebeat. Packaging is still done entirely from the OSS filebeat directory by making the build run
mage updatein x-pack/filebeat then customizing the packaging configuration to point to different dashboards, config, and fields.yml for the Elastic licensed packages.Long term we will build, test, and package the OSS and Elastic licensed Beats from their respective directories, but this gives us a smaller step in order to be able to release the X-Pack content before the build system is fully transitioned.