Accept multiple ingest pipelines in Filebeat

[ycombinator]

Motivated by #8852 (comment).

Starting with 6.5.0, Elasticsearch Ingest Pipelines have gained the ability to:

• run sub-pipelines via the pipeline processor, and
• conditionally run processors via an if field.

These abilities combined present the opportunity for a fileset to ingest the same logical information presented in different formats, e.g. plaintext vs. json versions of the same log files. Imagine an entry point ingest pipeline that detects the format of a log entry and then conditionally delegates further processing of that log entry, depending on the format, to another pipeline.

This PR allows filesets to specify one or more ingest pipelines via the ingest_pipeline property in their manifest.yml. If more than one ingest pipeline is specified, the first one is taken to be the entry point ingest pipeline.

Example with multiple pipelines

ingest_pipeline:
  - pipeline-ze-boss.json 
  - pipeline-plain.json
  - pipeline-json.json

Example with a single pipeline

This is just to show that the existing functionality will continue to work as-is.

ingest_pipeline: pipeline.json

Now, if the root pipeline wants to delegate processing to another pipeline, it must use a pipeline processor to do so. This processor's name field will need to reference the other pipeline by its name. To ensure correct referencing, the name field must be specified as follows:

{
  "pipeline" : {
    "name": "{< IngestPipeline "pipeline-plain" >}"
  }
}

This will ensure that the specified name gets correctly converted to the corresponding name in Elasticsearch, since Filebeat prefixes it's "raw" Ingest pipeline names with filebeat-<version>-<module>-<fileset>- when loading them into Elasticsearch.

[ycombinator]

There's still one TODO item left for this PR related to documentation but I think it's ready for a code review. Thanks folks!

[andrewkroh]

Is “root” synonmous with entrypoint in this context?

[ycombinator]

Yes, and I'm happy to change the terminology to whatever is clearest to most folks.

[dedemorton]

I like entry point better because "root" suggests that there is always a hierarchy. I'm not sure it that's true. Is it possible that devs might want to specify multiple pipelines as a way of breaking their ingest pipeline configs into smaller pipelines that encapsulate specific processing tasks? Also, can other pipelines besides the first one delegate processing? If so, I would avoid using "root".

[ycombinator]

jenkins, test this

[ycombinator]

@dedemorton I've added docs for this feature in df2488a27e171a8f10d8fac6766ae7a352dbf1a8 but I'm not sure about my language and structure/organization. I'd love for you to take a look, if you have some time. Thanks!

[ycombinator]

jenkins, test this

[ycombinator]

jenkins, test this

[ruflin]

What happens if a user uses a module with multiple pipelines against and older version of Elasticsearch?

[ycombinator]

What happens if a user uses a module with multiple pipelines against and older version of Elasticsearch?

I tested this PR with Elasticsearch 6.4.0, where neither the pipelines processor nor the if field on processors exist.

There are two related parts to this PR. One worked with ES 6.4.0, but the other did not. Specifically:

• The part where multiple ingest pipelines are specified under the ingest_pipeline key in the module's manifest.yml file will work with older versions of ES. All pipelines listed will get loaded into ES.

• The part where a pipeline uses the pipeline processor or the if processor field (both features introduced in ES 6.5) will not work with older versions of ES. Concretely, when Filebeat attempts to load such a pipeline into ES, ES will return a 400 with an error message that Filebeat emits to its log. FWIW, this can happen today (in master) as well if a pipeline is using a processor that has not been installed in ES as a plugin, e.g. the CSV processor plugin.

Note that the pipeline loading (within a fileset) is short-circuiting. For example, imagine that a module developer has specified 3 pipelines in the list under ingest_pipeline in the module's manifest.yml. Let's say the 2nd one in the list fails loading because it contains a pipeline processor, or an if somewhere, or for some other reason. The 1st pipeline in the list would've been successfully loaded into ES but the 3rd one will not get loaded. And obviously the 2nd one won't get loaded either because it caused the ES error.

I can see four options on how to proceed:

1. Don't load the pipelines (within a fileset) in a short-circuiting fashion. Instead, even if one pipeline fails to load for some reason, emit the error in the log, but continue to try and load the remaining pipelines in the list.

2. If a pipeline in the list fails to load, try to DELETE previously-loaded pipelines for the fileset (i.e rollback). This way either all pipelines within the fileset get loaded or none of them do.

3. Before attempting to load any pipelines for a fileset, ask Elasticsearch to validate all of them. Again, this way either all pipelines within the fileset get loaded or none of them do.

4. Keep it simple and keep the current short-circuiting approach. Depending on the error, users might need to upgrade ES or perform some other fix, such as loading a plugin, or filing a bug to get the pipeline definition in Filebeat fixed up. Once the fix is made, they can simply re-run Filebeat and any previously-erroring pipelines will get loaded into ES.

My personal preference would be option 1. It is essentially like option 4 but doesn't give up on the remaining pipelines either, so if multiple pipelines have different errors, at least the user can find out about them all at once. Option 2 could lead to an inconsistent state (if any of the DELETEs fail). Option 3 is probably the safest but also unjustifiably expensive.

[kvch]

I would not go with option 1, because it pollutes the ES instance of users with possibly unused pipelines. I prefer the solutions with rollbacks.

[ruflin]

Can you elaborate on why you think 3 is too expensive? I would expect checking for the pipelines does not happen too often so I would not be worried about the extra load.

I just realised this also touches and other problem: What if geo or user_agent are not installed? At the moment I think it does not get loaded as we only have one pipeline but not sure how good the error is. We already have requirements for the pipelines in the manifest, there we could also add requirements for the ES version and use option 3?

[ruflin]

@ycombinator I'm good with merging. But in case #9777 gets in before this one, a rebase would be nice.