Inconsistent Fleet Behavior when Removing Configurations from kibana.yml for Managed Agent Policies

[eyalkraft]

Kibana version: latest (8.15)

Elasticsearch version: latest (8.15)

Describe the bug:
We observed inconsistency in Fleet’s behavior when removing configurations from kibana.yml, specifically for managed Agent policies.
First we configured a Fleet Host, Fleet output, and a managed Agent Policy (using the host and output) through the kibana.yml. After starting Kibana we removed these configurations from kibana.yml.
After removing the Fleet Host and Fleet output configurations, the Agent policy remains, but the Fleet Host and Fleet output are deleted. This leads to the Fleet UI failing since the Agent policy references non-existing Fleet Host and output.

Steps to Reproduce:

1. Configure Kibana using kibana.yml with a managed Agent policy, Fleet Host, and Fleet output.
2. Remove the Fleet Host, Output and agent policy from kibana.yml.
3. Stop and start Kibana to re-read kibana.yml
4. Observe that the Fleet UI fails due to the missing references in the managed Agent policy.

kibana.yml example

# Kibana configuration file

# Enabling agentless mode

xpack.cloud.serverless.project_id: 'some_fake_project_id'

xpack.securitySolutionServerless.productTypes:
  - product_line: security
    product_tier: complete
xpack.ml.nlp.enabled: true  # Enable NLP when security and complete are selected

# Fleet and agentless settings
xpack.fleet.enableExperimental:
  - agentless

xpack.fleet.packages:
  - name: "cloud_security_posture"
    version: "latest"

server.versioned.versionResolution: oldest

-# Remove everything below

xpack.fleet.agentPolicies:
  - name: "Agentless"
    id: "agentless-policy"
    is_managed: true
    namespace: "default"
    fleet_server_host_id: "agentless-fleet-internal-host"
    data_output_id: "agentless-es-internal-output"
    monitoring_output_id: "agentless-es-internal-output"
    monitoring_enabled: ["logs", "metrics"]
    supports_agentless: true
    package_policies: []

xpack.fleet.fleetServerHosts:
  - id: "agentless-fleet-internal-host"
    name: "Agentless internal fleet server"
    is_default: false
    is_internal: true
    host_urls: ["https://internal-fleet-server-url/"]

xpack.fleet.outputs:
  - id: "agentless-es-internal-output"
    name: "Internal agentless output"
    type: "elasticsearch"
    is_default: false
    is_default_monitoring: false
    is_internal: true
    hosts: ["https://internal-es-url/"]

Expected Behavior:

1. Deletion of non-default fleet host and output from kibana.yml which result with their deletion in Kibana, should result with removal of references to this host and output from managed agent policies, just like it works for non-managed agent policies. They should be replaced with the default fleet host and output.
2. The Fleet UI should not crash if an agent policy references a non-existing Fleet Host or output. Instead, this specific Agent policy should display an error, while the remaining policies should be accessible and unaffected.

Screenshots (if relevant):

Provide logs and/or server output (if relevant):

Details

[2024-09-19T13:34:13.956+03:00][INFO ][status.plugins.fleet] fleet plugin is now available: Fleet setup failed

GET kbn:/api/fleet/agent_policies

{
    "item": {
        "id": "agentless-policy",
        "version": "WzM4NSwxXQ==",
        "space_ids": [],
        "monitoring_enabled": [
            "logs",
            "metrics"
        ],
        "inactivity_timeout": 1209600,
        "is_preconfigured": true,
        "data_output_id": "agentless-es-internal-output",
        "monitoring_output_id": "agentless-es-internal-output",
        "fleet_server_host_id": "agentless-fleet-internal-host",
        "schema_version": "1.1.1",
        "package_policies": [],
        "agents": 0,
        "namespace": "default",
        "name": "Agentless",
        "supports_agentless": true,
        "status": "active",
        "is_managed": true,
        "revision": 2,
        "updated_at": "2024-09-19T10:04:32.766Z",
        "updated_by": "system",
        "is_protected": false,
        "unprivileged_agents": 0
    }
}

Any additional context:

This bug was discovered as part of an expected migration of agentless on serverless from using preconfigured (kibana.yml) agent policy to using the agentless API.

• Slack thread

cc @kpollich @nchaulet

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)