Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Cloud Security] [Agentless] Create GHA Cloud Security workflow sanity tests for Agentless ESS deployments. #192182

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
+62 −99
Open

[Cloud Security] [Agentless] Create GHA Cloud Security workflow sanity tests for Agentless ESS deployments. #192182

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4f34dc5
Added Quality Gates test
seanrathier Sep 16, 2024
0c4c24f
Testing
seanrathier Sep 17, 2024
050ffa3
Testing
seanrathier Sep 17, 2024
6186f64
using the agentless-api test for quality-gates and PR tests
seanrathier Sep 17, 2024
a373194
reverting the agentless api test config
seanrathier Sep 17, 2024
a32f33d
omit loggin in as admin if running in cloud
seanrathier Sep 18, 2024
0614796
Added login as admin for serverless
seanrathier Sep 18, 2024
f5b4879
Merge branch 'main' into 10412-agentless-serverless-quality-gate
elasticmachine Sep 18, 2024
f71bebd
fixing type error
seanrathier Sep 20, 2024
3b9875c
Initial tests
seanrathier Sep 5, 2024
cd0d867
less flakey
seanrathier Sep 6, 2024
b575769
removed commented code
seanrathier Sep 6, 2024
66d9a16
fixing merge conflicts
seanrathier Sep 6, 2024
80d7617
removedthe health check
seanrathier Sep 12, 2024
e65815f
Fixed an unwanted merge of agentless_snaity
seanrathier Sep 12, 2024
0dd52d1
Fixed an error
seanrathier Sep 12, 2024
9808b3c
Fixed a typo
seanrathier Sep 12, 2024
0d5b3f2
Added a wait for page to load
seanrathier Sep 16, 2024
05c1974
better testing
seanrathier Sep 20, 2024
23ee4e5
removing commented code
seanrathier Sep 20, 2024
71a47a5
adding tag
seanrathier Sep 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
less flakey
  • Loading branch information
@seanrathier
seanrathier committed Sep 20, 2024
commit cd0d867510e161a7f32f55c8dc8ec8ae28547cb2
152 changes: 93 additions & 59 deletions x-pack/test/cloud_security_posture_functional/cloud_tests/agentless_sanity.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,13 @@
* 2.0.
*/

import { CLOUD_CREDENTIALS_PACKAGE_VERSION } from '@kbn/cloud-security-posture-plugin/common/constants';
import expect from '@kbn/expect';
import type { FtrProviderContext } from '../ftr_provider_context';
// eslint-disable-next-line import/no-default-export
export default function ({ getPageObjects, getService }: FtrProviderContext) {
const queryBar = getService('queryBar');
const testSubjects = getService('testSubjects');
const find = getService('find');
const queryBar = getService('queryBar');
const pageObjects = getPageObjects([
'common',
'cspSecurity',
Expand All @@ -21,81 +20,116 @@ export default function ({ getPageObjects, getService }: FtrProviderContext) {
'cisAddIntegration',
]);

const CIS_AWS_OPTION_TEST_ID = 'cisAwsTestId';

const AWS_SINGLE_ACCOUNT_TEST_ID = 'awsSingleTestId';

describe('Agentless cloud', function () {
let cisIntegration: typeof pageObjects.cisAddIntegration;
let cisIntegrationAws: typeof pageObjects.cisAddIntegration.cisAws;

before(async () => {
cisIntegration = pageObjects.cisAddIntegration;
cisIntegrationAws = pageObjects.cisAddIntegration.cisAws; // Start the usage api mock server on port 8081
describe('Agentless Cloud - Sanity Tests', function () {
describe('agentless agent health', function () {
Copy link
Contributor

@gurevichdmitry gurevichdmitry Sep 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if testing agent health through the UI is the most suitable approach for our case. The Fleet component isn't under our responsibility, and although you're using test objects for some parts, you're relying on CSS selectors and retrieving visible text for the verification steps, which is more critical, there's a risk that if the Fleet team changes something related to the UI, it could break our tests.

If we still want to pinpoint where our tests are failing—whether the issue is on the UI side or due to the agent's health—it might be better to use a Fleet API call to verify if the required agent is healthy.

In the next test suite, we are already querying agentless data. If that suite fails, it should be sufficient to help identify where the problem lies.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've since removed the health checks and decided to simplify this and only check that we receive findings. If we are receiving findings that should indicate a good health and working Agentless agent.

it(`should be healthy`, async () => {
const AGENTLESS_POLICY_NAME_PREFIX = 'Agentless policy for';
const AGENT_NAME_PREFIX = 'agentless-';
const HEALTHY_STATUS = 'Healthy';
const fleet = pageObjects.fleet;

await cisIntegration.navigateToAddIntegrationCspmWithVersionPage(
CLOUD_CREDENTIALS_PACKAGE_VERSION
);

await cisIntegration.clickOptionButton(CIS_AWS_OPTION_TEST_ID);
await cisIntegration.clickOptionButton(AWS_SINGLE_ACCOUNT_TEST_ID);

await cisIntegration.inputIntegrationName(integrationPolicyName);

// Click the first agentless policy
for (const option of options) {
if ((await option.getVisibleText()).includes(AGENTLESS_POLICY_NAME_PREFIX)) {
await option.click();
break;
}
}

// The agent is healthy
const agentHealthSpan = await find.byCssSelector(`span[title='${HEALTHY_STATUS}']`);
expect(agentHealthSpan).not.to.empty();

const agentTableLinks = await find.allByCssSelector('tbody a');

// The agent has enrolled with Fleet
const host = await agentTableLinks[0].getVisibleText();
expect(host).to.be.contain(AGENT_NAME_PREFIX);

// The integration policy is applied
const policy = await agentTableLinks[1].getVisibleText();
expect(policy).to.contain(AGENTLESS_POLICY_NAME_PREFIX);
});
});

after(async () => {
await pageObjects.cspSecurity.logout();
});
describe('agentless agent findings', function () {
it(`should have been collected`, async () => {
const findings = pageObjects.findings;

it(`should create agentless-agent`, async () => {
const integrationPolicyName = `cloud_security_posture-${new Date().toISOString()}`;
await cisIntegration.navigateToAddIntegrationCspmWithVersionPage(
CLOUD_CREDENTIALS_PACKAGE_VERSION
);
await findings.navigateToLatestFindingsPage();
await pageObjects.header.waitUntilLoadingHasFinished();

await cisIntegration.clickOptionButton(CIS_AWS_OPTION_TEST_ID);
await cisIntegration.clickOptionButton(AWS_SINGLE_ACCOUNT_TEST_ID);
await queryBar.setQuery('agent.name : *agentless*');
await queryBar.submitQuery();

await cisIntegration.inputIntegrationName(integrationPolicyName);
const agentlessFindingsRowsCount = await findings
.createDataTableObject('latest_findings_table')
.getRowsCount();

await cisIntegration.selectSetupTechnology('agentless');
await cisIntegration.selectAwsCredentials('direct');
expect(agentlessFindingsRowsCount).to.be.greaterThan(0);
});
});

await pageObjects.header.waitUntilLoadingHasFinished();
// Tech Debt: The following test is disabled until the delete agentless agent task is done
// https://github.com/elastic/kibana/issues/174598

await cisIntegration.clickSaveButton();
await pageObjects.header.waitUntilLoadingHasFinished();
// describe('create and delete Agentless agent', function () {
// let cisIntegration: typeof pageObjects.cisAddIntegration;
// let cisIntegrationAws: typeof pageObjects.cisAddIntegration.cisAws;

expect(await cisIntegrationAws.showPostInstallCloudFormationModal()).to.be(false);
// before(async () => {
// cisIntegration = pageObjects.cisAddIntegration;
// cisIntegrationAws = pageObjects.cisAddIntegration.cisAws; // Start the usage api mock server on port 8081
// });
// it(`should create agentless-agent for AWS single account`, async () => {
// const agentlessIntegrationAwsName = `cloud_security_posture-agentless-aws-sanity-${new Date().toISOString()}`;
// const agentlessAgentAwsPolicyName = `Agentless policy for ${agentlessIntegrationAwsName}`;

await cisIntegration.navigateToIntegrationCspList();
await pageObjects.header.waitUntilLoadingHasFinished();
// await cisIntegration.navigateToAddIntegrationCspmWithVersionPage(
// CLOUD_CREDENTIALS_PACKAGE_VERSION
// );

expect(await cisIntegration.getFirstCspmIntegrationPageIntegration()).to.be(
integrationPolicyName
);
expect(await cisIntegration.getFirstCspmIntegrationPageAgent()).to.be(
`Agentless policy for ${integrationPolicyName}`
);
});
// await cisIntegration.clickOptionButton(
// cisIntegration.testSubjectIds.CIS_AWS_OPTION_TEST_ID
// );
// await cisIntegration.clickOptionButton(
// cisIntegration.testSubjectIds.AWS_SINGLE_ACCOUNT_TEST_ID
// );

it(`should create default agent-based agent`, async () => {
const integrationPolicyName = `cloud_security_posture-${new Date().toISOString()}`;
// await cisIntegration.inputIntegrationName(agentlessIntegrationAwsName);

await cisIntegration.navigateToAddIntegrationCspmWithVersionPage(
CLOUD_CREDENTIALS_PACKAGE_VERSION
);
// await cisIntegration.selectSetupTechnology('agentless');
// await cisIntegration.selectAwsCredentials('direct');

await cisIntegration.clickOptionButton(CIS_AWS_OPTION_TEST_ID);
await cisIntegration.clickOptionButton(AWS_SINGLE_ACCOUNT_TEST_ID);
// await pageObjects.header.waitUntilLoadingHasFinished();

await cisIntegration.inputIntegrationName(integrationPolicyName);
// await cisIntegration.clickSaveButton();
// await pageObjects.header.waitUntilLoadingHasFinished();

await cisIntegration.clickSaveButton();
await pageObjects.header.waitUntilLoadingHasFinished();
// expect(await cisIntegrationAws.showPostInstallCloudFormationModal()).to.be(false);

expect(await cisIntegrationAws.showPostInstallCloudFormationModal()).to.be(true);
// await cisIntegration.navigateToIntegrationCspList();
// await pageObjects.header.waitUntilLoadingHasFinished();

const agentPolicyName = await cisIntegration.getAgentBasedPolicyValue();
// expect(await cisIntegration.getFirstCspmIntegrationPageIntegration()).to.be(
// agentlessIntegrationAwsName
// );
// expect(await cisIntegration.getFirstCspmIntegrationPageAgent()).to.be(
// agentlessAgentAwsPolicyName
// );

await cisIntegration.navigateToIntegrationCspList();
await pageObjects.header.waitUntilLoadingHasFinished();

expect(await cisIntegration.getFirstCspmIntegrationPageIntegration()).to.be(
integrationPolicyName
);
expect(await cisIntegration.getFirstCspmIntegrationPageAgent()).to.be(agentPolicyName);
});
// // Delete the agentless agent when the following task is done
// // https://github.com/elastic/kibana/issues/174598
// });
// });
});
}
6 changes: 3 additions & 3 deletions x-pack/test/cloud_security_posture_functional/cloud_tests/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@ import { FtrProviderContext } from '../ftr_provider_context';
export default function ({ loadTestFile }: FtrProviderContext) {
describe('Cloud Security Posture', function () {
loadTestFile(require.resolve('./agentless_sanity'));
seanrathier marked this conversation as resolved.
Show resolved Hide resolved
// loadTestFile(require.resolve('./dashboard_sanity'));
// loadTestFile(require.resolve('./benchmark_sanity'));
// loadTestFile(require.resolve('./findings_sanity'));
loadTestFile(require.resolve('./dashboard_sanity'));
loadTestFile(require.resolve('./benchmark_sanity'));
loadTestFile(require.resolve('./findings_sanity'));
});
}