Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] googleStorageUploadExt step #24048

Merged
merged 6 commits into from
Feb 16, 2021

Conversation

v1v
Copy link
Member

@v1v v1v commented Feb 15, 2021

What does this PR do?

Use the customised step to upload artifacts to Google Storage

Why is it important?

Known bug already in the built-in step provided by the Google Storage plugin

Issues

Requires elastic/apm-pipeline-library#867

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 15, 2021
@v1v v1v self-assigned this Feb 15, 2021
@v1v v1v added automation ci Team:Automation Label for the Observability productivity team labels Feb 15, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 15, 2021
Jenkinsfile Outdated Show resolved Hide resolved
.ci/packaging.groovy Outdated Show resolved Hide resolved
@elasticmachine
Copy link
Collaborator

elasticmachine commented Feb 15, 2021

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #24048 updated

  • Start Time: 2021-02-16T10:15:08.788+0000

  • Duration: 47 min 25 sec

  • Commit: 838d9a9

Test stats 🧪

Test Results
Failed 0
Passed 44071
Skipped 4511
Total 48582

Trends 🧪

Image of Build Times

Image of Tests

Steps errors 4

Expand to view the steps failures

x-pack/filebeat-build - Install Go/Mage/Python/Docker/Terraform 1.15.8
  • Took 0 min 7 sec . View more details on here
  • Description: .ci/scripts/install-tools.sh
x-pack/filebeat-build - Install Go/Mage/Python/Docker/Terraform 1.15.8
  • Took 0 min 0 sec . View more details on here
  • Description: .ci/scripts/install-tools.sh
Google Storage Download
  • Took 3 min 22 sec . View more details on here
  • Description: [2021-02-16T10:32:48.652Z] [Google Cloud Storage Plugin] Found 1 files to download from pattern: gs:
Checks if running on a Unix-like node
  • Took 0 min 0 sec . View more details on here

Log output

Expand to view the last 100 lines of log output

[2021-02-16T11:01:14.703Z] 721e0052aa64: Layer already exists
[2021-02-16T11:01:14.703Z] 4af4d8164c7a: Layer already exists
[2021-02-16T11:01:14.962Z] b2bfb40ecee3: Layer already exists
[2021-02-16T11:01:14.962Z] 00af10937683: Layer already exists
[2021-02-16T11:01:14.962Z] 07cfbfad3789: Layer already exists
[2021-02-16T11:01:14.962Z] c32da0d81269: Layer already exists
[2021-02-16T11:01:14.962Z] bda13f3d9383: Layer already exists
[2021-02-16T11:01:14.962Z] 3aa55ff7bca1: Layer already exists
[2021-02-16T11:01:15.530Z] 838d9a960939fa6c30778b348aaf2499c668337a: digest: sha256:d1f4057c76ebc935ccd794a023e5c6d891f3f24b0c1c145c422f6f10553bac11 size: 3034
[2021-02-16T11:01:15.896Z] Client: Docker Engine - Community
[2021-02-16T11:01:15.896Z]  Version:           20.10.3
[2021-02-16T11:01:15.896Z]  API version:       1.41
[2021-02-16T11:01:15.896Z]  Go version:        go1.13.15
[2021-02-16T11:01:15.896Z]  Git commit:        48d30b5
[2021-02-16T11:01:15.896Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-02-16T11:01:15.896Z]  OS/Arch:           linux/amd64
[2021-02-16T11:01:15.896Z]  Context:           default
[2021-02-16T11:01:15.896Z]  Experimental:      true
[2021-02-16T11:01:15.896Z] 
[2021-02-16T11:01:15.896Z] Server: Docker Engine - Community
[2021-02-16T11:01:15.896Z]  Engine:
[2021-02-16T11:01:15.896Z]   Version:          20.10.3
[2021-02-16T11:01:15.896Z]   API version:      1.41 (minimum version 1.12)
[2021-02-16T11:01:15.896Z]   Go version:       go1.13.15
[2021-02-16T11:01:15.896Z]   Git commit:       46229ca
[2021-02-16T11:01:15.896Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-02-16T11:01:15.896Z]   OS/Arch:          linux/amd64
[2021-02-16T11:01:15.896Z]   Experimental:     false
[2021-02-16T11:01:15.896Z]  containerd:
[2021-02-16T11:01:15.896Z]   Version:          1.4.3
[2021-02-16T11:01:15.896Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-02-16T11:01:15.896Z]  runc:
[2021-02-16T11:01:15.896Z]   Version:          1.0.0-rc92
[2021-02-16T11:01:15.896Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-02-16T11:01:15.896Z]  docker-init:
[2021-02-16T11:01:15.896Z]   Version:          0.19.0
[2021-02-16T11:01:15.896Z]   GitCommit:        de40ad0
[2021-02-16T11:01:15.896Z] Unable to find image 'alpine:3.4' locally
[2021-02-16T11:01:16.463Z] 3.4: Pulling from library/alpine
[2021-02-16T11:01:16.722Z] c1e54eec4b57: Pulling fs layer
[2021-02-16T11:01:16.981Z] c1e54eec4b57: Verifying Checksum
[2021-02-16T11:01:16.981Z] c1e54eec4b57: Download complete
[2021-02-16T11:01:17.241Z] c1e54eec4b57: Pull complete
[2021-02-16T11:01:17.241Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-02-16T11:01:17.241Z] Status: Downloaded newer image for alpine:3.4
[2021-02-16T11:01:19.438Z] + python .ci/scripts/pre_archive_test.py
[2021-02-16T11:01:29.414Z] Copy ./x-pack/metricbeat/build into build/x-pack/metricbeat/build
[2021-02-16T11:01:29.414Z] Copy ./x-pack/metricbeat/build/golang-crossbuild into build/x-pack/metricbeat/build/golang-crossbuild
[2021-02-16T11:01:29.414Z] Copy ./x-pack/elastic-agent/build into build/x-pack/elastic-agent/build
[2021-02-16T11:01:29.414Z] Copy ./x-pack/elastic-agent/build/golang-crossbuild into build/x-pack/elastic-agent/build/golang-crossbuild
[2021-02-16T11:01:29.414Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build
[2021-02-16T11:01:29.414Z] Copy ./x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build into build/x-pack/elastic-agent/build/package/elastic-agent-ubi8/elastic-agent-linux-amd64.docker/docker-build
[2021-02-16T11:01:29.414Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-02-16T11:01:29.414Z] Copy ./x-pack/filebeat/build/golang-crossbuild into build/x-pack/filebeat/build/golang-crossbuild
[2021-02-16T11:01:29.414Z] Copy ./x-pack/heartbeat/build into build/x-pack/heartbeat/build
[2021-02-16T11:01:29.414Z] Copy ./x-pack/heartbeat/build/golang-crossbuild into build/x-pack/heartbeat/build/golang-crossbuild
[2021-02-16T11:01:29.428Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24048/src/github.com/elastic/beats/build
[2021-02-16T11:01:29.730Z] + rm -rf ve
[2021-02-16T11:01:29.731Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-02-16T11:01:29.744Z] Recording test results
[2021-02-16T11:01:30.965Z] None of the test reports contained any result
[2021-02-16T11:01:30.973Z] [Checks API] No suitable checks publisher found.
[2021-02-16T11:01:31.319Z] Client: Docker Engine - Community
[2021-02-16T11:01:31.319Z]  Version:           20.10.3
[2021-02-16T11:01:31.319Z]  API version:       1.41
[2021-02-16T11:01:31.319Z]  Go version:        go1.13.15
[2021-02-16T11:01:31.319Z]  Git commit:        48d30b5
[2021-02-16T11:01:31.319Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-02-16T11:01:31.319Z]  OS/Arch:           linux/amd64
[2021-02-16T11:01:31.319Z]  Context:           default
[2021-02-16T11:01:31.319Z]  Experimental:      true
[2021-02-16T11:01:31.319Z] 
[2021-02-16T11:01:31.319Z] Server: Docker Engine - Community
[2021-02-16T11:01:31.319Z]  Engine:
[2021-02-16T11:01:31.319Z]   Version:          20.10.3
[2021-02-16T11:01:31.319Z]   API version:      1.41 (minimum version 1.12)
[2021-02-16T11:01:31.319Z]   Go version:       go1.13.15
[2021-02-16T11:01:31.319Z]   Git commit:       46229ca
[2021-02-16T11:01:31.319Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-02-16T11:01:31.319Z]   OS/Arch:          linux/amd64
[2021-02-16T11:01:31.319Z]   Experimental:     false
[2021-02-16T11:01:31.319Z]  containerd:
[2021-02-16T11:01:31.319Z]   Version:          1.4.3
[2021-02-16T11:01:31.319Z]   GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
[2021-02-16T11:01:31.319Z]  runc:
[2021-02-16T11:01:31.319Z]   Version:          1.0.0-rc92
[2021-02-16T11:01:31.319Z]   GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
[2021-02-16T11:01:31.319Z]  docker-init:
[2021-02-16T11:01:31.319Z]   Version:          0.19.0
[2021-02-16T11:01:31.319Z]   GitCommit:        de40ad0
[2021-02-16T11:02:31.963Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-02-16T11:02:32.023Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24048/src/github.com/elastic/beats
[2021-02-16T11:02:33.297Z] Running on worker-1244230 in /var/lib/jenkins/workspace/Beats_beats_PR-24048
[2021-02-16T11:02:33.337Z] [INFO] getVaultSecret: Getting secrets
[2021-02-16T11:02:33.434Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-02-16T11:02:35.363Z] + chmod 755 generate-build-data.sh
[2021-02-16T11:02:35.363Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24048/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24048/runs/4 FAILURE 2845191
[2021-02-16T11:02:35.363Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24048/runs/4/steps/?limit=10000 -o steps-info.json
[2021-02-16T11:02:48.319Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24048/runs/4/tests/?status=FAILED -o tests-errors.json
[2021-02-16T11:02:49.773Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24048/runs/4/log/ -o pipeline-log.txt

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 44071
Skipped 4511
Total 48582

@v1v v1v marked this pull request as ready for review February 16, 2021 10:14
@v1v v1v requested a review from a team as a code owner February 16, 2021 10:14
@v1v
Copy link
Member Author

v1v commented Feb 16, 2021

Build failures related to some environmental issues when downloading the docker-compose and also when fetching the source tarball

@v1v v1v merged commit 8064395 into elastic:master Feb 16, 2021
@v1v v1v deleted the feature/test-google-storage-ext branch February 16, 2021 12:25
v1v added a commit to v1v/beats that referenced this pull request Feb 16, 2021
v1v added a commit to v1v/beats that referenced this pull request Feb 16, 2021
…-arm

* upstream/master:
  [Metricbeat][Kubernetes] Extend state_node with more conditions (elastic#23905)
  [CI] googleStorageUploadExt step (elastic#24048)
  Check fields are documented for aws metricsets (elastic#23887)
  Update go-concert to 0.1.0 (elastic#23770)
  [Libbeat][New Processor] XML Decode (elastic#23678)
  Fix: bad substitution of API key (elastic#24036)
  [Filebeat] Add Pensando DFW Module (elastic#21063)
  [Filebeat] Check if processor is supported by ES version (elastic#23763)
  Syslog system tests: be more forgiving (elastic#24021)
v1v added a commit that referenced this pull request Feb 16, 2021
v1v added a commit to v1v/beats that referenced this pull request Feb 17, 2021
# Conflicts:
#	.ci/packaging.groovy
#	Jenkinsfile
v1v added a commit that referenced this pull request Feb 17, 2021
v1v added a commit to v1v/beats that referenced this pull request Feb 17, 2021
…dows-7

* upstream/master: (332 commits)
  Use ECS v1.8.0 (elastic#24086)
  Add support for postgresql csv logs (elastic#23334)
  [Heartbeat] Refactor config system (elastic#23467)
  [CI] install docker-compose with retry (elastic#24069)
  Add nodes to filebeat-kubernetes.yaml ClusterRole - fixes elastic#24051 (elastic#24052)
  updating manifest files for filebeat threatintel module (elastic#24074)
  Add Zeek Signatures (elastic#23772)
  Update Beats to ECS 1.8.0 (elastic#23465)
  Support running Docker logging plugin on ARM64 (elastic#24034)
  Fix ec2 metricset fields.yml and add integration test (elastic#23726)
  Only build targz and zip versions of Beats if PACKAGES is set in agent (elastic#24060)
  [Filebeat] Add field definitions for known Netflow/IPFIX vendor fields (elastic#23773)
  [Elastic Agent] Enroll with Fleet Server (elastic#23865)
  [Filebeat] Convert logstash logEvent.action objects to strings (elastic#23944)
  [Ingest Management] Fix reloading of log level for services (elastic#24055)
  Add Agent standalone k8s manifest (elastic#23679)
  [Metricbeat][Kubernetes] Extend state_node with more conditions (elastic#23905)
  [CI] googleStorageUploadExt step (elastic#24048)
  Check fields are documented for aws metricsets (elastic#23887)
  Update go-concert to 0.1.0 (elastic#23770)
  ...
v1v added a commit to v1v/beats that referenced this pull request Apr 30, 2021
v1v added a commit that referenced this pull request Apr 30, 2021
mergify bot pushed a commit that referenced this pull request Apr 30, 2021
This reverts commit 8064395.

(cherry picked from commit cdcbe36)
mergify bot pushed a commit that referenced this pull request Apr 30, 2021
This reverts commit 8064395.

(cherry picked from commit cdcbe36)
v1v added a commit that referenced this pull request Apr 30, 2021
This reverts commit 8064395.

(cherry picked from commit cdcbe36)

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
v1v added a commit that referenced this pull request Apr 30, 2021
This reverts commit 8064395.

(cherry picked from commit cdcbe36)

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
v1v added a commit that referenced this pull request May 4, 2021
v1v added a commit that referenced this pull request May 4, 2021
v1v added a commit that referenced this pull request May 4, 2021
v1v added a commit that referenced this pull request May 4, 2021
v1v added a commit that referenced this pull request May 4, 2021
v1v added a commit that referenced this pull request May 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
automation ci Team:Automation Label for the Observability productivity team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants