Using Bearer authorization for private gitlab terraform package registry

[javajawa]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

ghcr.io/renovatebot/renovate:37.59.8@sha256:5cc2c1681c1d43ebc2fd3e91792c780b3ecf4ad6b3e80a3562fb958e8979e296

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

What is your question?

GitLab's private terraform package registry requires the use of Authorization: Bearer [token] rather than Private-Token: [token] in order to query the API. When self hosting renovate on the same private gitlab instance, I have not been able to configure this correctly.

The private packages documentation seems to suggest there is a known solution to this, but I have been unable to get it working either directly as shown, or with alternate hostType to attempt to match the terraform packages.

"hostRules": [
  {
    "hostType": "maven",
    "matchHost": "https://gitlab.company.com/api/v4/packages/terraform",
    "token": "*****"
  }
]

I feel like I'm missing something obvious, but I'm not sure what, and I can only find tangentially related issues.

---

The underlying issue is very likely the way the token is presented -- using the configured token, I can replicate the following

$ curl -sS -f -H "PRIVATE-TOKEN: *****" -H "Content-Type: application/json" "https://gitlab.company.com/api/v4/packages/terraform/modules/v1/terraform/module-name/local/1.0.4"
curl: (22) The requested URL returned error: 401

$ curl -sS -f -H "Authorization: Bearer *****" -H "Content-Type: application/json" "https://gitlab.company.com/api/v4/packages/terraform/modules/v1/terraform/module-name/local/1.0.4"
{
  "name":"module-name/local",
  "provider":"local",
  "providers":["local"],
  "root":{"dependencies":[]},
  "source":"https://gitlab.company.com/terraform/modules/module-repo",
  "submodules":[],
  "version":"1.0.4",
  "versions":["1.0.4"]
}

Logs (if relevant)

Logs

+ export RENOVATE_ONBOARDING_CONFIG='{"extends": ["config:recommended"], "hostRules": [{"hostType": "terraform", "matchHost": "https://gitlab.company.com/api/v4/packages/terraform", "token": "****"}]}'
+ export RENOVATE_AUTODISCOVER_FILTER=terraform/infra
+ renovate --autodiscover

DEBUG: Using RE2 as regex engine
DEBUG: Parsing configs
DEBUG: Checking for config file in /usr/src/app/config.js
DEBUG: No config file found on disk - skipping
DEBUG: Converting GITHUB_COM_TOKEN into a global host rule
DEBUG: File config
       "config": {}
DEBUG: CLI config
       "config": {"autodiscover": true}
DEBUG: Env config
       "config": {
         "hostRules": [
           {"hostType": "github", "matchHost": "github.com", "token": "***********"}
         ],
         "extends": ["github>whitesource/merge-confidence:beta"],
         "repositoryCache": "true",
         "baseDir": "/tmp/renovate",
         "onboarding": false,
         "onboardingConfig": {
           "extends": ["config:recommended"],
           "baseBranches": ["ben"],
           "hostRules": [
             {
               "hostType": "terraform",
               "matchHost": "https://gitlab.company.com/api/v4/packages/terraform",
               "token": "***********"
             }
           ]
         },
         "requireConfig": "required",
         "optimizeForDisabled": true,
         "platform": "gitlab",
         "endpoint": "https://gitlab.company.com/api/v4",
         "token": "***********",
         "autodiscoverFilter": ["terraform/infra"],
         "gitAuthor": "Renovate Bot <renovate@company.com>",
         "ignorePrAuthor": true
       }
DEBUG: Combined config
       "config": {
         "hostRules": [
           {"hostType": "github", "matchHost": "github.com", "token": "***********"}
         ],
         "extends": ["github>whitesource/merge-confidence:beta"],
         "repositoryCache": "true",
         "baseDir": "/tmp/renovate",
         "onboarding": false,
         "onboardingConfig": {
           "extends": ["config:recommended"],
           "baseBranches": ["ben"],
           "hostRules": [
             {
               "hostType": "terraform",
               "matchHost": "https://gitlab.company.com/api/v4/packages/terraform",
               "token": "***********"
             }
           ]
         },
         "requireConfig": "required",
         "optimizeForDisabled": true,
         "platform": "gitlab",
         "endpoint": "https://gitlab.company.com/api/v4",
         "token": "***********",
         "autodiscoverFilter": ["terraform/infra"],
         "gitAuthor": "Renovate Bot <renovate@company.com>",
         "ignorePrAuthor": true,
         "autodiscover": true
       }
DEBUG: Adding trailing slash to endpoint
DEBUG: Found valid git version: 2.42.0
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: GitLab version is: 16.1.6-ee
DEBUG: Using configured gitAuthor (Renovate Bot <renovate@company.com>)
DEBUG: Adding token authentication for gitlab.company.com (hostType=gitlab) to hostRules
DEBUG: Using configured baseDir: /tmp/renovate
DEBUG: Using cacheDir: /tmp/renovate/cache
DEBUG: Using containerbaseDir: /tmp/renovate/cache/containerbase
DEBUG: Initializing Renovate internal cache into /tmp/renovate/cache/renovate/renovate-cache-v1
DEBUG: Commits limit = null
DEBUG: Setting global hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: Adding token authentication for gitlab.company.com (hostType=gitlab) to hostRules
DEBUG: validatePresets()
DEBUG: Autodiscovering GitLab repositories
DEBUG: Discovered 526 project(s)
DEBUG: Autodiscovered 525 repositories
DEBUG: Applying autodiscoverFilter
       "autodiscoverFilter": ["terraform/infra"]
DEBUG: Autodiscovered 1 repositories after filter
 INFO: Autodiscovered repositories
       "repositories": ["terraform/infra"]
DEBUG: Reinitializing hostRules for repo
DEBUG: Clearing hostRules
DEBUG: Adding token authentication for github.com (hostType=github) to hostRules
DEBUG: Adding token authentication for gitlab.company.com (hostType=gitlab) to hostRules
 INFO: Repository started (repository=terraform/infra)
       "renovateVersion": "37.59.8"
DEBUG: Using localDir: /tmp/renovate/repos/gitlab/terraform/infra (repository=terraform/infra)
DEBUG: PackageFiles.clear() - Package files deleted (repository=terraform/infra)
DEBUG: terraform/infra default branch = master (repository=terraform/infra)
DEBUG: Enabling Git FS (repository=terraform/infra)
DEBUG: Using http URL: https://gitlab.company.com/terraform/infra.git (repository=terraform/infra)
DEBUG: Resetting npmrc (repository=terraform/infra)
DEBUG: Resetting npmrc (repository=terraform/infra)
DEBUG: checkOnboarding() (repository=terraform/infra)
DEBUG: isOnboarded() (repository=terraform/infra)
DEBUG: findPr(renovate/configure, Configure Renovate, !open) (repository=terraform/infra)
DEBUG: findFile(renovate.json) (repository=terraform/infra)
DEBUG: Initializing git repository into /tmp/renovate/repos/gitlab/terraform/infra (repository=terraform/infra)
DEBUG: Performing blobless clone (repository=terraform/infra)
DEBUG: git clone completed (repository=terraform/infra)
       "durationMs": 2410
DEBUG: latest repository commit (repository=terraform/infra)
       "latestCommit": {
         "hash": "[redacted]",
         "date": "2024-02-08T16:48:31+00:00",
         "message": "[redacted]",
         "refs": "HEAD -> master, origin/master, origin/HEAD",
         "body": "[redacted]",
         "author_name": "Benedict Harcourt",
         "author_email": "[redacted]"
       }
DEBUG: Config file exists, fileName: renovate.json (repository=terraform/infra)
DEBUG: ensureIssueClosing() (repository=terraform/infra)
DEBUG: Repo is onboarded (repository=terraform/infra)
DEBUG: Found renovate.json config file (repository=terraform/infra)
DEBUG: Repository config (repository=terraform/infra)
       "fileName": "renovate.json",
       "config": {
         "extends": ["config:recommended"],
         "terraform": {"enabled": true},
         "terraform-version": {"enabled": true},
         "packageRules": [
           {
             "matchPaths": ["stacks/**"],
             "matchManagers": ["terraform", "terraform-version"],
             "groupName": "{{{ replace 'stacks/([^/]+)/.+' '$1' packageFileDir }}}-terraform"
           }
         ]
       }
DEBUG: migrateAndValidate() (repository=terraform/infra)
DEBUG: Config migration necessary (repository=terraform/infra)
       "oldConfig": {
         "extends": ["github>whitesource/merge-confidence:beta", "config:recommended"],
         "terraform": {"enabled": true},
         "terraform-version": {"enabled": true},
         "packageRules": [
           {
             "matchPaths": ["stacks/**"],
             "matchManagers": ["terraform", "terraform-version"],
             "groupName": "{{{ replace 'stacks/([^/]+)/.+' '$1' packageFileDir }}}-terraform"
           }
         ]
       },
       "newConfig": {
         "extends": ["mergeConfidence:all-badges", "config:recommended"],
         "terraform": {"enabled": true},
         "terraform-version": {"enabled": true},
         "packageRules": [
           {
             "matchFileNames": ["stacks/**"],
             "matchManagers": ["terraform", "terraform-version"],
             "groupName": "{{{ replace 'stacks/([^/]+)/.+' '$1' packageFileDir }}}-terraform"
           }
         ]
       }
DEBUG: Post-massage config (repository=terraform/infra)
       "config": {
         "extends": ["mergeConfidence:all-badges", "config:recommended"],
         "terraform": {"enabled": true},
         "terraform-version": {"enabled": true},
         "packageRules": [
           {
             "matchFileNames": ["stacks/**"],
             "matchManagers": ["terraform", "terraform-version"],
             "groupName": "{{{ replace 'stacks/([^/]+)/.+' '$1' packageFileDir }}}-terraform"
           }
         ]
       }
DEBUG: Setting hostRules from config (repository=terraform/infra)
DEBUG: Found repo ignorePaths (repository=terraform/infra)
       "ignorePaths": [
         "**/node_modules/**",
         "**/bower_components/**",
         "**/vendor/**",
         "**/examples/**",
         "**/__tests__/**",
         "**/test/**",
         "**/tests/**",
         "**/__fixtures__/**"
       ]
DEBUG: No vulnerability alerts found (repository=terraform/infra)
DEBUG: findIssue(Dependency Dashboard) (repository=terraform/infra)
DEBUG: No baseBranches (repository=terraform/infra)
DEBUG: extract() (repository=terraform/infra)
DEBUG: Setting current branch to master (repository=terraform/infra)
DEBUG: latest commit (repository=terraform/infra)
       "branchName": "master",
       "latestCommitDate": "2024-02-08T16:48:31+00:00"
DEBUG: Using file match: (^|/)tasks/[^/]+\.ya?ml$ for manager ansible (repository=terraform/infra)
[snip]
DEBUG: Using file match: ^\.woodpecker(?:/[^/]+)?\.ya?ml$ for manager woodpecker (repository=terraform/infra)
DEBUG: Matched 1 file(s) for manager dockerfile: support-tool/Dockerfile (repository=terraform/infra)
DEBUG: Matched 1 file(s) for manager gitlabci: .gitlab-ci.yml (repository=terraform/infra)
DEBUG: Matched 1 file(s) for manager gitlabci-include: .gitlab-ci.yml (repository=terraform/infra)
DEBUG: Matched 1 file(s) for manager pep621: support-tool/pyproject.toml (repository=terraform/infra)
DEBUG: Matched 1 file(s) for manager poetry: support-tool/pyproject.toml (repository=terraform/infra)
DEBUG: Matched 254 file(s) for manager terraform: [snip] (repository=terraform/infra)
DEBUG: Matched 103 file(s) for manager terraform-version: [snip] (repository=terraform/infra)

DEBUG: manager extract durations (ms) (repository=terraform/infra)
       "managers": {
         "dockerfile": 16,
         "gitlabci": 50,
         "gitlabci-include": 52,
         "pep621": 87,
         "poetry": 159,
         "terraform-version": 658,
         "terraform": 1832
       }
DEBUG: Found dockerfile package files (repository=terraform/infra)
DEBUG: Found gitlabci package files (repository=terraform/infra)
DEBUG: Found gitlabci-include package files (repository=terraform/infra)
DEBUG: Found pep621 package files (repository=terraform/infra)
DEBUG: Found terraform package files (repository=terraform/infra)
DEBUG: Found terraform-version package files (repository=terraform/infra)
DEBUG: Found 346 package file(s) (repository=terraform/infra)
 INFO: Dependency extraction complete (repository=terraform/infra, baseBranch=master)
       "stats": {
         "managers": {
           "dockerfile": {"fileCount": 1, "depCount": 1},
           "gitlabci": {"fileCount": 1, "depCount": 1},
           "gitlabci-include": {"fileCount": 1, "depCount": 1},
           "pep621": {"fileCount": 1, "depCount": 4},
           "terraform": {"fileCount": 239, "depCount": 810},
           "terraform-version": {"fileCount": 103, "depCount": 103}
         },
         "total": {"fileCount": 346, "depCount": 920}
       }
DEBUG: GET https://gitlab.company.com/api/v4/packages/terraform/modules/v1/terraform/package-name/local/versions = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=401 retryCount=0, duration=194) (repository=terraform/infra)
DEBUG: Datasource unauthorized (repository=terraform/infra)
       "datasource": "terraform-module",
       "packageName": "gitlab.company.com/terraform/package-name/local",
       "url": "https://gitlab.company.com/api/v4/packages/terraform/modules/v1/terraform/package-name/local/versions"

[viceice]

@rarkins it seems we should switch from that deprecated gitlab header to the official auth header?

that's probably breaking for someone who's using very outdated and unsupported Gitlab version

[rarkins]

Wonder what happens if we send the same token in both headers with every request

[viceice]

I think we should check if there is still a route which don't support bearer and is used by renovate.

[javajawa]

Hi!

How would you like to proceed, and how can I help?
Feels like you'd be leaning towards moving this into an issue that can be worked on; if there's somewhere I can easily see a list of gitlab endpoints in use I can also assist in checking Bearer auth compatability. (Unfortunately I've have finished upgrading old gitlab installs, so can't confirm backward compatibility as easily now)

[rarkins]

Next step is extensive research of the GitLab authentication support (which headers for which endpoints, and including if that changed on any recent releases). Without that from you or someone else, we can't convert to an issue.

[Jannik123]

I'm running into the same issue. @javajawa Have you found a workaround or solution? Thanks in advance!