feat(dependency): implement outputs fetching from gcs

[TPXP]

Description

This implements outputs fetching from the GCS backend, in a similar fashion to what has been implemented for S3.

I've also added some misc fixes:

• correct typo: steateBody -> stateBody
• drop unused Path parameter on GCS config (it's not supported)

Tested on my architecture, everything seems to work!

TODOs

Read the Gruntwork contribution guidelines.

• Update the docs.
• Run the relevant tests successfully, including pre-commit checks.
• Include release notes. If this PR is backward incompatible, include a migration guide.

Release Notes (draft)

Added support for the GCS backend for dependency output fetching (--terragrunt-fetch-dependency-output-from-state)

Migration Guide

None, this makes this error message go away (and speeds up dependency output state resolution)

ERRO[0001] FetchDependencyOutputFromState is not supported for backend gcs, falling back to normal method  prefix=[something]

[denis256]

Hello,
looks like goimports should be executed:

[INFO] Initializing environment for https://github.com/gruntwork-io/pre-commit.
Terraform fmt............................................................Passed
goimports................................................................Failed
- hook id: goimports
- files were modified by this hook

config/dependency.go
config/dependency.go
config/dependency.go


Exited with code exit status 1

[denis256]

It would be helpful to add integration tests for these changes. This will allow us to track and verify that the functionality works end-to-end in the future.

[TPXP]

Hello, thanks a lot for the prompt feedback 🙌

I forgot to run pre-commit indeed, my bad. I ran it again, it looks better now.

Regarding testing, I'm not sure how to proceed. I couldn't find tests for getTerragruntOutputJsonFromRemoteStateS3. The easiest seems to be to use an integration test with dependencies?

[denis256]

INFO [runner] linters took 52.66096551s with stages: goanalysis_metalinter: 52.642627041s 
config/dependency.go:946:20: Error return value of `reader.Close` is not checked (errcheck)
        defer reader.Close()
                          ^

[sonarcloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[TPXP]

Hello, thanks for getting back to me 😃

I looked into the CI and its tests a little more and ran goimports, golint as well as a few tests. Hopefully all is good now 🤞

[yhakbar]

Hey @TPXP ,

It looks like this PR needs to be rebased off master. Please mark the PR as ready for review when it is.

[TPXP]

Hello, thanks for getting back to me. I've rebased the PR on the latest changes, let me know if there's anything else I can do to help :)

[denis256]

Hello,
looks like PR still doesn't have any tests, it will be impossible to track if this functionality continues to work without automated tests

[TPXP]

Hey @denis256 , thanks for your reply.

I tried to adapt the AWS tests using this feature and added them to this PR. I can't say for sure if they work as I couldn't get them to run on my machine unfortunately (it seems the CI uses a magic run-go-tests program). Let me know how that looks 🙏

[denis256]

Looks like integration_gcp_test.go need formatting

pre-commit installed at .git/hooks/pre-commit
[INFO] Initializing environment for https://github.com/gruntwork-io/pre-commit.
Terraform fmt............................................................Passed
goimports................................................................Failed
- hook id: goimports
- files were modified by this hook

test/integration_gcp_test.go

[yhakbar]

This PR has gone stale, @TPXP . Would you be willing to resolve the merge conflicts, then address the formatting issue @denis256 raised?

[TPXP]

Hello @yhakbar, apologies for the delay. I took some time to address @denis256's feedback and fixed the merge conflicts.

• The code still works on my GCS setup
• It has a test (which I couldn't run as I don't have run-go-tests)
• pre-commit run --all and make run-lint pass

Feel free to edit my branch to speed up review 🙏

[yhakbar]

Tests are failing due to this issue. Once these changes are pulled in, we should be good to test again.

[TPXP]

Thanks for the testing @yhakbar , I've updated the code with your suggestion 👍

[denis256]

=== RUN   TestGcpOutputFromRemoteState
    integration_gcp_test.go:146: Copying fixtures/gcs-output-from-remote-state to /tmp/terragrunt-test3565337247
    integration_gcp_test.go:149: Error writing temp file to /tmp/terragrunt-test3565337247: open /tmp/terragrunt-test3565337247: is a directory
    panic.go:626: Deleting test GCS bucket terragrunt-test-bucket-yzmacx
    panic.go:626: Failed to list objects and versions in GCS bucket terragrunt-test-bucket-yzmacx: storage: bucket doesn't exist
--- FAIL: TestGcpOutputFromRemoteState (0.14s)

[TPXP]

Thanks for running the tests, something was wrong indeed. I fixed this line, hopefully the test should pass now 🤞

[denis256]

well... it still fails

=== RUN   TestGcpMockOutputsFromRemoteState
    integration_gcp_test.go:189: Copying fixtures/gcs-output-from-remote-state to /tmp/terragrunt-test3189224221
    integration_gcp_test.go:197: [terragrunt apply --terragrunt-fetch-dependency-output-from-state --auto-approve --terragrunt-non-interactive --terragrunt-working-dir /tmp/terragrunt-test3189224221/fixtures/gcs-output-from-remote-state/env1/app1 --terragrunt-disable-log-formatting]
time=2024-09-20T17:51:09Z level=error prefix=. msg=Error: Unsupported block type


time=2024-09-20T17:51:09Z level=error prefix=. msg=  on ../../terragrunt.hcl line 4, in remote_state:

time=2024-09-20T17:51:09Z level=error prefix=. msg=   4:   generate {

time=2024-09-20T17:51:09Z level=error prefix=. msg=

time=2024-09-20T17:51:09Z level=error prefix=. msg=Blocks of type "generate" are not expected here. Did you mean to define argument
"generate"? If so, use the equals sign to assign it a value.


    integration_gcp_test.go:197: Failed to run Terragrunt command 'terragrunt apply --terragrunt-fetch-dependency-output-from-state --auto-approve --terragrunt-non-interactive --terragrunt-working-dir /tmp/terragrunt-test3189224221/fixtures/gcs-output-from-remote-state/env1/app1' due to error: hcl.Diagnostics /tmp/terragrunt-test3189224221/fixtures/gcs-output-from-remote-state/terragrunt.hcl:4,3-11: Unsupported block type; Blocks of type "generate" are not expected here. Did you mean to define argument "generate"? If so, use the equals sign to assign it a value.
        /home/circleci/project/config/hclparse/file.go:70 (0x10077ac)
        	(*File).Decode: return errors.WithStackTrace(err)
        /home/circleci/project/config/config.go:937 (0x1609b3f)
        	decodeAsTerragruntConfigFile: if err := file.Decode(&terragruntConfig, evalContext); err != nil {
        /home/circleci/project/config/config.go:864 (0x16092af)
        	ParseConfig: terragruntConfigFile, err := decodeAsTerragruntConfigFile(ctx, file, evalContext)
        /home/circleci/project/config/config.go:773 (0x1608eb8)
        	ParseConfigFile.func1: config, err = ParseConfig(ctx, file, includeFromChild) //nolint:contextcheck
        /home/circleci/project/telemetry/metrics.go:42 (0x10a0793)
        	Time: return fn(ctx)
        /home/circleci/project/telemetry/telemetry.go:80 (0x1608aad)
        	ParseConfigFile.Telemetry.func2: return Time(ctx, name, attrs, fn)
        /home/circleci/project/telemetry/traces.go:38 (0x10a30dd)
        	Trace: return fn(ctx)
        /home/circleci/project/telemetry/telemetry.go:79 (0x1608a25)
        	ParseConfigFile: return Trace(ctx, name, attrs, func(ctx context.Context) error {
        /home/circleci/project/config/config.go:730 (0x16087f3)
        	ParseConfigFile: err := telemetry.Telemetry(ctx, ctx.TerragruntOptions, "parse_config_file", map[string]interface{}{
        /home/circleci/project/config/include.go:96 (0x16276f7)
        	parseIncludedConfig: return ParseConfigFile(ctx, includePath, includedConfig)
        /home/circleci/project/config/include.go:128 (0x162790a)
        	handleInclude: parsedIncludeConfig, err = parseIncludedConfig(ctx, &includeConfig)
        /home/circleci/project/config/config.go:880 (0x1609310)
        	ParseConfig: mergedConfig, err := handleInclude(ctx, config, false)
        /home/circleci/project/config/config.go:773 (0x1608eb8)
        	ParseConfigFile.func1: config, err = ParseConfig(ctx, file, includeFromChild) //nolint:contextcheck
        /home/circleci/project/telemetry/metrics.go:42 (0x10a0793)
        	Time: return fn(ctx)
        /home/circleci/project/telemetry/telemetry.go:80 (0x1608aad)
        	ParseConfigFile.Telemetry.func2: return Time(ctx, name, attrs, fn)
        /home/circleci/project/telemetry/traces.go:38 (0x10a30dd)
        	Trace: return fn(ctx)
        /home/circleci/project/telemetry/telemetry.go:79 (0x1608a25)
        	ParseConfigFile: return Trace(ctx, name, attrs, func(ctx context.Context) error {
        /home/circleci/project/config/config.go:730 (0x16087f3)
        	ParseConfigFile: err := telemetry.Telemetry(ctx, ctx.TerragruntOptions, "parse_config_file", map[string]interface{}{
        /home/circleci/project/config/config.go:720 (0x160871b)
        	ReadTerragruntConfig: return ParseConfigFile(parcingCtx, terragruntOptions.TerragruntConfigPath, nil) //nolint:contextcheck
        /home/circleci/project/cli/commands/terraform/action.go:101 (0x184dcc5)
        	runTerraform: terragruntConfig, err := config.ReadTerragruntConfig(ctx, terragruntOptions, config.DefaultParserOptions(terragruntOptions))
        /home/circleci/project/cli/commands/terraform/action.go:83 (0x184dabc)
        	Run: return runTerraform(ctx, opts, new(Target))
        /home/circleci/project/cli/commands/terraform/command.go:47 (0x1e48a4d)
        	NewApp.NewCommand.Action.func5: return Run(ctx.Context, opts.OptionsFromContext(ctx))
        /home/circleci/project/cli/app.go:239 (0x1e4a647)
        	runAction.func2: return action(cliCtx)
        /home/circleci/go/pkg/mod/golang.org/x/sync@v0.8.0/errgroup/errgroup.go:78 (0x1603036)
        	(*Group).Go.func1: if err := f(); err != nil {
        /usr/local/go/src/runtime/asm_amd64.s:1695 (0x47aae1)
        	goexit: BYTE	$0x90	// NOP
        
        
        Stdout: (see log output above)
        
        Stderr: (see log output above)
    panic.go:626: Deleting test GCS bucket terragrunt-test-bucket-3f2cvv
    panic.go:626: Failed to list objects and versions in GCS bucket terragrunt-test-bucket-3f2cvv: storage: bucket doesn't exist
--- FAIL: TestGcpMockOutputsFromRemoteState (0.22s)