https://github.com/w1mvy/tfmigrate/tree/gcs-history
I added fake-gcs-server to docker-compose and see how it works. The commit is here.
$ docker-compose run --rm tfmigrate /bin/bash
bash-5.0# terraform init -from-module=../../test-fixtures/storage_gcs/
Initializing the backend...
Successfully configured the backend "gcs"! Terraform will automatically
use this backend unless the backend configuration changes.
╷
│ Error: Failed to get existing workspaces: querying Cloud Storage failed: storage: bucket doesn't exist
│
│
╵
An error occurred during the terraform init stage. I only did a quick check on this behavior below.
Once I set my sights on tfmigrate only, I made a change on the terraform side to look at AWS.
The commit is here
$ docker-compose run --rm tfmigrate /bin/bash
Starting tfmigrate_fake-gcs-server_1 ... done
Creating tfmigrate_tfmigrate_run ... done
bash-5.0# mkdir -p tmp/dir1 && cd tmp/dir1
bash-5.0# terraform init -from-module=../../test-fixtures/storage_gcs/
Copying configuration from "../../test-fixtures/storage_gcs/"...
Initializing the backend...
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/null v3.1.0...
- Installed hashicorp/null v3.1.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.54.0...
- Installed hashicorp/aws v3.54.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
execute terraform apply, edit main.tf, and confirm that there is a difference in terraform plan
bash-5.0# terraform plan
null_resource.foo4: Refreshing state... [id=8919313986794465716]
null_resource.foo3: Refreshing state... [id=8174655520624044808]
null_resource.foo1: Refreshing state... [id=2060914622096163276]
null_resource.foo2: Refreshing state... [id=8696412287786484432]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
bash-5.0#
bash-5.0# vi main.tf
bash-5.0# terraform plan
null_resource.foo4: Refreshing state... [id=8919313986794465716]
null_resource.foo3: Refreshing state... [id=8174655520624044808]
null_resource.foo2: Refreshing state... [id=8696412287786484432]
null_resource.foo1: Refreshing state... [id=2060914622096163276]
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create
- destroy
Terraform will perform the following actions:
# null_resource.bar1 will be created
+ resource "null_resource" "bar1" {
+ id = (known after apply)
}
# null_resource.foo1 will be destroyed
- resource "null_resource" "foo1" {
- id = "2060914622096163276" -> null
}
Plan: 1 to add, 0 to change, 1 to destroy.
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform
apply" now.
Create .tfmigrate.hcl since terraform init -from-module doesn't seem to copy the dot file.
bash-5.0# ls .tfmigrate.hcl
ls: .tfmigrate.hcl: No such file or directory
bash-5.0# cp ../../test-fixtures/
backend_s3/ fake-gcs-server/ storage_gcs/ storage_s3/
bash-5.0# cp ../../test-fixtures/storage_gcs/.tfmigrate.hcl .
I ran tfmigrate apply and confirmed that there are no more differences in the tfmigrate plan and terraform plan.
bash-5.0# tfmigrate plan
2021/08/18 08:52:42 [INFO] [runner] unapplied migration files: [mv_bar1.hcl]
2021/08/18 08:52:42 [INFO] [runner] load migration file: tfmigrate/mv_bar1.hcl
2021/08/18 08:52:42 [INFO] [migrator] start state migrator plan
2021/08/18 08:52:43 [INFO] [migrator@.] terraform version: 1.0.4
2021/08/18 08:52:43 [INFO] [migrator@.] initialize work dir
2021/08/18 08:52:50 [INFO] [migrator@.] switch to remote workspace default
2021/08/18 08:52:54 [INFO] [migrator@.] get the current remote state
2021/08/18 08:52:57 [INFO] [migrator@.] override backend to local
2021/08/18 08:52:57 [INFO] [executor@.] create an override file
2021/08/18 08:52:57 [INFO] [migrator@.] creating local workspace folder in: terraform.tfstate.d/default
2021/08/18 08:52:57 [INFO] [executor@.] switch backend to local
2021/08/18 08:53:07 [INFO] [migrator@.] compute a new state
2021/08/18 08:53:08 [INFO] [migrator@.] check diffs
2021/08/18 08:53:16 [INFO] [executor@.] remove the override file
2021/08/18 08:53:16 [INFO] [executor@.] remove the workspace state folder
2021/08/18 08:53:16 [INFO] [executor@.] switch back to remote
2021/08/18 08:53:26 [INFO] [migrator] state migrator plan success!
bash-5.0# tfmigrate apply
2021/08/18 08:53:41 [INFO] [runner] unapplied migration files: [mv_bar1.hcl]
2021/08/18 08:53:41 [INFO] [runner] load migration file: tfmigrate/mv_bar1.hcl
2021/08/18 08:53:41 [INFO] [migrator] start state migrator plan phase for apply
2021/08/18 08:53:41 [INFO] [migrator@.] terraform version: 1.0.4
2021/08/18 08:53:41 [INFO] [migrator@.] initialize work dir
2021/08/18 08:53:49 [INFO] [migrator@.] switch to remote workspace default
2021/08/18 08:53:52 [INFO] [migrator@.] get the current remote state
2021/08/18 08:53:56 [INFO] [migrator@.] override backend to local
2021/08/18 08:53:56 [INFO] [executor@.] create an override file
2021/08/18 08:53:56 [INFO] [migrator@.] creating local workspace folder in: terraform.tfstate.d/default
2021/08/18 08:53:56 [INFO] [executor@.] switch backend to local
2021/08/18 08:54:06 [INFO] [migrator@.] compute a new state
2021/08/18 08:54:07 [INFO] [migrator@.] check diffs
2021/08/18 08:54:15 [INFO] [executor@.] remove the override file
2021/08/18 08:54:15 [INFO] [executor@.] remove the workspace state folder
2021/08/18 08:54:15 [INFO] [executor@.] switch back to remote
2021/08/18 08:54:26 [INFO] [migrator] start state migrator apply phase
2021/08/18 08:54:26 [INFO] [migrator] push the new state to remote
2021/08/18 08:54:30 [INFO] [migrator] state migrator apply success!
2021/08/18 08:54:30 [INFO] [runner] add a record to history: mv_bar1.hcl
2021/08/18 08:54:30 [INFO] [runner] save history
2021/08/18 08:54:30 [INFO] [runner] history saved
bash-5.0# terraform plan
null_resource.foo4: Refreshing state... [id=8919313986794465716]
null_resource.foo3: Refreshing state... [id=8174655520624044808]
null_resource.foo2: Refreshing state... [id=8696412287786484432]
null_resource.bar1: Refreshing state... [id=2060914622096163276]
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
bash-5.0#
Both terraform and tfmigrate use cloud.google.com/go/storage v1.10.0, but only the terraform side fails.
The following is the part where the bucket is read in terraform init. It is calling Objects and throwing an error at L.31.
tfmigrate calls Object to check the existence of the bucket.
Also, Incloud.google.com/go/storage, there is a recent fix for cases using simulators such as googleapis/google-cloud-go#4616.
I created a sample code that calls Objects and Objects against fake-gcs-server, and tried it while changing the version of cloud.google.com/go/storage.
package main
import (
"context"
"fmt"
"io/ioutil"
"log"
"cloud.google.com/go/storage"
)
func main() {
ctx := context.Background()
client, err := storage.NewClient(ctx)
if err != nil {
log.Fatal(err)
}
bucketName := "sample-bucket"
fileKey := "hoge/fuga"
bkt := client.Bucket(bucketName)
data, err := downloadFile(client, bucketName, fileKey)
if err != nil {
log.Fatal(err)
}
fmt.Printf("contents of %s/%s: %s\n", bucketName, fileKey, data)
// https://github.com/hashicorp/terraform/blob/8ff7e65bd6c159cdc32099f474314eb137cbf796/internal/backend/remote-state/gcs/backend_state.go#L28-L51
objs := bkt.Objects(ctx, &storage.Query{
Delimiter: "/",
Prefix: "hoge",
})
attrs, err := objs.Next()
fmt.Printf("attrs : %v\n", attrs)
if err != nil {
fmt.Errorf("querying Cloud Storage failed: %v", err)
}
fmt.Println("finished")
}
// use fake-gcs-server example https://github.com/fsouza/fake-gcs-server/blob/a75795d305aa5920f6c0b98eb5367c191c6623e8/examples/go/main.go#L69-L76
func downloadFile(client *storage.Client, bucketName, fileKey string) ([]byte, error) {
reader, err := client.Bucket(bucketName).Object(fileKey).NewReader(context.TODO())
if err != nil {
return nil, err
}
defer reader.Close()
return ioutil.ReadAll(reader)
}Objects succeeds, but the list cannot be retrieved by Objects.
Therefore, it is assumed that the error occurs only on the terraform side.
$ go mod tidy && go run example.go
contents of sample-bucket/hoge/fuga:
attrs : <nil>
finished
Neither of them succeeds.
The hostname given by STORAGE_EMULATOR_HOST disappears.
$ go mod tidy && go run example.go
2021/08/19 14:45:53 Get "http:///sample-bucket/hoge/fuga": http: no Host in request URL
exit status 1
Both will succeed. If this version is used, it will be possible to use it to check the behavior of the terraform side.
$ go mod tidy && go run example.go
attrs : &{ false false 0001-01-01 00:00:00 +0000 UTC [] 0 [] 0 map[] 0 0 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC hoge/ 0001-01-01 00:00:00 +0000 UTC}
finished