[GH-ISSUE #42 ] Can&#39;t deal with GitLab #16

@jayor96 commented on GitHub (Aug 21, 2023):

if you replace all "github" with "gitlab" in file modules/gitlibs.py and then change module source definition from "git@gitlab...." to "git::ssh://git@gitlab......" in your terraform code, it should work. Also keep in mind that if you use modules from branch via ?ref=... it will not work and you should comment out ?ref part and change default branch in your repo to one which you want to pull.

@jayor96 commented on GitHub (Aug 21, 2023): if you replace all "github" with "gitlab" in file modules/gitlibs.py and then change module source definition from "git@gitlab...." to "git::ssh://git@gitlab......" in your terraform code, it should work. Also keep in mind that if you use modules from branch via ?ref=... it will not work and you should comment out ?ref part and change default branch in your repo to one which you want to pull.

BreizhHardware commented

@patrickchugh commented on GitHub (Aug 21, 2023):

thanks for the feedback will work on this fix soon @FransUrbo

@patrickchugh commented on GitHub (Aug 21, 2023): thanks for the feedback will work on this fix soon @FransUrbo

BreizhHardware commented

@patrickchugh commented on GitHub (Aug 21, 2023):

@FransUrbo Can you do a git pull and try with the latest version?

@patrickchugh commented on GitHub (Aug 21, 2023): @FransUrbo Can you do a `git pull` and try with the latest version?

BreizhHardware commented

@FransUrbo commented on GitHub (Aug 22, 2023):

@jayor96 I have A LOT of modules! git@ is a valid source address, so not keen on changing that. Also, using ?ref= is also a valid source, how else would you be able to have "versioning" of your IaC?

As for latest main, I'm getting this error now:

Preflight check..
  dot command detected: /opt/homebrew/bin/dot
  gvpr command detected: /opt/homebrew/bin/gvpr
  git command detected: /usr/bin/git
  Added Source Location: /home/turbo/src/module
  Parsing //home/turbo/src/module/state.tf
  Parsing /home/turbo/src/module/main.tf
    Found 1 module stanza(s)
Loading Sources..
  Downloading External Module: git@gitlab.com:my-repository//module?ref=main
    Cloning from Terraform registry source: git@gitlab.com:my-repository//module?ref=main
0it [00:00, ?it/s]
ERROR: Unable to call Git to clone repository! Check git and SSH fingerprints and keys are correct and ensure the URL git@gitlab.com:my-repository//module?ref=main is reachable via CLI.

Looking at the code, the modules/gitlibs.py:clone_files() -> git.Repo.clone_from() is failing. Not really a Python programmer, so not sure how that's supposed to work..

I just did a terraform get -update which worked just fine.

But @jayor96 might be on to something here.. When doing the get, I noticed (which I haven't noticed before, strangely! :) ) that TF say:

Downloading git::ssh://git@gitlab.com/my-repository?ref=main for my-module...

So I wonder if TF is "prefixing" a git@... source (which is a valid GIT source) with git::ssh:// and maybe that's why TV isn't working?

@FransUrbo commented on GitHub (Aug 22, 2023): @jayor96 I have A LOT of modules! `git@` is a valid source address, so not keen on changing that. Also, using `?ref=` is also a valid source, how else would you be able to have "versioning" of your IaC? As for latest `main`, I'm getting this error now: ``` Preflight check.. dot command detected: /opt/homebrew/bin/dot gvpr command detected: /opt/homebrew/bin/gvpr git command detected: /usr/bin/git Added Source Location: /home/turbo/src/module Parsing //home/turbo/src/module/state.tf Parsing /home/turbo/src/module/main.tf Found 1 module stanza(s) Loading Sources.. Downloading External Module: git@gitlab.com:my-repository//module?ref=main Cloning from Terraform registry source: git@gitlab.com:my-repository//module?ref=main 0it [00:00, ?it/s] ERROR: Unable to call Git to clone repository! Check git and SSH fingerprints and keys are correct and ensure the URL git@gitlab.com:my-repository//module?ref=main is reachable via CLI. ``` Looking at the code, the `modules/gitlibs.py:clone_files() -> git.Repo.clone_from()` is failing. Not really a Python programmer, so not sure how that's supposed to work.. I just did a `terraform get -update` which worked just fine. But @jayor96 might be on to something here.. When doing the `get`, I noticed (which I haven't noticed before, strangely! :) ) that TF say: ``` Downloading git::ssh://git@gitlab.com/my-repository?ref=main for my-module... ``` So I wonder if TF is "prefixing" a `git@...` source (which is a valid GIT source) with `git::ssh://` and maybe that's why TV isn't working?

BreizhHardware commented

@FransUrbo commented on GitHub (Aug 22, 2023):

I put a debug in modules/gitlibs.py:get_clone_url() and I see that githubURL is:

git@gitlab.com:my-repository//my-module?ref=main

Which might not be a valid GIT url! I'm not entirely sure how TF does it, how it does get the right repo, right directory (within the repo) AND the right branch/tag/commit..

On the other hand, ALL my modules, through all levels, are all in the .terraform/modules. Might be something for the/a future work/release - get the modules from there instead of checking them out..

I'm not sure TV would work anyway in my case. I have several (actually four!!) levels/layers of modules.

It starts with the "TOP" (what we call the "Customer") repo. That basically only sets up the state etc and calls a "wrapper" module. That in turns calls the "application" modules (our IaC consists of several individual apps that we've developed). Those app modules in turn calls all the "service" modules. Those are the ones that actually DO something - create all the resources in AWS (S3, RDS, EC2 etc etc)..

@FransUrbo commented on GitHub (Aug 22, 2023): I put a debug in `modules/gitlibs.py:get_clone_url()` and I see that `githubURL` is: ``` git@gitlab.com:my-repository//my-module?ref=main ``` Which might not be a valid GIT url! I'm not entirely sure how TF does it, how it does get the right repo, right directory (within the repo) AND the right branch/tag/commit.. On the other hand, ALL my modules, through all levels, are all in the `.terraform/modules`. Might be something for the/a future work/release - get the modules from there instead of checking them out.. I'm not sure TV would work anyway in my case. I have several (actually four!!) levels/layers of modules. It starts with the "TOP" (what we call the "Customer") repo. That basically only sets up the state etc and calls a "wrapper" module. That in turns calls the "application" modules (our IaC consists of several individual apps that we've developed). Those app modules in turn calls all the "service" modules. Those are the ones that actually DO something - create all the resources in AWS (S3, RDS, EC2 etc etc)..

BreizhHardware commented

@jayor96 commented on GitHub (Aug 22, 2023):

The problem is that terravision got its own logic to fetch modules which is different than in terraform and it can't work with branches and normal git syntax. I was just experimenting with one of my smaller infra projects to get some output to see if its working but even after I fix all git related issues I get into problems with python itself and how it declare variables internally

@jayor96 commented on GitHub (Aug 22, 2023): The problem is that terravision got its own logic to fetch modules which is different than in terraform and it can't work with branches and normal git syntax. I was just experimenting with one of my smaller infra projects to get some output to see if its working but even after I fix all git related issues I get into problems with python itself and how it declare variables internally

BreizhHardware commented

@patrickchugh commented on GitHub (Aug 22, 2023):

@FransUrbo @jayor96 can you provide me some sample Terraform to try to reproduce your problem? This should not be difficult to fix.

@patrickchugh commented on GitHub (Aug 22, 2023): @FransUrbo @jayor96 can you provide me some sample Terraform to try to reproduce your problem? This should not be difficult to fix.

BreizhHardware commented

@patrickchugh commented on GitHub (Aug 26, 2023):

@FransUrbo Please git pull the latest code and try again. I am able to use gitlab with both ssh and https.

terravision draw --source git@gitlab.com:patrickchugh/test.git --show --debug

Preflight check..
dot command detected: /usr/local/bin/dot
gvpr command detected: /usr/local/bin/gvpr
git command detected: /usr/local/bin/git
Loading Sources..
Downloading External Module: git@gitlab.com:patrickchugh/test.git
Cloning from Terraform registry source: git@gitlab.com:patrickchugh/test.git
Added Source Location: git@gitlab.com:patrickchugh/test.git
Parsing /Users/patrick/.terravision/module_cache/git@gitlab.com_patrickchugh/test.git;main;/main.tf
Found 5 resource stanza(s)
Parsing /Users/patrick/.terravision/module_cache/git@gitlab.com_patrickchugh/test.git;main;/network.tf
Found 2 resource stanza(s)
Parsing /Users/patrick/.terravision/module_cache/git@gitlab.com_patrickchugh/test.git;main;/sg.tf
Found 1 resource stanza(s)
.....

Final Graphviz Input Dictionary
{
"aws_az.az": [
"aws_subnet.mysubnet"
],
"aws_cloudwatch_log_group.cloudwatch": [],
"aws_group.shared_services": [
"aws_cloudwatch_log_group.cloudwatch"
],
"aws_iam_policy.policy": [],
"aws_iam_policy_attachment.test-attach": [
"aws_iam_policy.policy"
],
"aws_iam_role.another_role": [],
"aws_iam_role.iam_for_lambda": [
"aws_iam_policy_attachment.test-attach",
"aws_lambda_function.test_lambda"
],
"aws_lambda_function.test_lambda": [
"aws_cloudwatch_log_group.cloudwatch"
],
"aws_security_group.allow_tls": [
"aws_lambda_function.test_lambda"
],
"aws_subnet.mysubnet": [
"aws_security_group.allow_tls"
],
"aws_vpc.main": [
"aws_az.az"
]
}

Rendering Architecture Image...
Output file: /Users/patrick/git/terravision/architecture.dot.png
Completed!

@patrickchugh commented on GitHub (Aug 26, 2023): @FransUrbo Please git pull the latest code and try again. I am able to use gitlab with both ssh and https. terravision draw --source git@gitlab.com:patrickchugh/test.git --show --debug Preflight check.. dot command detected: /usr/local/bin/dot gvpr command detected: /usr/local/bin/gvpr git command detected: /usr/local/bin/git Loading Sources.. Downloading External Module: git@gitlab.com:patrickchugh/test.git Cloning from Terraform registry source: git@gitlab.com:patrickchugh/test.git Added Source Location: git@gitlab.com:patrickchugh/test.git Parsing /Users/patrick/.terravision/module_cache/git@gitlab.com_patrickchugh/test.git;main;/main.tf Found 5 resource stanza(s) Parsing /Users/patrick/.terravision/module_cache/git@gitlab.com_patrickchugh/test.git;main;/network.tf Found 2 resource stanza(s) Parsing /Users/patrick/.terravision/module_cache/git@gitlab.com_patrickchugh/test.git;main;/sg.tf Found 1 resource stanza(s) ..... Final Graphviz Input Dictionary { "aws_az.az": [ "aws_subnet.mysubnet" ], "aws_cloudwatch_log_group.cloudwatch": [], "aws_group.shared_services": [ "aws_cloudwatch_log_group.cloudwatch" ], "aws_iam_policy.policy": [], "aws_iam_policy_attachment.test-attach": [ "aws_iam_policy.policy" ], "aws_iam_role.another_role": [], "aws_iam_role.iam_for_lambda": [ "aws_iam_policy_attachment.test-attach", "aws_lambda_function.test_lambda" ], "aws_lambda_function.test_lambda": [ "aws_cloudwatch_log_group.cloudwatch" ], "aws_security_group.allow_tls": [ "aws_lambda_function.test_lambda" ], "aws_subnet.mysubnet": [ "aws_security_group.allow_tls" ], "aws_vpc.main": [ "aws_az.az" ] } Rendering Architecture Image... Output file: /Users/patrick/git/terravision/architecture.dot.png Completed!

BreizhHardware commented

@patrickchugh commented on GitHub (Aug 26, 2023):

@jayor96 Can you try again with the latest fixes I made ? git pull the main branch

@patrickchugh commented on GitHub (Aug 26, 2023): @jayor96 Can you try again with the latest fixes I made ? git pull the main branch

BreizhHardware commented

@jayor96 commented on GitHub (Aug 28, 2023):

@patrickchugh Hello, pull of code works now thanks. But I still hit this error
Unhandled error: <class 'TypeError'>, argument of type 'int' is not iterable, <traceback object at 0x106ad5c40>
Should I open issue with it ? I'm afraid that without some debug mode I can't provide variable which is causing issue as my infra is large and complex

@jayor96 commented on GitHub (Aug 28, 2023): @patrickchugh Hello, pull of code works now thanks. But I still hit this error Unhandled error: <class 'TypeError'>, argument of type 'int' is not iterable, <traceback object at 0x106ad5c40> Should I open issue with it ? I'm afraid that without some debug mode I can't provide variable which is causing issue as my infra is large and complex

BreizhHardware commented

@patrickchugh commented on GitHub (Aug 28, 2023):

@jayor96 I will be updating the code to produce more meaningful debug you can try again then

@patrickchugh commented on GitHub (Aug 28, 2023): @jayor96 I will be updating the code to produce more meaningful debug you can try again then

BreizhHardware commented

@jayor96 commented on GitHub (Aug 28, 2023):

@patrickchugh Great ! Thanks for info

@jayor96 commented on GitHub (Aug 28, 2023): @patrickchugh Great ! Thanks for info

BreizhHardware commented

@patrickchugh commented on GitHub (Aug 28, 2023):

@jayor96 I made some changes to the code now to show more verbose debug messages you can paste the last few relevant lines from the output if necessary. Let me know if it works?

@patrickchugh commented on GitHub (Aug 28, 2023): @jayor96 I made some changes to the code now to show more verbose debug messages you can paste the last few relevant lines from the output if necessary. Let me know if it works?

BreizhHardware commented

@jayor96 commented on GitHub (Aug 28, 2023):

@patrickchugh I can't give you full outcome as it contains sensitive data, but here is cleared error which i get

ERROR: LB : aws_lb_target_group.main count = 0 (Error in calling function True ? ERROR!_length([ 'something': 'likethis', 'random':5, ..... ]) : 0))
Unhandled error: <class 'TypeError'>, argument of type 'int' is not iterable, <traceback object at 0x1061a3d40>

@jayor96 commented on GitHub (Aug 28, 2023): @patrickchugh I can't give you full outcome as it contains sensitive data, but here is cleared error which i get ERROR: LB : aws_lb_target_group.main count = 0 (Error in calling function True ? ERROR!_length([ 'something': 'likethis', 'random':5, ..... ]) : 0)) Unhandled error: <class 'TypeError'>, argument of type 'int' is not iterable, <traceback object at 0x1061a3d40>

BreizhHardware commented