mirror of
https://github.com/patrickchugh/terravision.git
synced 2026-05-09 08:25:25 +02:00
[GH-ISSUE #2] Bug: does not work with local modules #3
Labels
No labels
bug
enhancement
enhancement
good first issue
good first issue
good first issue
pipeline
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/terravision#3
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @sterliakov on GitHub (Mar 22, 2023).
Original GitHub issue: https://github.com/patrickchugh/terravision/issues/2
Consider the following definition (simplified version of my setup):
So that
Then execute
terravision:This crashes with the following error:
Well, let's fix this minor issue:
Build and run again to get:
Well, maybe
all_localsare not that important, if they are never populated? Let's try:Build and run again to get even more spurious error:
Hm?
mainmodule does not reference this variable. Okay, let's define it (though please note that it's not a workaround, because same source folder may be used several times with different variables, and so the module vs file distinction is crucial):Still no avail:
Testcase? Okay, let's add it to pyinstaller and modify the source path (without pyinstaller image files are not found). After that we're almost done:
... except the fact that
stat /tmp/result.png.dot.pnggives "No such file or directory".At this point I decided to give up and ask you for some guidance. Could you help resolving this issue? (in fact I'm using the same source for several modules, and the overall architecture contains many modules with several files in each).
@patrickchugh commented on GitHub (Mar 22, 2023):
Thanks this is amazing feedback I think most of this will be fixed when I push up the latest changes. Sent from my Galaxy
-------- Original message --------From: sterliakov @.> Date: 22/3/2023 16:31 (GMT+05:30) To: patrickchugh/terravision @.> Cc: Subscribed @.***> Subject: [patrickchugh/terravision] Bug: does not work with local modules (Issue #2)
Consider the following definition (simplified version of my setup):
mkdir -p /tmp/example/modules/mod
cat > /tmp/example/main.tf << EOF
terraform {
required_version = ">= 0.12.4"
}
provider "aws" {
region = "us-west-1"
}
module "mymod" {
source = "./modules/mod"
prefix = "prefix"
}
EOF
cat > /tmp/example/modules/mod/mod.tf << EOF
resource "aws_cloudwatch_event_rule" "main" {
name = "${var.prefix}-event-rule"
description = "Some event"
}
EOF
cat > /tmp/example/modules/mod/variables.tf << EOF
variable "prefix" {
type = "string"
}
EOF
So that
$ tree /tmp/example
/tmp/example
├── main.tf
└── modules
└── mod
├── mod.tf
└── variables.tf
2 directories, 3 files
Then execute terravision:
cd /tmp
git clone https://github.com/patrickchugh/terravision
cd terravision
python -mvenv venv
. venv/bin/activate
pip install pytest pyinstaller click python-hcl2 gitPython graphviz requests tqdm pyyaml
python -m PyInstaller terravision.py --onefile --add-data resource_images:resource_images --add-data hcl2:hcl2
PATH="/tmp/terravision:/tmp/terravision/dist:$PATH"
/tmp/terravision_1/dist/terravision draw --source /tmp/example/
This crashes with the following error:
Preflight check..
dot command detected: /usr/bin/dot
gvpr command detected: /usr/bin/gvpr
git command detected: /usr/bin/git
Added Source Location: /tmp/example/
Reading Terraforms..
Parsing /tmp/example/main.tf
Found 1 module stanza(s)
Added Source Location: /tmp/example/modules/mod
Traceback (most recent call last):
File "terravision.py", line 163, in
File "click/core.py", line 1130, in call
File "click/core.py", line 1055, in main
File "click/core.py", line 1657, in invoke
File "click/core.py", line 1404, in invoke
File "click/core.py", line 760, in invoke
File "terravision.py", line 112, in draw
File "terravision.py", line 19, in compile_tfdata
File "modules/fileparser.py", line 172, in parse_tf_files
File "modules/fileparser.py", line 95, in handle_module
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/_MEI0vIsuN/modules'
[422119] Failed to execute script 'terravision' due to unhandled exception!
Well, let's fix this minor issue:
cd /tmp/terravision
cat > patch1.patch << EOF
diff --git a/modules/fileparser.py b/modules/fileparser.py
index 25c5cdf..e8513c7 100644
--- a/modules/fileparser.py
+++ b/modules/fileparser.py
@@ -23,6 +23,7 @@ annotations = dict()
temp_dir = tempfile.TemporaryDirectory(dir=tempfile.gettempdir())
abspath = os.path.abspath(file)
dname = os.path.dirname(abspath)
+Path(dname).mkdir(exist_ok=True, parents=True)
MODULE_DIR = str(Path(Path.home(), ".terravision", "module_cache"))
if not os.path.exists(MODULE_DIR):
os.makedirs(MODULE_DIR)
EOF
git apply patch1.patch
Build and run again to get:
Preflight check..
dot command detected: /usr/bin/dot
gvpr command detected: /usr/bin/gvpr
git command detected: /usr/bin/git
Added Source Location: /tmp/example/
Reading Terraforms..
Parsing /tmp/example/main.tf
Found 1 module stanza(s)
Added Source Location: /tmp/example/modules/mod
Parsing /tmp/example/modules/mod/mod.tf
Found 1 resource stanza(s)
Parsing /tmp/example/modules/mod/variables.tf
Found 1 variable stanza(s)
Processing Variables..
Parsing locals...
Traceback (most recent call last):
File "terravision.py", line 163, in
File "click/core.py", line 1130, in call
File "click/core.py", line 1055, in main
File "click/core.py", line 1657, in invoke
File "click/core.py", line 1404, in invoke
File "click/core.py", line 760, in invoke
File "terravision.py", line 112, in draw
File "terravision.py", line 23, in compile_tfdata
File "modules/interpreter.py", line 188, in extract_locals
KeyError: 'all_locals'
[422926] Failed to execute script 'terravision' due to unhandled exception!
Well, maybe all_locals are not that important, if they are never populated? Let's try:
cd /tmp/terravision
cat > patch2.patch << EOF
diff --git a/modules/interpreter.py b/modules/interpreter.py
index 8d8fc36..6292af3 100644
--- a/modules/interpreter.py
+++ b/modules/interpreter.py
@@ -185,7 +185,7 @@ def extract_locals(tfdata):
click.echo("\n Parsing locals...")
module_locals = dict()
# Remove array layer of locals dict structure and copy over to final_locals dict first
if ";" in file:
modname = file.split(";")[1]
else:
EOF
git apply patch2.patch
Build and run again to get even more spurious error:
Preflight check..
dot command detected: /usr/bin/dot
gvpr command detected: /usr/bin/gvpr
git command detected: /usr/bin/git
Added Source Location: /tmp/example/
Reading Terraforms..
Parsing /tmp/example/main.tf
Found 1 module stanza(s)
Added Source Location: /tmp/example/modules/mod
Parsing /tmp/example/modules/mod/mod.tf
Found 1 resource stanza(s)
Parsing /tmp/example/modules/mod/variables.tf
Found 1 variable stanza(s)
Processing Variables..
Parsing locals...
ERROR: No variable value supplied for ${var.prefix} but it is referenced in module main
Consider passing a valid Terraform .tfvars variable file with the --varfile parameter
Hm? main module does not reference this variable. Okay, let's define it (though please note that it's not a workaround, because same source folder may be used several times with different variables, and so the module vs file distinction is crucial):
echo 'prefix = "prefix"' >> test.tfvars
/tmp/terravision/dist/terravision draw --source /tmp/example/ --varfile /tmp/terravision/test.tfvars
Still no avail:
Preflight check..
dot command detected: /usr/bin/dot
gvpr command detected: /usr/bin/gvpr
git command detected: /usr/bin/git
Added Source Location: /tmp/example/
Reading Terraforms..
Parsing /tmp/example/main.tf
Found 1 module stanza(s)
Added Source Location: /tmp/example/modules/mod
Parsing /tmp/example/modules/mod/mod.tf
Found 1 resource stanza(s)
Parsing /tmp/example/modules/mod/variables.tf
Found 1 variable stanza(s)
Processing Variables..
Parsing locals...
Locals list :
Module list :
example/main.tf: mymod.source
Resource list :
mod/mod.tf: aws_cloudwatch_event_rule.main
Data list :
Variable List:
Conditional Resource List:
Computing Relations between 1 out of 1 resources...
Unprocessed Graph Dictionary:
{
"aws_cloudwatch_event_rule.main": []
}
Final Graphviz Input Dictionary
{
"aws_cloudwatch_log_group.cloudwatch": [],
"aws_group.shared_services": [
"aws_cloudwatch_log_group.cloudwatch"
]
}
Traceback (most recent call last):
File "terravision.py", line 163, in
File "click/core.py", line 1130, in call
File "click/core.py", line 1055, in main
File "click/core.py", line 1657, in invoke
File "click/core.py", line 1404, in invoke
File "click/core.py", line 760, in invoke
File "terravision.py", line 112, in draw
File "terravision.py", line 36, in compile_tfdata
FileNotFoundError: [Errno 2] No such file or directory: 'testcase-wordpress.json'
[424913] Failed to execute script 'terravision' due to unhandled exception!
Testcase? Okay, let's add it to pyinstaller and modify the source path (without pyinstaller image files are not found). After that we're almost done:
/tmp/terravision_1/dist/terravision draw --source /tmp/example/ --varfile /tmp/terravision_1/test.tfvars --outfile /tmp/result.png
Preflight check..
dot command detected: /usr/bin/dot
gvpr command detected: /usr/bin/gvpr
git command detected: /usr/bin/git
Added Source Location: /tmp/example/
Reading Terraforms..
Parsing /tmp/example/main.tf
Found 1 module stanza(s)
Added Source Location: /tmp/example/modules/mod
Parsing /tmp/example/modules/mod/mod.tf
Found 1 resource stanza(s)
Parsing /tmp/example/modules/mod/variables.tf
Found 1 variable stanza(s)
Processing Variables..
Parsing locals...
Locals list :
Module list :
example/main.tf: mymod.source
Resource list :
mod/mod.tf: aws_cloudwatch_event_rule.main
Data list :
Variable List:
Conditional Resource List:
Computing Relations between 1 out of 1 resources...
Unprocessed Graph Dictionary:
{
"aws_cloudwatch_event_rule.main": []
}
Final Graphviz Input Dictionary
{
"aws_cloudwatch_log_group.cloudwatch": [],
"aws_group.shared_services": [
"aws_cloudwatch_log_group.cloudwatch"
]
}
Rendering Architecture Image...
gvpr: could not open /tmp/_MEIrB0Tbr/shiftLabel.gvpr for reading
Output file: /tmp/result.png.dot.png
Completed!
... except the fact that stat /tmp/result.png.dot.png gives "No such file or directory".
At this point I decided to give up and ask you for some guidance. Could you help resolving this issue? (in fact I'm using the same source for several modules, and the overall architecture contains many modules with several files in each).
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: @.***>
@sterliakov commented on GitHub (Mar 22, 2023):
Great, thanks! The tool looks very promising (and unique: I haven't found other working solutions yet). Please feel free to ping me if you need any maintenance help - I'd be glad to contribute.
@patrickchugh commented on GitHub (Apr 2, 2023):
@sterliakov Can you give it a try now with the latest code? Please don't compile it using Pyinstaller just run it from the python script as per the updated README for now. The EXE build needs an overhaul to improve startup times.
@sterliakov commented on GitHub (Apr 2, 2023):
Hm, sorry, I have no way to run exe, I'm on Linux and spinning up a vm and
installing everything on windows will be a hell for many hours:(
On Sun, Apr 2, 2023, 12:31 Patrick Chugh @.***> wrote:
@patrickchugh commented on GitHub (Apr 2, 2023):
That's fine just don't run pyinstaller to produce a binary on Linux just run it using the python interpreter Sent from my Galaxy
-------- Original message --------From: sterliakov @.> Date: 2/4/2023 20:02 (GMT+08:00) To: patrickchugh/terravision @.> Cc: Patrick Chugh @.>, Comment @.> Subject: Re: [patrickchugh/terravision] Bug: does not work with local modules (Issue #2)
Hm, sorry, I have no way to run exe, I'm on Linux and spinning up a vm and
installing everything on windows will be a hell for many hours:(
On Sun, Apr 2, 2023, 12:31 Patrick Chugh @.***> wrote:
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>
@sterliakov commented on GitHub (Apr 3, 2023):
Sorry for the late reply, still no success:(
cicd_ml_codebuildis one of my module names. The problem is thatmodules.interpreter.extract_localskeeps thinking thatmodnameis"main", even for modules (because no copy to a location with;happened? Not sure here)@patrickchugh commented on GitHub (Apr 5, 2023):
Which version distro and version of Linux are you using? It works fine on my Mac
./terravision.py draw --source /tmp/example/
Preflight check..
dot command detected: /opt/local/bin/dot
gvpr command detected: /opt/local/bin/gvpr
git command detected: /usr/bin/git
Added Source Location: /tmp/example/
Reading Terraforms..
Parsing /tmp/example/main.tf
Found 1 module stanza(s)
Added Source Location: /private/tmp/example/modules/mod
Parsing /private/tmp/example/modules/mod/mod.tf
Found 1 resource stanza(s)
Parsing /private/tmp/example/modules/mod/variables.tf
Found 1 variable stanza(s)
Processing Variables..
Parsing locals...
Locals list :
Module list :
example/main.tf: mymod.source
Resource list :
mod/mod.tf: aws_cloudwatch_event_rule.main
Data list :
Variable List:
Conditional Resource List:
Computing Relations between 1 out of 1 resources...
Unprocessed Graph Dictionary:
{
"aws_cloudwatch_event_rule.main": []
}
Final Graphviz Input Dictionary
{
"aws_cloudwatch_log_group.cloudwatch": [],
"aws_group.shared_services": [
"aws_cloudwatch_log_group.cloudwatch"
]
}
Rendering Architecture Image...
Output file: /Users/parchugh/git/terravision/architecture.dot.png
Completed!
@patrickchugh commented on GitHub (Apr 5, 2023):
Could you share your terraform for me to check it out?
@patrickchugh commented on GitHub (May 14, 2023):
Closing as no response from OP