Skip to content

Environment/depfile: fix bug with Git hash versions (attempt #2)#37560

Merged
haampie merged 9 commits intospack:developfrom
scheibelp:bugfix/depfile-hash-eq-ver2
Jul 17, 2023
Merged

Environment/depfile: fix bug with Git hash versions (attempt #2)#37560
haampie merged 9 commits intospack:developfrom
scheibelp:bugfix/depfile-hash-eq-ver2

Conversation

@scheibelp
Copy link
Copy Markdown
Member

@scheibelp scheibelp commented May 8, 2023

Replaces #36642

If a spec contains a hash=number version, this was resulting in malformed Makefiles generated by spack env depfile (the target cannot include an un-escaped '=' character). This

  • Introduces regular expressions to remove special characters from the spec component of generated makefile targets
  • Replaces all references to Specs in the MakefileModel (and associated objects) with references to a limited interface to avoid similar issues in the future

TODO

  • Do one-time regular expression compilation

@spackbot-app spackbot-app bot added commands core PR affects Spack core functionality environments tests General test capability(ies) labels May 8, 2023
@scheibelp scheibelp changed the title Environment/depfile: fix bug with Git hash versions (attempt #2) [WIP] Environment/depfile: fix bug with Git hash versions (attempt #2) May 8, 2023
@scheibelp scheibelp changed the title [WIP] Environment/depfile: fix bug with Git hash versions (attempt #2) Environment/depfile: fix bug with Git hash versions (attempt #2) May 8, 2023
@scheibelp scheibelp requested a review from haampie May 9, 2023 00:59
@haampie
Copy link
Copy Markdown
Member

haampie commented May 9, 2023

I noticed that this escaping problem is not limited to depfile :( apply this diff

diff --git a/var/spack/repos/builtin/packages/zlib/package.py b/var/spack/repos/builtin/packages/zlib/package.py
index 16341ecbaf..0e3e580aed 100644
--- a/var/spack/repos/builtin/packages/zlib/package.py
+++ b/var/spack/repos/builtin/packages/zlib/package.py
@@ -22,6 +22,7 @@ class Zlib(MakefilePackage, Package):
     homepage = "https://zlib.net"
     # URL must remain http:// so Spack can bootstrap curl
     url = "http://zlib.net/fossils/zlib-1.2.11.tar.gz"
+    git = "https://github.com/madler/zlib.git"
 
     version("1.2.13", sha256="b3a24de97a8fdbc835b9833169501030b8977031bcb54b3b3ac13740f846ab30")
     version(

and install zlib@04f42ceca40f73e2978b50e93806c2a18c1281fc (which resolves to =1.2.13).

It generates a broken Makefile due to = in the install prefix; install fails.

So... I wonder if a safe Spec.format function would be good to have in general? We'll need it in projections.

@scheibelp
Copy link
Copy Markdown
Member Author

I wonder if a safe Spec.format function would be good to have in general?

The concept of "safe" can change depending on the context, so I'm not sure if there's a one-size-fits-all definition of safe (perhaps there is some "very safe" format which would satisfy all needs). If you are proposing defining a Spec.safe_format function I'd prefer to defer that until a few use cases have been aggregated.

I also think most of the work is in knowing where to call it vs. whether to have it. In the case you brought up, it sounds like Spec.prefix (or actually DirectoryLayout.relative_path_for_spec) should be redefined to safely format the Spec (this gets back to context-specific notion of safety though, since = is safe in paths and unsafe in makefiles, whereas you can escape \= in makefiles but not necessarily elsewhere) i.e. our prefixes would never contain =.

unsafe_result = self.spec.format(format_str)
if not MakefileSpec._pattern:
MakefileSpec._pattern = re.compile(r"[^A-Za-z0-9_.-]")
return MakefileSpec._pattern.sub("_", unsafe_result)
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@haampie if I understand from our conversation this morning, you are saying this re.sub call could be moved into Spec.format itself (i.e. it would take another parameter like filter_regex, in which case that provides a way to resolve zlib's =-in-prefix issue). That is true but that consolidation doesn't seem to provide much benefit (i.e. moving that into Spec doesn't eliminate work on the caller's part other than that they wouldn't have to import re anymore).

return True


class MakefileSpec(object):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
class MakefileSpec(object):
class MakefileSpec:

else:
return orig_format(obj, format_str)

monkeypatch.setattr(spack.spec.Spec, "format", _contrived_spec_format)
Copy link
Copy Markdown
Member

@haampie haampie Jul 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, but I still don't find this test acceptable. Don't mock internals, they may be relied on elsewhere, and only make refactoring hard. I'll just push an integration/regression test for git version, and a unit test for your formatter.

@haampie haampie force-pushed the bugfix/depfile-hash-eq-ver2 branch from 50d48f8 to 140d8d1 Compare July 17, 2023 10:16
@haampie haampie force-pushed the bugfix/depfile-hash-eq-ver2 branch from 140d8d1 to b645f22 Compare July 17, 2023 10:19
@haampie haampie enabled auto-merge (squash) July 17, 2023 10:20
@haampie haampie merged commit 31431f9 into spack:develop Jul 17, 2023
eugeneswalker pushed a commit to eugeneswalker/spack that referenced this pull request Jul 22, 2023
mpokorny pushed a commit to mpokorny/spack that referenced this pull request Sep 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

commands core PR affects Spack core functionality environments tests General test capability(ies)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants