Skip to content

Tests: Do not remove filename from boost filesystem path for corner cases#10976

Merged
chriseth merged 1 commit intodevelopfrom
fix-boost-filesystem-bug
Feb 23, 2021
Merged

Tests: Do not remove filename from boost filesystem path for corner cases#10976
chriseth merged 1 commit intodevelopfrom
fix-boost-filesystem-bug

Conversation

@bshastry
Copy link
Copy Markdown
Contributor

Fixes #10713

Also see boostorg/filesystem#176 for upstream issue

@bshastry bshastry requested a review from cameel February 18, 2021 15:44
@bshastry bshastry force-pushed the fix-boost-filesystem-bug branch 2 times, most recently from 5ae448c to 64f5b91 Compare February 18, 2021 16:08
@bshastry
Copy link
Copy Markdown
Contributor Author

For all the hacks required elsewhere, I'm wondering if this fix merits a test for a bug in boost filesystem.

Comment thread scripts/splitSources.py Outdated
Comment thread test/stopAfterParseTests.sh
@bshastry bshastry force-pushed the fix-boost-filesystem-bug branch from 64f5b91 to 55f2abf Compare February 18, 2021 16:37
Comment thread test/libsolidity/syntaxTests/imports/boost_filesystem_bug.sol
Comment thread libsolutil/CommonIO.cpp
boost::filesystem::path result(_reference);
result.remove_filename();

// If filename is "/", then remove_filename() throws.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

absolutePath was intended to be used on file names, maybe we should check that it does not end in / instead?

Copy link
Copy Markdown
Contributor Author

@bshastry bshastry Feb 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ending with / does not seem to be the problem but containing at least 4 / only and nothing else seems to be.

Examples

////
//////
/////////

(Edit: For the sequences listed above, the parsed filename is always "/". Alternatively, we could fix the call sites by checking if the absolutePath contains at least 4 and only / but that is a little more elaborate)

I actually fuzzed the remove_filename API with random path strings and it turns out the fix is rather robust at least for ASCII input sequences. Not sure what the underlying root cause is.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @chriseth that requiring _path to always contain a file name would be a nice way to side-step the issue.

Looks like the only place where we use absolutePath() is in CompilerStack::loadMissingSources(). Currently it tries to load the file listed in an import without checking if it's actually a file but for directories it fails anyway:

Error: Source "/tmp/xyz" not found: Not a valid file.
 --> /tmp/test.sol:1:1:
  |
1 | import "./xyz/";
  | ^^^^^^^^^^^^^^^^

We could make it fail immediately if the path ends in the OS-specific directory separator and then absolutePath() could just assert that _path contains a file name as a precondition.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could make it fail immediately if the path ends in the OS-specific directory separator

The following test currently passes. Are you suggesting that it should fail instead because import string ends with a dir separator?

==== Source: a/b/c.sol ====
contract C {}
==== Source: a/b/d.sol ====
import "./c.sol/";
contract D is C {}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boost::fs is surprisingly robust except for the //// corner case. For example, the following test also passes

==== Source: a/b/c.sol ====
contract C {}
==== Source: a/b/d.sol ====
import ".//c.sol////";
contract D is C {}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, yeah, I think this should be an error. To me it looks as if c.sol/ is a directory and if I saw this in someone's code I'd immediately start wondering what's going on, what does it mean to import a whole directory :)

Though the fact that it works right now complicates things. This makes it a breaking change so we still need some other workaround for 0.8.x. The one you did in this PR is good enough in my opinion. But let's see what @chriseth says.

Copy link
Copy Markdown
Collaborator

@cameel cameel Feb 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boost::fs is surprisingly robust except for the //// corner case.

I think this has something to do with root names. For example in //a/b/c, the path is actually /b/c and //a is the root name. So it has all kinds of special cases for // at the beginning of the path. This is even more relevant on Windows where something like C: or D: could be a root name. See for example the table under boost:filesystem::absolute(). The only reason it needs a table is the existence of root names :)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, yeah, I think this should be an error. To me it looks as if c.sol/ is a directory and if I saw this in someone's code I'd immediately start wondering what's going on, what does it mean to import a whole directory :)

True, that would be a red flag for sure.

Though the fact that it works right now complicates things. This makes it a breaking change so we still need some other workaround for 0.8.x. The one you did in this PR is good enough in my opinion. But let's see what @chriseth says.

Agreed, I was also mainly concerned of unknowingly breaking something by disallowing certain (albeit weird) forms of import path strings.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boost::fs is surprisingly robust except for the //// corner case.

I think this has something to do with root names. For example in //a/b/c, the path is actually /b/c and //a is the root name. So it has all kinds of special cases for // at the beginning of the path. This is even more relevant on Windows where something like C: or D: could be a root name. See for example the table under boost:filesystem::absolute(). The only reason it needs a table is the existence of root names :)

Interesting. Today I learnt :-)

Copy link
Copy Markdown
Collaborator

@cameel cameel Feb 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have created an issue for disallowing the trailing slash (#10980) and added it to the design backlog so that in this PR we can focus on doing just a basic, non-breaking fix.

Comment thread test/stopAfterParseTests.sh
Comment thread test/libsolidity/syntaxTests/imports/boost_filesystem_bug.sol
Comment thread scripts/ASTImportTest.sh
Comment thread libsolutil/CommonIO.cpp
Comment thread libsolutil/CommonIO.cpp
@bshastry bshastry force-pushed the fix-boost-filesystem-bug branch from 0bca759 to 8484629 Compare February 19, 2021 10:43
@bshastry bshastry mentioned this pull request Feb 19, 2021
@bshastry bshastry force-pushed the fix-boost-filesystem-bug branch 2 times, most recently from 3aa40d3 to 1280ca5 Compare February 19, 2021 11:16
Comment thread test/libsolidity/syntaxTests/imports/boost_filesystem_bug.sol Outdated
Comment thread scripts/ASTImportTest.sh Outdated
@bshastry bshastry force-pushed the fix-boost-filesystem-bug branch 3 times, most recently from feff645 to 6d58f79 Compare February 19, 2021 14:02
@bshastry bshastry force-pushed the fix-boost-filesystem-bug branch from 6d58f79 to 1ddfc74 Compare February 22, 2021 09:33
Copy link
Copy Markdown
Collaborator

@cameel cameel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the fix is fine so if @chriseth has no objections (regarding #10976 (comment)) I'd say to merge it.

@chriseth chriseth merged commit 02b27bd into develop Feb 23, 2021
@chriseth chriseth deleted the fix-boost-filesystem-bug branch February 23, 2021 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Testing] Seg fault probably due to boost::filesystem

3 participants