layer: add hardlink handling information #336

vbatts · 2016-09-21T19:45:08Z

Fixes: #306

Signed-off-by: Vincent Batts [email protected]

vbatts · 2016-09-21T19:45:44Z

View the document: https://github.com/vbatts/oci-image-spec/blob/hardlink-verbiage/layer.md

wking · 2016-09-21T20:04:39Z

layer.md

+Non-directory files with linkcount greater than 1 have hardlinks.
+The corresponding files that share the link with the > 1 linkcount may be outside the directory that the changeset is being produced from, in which case the `linkname` is not recorded in the changeset.
+
+Hardlinks are stored in a tar archive with type of `'1'`, per the [Basic Tar Format][tar-standard].


Linking to the GNU docs makes it less clear where this fits in the basket of tar standards ;). libarchive's tar(5) attributes this typeflag to ustar (IEEE Std 1003.1-1988). IEEE Std 1003.1-1988 has been superseded by IEEE Std 1003.1-2008, which seems to have been superseded by IEEE Std 9945-2009. The IEEE versions are paywalled, so I'm not sure what's inside. But it's probably better to mention ustar and successors here, and possibly link to the libarchive man page for more historical context, instead of linking to the GNU docs.

You drop so many links and words, but they still resolve to the same thing, that the typeflag is '1'

philips

LGTM

philips · 2016-09-21T22:09:20Z

LGTM

philips · 2016-09-21T22:35:50Z

@vbatts this upset the pdf build somehow

stevvooe · 2016-09-22T00:28:03Z

layer.md

+Not all filesystems support hardlinks (e.g. [FAT](https://en.wikipedia.org/wiki/File_Allocation_Table)).
+
+Hardlinks are possible with all [file types](#file-types) except `directories`.
+Non-directory files with linkcount greater than 1 have hardlinks.


Might be good to clarify that these must also have the same inode and devid. It looks something like https://github.com/stevvooe/continuity/blob/master/hardlinks_unix.go#L11 in practice.

yeah, I got at that in the pseudocode, but should put it in a sentence here too

vbatts · 2016-09-22T19:38:48Z

layer.md


+#### Hardlinks
+
+Hardlinks are a [POSIX concept](http://pubs.opengroup.org/onlinepubs/9699919799/functions/link.html) for having one or more directory entries for the same file on the same device.


@stevvooe i do say part of that here. So, make it more explicit?

@vbatts I still think that condition needs to be added to the clause above. The issue is that two files with the same inode but different devices ids are two separate hardlinks.

On Thu, Sep 22, 2016 at 02:14:30PM -0700, Stephen Day wrote:

@vbatts I still think that condition needs to be added to the clause
above. The issue is that two files with the same inode but different
devices ids are two separate hardlinks.

He covers that in Markdown in the next paragraph. Are you just asking for “file” → “inode” here?

wking · 2016-09-22T19:43:49Z

On Thu, Sep 22, 2016 at 12:32:47PM -0700, Vincent Batts wrote:

+Hardlinks are stored in a tar archive with type of '1', per the [Basic Tar Format][tar-standard].

You drop so many links and words, but they still resolve to the same
thing, that the typeflag is '1'

And the tar flavor must be ustar-compatible. If we depend on a
versioned spec like IEEE Std 1003.1-1988 (linking to the IEEE page
and/or the libarchive tar(5)), then we have a stable requirement. If
we link to the GNU docs, and not even a particular version of the GNU
docs, things become too fluid for my taste. For example, that page
currently has:

BEWARE BEWARE BEWARE that the following information is still
boiling, and may change. Even if the OLDGNU format description
should be accurate, the so-called GNU format is not yet fully
decided.

above some newer structures. And it also has (at the very bottom):

For references, see ISO/IEC 9945-1:1990 or IEEE Std 1003.1-1990,
pages 169-173 (section 10.1) for Archive/Interchange File Format;
and IEEE Std 1003.2-1992, pages 380-388 (section 4.48) and pages
936-940 (section E.4.48) for pax - Portable archive interchange.

So I think we should take that advice and reference one of those
specs.

vbatts · 2016-09-22T20:04:59Z

updated. PTAL.

wking · 2016-09-22T20:07:54Z

layer.md

+
+Hardlinks are possible with all [file types](#file-types) except `directories`.
+Non-directory files have hardlinks when the linkcount is greater than 1.
+Hardlinked files are on a same device (i.e. comparing Major:Minor pair), and have the same inode.


nit: No need for the comma here; “hardlinked files” is the subject of both portions.

wking · 2016-09-22T20:08:39Z

layer.md

+Hardlinked files are on a same device (i.e. comparing Major:Minor pair), and have the same inode.
+The corresponding files that share the link with the > 1 linkcount may be outside the directory that the changeset is being produced from, in which case the `linkname` is not recorded in the changeset.
+
+Hardlinks are stored in a tar archive with type of '`1`', per the [GNU Basic Tar Format][gnu-tar-standard] and [libarchive(5)][libarchive].


nit: Quoting and backticking seems strange. Can we just backtick or just quote?

Also, the page you are linking is libarchive's tar(5), not libarchive(5). I think you want a line like:

Hardlinks are stored in a tar archive with type of `1` as specified by [IEEE Std 1003.1-1988][ieee1003.1-1998] (“ustar”, discussed [here][gnu-tar-standard] and [here][tar.5-ustar]). [ieee1003-1998]: http://standards.ieee.org/findstds/standard/1003.1-1988.html [gnu-tar-standard]: http://www.gnu.org/software/tar/manual/html_node/Standard.html [tar.5-ustar]: https://github.com/libarchive/libarchive/wiki/ManPageTar5#POSIX_ustar_Archives

like you said that is paywalled. I'm not going to use that reference for now. The other docs include their own citing.

vbatts · 2016-09-23T18:34:46Z

updated. PTAL

wking · 2016-09-23T19:00:42Z

On Fri, Sep 23, 2016 at 11:33:05AM -0700, Vincent Batts wrote:

like you said that is paywalled. I'm not going to use that reference
for now. The other docs include their own citing.

The other docs cite many references, and I'm not sure which one we're
requiring (suggesting?) support for here. From tar(5), it looks like
the old tar from Version 7 AT&T UNIX already supported linkflag 1
(typeflag in later specs) for hardlinks, and that all subsequent
versions discussed in tar(5) support that too. IEEE Std 1003.1-2001's
pax requires linkpath to be UTF-8, although this doesn't seem to have
been a requirement before. The GNU format (with an unversioned spec?)
supports the K typeflag to give linknames > 100 chars for subsequent
entries. Which of those approaches should image-spec-compatible
libraries use? Are K typeflags valid? Are non-UTF-8 linknames valid?
Are we punting to the tar library to pick the tar format based on
magic values? If so, can we just say that? If not, can we say which
version of tar we're requiring?

wking · 2016-09-23T19:13:02Z

If we want a freely-accessible standard, I'd suggest pax (IEEE Std
1003.1, 2013 Edition) 1. It looks like Go has supported it since
v1.1 [2,3].

vbatts · 2016-09-23T19:37:52Z

On 23/09/16 12:00 -0700, W. Trevor King wrote:

The other docs cite many references, and I'm not sure which one we're
requiring (suggesting?) support for here. From tar(5), it looks like
the old tar from Version 7 AT&T UNIX already supported linkflag 1
(typeflag in later specs) for hardlinks, and that all subsequent
versions discussed in tar(5) support that too. IEEE Std 1003.1-2001's
pax requires linkpath to be UTF-8, although this doesn't seem to have
been a requirement before. The GNU format (with an unversioned spec?)
supports the K typeflag to give linknames > 100 chars for subsequent
entries. Which of those approaches should image-spec-compatible
libraries use? Are K typeflags valid? Are non-UTF-8 linknames valid?
Are we punting to the tar library to pick the tar format based on
magic values? If so, can we just say that? If not, can we say which
version of tar we're requiring?

Really, what are you trying to accomplish with responses like this?
It is not constructive.

wking · 2016-09-23T20:01:17Z

On Fri, Sep 23, 2016 at 12:37:53PM -0700, Vincent Batts wrote:

On 23/09/16 12:00 -0700, W. Trevor King wrote:

The other docs cite many references, and I'm not sure which one we're
requiring (suggesting?) support for here. From tar(5), it looks like
the old tar from Version 7 AT&T UNIX already supported linkflag 1
(typeflag in later specs) for hardlinks, and that all subsequent
versions discussed in tar(5) support that too. IEEE Std 1003.1-2001's
pax requires linkpath to be UTF-8, although this doesn't seem to have
been a requirement before. The GNU format (with an unversioned spec?)
supports the K typeflag to give linknames > 100 chars for subsequent
entries. Which of those approaches should image-spec-compatible
libraries use? Are K typeflags valid? Are non-UTF-8 linknames valid?
Are we punting to the tar library to pick the tar format based on
magic values? If so, can we just say that? If not, can we say which
version of tar we're requiring?

Really, what are you trying to accomplish with responses like this?
It is not constructive.

The idea with a spec like this is to define behavior so that different
implementations can interoperate reliably. When there is an interop
problem betweem implementations A and B, it should be clear from the
spec whether implementation A is broken, implementation B is broken,
or the spec is insufficiently clear. As it stands, if implementation
A produced a spec with a K typeflag and implementation B died unpacking
it, I am not clear whose fault that is. That's a problem with the
spec.

If we require support for the pax format as defined by IEEE Std
1003.1, 2013 Edition [1](this would be my preference), then it is
clearly implementation A which is broken [edit: changed B → A. pax has “The letters 'A' to 'Z', inclusive, are reserved for custom implementations”].

If we explicitly make the tar version unspecified or
implementation-defined, then we're saying that interop issues like B's
choking on A's layer are not a problem we're trying to solve in this
spec.

But by doing neither of those, layer interop is just confusing.

stevvooe · 2016-09-23T22:53:48Z

layer.md

+Not all filesystems support hardlinks (e.g. [FAT](https://en.wikipedia.org/wiki/File_Allocation_Table)).
+
+Hardlinks are possible with all [file types](#file-types) except `directories`.
+Non-directory files have hardlinks when the linkcount is greater than 1.


Non-directory files are considered "hardlinked" when their link count is greater than 1.

stevvooe · 2016-09-23T22:54:34Z

@vbatts I read it through again it looks good.

Fixes: opencontainers#306 Signed-off-by: Vincent Batts <[email protected]>

vbatts · 2016-09-24T12:41:08Z

updated.

On Fri, Sep 23, 2016 at 6:53 PM, Stephen Day [email protected]
wrote:

@stevvooe requested changes on this pull request.

In layer.md
#336 (review)
:

Sparse files SHOULD NOT be used because they lack consistent support across tar implementations.

+#### Hardlinks
+
+Hardlinks are a POSIX concept for having one or more directory entries for the same file on the same device.
+Not all filesystems support hardlinks (e.g. FAT).
+
+Hardlinks are possible with all file types except directories.
+Non-directory files have hardlinks when the linkcount is greater than 1.

Non-directory files are considered "hardlinked" when their link count is
greater than 1.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#336 (review),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEF6UiCtRA8V1BIIWfKbf7IxTRn71d6ks5qtFiCgaJpZM4KDM80
.

stevvooe · 2016-09-26T18:46:01Z

LGTM

philips · 2016-09-27T05:27:47Z

LGTM

(aside: on mini-vacation so I won't be back into the full swing until Thursday)

wking reviewed Sep 21, 2016

View reviewed changes

philips approved these changes Sep 21, 2016

View reviewed changes

philips added this to the v1.0.0-rc1 milestone Sep 21, 2016

stevvooe requested changes Sep 22, 2016

View reviewed changes

vbatts commented Sep 22, 2016

View reviewed changes

vbatts force-pushed the hardlink-verbiage branch from 4daaa1d to 2be6ecf Compare September 22, 2016 20:04

wking reviewed Sep 22, 2016

View reviewed changes

vbatts force-pushed the hardlink-verbiage branch from 2be6ecf to f91b794 Compare September 23, 2016 18:34

wking mentioned this pull request Sep 23, 2016

layer: Require ustar (originally from IEEE Std 1003.1-1988 but linking IEEE Std 1003.1-2013) #342

Closed

stevvooe requested changes Sep 23, 2016

View reviewed changes

layer: add hardlink handling information

8cd0673

Fixes: opencontainers#306 Signed-off-by: Vincent Batts <[email protected]>

vbatts force-pushed the hardlink-verbiage branch from f91b794 to 8cd0673 Compare September 24, 2016 12:40

stevvooe approved these changes Sep 26, 2016

View reviewed changes

philips merged commit e15b9c0 into opencontainers:master Sep 27, 2016

vbatts deleted the hardlink-verbiage branch September 27, 2016 17:01


		#### Hardlinks

		Hardlinks are a [POSIX concept](http://pubs.opengroup.org/onlinepubs/9699919799/functions/link.html) for having one or more directory entries for the same file on the same device.

layer: add hardlink handling information #336

layer: add hardlink handling information #336

Uh oh!

Conversation

vbatts commented Sep 21, 2016

Uh oh!

vbatts commented Sep 21, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philips left a comment

Choose a reason for hiding this comment

Uh oh!

philips commented Sep 21, 2016 • edited by caniszczyk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philips commented Sep 21, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wking commented Sep 22, 2016

Uh oh!

vbatts commented Sep 22, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vbatts commented Sep 23, 2016

Uh oh!

wking commented Sep 23, 2016

Uh oh!

wking commented Sep 23, 2016

Uh oh!

vbatts commented Sep 23, 2016

Uh oh!

wking commented Sep 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stevvooe commented Sep 23, 2016

Uh oh!

vbatts commented Sep 24, 2016

@stevvooe requested changes on this pull request.

Uh oh!

stevvooe commented Sep 26, 2016 • edited by caniszczyk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

philips commented Sep 27, 2016 • edited by caniszczyk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

philips commented Sep 21, 2016 •

edited by caniszczyk

Loading

wking commented Sep 23, 2016 •

edited

Loading

stevvooe commented Sep 26, 2016 •

edited by caniszczyk

Loading

philips commented Sep 27, 2016 •

edited by caniszczyk

Loading