@dlongley if you don't mind, I'll change the title on this issue and add algorithm steps to handle it, so that it is complete.

As you note, we'll need to update syntax a bit to allow for this.

if you don't mind, I'll change the title on this issue and add algorithm steps to handle it, so that it is complete.

Sure and thanks, @gkellogg!

@dlongley see if you like the spec text associated with the feature.

@gkellogg,

On the last call, we discussed what might happen if someone nullified the context via a type-scoped context. This resulted in me changing my implementation to handle this case. I think the implementation is actually simpler:

digitalbazaar/jsonld.js@1c00ec3

In short, whenever a type-scoped context is being processed, we save the previous active context. Then, when we call revertTypeScopedContext (instead of revertTypeScopedTerms), which restores the context to whatever it was prior to processing the type-scoped context. This will also make sure to revert any changes to @base, @vocab, etc. that were made just for the typed object.

To handle property-scoped contexts that come up within a type-scoped context, we process them against the stored previous context (updating it), rather than against the current active context, as they occur "within" and apply "according to" that previous context, not the type-scoped one.

At first, I was thinking that we would want the property-scoped contexts to be processed against the type-scoped context, but in practice, this is already confusing. For example, if you set @protected or @version on a context, then any scoped contexts do not "automatically" have this flag set, rather, they must be set on their own. Further confusion can arise when mixing multiple types that use multiple type-scoped contexts -- nested scoped contexts should not make assumptions about how/which existing prefix definitions will manifest. Therefore, I think that scoped contexts should be considered "self-contained", specifically declaring their own prefixes, versions, protected flags, etc. (any context-wide flags they use within their own context definition). We can mention this in the best practices doc.

With the other change that the group approved on the last call (allow redefinitions of protected terms so long as the definitions match, modulo the protected flag itself), redefining protected prefixes to the same thing within scoped contexts is not a problem.

We still need to add tests for type-scoped context nullification, @base, and @vocab changes -- and I think we should add one for a type-scoped context that has multiple property-scoped contexts within it to ensure we're covered.

@gkellogg,

JFYI -- We've also discovered that type-scoped nullification has some other weird properties, such that it wipes out the @type definition for the object that brought that scoped context in to begin with! So if a type-scoped context was brought in via the type Foo and that context does nullification, then the output will treat Foo like it doesn't exist (it will treat it as a relative IRI). A similar problem could occur even without nullification, where the type-scoped context redefines its own type to be something else!

It seems like that's quite unexpected and potentially even evil -- and this problem is totally orthogonal to the rest of these other patches (it's a separate bug that's been around that no one has detected yet). The good news is that, with the above change I mentioned, we have access to the previousContext ... which means we can interpret @type values within the object against the previous context, which is probably the expected behavior. We're working on a patch and tests for that right now.

Basically, this seems that we call revertTypeScopedContext on entry to compact, before handling @context or a property-scoped context, where before it was done after previously. On expand, we use any saved context for creating the property-scoped contexts.

For dealing with the context-clearing issue, I'm not sure I follow your processing. On expansion, the first thing I do is revert to the previous context, which would be null in this case. Are you saying there's special processing for @type, which continues to use the context before reverting? Does it do this only if the active context is null? Or, is there something else going on.

Also, the minutes said something about wildcarding properties, but I didn't understand what this might refer to.

Also, Expand tpr23-25 and tpr27 relate to changing a protected term definition that is equivalent, which should be in a different PR.

Protected term redefinition tests moved here: #92

@gkellogg,

On expansion, the first thing I do is revert to the previous context, which would be null in this case.

I'll have more to say a bit later, but the "previous context" should be whatever the context was before the type-scoped context was processed. That means that if a type-scoped context happens to be an array of contexts, the "previous context" does not change as you process each one in the array as they all apply as "type-scoped" contexts. Rather, it is only set once. Perhaps we need a better name than "previous context". Your comment about the "previous context" being null makes me think your implementation is updating it as it goes through the type-scoped array of contexts, which it should not be doing.

Our implementation has also added some extra processing to ensure that values and subject references that occur within a typed object are expanded and compacted using the appropriate type-scoped context, rather than erroneously reverting. There's a test added to cover that case.

@gkellogg,

So, as mentioned above, we set the previous context once at the start of context processing, prior to processing any contexts in an array of contexts. Then, when expanding @type values, we use that previous context if it's set:

digitalbazaar/jsonld.js@5e4e2e2

That ensures that the @type values are consistent, even when using type-scoped contexts that might clear or change the definition of the terms used.

During compaction, we ensure we keep a reference to the current active context to find all of the type-scoped contexts as we process them to produce a new active context. Then we do something analogous to expansion where we use the previous context, if present, to compact any values for @type:

digitalbazaar/jsonld.js@42392d5

Lastly, also during expansion, we make sure that values and subject references are processed against the type-scoped context, as they aren't really "nested" values:

digitalbazaar/jsonld.js@2ee232a

With these changes, we pass the tests we've submitted here and under #92 and #93.

@gkellogg,

JFYI, we're still working through a bug on value expansion, will let you know when we've gotten past that.

@gkellogg, in order to expand values properly, we needed to pass in the type-scoped context and use it if present. We've added tests that expand and compact values of typed-object properties.

See: digitalbazaar/jsonld.js@55d013d

@gkellogg,

Another change for you to be aware of:

Since we ensure that when we combine property-scoped contexts with type-scoped contexts that we processed the property-scoped contexts against the previous context (as opposed to the type-scoped ones that will be reverted), we also need to ensure that when we expand or compact IRIs we do the same. This patch does that:

digitalbazaar/jsonld.js@074527e

We've added tests for this as well.

Hopefully we're not making the algorithms full of special cases to handle this. In spite of the unexpected results, the previous was reasonably simple.

I'll look at this more tomorrow.

@gkellogg,

Hopefully we're not making the algorithms full of special cases to handle this.

I don't think we've got too much of that (any more than usual for other features), but I also suspect we could unify and simplify a bit. Right now I've just got to make sure it works and the features are composable, which I believe we're in good shape for now. I have to ship our implementation to get the VCWG @context updated (which composes all these new JSON-LD 1.1 features together) and its related test suite running.

In spite of the unexpected results, the previous was reasonably simple.

Yes, but of course a simple algorithm that yields unexpected results is its own set of problems :). Also, some of the changes we've made had to do with things that weren't tested previously and did not work as is (e.g., nullification/changing type terms). I do think that changes made (which, at the end of the day weren't too numerous) can be boiled down to some organizing principles that could be potentially simplified in algorithms/implementations.

Essentially, when processing contexts:

1. Type-scoped contexts must track previous contexts
2. Property-scoped contexts that are composed with type-scoped contexts must be applied to that previous context

When expanding/compacting:

1. We must revert type-scoped contexts when traversing out of the typed-object into another node object
2. We must use the aforementioned previous context when the active context was produced from a property-scoped context in conjunction with a type-scoped context

I'll look at this more tomorrow.

Thanks, @gkellogg!

@dlongley: I've updated the algorithms.

I do not get your results for compaction c025:

I do not get the same compacted result because the term definition for "nested" includes "@type": "@id". This prevents term selection from selecting this term, because IRI Compaction ends up setting type/language to @language, which doesn't get entries created in the inverse context when @type: @id. It doesn't seem to be what the test is testing (so we might just remove that from the term definition), but it points to a some other difference in the term selection algorithms.

As I mentioned above, the minutes said something about wildcarding properties, but I didn't understand what this might refer to. Otherwise, with an exception of c025, I think work is complete here, and per the WG discussion, we can merge it when you're happy with the spec text.

@gkellogg,

@dlongley: I've updated the algorithms.

Thanks!

I do not get your results for compaction c025:
I do not get the same compacted result because the term definition for "nested" includes "@type": "@id".

The value for nested is a node object, more specifically, a node object in a graph container, so it should compact using that term. Does your implementation compact to the nested term if you remove all the type scoping and just use a flat context? (I believe it should, and therefore should behave the same way when using type-scoped contexts):

{
  "@context": {
    "@version": 1.1,
    "type": "@type",
    "Outer": {
      "@id": "ex:Outer"
    },
    "nested": {
      "@id": "ex:nested",
      "@type": "@id",
      "@container": "@graph"
    },
    "Inner": {
      "@id": "ex:Inner"
    }
  }
}

As I mentioned above, the minutes said something about wildcarding properties, but I didn't understand what this might refer to.

I think we probably need to open a new issue for that. Essentially, now you can use property-scoped term definitions within a type-scoped term definition to cause terms to be defined beyond the type-scoped object with fine grained composability and without clashes w/@protected terms. This is great and what we want, but @azaroth42 has some cases where the same property-scoped context would be used across many properties in the same type-scoped context. Therefore, there's a desire to find a way to do this with some shorthand/sugar ("wildcard") that allows for pulling in a context that will define all of those properties without having to tediously declare all of those properties with the same @context.

This is related to:

2b. There is a shortcut suggestion to allow type scoped contexts to have a method to say that all properties of the instance should be treated as if they were property-scoped to the same context. This seems to be the right fix.

IIIF/api#1846

Otherwise, with an exception of c025, I think work is complete here, and per the WG discussion, we can merge it when you're happy with the spec text.

Sounds good. I looked over the text earlier and it looked good, but perhaps something is missing given c025 (which I think has correct output as is). I'll take a closer look when I can.

@gkellogg,

Yeah, I don't see one more change to the IRI compaction/expansion algorithms in the preview that is necessary. When expanding or compacting an IRI, if the context is from a property-term-scoped context that has been combined with a type-scoped context, we use the previous context to do the expansion/compaction (i.e., as the active context going forward). This could be added as a new step 3 in IRI expansion and step 2 in IRI compaction. My implementation sets a flag on the active context when the context is from a property-term-scoped context and then uses the presence of a previous context to check for this condition:

digitalbazaar/jsonld.js@074527e

I suspect that will resolve the difference with test c025.

I’ll check my implementation again, but I think I’ve hit everything that’s in the spec.

The c025 difference is due to the term selection algorithm, the proper context is loaded when calling Compact IRI. I’ll detail the inverse context and what’s in the call to term select. It has type/language set to language, which is empty in the inverse context, so the term doesn’t get selected. Remove @type: @id from the term definition, and it is selected.

@gkellogg,

I’ll check my implementation again, but I think I’ve hit everything that’s in the spec.

But I think the spec is missing the change that handles IRI expansion/compaction when you've got a property-term-scoped context inside a type-scoped context (see: #89 (comment)). In this case, context processing updates the previous context, which means that IRI expansion/compaction must use that context. All that's necessary is to check for that case and then assign the active context to the previous context before continuing on with those algorithms. That will cause the proper inverse context to be used, etc.

The issue doesn't seem related to selecting a context in compact/expand IRI, as I do end up with the correct term definition when compacting; I'm not sure why the code you cite might be needed, as I'm producing all the appropriate results otherwise.

What's going on when trying to do the compact IRI on ex:nested is more the issue. The inverse context entry I produce looks like the following:

"ex:nested": {
  "@graph": {
    "@language": {},
    "@type": {"@id": "nested"},
    "@any": {"@none": "nested"}
  }
}

This seems correct according to the algorithm in the API.

Because nested has a type mapping, step 3.10 of the Inverse Context algorithm creates the "@id": "nested"}, but @language remains empty.

In compact IRI, I have the following arguments to the Term Selection algorithm:

container mapping = ["@graph", "@graph@set", "@set", "@graph@index", "@graph@index@set", "@graph@id", "@graph@id@set", "@index", "@index@set", "@none", "@index", "@index@set"]

type/language = "@language"

preferred values = ["@null", "@none", "@any"]

It's because type/language is @language, that it doesn't find the entry in the inverse context. type/language is initialized to @language in the beginning of the algorithm, and never gets set to anything else. This is why removing @type: @id from the term definition allows it to work, because the inverse context will have entries for both @type and @language. Your implementation must either create the inverse context differently, or have different logic to set type/language.

As for the code to expand/compact IRI to use the previous context, can you point me to a test which should fail? It must be doing the equivalent logic elsewhere for the results to be the same, aside form c025.

@gkellogg,

My inverse context entry for ex:nested is the same as yours. However, my type/language is different. This is the value I have that is passed into the Compact IRI Algorithm when choosing the term for ex:nested:

{
  "@graph": [
    {
      "@type": [
        "ex:Inner"
      ],
      "ex:foo": [
        {
          "@value": "bar"
        }
      ]
    }
  ]
}

Which results in type/language = "@type" and preferred values = ["@id"].

Do you have a different value?

@gkellogg,

I'm looking at the IRI Compaction algorithm in the spec and it looks like it has changed in a breaking way from 1.0. I'm trying to tease out the differences now.

@gkellogg,

Yes, there's a new conditional fork for graph objects (step 2.8) in the 1.1 algorithm. Before that conditional was added, graph objects would fall through to the last step (step 2.9) and be detected as a non-value, which would cause steps 2.9.2 and 2.9.3 to run. These two steps are never run now for graph objects, so they need to be copied (or otherwise rearranged to run) into the end of 2.8. Otherwise the type/language is set to @language (as you're seeing in your implementation) for graph objects, which is clearly incorrect.

EDIT: It looks like only 2.9.2 must be copied over since @set is already added to containers in the new section.

@gkellogg,

The issue doesn't seem related to selecting a context in compact/expand IRI, as I do end up with the correct term definition when compacting; I'm not sure why the code you cite might be needed, as I'm producing all the appropriate results otherwise.

Yes, I don't think it's needed. I've looked more closely at your changes to the compaction algorithm and it applies a simplification my implementation doesn't have. It removes the early (step 1) processing of property scoped contexts and moves it down to a new step 6 after any type-scoped context has been reverted. I'll likely update my implementation to do this when I get the chance.

So it sounds like the only issue is the above missing step that will set type/language to @type properly for @graph. If you agree and that change fixes your test, I'll approve for merging.

Good catch, yes it struck me that the graph path wasn’t examining more properties, but it was changed so long ago. I’ll update my implementation and the spec.

Okay, I think that did it. See if you think the new steps in IRI Compaction are correct. Then, I think we're good to merge.

@gkellogg,

I think we're good here too. The only thing that looks a bit odd to me is that @set gets added to containers three separate times after that latest commit. I'm pretty sure that the way it's used in the Term Selection algorithm would make this unnecessary or erroneous. It should probably only be added at the very end, as it's the least specific choice? I defer to you on that as I still haven't updated my implementation to the latest version of that algorithm.

Everything else looks good. Thanks!

The last @set was errant, and the one before, as you note, was redundant.