Issue using FillPath with a recursion structure and hidden fields #1671
Description
Overview
In Dagger we have an action that reads a json/yaml document and converts leaf values into secrets (a struct with a reference to a value in memory). So consider the following example, decrypted from SOPS:
FOO: bar
ONE:
TWO:
THREE: one-hundred-twenty-threeWe want to convert the strings bar and one-hundred-twenty-three to secrets, while keeping their path to be referenced in other actions.
We've been using the current CUE type for that action's output:
output: #Secret | {[string]: output}It has been working ok, but now we're seeing some issues when trying to use FillPath.
Current scenario
Consider this repro, running in cuelang.org/go v0.4.3:
package main
import (
"fmt"
"cuelang.org/go/cue"
"cuelang.org/go/cue/cuecontext"
"cuelang.org/go/cue/format"
)
const source = `
#Secret: {
$secret: _id: string
}
a: {
// original scenario
output: #Secret | {[string]: output}
// attempted solution
// output: _#out
// _#out: #Secret | {[string]: _#out}
// workaround to unblock
// output: _
}
b: a.output.FOO
c: a.output.ONE.TWO.THREE
`
const output = `
FOO: $secret: _id: "100"
ONE: TWO: THREE: $secret: _id: "123"
`
const pkg = "core"
var ctx *cue.Context
func main() {
ctx = cuecontext.New()
src := NewValue(source)
out := NewValue(output)
// Debug("SOURCE", &src)
final := src.FillPath(cue.ParsePath("a.output"), out)
Debug("FINAL", &final)
if final.Err() != nil {
return
}
b := Lookup(final, "b")
c := Lookup(final, "c")
fmt.Printf("===> bId=%q; cId=%q\n", b, c)
}
func NewValue(s string) cue.Value {
v := ctx.CompileString(s, cue.ImportPath(pkg))
if v.Err() != nil {
panic(v.Err())
}
return v
}
func Lookup(val cue.Value, field string) string {
v := val.LookupPath(cue.MakePath(cue.Str(field), cue.Str("$secret"), cue.Hid("_id", pkg)))
if !v.Exists() || !v.IsConcrete() {
fmt.Printf("ERROR: %q: %v\n\n", field, v)
return ""
}
s, err := v.String()
if err != nil {
panic(err)
}
return s
}
func Debug(label string, v *cue.Value, opts ...cue.Option) {
b, err := format.Node(
v.Eval().Syntax(opts...),
format.UseSpaces(4),
format.TabIndent(false),
)
if err == nil {
fmt.Printf("%s\n%.*s\n%s\n\n", label, len(label), "================================", b)
}
}The output:
FINAL
=====
_|_ // a.output: 4 errors in empty disjunction: (and 4 more errors)
In our more complex setup we're seeing structural cycle issues, as seen in dagger/dagger#1867:
actions.good.output.Password.Password: structural cycle
So it seems like keys are being repeated.
Attempted solution
I attempted a solution (changeset) that doesn't reference back to output:
output: _#out
_#out: #Secret | {[string]: _#out}But now we get empty disjunction issues:
FINAL
=====
{
#Secret: {
$secret: {}
}
a: {
output: {
FOO: {
$secret: {}
} | {
$secret: {
$secret: {}
} | {}
}
ONE: {
TWO: {
THREE: {
$secret: {}
} | {
$secret: {
$secret: {}
} | {}
}
}
}
}
}
b: {
$secret: {}
} | {
$secret: {
$secret: {}
} | {}
}
c: {
$secret: {}
} | {
$secret: {
$secret: {}
} | {}
}
}
ERROR: "b": _|_ // field not found: $secret
ERROR: "c": _|_ // field not found: $secret
===> bId=""; cId=""
The issue, I think, is because the _id field, since it's hidden, it's not being enough to resolve the disjunction.
It works if "id" is visible
diff --git a/main.go b/main.go
index 72da1f4..df4cb43 100644
--- a/main.go
+++ b/main.go
@@ -10,7 +10,7 @@ import (
const source = `
#Secret: {
- $secret: _id: string
+ $secret: id: string
}
a: {
@@ -30,8 +30,8 @@ c: a.output.ONE.TWO.THREE
`
const output = `
-FOO: $secret: _id: "100"
-ONE: TWO: THREE: $secret: _id: "123"
+FOO: $secret: id: "100"
+ONE: TWO: THREE: $secret: id: "123"
`
const pkg = "core"
@@ -68,7 +68,7 @@ func NewValue(s string) cue.Value {
}
func Lookup(val cue.Value, field string) string {
- v := val.LookupPath(cue.MakePath(cue.Str(field), cue.Str("$secret"), cue.Hid("_id", pkg)))
+ v := val.LookupPath(cue.MakePath(cue.Str(field), cue.Str("$secret"), cue.Str("id")))
if !v.Exists() || !v.IsConcrete() {
fmt.Printf("ERROR: %q: %v\n\n", field, v)
return ""Output:
FINAL
=====
{
#Secret: {
$secret: {
id: string
}
}
a: {
output: {
FOO: {
$secret: {
id: "100"
}
}
ONE: {
TWO: {
THREE: {
$secret: {
id: "123"
}
}
}
}
}
}
b: {
$secret: {
id: "100"
}
}
c: {
$secret: {
id: "123"
}
}
}
===> bId="100"; cId="123"
Workaround
To work around this, I can abandon the CUE definition and just use whatever's filled in runtime:
output: _Which outputs correctly:
FINAL
=====
{
#Secret: {
$secret: {}
}
a: {
output: {
FOO: {
$secret: {}
}
ONE: {
TWO: {
THREE: {
$secret: {}
}
}
}
}
}
b: {
$secret: {}
}
c: {
$secret: {}
}
}
===> bId="100"; cId="123"
But that's not ideal.
Question
So is there a way to define output that makes this structure work?
/cc @jlongtine