v0.3.0-alpha3

This release mainly introduces per-file inclusion based on tags, bug fixes for bugs introduced in the new evaluator, and tests to close issues raised on the old evaluator that are now verified to be fixed.

Language extensions

Attribute declarations in preamble

Attribute declarations are now allowed before the package clause and import declarations. This is used to implement file-wide inclusion.

Any scalar now allowed in interpolations

This includes boolean and byte values. Boolean values are represented as their JSON value. In string interpolations, bytes are converted to valid UTF-8 by replacing the maximal subpart of ill-formed subsequences with a single replacement character (W3C encoding standard).

Whole file inclusion

CUE now allows files to be conditionally included in a build. To conditionally include a file in a CUE program, one can now write:

@if(prod)

package acmecorp

and use the -t prod flag to cause this file to be included. The expression can be any valid CUE expression using only boolean operators and boolean values.

See cue help injection for more details.

Closedness rewrite

A large number of bug fixes related to an incorrect “field not allowed” error are now in. The algorithm for closedness has been significantly simplified and streamlined based on the benefit of hindsight.

Note that one may still have a misleading “field not allowed” error if a field is disallowed in one disjunct and allowed in another, but where there is another field in conflict. Right now only one of the errors is reported.

CUE API

New evaluator

The CUE API is now moved to the new evaluator.

Package dependencies

Builtins now live in their own package and are no longer copied into the cue API. This was part of the effort to break the implementation into logical pieces, instead of having one large package.

This has the additional advantage that each builtin package now only lives in its own directory and is not copied. This makes maintenance easier and also removes some of the limitations imposed by the old implementation.

Unfortunately, this means that custom loader implementations (ones that do not use cue/load for loading packages) will have to include

import 	_ "cuelang.org/go/pkg"

somewhere in the package to trigger linking in the builtin packages.

cue tool

This release also marks the first steps of moving functionality from the cue command line tool to the CUE APIs. The intention is that the cue tool will be mostly a wrapper around core functionality present in the API. Other tooling could provide a different “view” without being second-class citizens in the ecosystem and while providing consistency between such tools.

The first step towards this is the injection mechanism. The existing parts of this were moved to the cue/load package. The new file inclusion functionality, meanwhile, was directly implemented in cue/load.

The next step will be to move non-CUE file support and packaging mechanisms to cue/load and friends. Then next up is the tooling layer, starting with an API for dependency analysis.

Next steps

There are two outstanding issues on the table to come to a beta release:

Disjunctions

Although there are no known semantic bugs related to disjunctions, the code needs some restructuring to improve error messages and to fix some performance issues. This is slated for the alpha4 release.

Structural cycles

There are still some bugs outstanding related to structural cycles. The plan is to address all of these in an alpha5 (or possibly beta) release.

Backwards compatibility

Go version

Support for go1.12 was dropped. In exchange, this release is verified to build with Go’s tip (1.16).

CUE API

The CUE API has now fully been moved over to use the new evaluator. The old evaluator code has been removed.

• cue.Merge: the purpose of Merge is to unify the underlying instances as data. It now no longer also unifies definitions. Bug fixes in closedness exposed this previously unnoticed oversight.

Custom loader implementations (ones that do not use cue/load for loading packages) will have to include

import 	_ "cuelang.org/go/pkg"

somewhere in the package to trigger linking in the builtin packages.

Language changes

CUE code that previously expected a boolean or bytes value substituted in a string interpolation to fail my now break.

cue cmd

Commands that operate on multiple instances simultaneously (e.g. cue cmd foo ./...) can no longer reference definitions in the underlying commands. For more info see cue.Merge in the API section.

Changelog

d69319c all: add missing copyright
4c10692 all: fix four previously ignored errors
6c27cef ci: drop go1.12 add go1.15
c3f30d8 ci: name dispatch workflow differently to default branch push workflow
1c54297 cmd/cue/cmd: fix get go package handling
1e8906a cmd/cue/cmd: move injection mechanism to cue/load
faf617c cmd/cue/cmd: verify issue 236 is fixed
e71970c cmd/cue/cmd: verify issue 322 is fixed
8776561 cmd/cue/cmd: verify issue 425 is fixed
ba5708b cmd/cue: get go: add --package flag
1a9b88d cmd/cue: get go: use constant.Value.ExactString to extract constant
f25c04d cmd/help: fix example in doc
089f461 cue/load: add support for build tags
bc101b3 cue/load: add tool and test files to BuildFiles if requested
20c637a cue/load: relax places where @tag is allowed
bd3dd75 cue/load: rewrite pre-resolved files after injection
c264475 cue/parser: allow attributes before package and import clauses
4cce6c4 cue: apply wrapping to all errors in list
d2fdbf0 cue: clean up handling of preamble declarations
80e70b9 cue: fix instance error handling
c76a530 cue: implement fill in terms of Value.Fill
845df05 cue: make internal cue port main API implementation
abce145 cue: move to external builtin packages
ce28b20 cue: prepare to hoist builtin code
9acd624 cue: prepare to hoist builtins, step 2
40a6bcd cue: remove dead code
6f84d11 deps: upgrade to github.com/rogpeppe/go-internal v1.6.1
edea6f3 doc/ref/spec.md: fix len count for structs
043c534 doc/ref/spec: clarification on cycles
dd0fa88 doc/ref/spec: consider definitions for closedness
c7c14fd doc/tutorial/kubernetes: remove a duplicate kind field
e3a03a1 internal/core/adt: exclude definitions in data conversion
db18b37 internal/core/adt: prevent nil interface bug for Source
f5fa009 internal/core/adt: remove all closedness IDs in ToData
30ca062 internal/core/adt: support interpolation of bytes and bool
df01042 internal/core/compile: better error message for user-defined errors
d55e2b5 internal/core/eval: allow lists resulting from dependent expressions
c72ce5d internal/core/eval: fix empty data vertex bug
0b57e43 internal/core/eval: fix interpolation of numbers
e0c19d1 internal/core/eval: fix issue 507
304b02e internal/core/eval: nested comprehension fix
62528c3 internal/core/eval: rewrite of closedness algorithm
dc2401c internal/core/eval: verify issue 258 is fixed
ddd99d5 internal/core/eval: verify issue 293 is fixed
ea5d964 internal/core/eval: verify issue 295 is fixed
480e1a1 internal/core/eval: verify issue 299 if fixed
541bf03 internal/core/eval: verify issue 306 is fixed
5905c22 internal/core/eval: verify issue 332 is fixed
121ace7 internal/core/eval: verify issue 342 is fixed
07ce2f6 internal/core/eval: verify issue 353 is fixed
4ecec98 internal/core/eval: verify issue 370 is fixed
fbf9de3 internal/core/eval: verify issue 398 is fixed
c6ade67 internal/core/export: fix bug 473
b9b2c54 internal/core/export: better handling of generated CUE
1e1fe6f internal/core/export: handle scalar values in adt.Vertex
67d1d42 internal/core/export: suport default picking
df9468e internal/core/export: verify issue 349 is fixed
60c207f internal/core/runtime: collect errors for all files in resolve
a6ce869 internal/core: finalize list in slice
148f67a internal/core: fix incorrect error handling
3d863fc internal/core: move CloseID from Environment to Conjunct
bf882b0 pkg/internal: copy cue builtin code to separate package
dda7de4 pkg/math: fix issue 418
4f4b7a3 pkg/tool/file/append: create file if none exists.
c8a4514 pkg/tool/http: fix handling of errors
248794e pkg: generate per-directory tests from builtin_test.go
1bcb73a update go-internal to pull in fix for Go 1.16

v0.3.0-alpha2

DO NOT USE THIS RELEASE

v0.3.0-alpha1

This release implements the long-anticipated move of cue to the new evaluator. This particular alpha release only switches over the cue command. A follow-up release will switch over the API as well. This allows people to upgrade to the tool without having to worry about API breakage just yet.

Note that this is an alpha release. This release includes an almost complete rewrite of CUE core. Although it will see many benefits, it can be expected to need some more ironing out. This includes new bugs in semantics and performance. But it also means dealing with bugs in the old evaluator that are now uncovered.

Overview of new capabilities

This release of the new evaluator mostly aims to address old bugs and doesn’t introduce many new things. However, this still means it addresses some important things:

Considerable performance improvements. More on that later.
Many bug fixes: the new evaluator is designed with embeddings, definitions, and optional constraints in mind, instead of them being an afterthought.
Structural cycle detection: this should put an end to stack overflows (but see caveats below).
Non-monotonic constraints: this allows things like structs.MinFields(3) to work. This also enables the implementation of things like not (for JSON Schema support), association lists, and all sorts of other goodness.

A more official ticking off the box of existing issues is still forthcoming.

Under the hood it has an almost complete implementation of the following features:

JSON Schema semantics for pattern constraints (bulk optional fields) and additional constraints (allowing ...T in structs as well).
Embedded scalars: embedding scalars is allowed at the top-level, but now would be allowed anywhere: with the same rules that apply at the top-level. For instance, { "foo", #field: 2 } would be allowed.

These features will be enabled in a later release (in most cases by simply relaxing the parser), once they are better tested.

Under the hood, the new evaluator has been broken up into about 10 components, instead of being one large monolith, separating out clear responsibilities. The largest package is now about 1/15th the size of the original implementation. This clarifies dependencies and responsibilities.

Things that are known to be broken or incompatible

Tooling overlay

Previously, *_tool.cue files were interpreted in a different namespace that lived as an overlay over the other configurations. This was hard to explain for users and also results in many awkward situations, like the inability to print _tool.cue files.

This functionality is not implemented in the new evaluator. This means the command section is merged in with the other files. This may cause clashes.

The idea behind not implementing it is that the new #-style definitions, which live in a different namespace, allow for another, better approach: commands can simply be declared as definitions to avoid clashes with regular fields.

For cases where the user also has no control over the definition names, the idea is to allow for a convention where a tooling logic can be in a separate pkg_tool package to accompany a regular package, not unlike a foo_test package in Go. To CUE this would then just be a regular package. Feedback appreciated.

if comprehension fix

The old evaluator used to silently ignore an error of the condition of an if clause, making it the same meaning as false. This was incorrect and led to several bugs. The new evaluator now treats errors as errors.

If the old behavior was desired, one can achieve the same result using disjunctions. One such case was:

foo: string
if len(foo) > 0 {
}

The old (faulty) behavior can be simulated writing this as follows:

foo: string
if *(len(foo) > 0) | false {
}

Disjunction bug fix

In the old evaluator, sometimes allowed a selector to index into an unresolved disjunction, causing (null | { c: 2 }).c to resolve. This is against the spec and is always illegal in the new evaluator. So one may find cases where one will have to write (d&{}).c to first resolve the disjunction.

Error messages

Please don’t send bug reports on faulty error messages for this release.

The new evaluator implements a different approach to error message generation. This allows the creation of very tailored and detailed error messages. It is just not done yet for the most part. Net effect:
a lot of the line number information is missing,
the path is sometimes missing (but also improved in some cases, as the path information is now more reliable),
some of the error messages are still sloppy.

Reentrancy limitations are now enforced

This means CUE should no longer crash on structural cycles.

Please submit bug reports if CUE hangs in an infinite loop or has a stack overflow

The spec disallows structural cycles. This includes reentrancy for the form

foo: { n: int, out: foo&{n:2}.out+1 }

The new evaluator also has some performance stats built in. This showed that CUE really isn’t great at doing reentrancy and will quickly venture into unexpected exponential behavior.

CUE’s strength is to combine results of computation, not do the computation itself. So instead, the way forward for CUE is to make it easier to shell out to other languages like Go, Python, Javascript and/or WASM.

That said, at the moment shelling out to other languages can only be done by creating custom invocations in the tooling layer. A convenient way to do this is not yet supported.

For the time being, we could consider supporting a structural cycle allowance e.g. as a command line flag or environment variable that allows a certain number of structural cycles to be ignored by CUE. This could help people transition off the use of reentrancy. Feedback appreciated.

Printing

Printing of CUE values has been completely rewritten and as a result will result in the inevitable differences. CUE now has a stricter distinction between values and expressions. This may cause printed values to be rendered differently. Also, printing of definitions still needs some work and can be expected to have some bugs.

There will always be the inevitable cosmetic changes, but:

please report where changes of printing results in bad incompatibilities.

Field order

CUE used to print fields in the order that a field name was declared anywhere. This often resulted in fields being sorted in schema order, but this was clearly not always the case.

The new evaluator does a topologic sort on fields before printing them. This means that fields will observe the order of fields of all structs that combine into a result. Insofar there remains ambiguity, the fields will be combined arbitrarily. In case different structs give conflicting orderings, the result is undefined.

The algorithm does not yet take the relative ordering of embedded structs into account.

tools/trim

Trim relied on very intricate behavior of the API as well as various deprecated API features. It made assumptions that no longer held as the language progressed even in the old evaluator.

The code has been adjusted to use the new APIs. It now often does not remove the top-level element that was inserted by a comprehension. Otherwise it should largely give the same results. The new evaluator also fixes some inconsistencies that fixes known bugs in the old evaluator that failed to remove some fields in some cases.

Other things to watch out for

Cycle handling

One of the benefits of the “always evaluate everything” approach is that CUE can annotate a result of evaluation with cycle points. This means it is possible to rely on a single point where cycles are detected. This is great, as cycle detection is complex! Previously, stack overflow detection was implemented in many components, each with their own peculiarities.

This means that cycle detection has been removed from many components. As a result, there may be bugs in the assumptions in components like the printer, for instance, that will still result in cycles. For instance, printing debug strings of values that are being evaluated may be treacherous. In general, though, the idea is that the evaluator clearly marks cyclic points in the graph, forcing an API to make a conscious decision about descending into a cycle.

Please submit bug reports if CUE hangs in an infinite loop or has a stack overflow

Note that although cycle detection is complex, it is not expensive. The implementation is based on unification algorithms that have cycle detection as a nearly free side-effect. The cost is currently minimal and can probably be improved still.

Performance characteristics

CUE now always evaluates an entire configuration value. This is required by the spec to be fully correct. It also simplifies things greatly, and is paramount in supporting certain new capabilities. It does mean that the API may have very different performance characteristics. cue.Unify may do more than expected, while cue.Validate now just collects errors from an evaluated tree so may run a lot faster.

An aim of the new evaluator was to remove any gratuitous exponential computation. The old evaluator could run into exponential behavior for aliases, for instance. The new algorithm is a non-copying unification algorithm: it therefore no longer needs to descend the graph to make copies. This was a performance quagmire in the old evaluator.

The new evaluator has been designed to allow for O(n) execution under certain circumstances (not using comprehensions or disjunctions without discriminators). It has not been implemented that way though! Currently the performance characteristics can be expected to be strictly better than the old implementation, though, barring having a worse constant in some cases.

_That said, there may still be performance bu...

v0.2.2

This is the last release before commencing the move to the new evaluator.

Backwards incompatible changes

yaml.Validate

This now uses Unification again. This brings it in line with the json package.

This change was done to facilitate moving to the new evaluator. This seems no longer necessary. Either way, using subsumption for validation seems wrong. It is better if the desired validation can be specified in the language itself.

Language feature support override

This is the last version that supports :: under the hood and the last release that will allow moving from :: to new-style definitions using cue fix. The cue tool now supports the CUE_SYNTAX_OVERRIDE environment variable to specify the features to allow at a specific version. So even though this feature disallows :: by default, it can be allowed again by setting this environment variable to v0.1.0, for instance.

This will not re-enable features that have been permanently removed.

OpenAPI fixes

OpenAPI is not always compatible with JSON Schema. For instance, it may use an older version. CUE used the new interpretation of exclusive ranges, whereas OpenAPI v3.0.0 and before required the old interpretation. Both interpretations are now supported and are switched based on the selected version (an API feature).

cue completion

This release exposes the Cobra command completion feature. This is currently only tested for bash and is known to not work, for instance, for Fish.

cue get go

This will now generate unexported constants using the _# prefix.

API

Fill

Now allows a cue.Value to be set anywhere in a Go value.

Builtins

list.SliceRunes

Allow slicing of strings per rune.

Embedding in NewStruct

Added ast.Embed helper to specify embeddings in ast.NewStruct.

Testing

Most txtar-based tests now support the CUE_UPDATE environment variable for updating tests. This allows running

CUE_UPDATE=1 go test ./...

to update all these tests at once. It is not possible to use the --update flag for a bulk update, because not all tests implement it, causing such tests to fail.

Changelog

29fa119 cmd/cue/cmd: allow overriding syntax limitation
ab43a15 cmd/cue/cmd: include tool and test files in fix
94db2eb cmd/cue/cmd: include unexported constants for get go
f0f62b8 cue/ast: allow use of Embed in NewStruct
10fbc21 cue/ast: fix typo
a20e1d4 cue/errors: don't equate error messages without line info
0ccdee2 cue/gen.go: fix bug in generation.
1784fca cue: add test cases
097cf16 cue: allow Value to be set at any path in Fill
8f07f84 cue: print hex numbers correctly
ed61df2 cue: return empty struct for disallowed unification
21ca371 doc/ref/spec.md: alternative definitions for closedness and friends
ecfbd72 encoding/openapi: date-time & string format
616d9e8 encoding/openapi: fix generation bug in CRD path
d3c0096 encoding/openapi: update tests
c615c91 encoding/yaml: fix bug where an empty document is not treated as null
9ebfa80 feat: add completion command
16a2b89 internal/cuetxtar: add Load function
97b215f internal/cuetxtar: allow CUE_UPDATE env variable.
8a0ac3e internal: add ToStruct helper.
57f8242 pkg/encoding/yaml: validate concreteness instead of instance of for Validate
f01cfc5 pkg/strings: add SliceRunes

Prep for new evaluator and JSON Schema/OpenAPI fixes

Disabling of previously deprecated language features

The new evaluator will no longer support some of the legacy syntax. Also, it will adopt JSON Schema semantics for bulk optional fields (to be called pattern constraints) and will have more flexible semantics for ellipsis to allow encoding “additionalConstraints” more easily.

To allow for a smooth transition, this release disables the older deprecated semantics and syntax to remind people to run cue fix to upgrade their configurations, as cue fix won’t work for these deprecated features once the new evaluator comes around.

API changes

To facilitate moving the API as well, LookupDef now automatically adds a # prefix if a field cannot be found and is not a valid definition in the new syntax. This is an unambiguous addition.

JSON Schema and OpenAPI fixes

Exclusive minimum and maximum legacy support

The current OpenAPI implementation used the current JSON Schema semantics for representing exclusive minima and maxima. OpenAPI v3.0.0, however, mandates the use of the deprecated encoding. The OpenAPI generator now generates such constraints using the legacy syntax. The OpenAPI generation API now has a Version field that allows enabling the new behavior if set to 3.1.0.

As for now, the JSON schema reader will accept the legacy encoding unconditionally in addition to accepting the standard encoding.

$ref changes

OpenAPI defines that a$ref replaces all other fields in a schema. The JSON Schema RFC seems to allow at least some fields to be alongside $ref and there are some side comment notes that seem to imply that this is explicitly allowed. In order to avoid any confusion, however, $ref is now wrapped in an allOf if it may otherwise cause a conflict.

JSON Schema model fixing for mixed type fields

The original JSON Schema made the erroneous assumption that a constraint for a certain type implies that the value must be of this type. This is not the case. So

type: [string, integer]
minimum: 3
maxLengh: 4

will now be encoded as

strings.MaxRunes(4) | (int & >=3)

cue get go

The cue get go tool now also converts Go constraints encoding the field tags. This bring is in line with the API converters.

The tool now also has a --local flag to allow generating files in the Go package directory, rather than the corresponding cue.mod/gen subdirectory. This is not set to the default as to not break existing behavior.

Changelog

ddc8eb4 .github: disable generation of schema
083afe7 all: update most tests to not use ::
78cc785 ci: fix vendor path of GitHub workflow definitions
c47316f ci: only run gorelease check on latest versions of Go
2f69bfa ci: re-add Go 1.12.x to the build matrix
019060d cmd/cue/cmd: extract cue constraints in tags in get go
4d31efd cmd/cue/cmd: get go: import files to cue module if in main module
8bfdfe3 cue/ast/astutil: add Expr to File conversion (ToFile)
f5315d4 cue/ast: don't require ellipsis to be at end of struct.
197f4c8 cue/ast: introduce NewNull()
53d18cc cue: allow omitting leading "#" for LookupDef
7efeb70 cue: deprecate :: and combining bulk optional fields with other fields
cb21170 encoding/gocodec: Validate helper function
7aa2eb7 encoding/jsonschema: add support for OpenAPI nullable
a91b869 encoding/jsonschema: constraints don't imply types
eb7706b encoding/jsonschema: fix printing of error message
91d267a encoding/jsonschema: record more line information
50b9067 encoding/openapi: wrap $ref in allOf when needed.
2a8b4ed encoding/openapi|jsonschema: allow bool for exclusiveNum
3e62450 go.mod: use github.com/rogpeppe/go-internal/testscript
519db58 internal/encoding: don't simplify interpreted encodings
1d213a3 pkg/list: remove use of :: (Incompatible!!!)
5515ee9 tools/fix: move fix functionality to separate package

Language Changes and Streamlined CLI semantics

This release makes several minor and bigger language changes. The changes are a result of analyzing CUE’s interoperability with other languages and what is needed for this. It is expected that these are the last major changes and that this fixes the look and feel of the language. In addition to the usual cue fmt-based rewrites, we now also provide a cue fix command to aid in the transition. Also, the old formats are still supported for now. Minor language changes and additions to the language are still planned on the path to stability.

The new language changes allow for a simpler JSON Schema mapping. This, in turn, allows for a more straightforward combination of schema and data on the command line.

New syntax for definitions

Before, definitions were indicated with a double colon ::. They lived in the same namespace as regular fields. This property complicated defining automated conversions of some other data formats to CUE. For instance, JSON Schema keeps separate sections for schema and definitions. As there is no enforced convention, as in Go, for naming the two kinds differently, one could not map these to the same struct. This forced schemas to be moved to another location, which turned out to be cumbersome and unnatural.

Another issue with the old syntax is that one could not determine from a reference a.b whether b would refer to a definition or regular field. Not a big problem per se, but it lacked clarity.

As of v0.2.0, definitions are denoted with special identifier starting with a #. As this was not a legal identifier before, such identifiers can safely be used exclusively for definitions. Effectively, definitions now have their own namespace. Other than that, definitions work as before. They still close a struct, they are not output during export, and you simply use their identifier to refer to them.

Before:

A :: b: int
b: c :: string

D: A
E: b.c

After

#A: b: int
b: #c: string
D: #A
E: b.#c

The #c notation is just an identifier: no special syntax is needed to handle them. As before, "#foo" still denotes a regular field with a name starting with #.

Using an initial # to distinguish between definitions and regular fields was inspired by Go's use of initial casing to distinguish between exported and non-exported identifiers. Indeed because CUE is a JSON superset and interop focused, there is little control over casing. As we explain later in the Hidden Fields section, we also resurrect _ as a means of excluding fields and definitions from output.

The new notation may take a bit of getting used to, but the increased readability and the flexibility from the separate namespaces are big wins. So far the feedback has been overwhelmingly positive.

The Go, Protobuf, JSON Schema, and OpenAPI mappings, as well as the tutorials, have been ported to use #-style definitions. Please beware that if one has other templates based on this they need to be rewritten as well.

Limitations

The new definition syntax is more restricted than the old syntax. In general, it is no longer allowed to have dynamic definition names. For instance, the syntax disallows using interpolations for creating names. Also bulk optional fields only apply to regular fields. We think this actually benefits static analysis and is overall a good change. If need be, though, the old semantics can be simulated by containing a struct within a definition used as a map. In case this proves to be too limiting, we have a possible language change up our sleeves that would allow dynamic definitions again.

Old style definitions will keep being supported up to the next minor release. Use cue fix to rewrite current files (see below).

API changes

The current API assumed definitions and fields lived in the same namespace. The current API is therefore inherently broken. The Struct.FieldByName has been deliberately broken to force users to disambiguate. To evaluate a reference, it is recommended to use Dereference instead of passing the result of Reference to Lookup.

There are some good ideas to make the API considerably simpler and more powerful. This is contemplated to be built on top of the new adt package, which is developed as part of the evaluator rewrite, which is planned for the next minor release.

cue fix

To aid in the conversion to #-style definitions, the cue fix command is introduced. This command does a module-wide update of packages by default, but allows specifying individual files and packages.

This is no longer piggybacked on cue fmt. Unlike previous rewrites, the definition rewrite requires evaluation. In addition, not all current definitions are legal in the new syntax. The converter will not remap those. Also, unfortunately, due to API limitations, the converter can also not handle all legal cases. We hope that such cases will be rare, though.

Cases that cannot be handled are clearly marked with locations that may reference the definition included. Definitions with issues have are marked with a deliberately obscure @tmpNoExportNewDef attribute, indicating they need manual fixing

There is also a possibility there will be undetected breakage. This may happen if a selection is made in a disjunction where the field may be either a definition or regular field. The fact that this is even possible indicates that the old model was probably too flexible and possibly more bug prone. With new-style definitions such shenanigans are no longer possible.

Unfortunately, cue fix currently does not handle updating multiple packages within the same directory at once unless manually specified. We intend to allow this at some point.

Tool semantics changes

Previously, the cue tool handled schemas and data differently. In practice, the distinction is not all that clear, making this distinction somewhat “forced”. The cue tool now treats data and schemas as the same thing and always unifies them without distinction by default. This greatly simplifies its use and makes it a very powerful tool for operating directly on non-CUE schema.

There are two exceptions to the new “always unify all files” rules: 1) with cue vet multiple data files are still individually verified against a single schema. 2) the -d option still separates schema from data files. One could argue that the latter isn’t really necessary anymore, as we will see next.

JSON Schema mapping

The following command

cue export schema.json data.yaml

converts the schema.json (which has a proper value for $schema to detect the format) to a schema and data.yaml to data, and unifies the results. This will naturally fail if the contents of data doesn’t correspond to the schema. There is no special logic needed to detect these cases, other than knowing when to interpret a data file as JSON Schema.

Note that #-style definitions were key to make this possible. Without them, there would be no obvious way to map JSON Schema to the root of a config as merging fields and schema to the same namespace could result in conflicts.

OpenAPI merging

Another example of the new capabilities is that one can now merge schema as one would merge data. For instance, the following command merges two OpenAPI files and then outputs it again as OpenAPI:

cue def openapi: file1.json file2.json --out openapi

The user won’t even see the intermediate CUE.

Hidden fields

This release also officially resurrects hidden fields (identifiers written as, for example, _foo). They were removed from the spec as with the introduction of definitions they were believed to be an incongruent feature. They fit, however, quite nicely with the new-style definitions. They were still in use and proved to be more useful than expected. As they were never really removed from the implementation nothing changes. However, their comeback is now official.

For clarity, both hidden fields and definitions are not shown in exported output. They serve different functions, though.

A definition defines a new composite type, the most general specification of something. They are at the other end of the spectrum to concrete instances. As definitions are supposed to fully define a schema, CUE can use them to catch typos in field names by detecting fields that are not supposed to be there.

A hidden field can be any value and is not subjected to such scrutiny. They are used to define fields that are not converted to output without making them a definition.

Although this is not yet enforced, hidden fields are local to a package. To make the matrix complete identifiers starting with _# denote hidden definitions, or definitions local to a package. These can be used today, although this restriction is also not yet enforced.

Streamlined syntax for list comprehension

List comprehensions are now written as

[ for x in src { x + 1 } ]

This brings them in line with field comprehensions, where the value also comes after the comprehension clauses. Overall, this greatly increases readability.

This also harmonized the syntax. It simplifies both spec and implementation, and prepares for some of the constructs mentioned in the query proposal.

The one thing that may require some getting used to is the curly braces alluding to the value being a struct. If the curly braces contain a single scalar value, however, it is promoted to be the result of an iteration. CUE users may already be familiar with this construct at the top-level scope, where “embedded scalars” are the final result of evaluation. This construct was originally invented to allow omitting curly braces, while still being a superset of JSON (JSonnet uses the same trick). It turns out, however, that this trick can be quite useful when generalized to apply to any struct, not just the top-level one. We continue to investigate this possibility. Con...

JSON Schema and OpenAPI fixes and improvements

This release contains several fixes and enhancements related to CUE conversions.

It fixes several issues with the related to JSON Schema conversion, including an incorrect implementation of the JSON Schema spec. Most notably, one may not infer from a constraint for a certain type that a value must be of that type. This means that if a schema, for instance, defines a properties section, but does not specify the type to an object, the resulting CUE value will be a disjunction from this object and all non-object top-level types. This is not a bug.

Another change is that the cue tool now may silently ignore imprecise conversions, as long as the produced result does not give false negatives. The --strict option will emit the original errors.

Changelog

0059b2b cmd/cue/cmd: allow lossy conversion by default and add --strict option
a38e811 cmd/cue/cmd: fix bug with help flag
b99d364 encoding/jsonschema: add several validators
bcc19ce encoding/jsonschema: encode minItems differently
78e21c1 encoding/jsonschema: fix type exclusion logic
ed4fc4d encoding/jsonschema: wrap comment lines
904f561 encoding/openapi: remove unused code
9b62b36 encoding/protobuf: fix required option parsing
efb89b8 github: fix release.on to push.tags

Stability and cosmetic improvements, can use CUE as OpenAPI encoding

The release fixes a critical regression that affected cmd eval as well as various other minor regressions and documentation and formatting issues.

Support for space-separated labels and old-style field comprehensions is now disabled by default. The intention is to remove these entirely soon.

This release now also allows OpenAPI inputs to be expressed in CUE itself, instead of just JSON or YAML.

Changelog

6f37a71 cmd/cue/cmd: fail earlier on faulty flags
83b0996 cmd/cue/cmd: fix newline discrepancies
cd2b3ef cmd/cue/cmd: fix regression that allowed space-separated fields again
d6fc6be cmd/cue/cmd: link commands help text
c5fc236 doc/tutorial/kubernetes: update to minimize diffs down the line
8dcec3c internal/encoding: allow CUE as OpenAPI input
d6f1eec internal: fix crash affecting eval

Streamlined and enhanced CLI capabilities

This release introduces various improvements to the cue command, including a streamlined CLI interface and improved interoperability between different file formats.

Many of the command-line flags were previously rather arbitrarily implemented for some commands and not others. This release fixes most of that. This release also makes it possible to convert between different formats, such as YAML, JSON, JSON Schema (input only), OpenAPI, and Protbuf (input only).

This release also starts a new versioning strategy. CUE exists of several fairly independent parts. Each next milestone focusses on a one of these parts. This round it was the cue command-line tool. Next round will be the evaluator. Each next minor version will correspond to one such milestone. This is not to say that upgrades and bug fixes cannot happen to other parts within a cycle—they can—but it helps focus.

The new strategy will also make it easier to do intermittent releases. We will use the minor releases to signify big additions and backwards-incompatible changes. Patch releases may include backwards-incompatible changes that include bug fixes or disabling previously deprecated features. We intent to follow this trategy until providing a general backwards-compatibility guarantee for v1.0.0. As usual, we intend to provide a smooth migration path for each incompatible language change.

As of this release, a Docker image is generated as part of the release process.

Define file types

A frequently asked feature was to be able to interpret files as anything. CUE now allows qualifiers to "cast" a file to any supported format. For instance,

$ cue eval json: foo.data

interprets foo.data as JSON and then prints it as CUE. This also shows that the input for eval no longer needs to be CUE but can be any format that is convertible to CUE.

This means that CUE can now also be used between non-CUE formats. For instance,

$ cue export foo.json -o foo.yaml

will convert JSON to YAML.

Packages can now be qualified by package name (for instance ./foo:bar), not just directory.

Run cue help inputs and cue help filetypes for more information.

Streamlined flags

The import command already had a set of flags that maps JSON and YAML files to a prescribed location within a CUE configuration. These flags are now also available for many other commands so that JSON and YAML files, as well as other formats, can be processed by the cue command without first having to convert them to CUE.

For instance,

$ cue eval -l 'strings.ToLower(kind)' -l name foo.yaml

puts any document in foo.yaml at a path based on its kind and name fields.

Also the output flags (file and format), among others, have been harmonized across commands. This sometimes led to backwards-incompatible changes.

Run cue help flags for more information.

CUE as a data format

The qualifier mechanism can also be used to restrict what language constructions can be used in .cue files. Most notably,

$ cue eval data: foo.cue

specifies that the file foo.cue may only contain data fields, but no references, definitions, builtin invocations, or any other CUE construct other than simple data. In other words, this allows CUE to be used as an alternative JSON or YAML format.

In the future we may add a file extension to indicate this CUE dialect.

New cue def command

The cue eval command was configured to generate cue that seemed pleasing to read to humans, but it would generate something that is neither useful as data nor as a schema. cue export could already be used to generate data, but there was nothing to conveniently generate a schema.

cue def is also the command of choice converting schema-based formats to CUE.

Value injection

Many users have requested a convenient way to defining environment-specific variants of configurations and ways to inject values based on environemt variables and command line flags.

We have added support for environment variables at various points in the tooling layer. Due to their non-hermetic nature, though, environment variables are not an appropriate mechanism for CUE configurations in general.

An important criterion for allowing such injection was retaining the ability to analyze configurations and make intent of how to use a configuration explicit. Simultaneously, we did not want to impose on the structure of a configuration.

Our solution is to use attributes to explicitly mark injection points. A field that with a "tag" attribute of the following form:

environment: "prod" | "staging" @tag(env)

can be set to "prod" on the command line using -t env=prod. The tag attribute may also explicitly define some shorthands. For instance, by specifying

environment: "prod" | "staging" @tag(env,short=prod|staging)

the same can be accomplished using -t prod.

This mechanism makes it very clear to both the user and the cue tooling which non-concrete values may be set by the user and which should be resolved by values within a configuration.

In the future we may consider to also allow whole-file injection analogous to Go build tags.

This release does not include support for command-line flags in the tooling layer. This is still being considered for a future release.

Run cue help injection for more information.

OpenAPI support

The cue tool now has basic support for converting OpenAPI definitions to CUE. This is a first stab, does not yet support external references, and is not as feature rich as the CUE-to-OpenAPI Go API that is used by Istio to generate its OpenAPI. But it should be a good start.

OpenAPI definitions can be imported using cue import or just used on the fly using cue eval or cue def. OpenAPI can be defined in JSON or YAML. For instance:

$ cue def openapi: foo.yaml

will interpret the foo.yaml file as OpenAPI and generate corresponding CUE.

It is also possible to convert CUE to OpenAPI. OpenAPI can be represented in JSON, YAML, or even CUE. For instance,

$ cue def ./pkg -out openapi+yaml

writes the evaluated CUE as OpenAPI in YAML format to stdout.

JSON Schema support

The cue tool has very basic support for mapping JSON Schema to CUE. This is still experimental and the converse (mapping CUE to JSON Schema) is not yet supported. The typical conventions followed in OpenAPI make it easy to find a mapping from OpenAPI to CUE. The same does not hold for JSON Schema. This is partly complicated by fields and definitions sharing the same namespace in CUE. Also, the top-level schema of JSON Schema is unnamed, making it hard to find an appropriate destination.

There is a proposal for CUE to have definitions and fields live in a separate namespace. This will make such a mapping considerably easier. For now, consider the mapping experimental.

As with OpenAPI, external references and anchors are not yet supported.

Auto detection of file types

The cue tool now interprets files with a .json, .yaml and .yml extension as either data, OpenAPI, or JSON Schema based on markers in the data. For JSON Schema this is a $schema field that refers to a schema on json-schema.org. OpenAPI is selected if a file contains a top-level openapi field with major version 3, an info.title, and an info.version field.

The use of qualifiers disables auto detection. So auto detection for a JSON file, for instance, can be disabled using a json: or data: qualifier.

cue import modes

The cue import tool now has import modes that specify which kinds of files to import. The default is the old default: JSON and YAML files, but with auto detection enabled.

Run cue help import for more details.

Protobuf support

The new cue import proto command can import Protobuf definitions.

A successful run imports all transitive dependencies of a .proto file into a CUE module: .proto files located within the module are converted to a CUE package at that location. The package name is derived from the go_package option, if present. Imported .proto files that can be found in a given include directory are put in the corresponding path in the cue.mod/gen directory.

The existing Protobuf API has been modified to generate CUE types hierarchically (instead of mapping all files to the top level) to make them closely correspond to the Protobuf definitions.

Protobuf definitions can be annotated with CUE-specific options to specify additional constraints for fields.

Command and Task changes

The task section in commands is now optional. A task can be included anywhere in a command. So commands with a single task can now be written as

import "tool/cli"

command: echo: cli.Print & {
   text: "Hello world!"
}

The cue tool will look for tasks anywhere nested within the run command where each task is considered a leaf node in this nesting.

Tasks can also be defined at the top of the file. These will only be run if they are referenced by a task defined in an executed command

Sometimes task must run after another task, even though it does not depend on any of its outputs. The $after field can be used in that case to explicitly mark a dependency.

See https://pkg.go.dev/cuelang.org/go@v0.1.0/pkg/tool?tab=doc for more information.

Vet changes

In addition to the flag restructuring, it is now possible to check a data file against a CUE package.

A backwards-incompatible change is that one now needs to use the -d flag, instead of the -e flag, to select a schema against which to compare. The -e flag was used by other commands to mean something different and this was cleaned up for consistency sake.

Trim safeguard

The cue trim command now checks ...

Bug fixes and stability improvements

This release focusses on bug fixes and stability improvements.

There is one language change, and that is to allow the $ character in identifiers. The main reasons for this is that it is a commonly used character in identifiers in JSON. The single $ is currently reserved.

Changelog

6f9c049 Add YAML support to export command
503fecd Decode null as NULL token in yaml decoder
c4f74cc Revert "cue: fix lookup for cue API"
8a043dc Use baseValue from source in binOp to make gc working
4dc023b Use cache during walk
170697a cmd/cue/cmd: also make field optional if it is a pointer
d372ba8 cmd/cue/cmd: convert package block comments in cue get go
d100ae6 cmd/cue/cmd: small help command fixes
a9fdbe7 cmd/cue/cmd: track use of indexing op for dependencies
f16c065 cue/format: improve indentation of in comprehensions
8e72f0b cue/load: cleanup and made code more defensive
ff96e8d cue/parser: improve error message for misused keywords
8a2df96 cue/scanner: allow $ as a letter in identifiers
b3cde9d cue: Reference handles concrete values for index operations
3a2e63b cue: always check for errors before calling builtin functions
630935e cue: detect errors in struct when comparing to bottom
afdbec3 cue: disallow references to string labels (per spec)
5c2d937 cue: don't turn of manifesting in normalization
324cc0f cue: don't validate optional fields
279ca6a cue: fix bug related to non-structural direct self-references
13628f1 cue: fix compilation bug using $foo identifiers
3d0b204 cue: fix crash on alias use in lists
40d2667 cue: fix exporting of label aliases
c937e7a cue: fix lookup
353ede2 cue: fix lookup for cue API
8bd2426 cue: fix lookup for cue API
52b04d4 cue: fix optional handling in subsumption
6cd62ea cue: ignore optional fields for Equals
5a0a3f2 cue: no quotes needed for identifiers with $
d256a2c cue: prevent regression to bug in older version
030b9b7 cue: remove uses of log.Panicf
d7d2837 describe the go get step before cue get go in the tutorial
c8d6c39 doc/ref/spec.md: clarify newline behavior for multiline strings
c6e5d17 doc/ref/spec.md: fix typo
4d29dde doc/ref/spec.md: simplify attributes
0e038f6 doc/tutorial/kubernetes: update to latest
3960473 doc/tutorial/kubernetes: update to use new syntax consistently
8f9ef31 doc/tutorial/kubernetes: update to v1 versions
815dca2 doc/tutorial: fixing bad link in string literals
80ffe27 doc/tutorial: removing duplicate word in bounds
5b4fa8b doc: fix typos in language spec
085ce9c fix typos in docs
31cd2b7 internal/diff: first stab add diffing functionality
5cb6ffe internal/yaml: import null value properly
7fdff9c pkg/math/bits: complete operations
7334a13 pkg/math: use higher precision implementations
6ffe55d tools/trim: fix spurious removal of top-level declarations
cba0372 tools/trim: move trim logic to package