Newline-Delimited Inner Tables

[eksortso]

This PR defines a new inner table value syntax. It is compatible with TOML v0.5.0 and the standard to date, and I hope to include it before the release of v1.0. This closes discussion on #525, though further comments are invited.

[std]
key1 = "std.value1"
key2 = "std.value2"

# An inner table may be defined anywhere that an inline table is allowed.
# To open the table, end a line with "{". Comments and whitespace are optional.
table3 = {
    # Key/value pairs are separated with newlines. Commas are not permitted.
    subkeyA = "std.table3.valueA"
    subkeyB = "std.table3.valueB"
    subkeyC = "std.table3.valueC"

    # Inline table values and dotted key paths are allowed. Nested inner tables
    # are also permitted, but ought to be discouraged.
    subZ = {A = "std.table3.subZ.valueA", keyB = "std.table3.subZ.valueB"}
    subkey_0.subsubkey_0 = "std.table3.subkey_0.subsubvalue_0"
}
# To close the table, begin a line with "}", with optional whitespace.

# Assignments to the standard table may resume afterwards.
key4 = "std.value4"

README.md still needs to be updated, and I would like to include some useful
examples in appropriate places.

[ChristianSi]

I'm not strictly against this, but I doubt it meets TOML's goal of being "minimal". Since we already have regular and inline tables, introducing yet another table type smacks of Perl's "There's more than one way to do it" (TIMTOWTDI) – not necessarily a bad thing, but the very opposite of minimalism. Moreover, the case for inner tables seems week, since converting them to regular tables is trivial. Consider your example rewritten:

[std]
key1 = "std.value1"
key2 = "std.value2"
key4 = "std.value4"  # all contents of a table must be keep together -- not a bad
                     # thing for readability

[std.table3]  # Full table name instead of local key part required -- might be a bit
              # verbose sometimes but also advantageous for readability since it's
              # always very clear where in the table hierarchy you are
# Contents of  [std.table3] omitted -- identical to above example
# Closing "}" is NOT required

[eksortso]

It is intended to behave similarly to how inline tables work. The key/value pair syntax is exactly the same as that used in regular table syntax. And converting to a regular table prevents the table from being introduced surrounded by an appropriate context. Refer back to #525 for more substantial examples where context is maintained with the use of inner tables. In addition, defining an array of tables with this syntax is much less confusing than with the current syntax, and I'm not alone in making that assessment.

[bitwalker]

I was curious, so locally I changed my parser to allow newlines inside inline table braces, while still requiring commas between key/value pairs, like the following (based on the OP):

[std]
key1 = "std.value1"
key2 = "std.value2"

table3 = {
    subkeyA = "std.table3.valueA",
    subkeyB = "std.table3.valueB",
    subkeyC = "std.table3.valueC",

    subZ = {
        A = "std.table3.subZ.valueA", 
        keyB = "std.table3.subZ.valueB"
    },
    subkey_0.subsubkey_0 = "std.table3.subkey_0.subsubvalue_0"
}

key4 = "std.value4"

It seems to me that this alone at least makes inline tables a little nicer (you can put each element on it's own line), while basically preserving the 0.5.0 spec with one small adjustment (allowing newlines within the braces).

The downside of course is that you can basically convert an entire TOML document into one big inline table, and hey, now you've got JSON+, and it barely looks like TOML, which seems to kind of defeat the point. That said, inline tables are already there, the really artificial limitation of disallowing newlines in them seems like a fairly weak attempt to keep people from using inline tables that way, which doesn't seem like a good justification. Additionally, if the point was to avoid more than one way of doing things, why do inline tables exist in the first place. The answer of course is convenience, and this is basically debating where the line is - with that in mind, I'd suggest tweaking this proposal to simply allow newlines, which for just about any parser that should be a really really trivial change. It provides an easy way to make inline tables "prettier" when desired, without making the spec go out of it's way to allow dropping commas as well.

[eksortso]

@bitwalker Much of this stuff was hashed out in #525, if not earlier. Since I don't want TOML to just be JSON with INI-like tables (and inline tables are basically stealth JSON), I tried my hardest to avoid the obvious problems with potential multiline, inline, table expressions:

• This syntax explicitly allows at most one key/value pair per line. Same as regular tables. Not the same if we just allowed newlines in inline tables.
• Newlines separate key/value pairs, not commas. For multi-line table definitions, this is the correct way. We can save the commas for arrays.
• Comments need to be allowed before the newlines. (I take it that you discovered this need already.)
• And if, say, we permitted an optional comma after key/value pairs in inner tables, what prevents us from permitting the same in regular tables? Just kill that thought. No commas.

So that's essentially why inline tables and multi-line inner tables need separate syntaxes, even though I wish them to be treated as the same thing: table values. The ABNF here explicitly calls them that. Their use is intended to be fairly limited, but it's up to configuration designers to see to that.

One last thing, and sorry to have to point this out, but your parser appears to be putting key4 inside the root table, and not inside std, where it belongs.

[bitwalker]

@eksortso I think you misunderstood my point (but thanks for pointing out the bug in my experiment, I was tweaking things and introduced that by accident), which is that commas should be required, not optional, simply allowing newlines (and yes, whitespace/comments) in inline tables. I would have chimed in on #525, but I missed that discussion unfortunately, so I'm having to chime in now :(

This syntax explicitly allows at most one key/value pair per line. Same as regular tables. Not the same if we just allowed newlines in inline tables.

Is there a particular value in allowing only one key/value pair per line? I'm not clear on the benefit of this particular point, since inline tables explicitly allow multiple key/values on a single line. Now we're introducing another table variant which only allows one per line, like regular tables, but on the surface appears and works similar to an inline table (i.e. used in a value position, uses braces as delimiter). I find it unnecessarily confusing to have effectively three variants of table syntax to accomplish the same thing.

Newlines separate key/value pairs, not commas. For multi-line table definitions, this is the correct way. We can save the commas for arrays.

Newlines don't separate key/value pairs in inline tables, commas do - and this syntax/proposal looks an awful lot like simply multi-line inline tables. Using commas to delimit key/values, and allowing newlines in inline tables, effectively builds on syntax people are already comfortable with in the existing specification, does not add a new variant of table syntax, and solves the problem statement in #525 (i.e. "But there does exist a real need, found in the wild, for tables to be defined, mid-section, in order to maintain a configuration file's logical flow with a minimal amount of repetition.") . As an additional benefit, the grammar change for this is very minimal, making it much easier to introduce between 0.5.0 and 1.0.

I saw your statement in the other thread that allowing multi-line inline tables is too JSON-like, and your statement in your last reply to that effect, but I fail to see how this proposal is meaningfully different if that is the intent - it differentiates in a way that is superficial (i.e. using newlines rather than commas as a delimiter in a syntax which is intended to represent an inline object). Syntax already exists in the current spec for this purpose, and the actual problem people have with that syntax is that it reads poorly due to the single-line requirement. Additionally, there are other parallels to the other inline data structure, arrays, namely that multi-line arrays are supported by simply allowing newlines between elements, but still requiring commas. If, as stated in #525, there is a need to allow multi-line tables to be defined inline, then I think keeping the specification simpler is a better way to move forward - it is less cognitive overhead (only having to remember the rules for two table syntaxes rather than two + a third which is a blend), and is easier to parse.

Comments need to be allowed before the newlines. (I take it that you discovered this need already.)

I didn't do anything special in my parser to support comments before the newlines, it just worked because of the way it is structured, but yes, the implication would be that if newlines are allowed, comments could easily be introduced on their own lines or at the end of lines - the parser simply discards either of those, rather than erroring on them like it would today, and pays attention only to keys, values, and delimiters (open/close, and separators).

And if, say, we permitted an optional comma after key/value pairs in inner tables, what prevents us from permitting the same in regular tables? Just kill that thought. No commas.

To be clear, my proposal is that commas continue to be required in inline tables, and only in inline tables, the way they are today in 0.5.0. I don't see any reason why you would add them to regular tables.

So that's essentially why inline tables and multi-line inner tables need separate syntaxes, even though I wish them to be treated as the same thing: table values. The ABNF here explicitly calls them that. Their use is intended to be fairly limited, but it's up to configuration designers to see to that.

I agree they need distinct syntaxes, but I don't agree that we need yet another new syntax for tables, I think we already have the syntax we need. I do agree that using inline tables should be reserved for cases where they are needed, but I don't think the specification should artificially restrict the formatting of inline tables to make it uncomfortable to use them, in the hopes that this will drive people to find other ways of expressing their problem. If someone is reaching for inline tables, it is because the regular table syntax is already not a good fit, and if formatting those inline tables requires multiple lines to be more readable, I think the specification should support that - ultimately what we all want is a readable, clear, and unambiguous document format.

I apologize for coming late to the party, if I didn't firmly believe this was worth pushing back on, I'd just leave it. For what it's worth, I'm watching the repo now so I can participate earlier on in future discussions.

[eksortso]

Since nobody cares about this feature, I am abandoning it. If I couldn't even garner the verbal support of the users who raised the issues that led to it, it is not worth my time or effort.

For now, anyway. If the same issues resurface after TOML 1.0 comes out, then I'll try again.

[pradyunsg]

Thanks @eksortso for spending the time and effort to investigate this! :)

[LongTengDao]

@eksortso I think multi-line inline table is a really good proposal! Maybe it could be free for users about using comma

[eksortso]

The thought of mixing commas and newlines causes me concern still, but thanks for supporting the main idea.