Sub-index pages for specific categories of PEPs (Packaging, etc)

[pradyunsg]

See https://discuss.python.org/t/14111/, where this was discussed initially and where folks have been generally in favour of this.

It would be valuable to be able to list PEPs related to Python Packaging, in a single location in addition to having them in the PEP 0 index. While packaging.python.org could have such a list (and it does, to an extent), that has an additional maintainance burden (it needs to be updated each time a new PEP is proposed). It would be more beneficial to be able to provide such a listing under peps.python.org. This has discoverability advantages, will be easier to maintain (since the PEP sources themselves will note that it's a Packaging-related PEP) and avoids needing to update two spots when a new Packaging-related PEP is proposed.

[pradyunsg]

A concrete proposal for implementing this, roughly based on the concensus in the aforementioned thread, is:

• Add an optional Track header for PEPs, that can have only one value for the initial implementation: Packaging.
  • Add this header to existing Python Packaging PEPs.
• Add a track-specific index page, that lists all PEPs with that track: peps.python.org/track/packaging
  • This page should have a different header and introduction section than PEP 0.

The Track header can be expanded to other values in the future (Typing and Release are examples of proposed ideas from the aforementioned thread).

Personally, I think it's valuable to implement this for just Packaging as an initial pass since it's a viable "minimum" implementation of this mechanism and... perhaps more importantly... avoids an extended discussion on what these other values should be allowed/updated for. :)

[AA-Turner]

I'm supportive of the idea.

Implementation-wise it wouldn't be challenging & happy to pick it up if we agree.

I'd suggest using the index namespace, as it's recognised that PEP 0 is the index, and this is in effect a 'sub-index' -- so peps.python.org/pep-0000/packaging.

A

[brettcannon]

I support this.

I'd suggest using the index namespace, as it's recognised that PEP 0 is the index, and this is in effect a 'sub-index' -- so peps.python.org/pep-0000/packaging

I actually don't know if most people realize there is a PEP 0 and that it's the index.

[pradyunsg]

I'd suggest using the index namespace, as it's recognised that PEP 0 is the index, and this is in effect a 'sub-index' -- so peps.python.org/pep-0000/packaging

I don't have strong feelings on the exact URL, so whatever PEP editors prefer works for me!

[AA-Turner]

I actually don't know if most people realize there is a PEP 0 and that it's the index.

Fair enough -- I suppose I'm approaching the issue somewhat with an implementation bias, too.

But if that's the worst thing we can argue about... 😉

A

[hugovk]

I also support this!

peps.python.org/pep-0000/packaging looks a little odd as well, something like peps.python.org/track/packaging feels a lot more intuitive.

[pradyunsg]

Looking at the implementation generating the index page, currently, PEP 0 is generated in pep_sphinx_extensions/ by adding a hook that generates the entire file in-memory. This file is then populated with the relevant content, as generated in PEPIndexWriter, written and then added to the Sphinx's various internal lists to ensure that it builds it appropriately.

It would be feasible to have pep-0000.rst be a regular file that's checked into the repository, if it used a custom reST directive that populates the index part of the file. The header and introduction can then be kept in the source, as prose, outside of the Python code (they're in a writer.py file currently).

This could make having a separate index easier, since the parts that need to differ (header and introduction) can live outside the Python generator, which would also (possibly?) simplify to "only" have a directive for the index generation -- i.e. hook into Sphinx's build system to a lesser extent.

I'm not super familiar with the Sphinx-based PEP generation code though and what constraints it was written with, so if this is not feasible for some reason, I apologise for my ignorance. 😅

PS: PEPIndexWriter doesn't actually write the file. It might be a good idea to rename it to PEPIndexGenerator or something. :)

[JelleZijlstra]

Another bikeshed :)

"Track" may not be the best term, because it's really only applicable to packaging, where PEPs go through a very different process from other PEPs. Typing PEPs are on the same "track" as regular PEPs, because they get discussed on python-dev and approved by the SC just like any other PEP. Alternative suggestion: "Area". Then we could have headers like "Area: Packaging", "Area: Typing", "Area: Release", and automatically generate pages for each of these areas.

For now we'd start with just "Area: Packaging", and if the community is interested in a new Area, we could add it later.

[hugovk]

There is a bit of overlap in naming here:

The PEP Type can be Standards Track, Informational, or Process.

https://peps.python.org/pep-0001/#pep-types

Track is only applied to one of the types, but track does also feel the right name for this new field. Maybe it doesn't matter?

[AA-Turner]

Topic? Area also seems reasonable.

We could also be very boring and have Index:.

A

[AA-Turner]

I re-titled the issue as although packaging will be first, the design needs to be sufficiently general for other potential sub-indicies.

A

[Rosuav]

+1 on having subindexes, not overly fussed about keyword. I like "index", honestly, but "track" is fine too.

[pradyunsg]

As long as we don't expand the scope for the first implementation beyond packaging, that works for me.

I do notice that there's two aspects (header name and URL) and we've basically headed straight into bikeshedding them both. I guess that's a good sign (at least, folks aren't opposed to the core idea!). Since we're bikeshedding, listed in order of my personal preference:

• Track: has an overlap with Standards Track, first proposed for something like this in 2019 (https://discuss.python.org/t/2645/4 ¹) and was also used throughout the discussion that led to this issue². Familiarity bias likely applies here.
• Area: works for me -- it's fairly similar to Track, except doesn't overlap with any existing vocabulary for PEPs (which probably doesn't matter).
• Topic: also works... but feels less appropriate than Track or Area.
• Index: feels odd to me, it's going to be listed in at-least two indexes.

I'll personally avoid engaging in futher bikesheding though -- all the options proposed so far are all good-enough as far as I am concerned. :)

---

Both @AA-Turner (in this issue) and @CAM-Gerlach (in the d.p.o thread) have said that they'd be willing to implement this -- so instead of adding my name to the hat, I'll share this tiny bit of incomplete work I've done already, adding the Track header and exposing it via the JSON "API" and let one of them pick up the rest of the work. :)

Footnotes

1. This thread was a governance thread and PEP 609 has happened since then. ↩

2. Which makes me think it was inituitive to the group that was having this discussion. ↩

[CAM-Gerlach]

I originally proposed Category on the Discourse thread, but later went with Track that others suggested since it was more specific to the packaging use case and didn't implicitly require categorizing other PEPs (or it would look a little out of place) and implied the differences in process surrounding them, while it could be extended to other specialized subdomains (typing, etc) if desired in the future. But that's somewhat moot if we want to apply it to other topic areas too beyond packaging in the immediate future, and also to some extent if we don't actually display it in the headers on individual rendered PEPs, and "Track" limits extensibility to other domains that don't have a modified process. So I'd prefer Category or Topic.

Comments on naming:

• +1 Topic, IMO best captures what it seems like people want this to represent in the general case—the the topic area of the PEP
• +1 Category is similar to Topic, but seems a little more "weighty" which may or may not be what we want to imply, but people might prefer it
• -0.5 Track implies a difference in process (true of packaging, but not all others) and also conflicts with "Standards Track"
• -1 Area I find very vague and a lot less obvious, clear and specific as to what it means, as it has several other meanings and is only unambiguous in the context of phrases like e.g. "topic area"...so why not just use something more specific, like Topic or Category?
• -1 Index seems very odd in the context of a PEP header, because the fact that it displays in a sub-index is really an implementation detail, not high-level metadata. It would only make sense if we hide it in the rendered PEP, and even so could confuse authors.

As for the URL, I think all but one of us here and on the discourse thread more or less agreed on peps.python.org/<headername>/<topicname>, which is clearest and easiest to type, and doesn't rely on implementation details, so I think the main question to be resolved is the header name.

I analyzed and outlined the specifics of what a MVP implementation would require on the Discourse thread, basically:

• Add the header to those parsed in parser.PEP.__init__
• Filter the PEPs by track and feed them to the PEP generation and writing code in a loop in pep_index_generator.create_pep_zero, plus modifying the filepath generation logic accordingly
• Tweak writer.PEPZeroWriter.write_pep0 to accept a track arg that controls using an appropriately modified intro paragraph
• Add it to any hardcoded logic elsewhere (e.g. JSON generation)

and it seems like @pradyunsg has already starting on implementing that. The only really non-trivial part is the subindex generation, which requires a few changes to the current logic.

Additionally, the following would also need updates:

• List and describe it in the header lists in PEP 1, PEP 12 and the template
• Add a simple linting check to the .pre-commit-config.yaml that it has a valid value (currently just Packaging, unless we want to decide on additional standard categories/topics now)
• Add it to the relevant PEPs

I agree with the overall rationale to check PEP 0 into the repo (again) and just inject the specific parts needed, if that makes things simpler and we'd otherwise have to add the new subpages multiple places, though it would make generating sub-indexes perhaps a bit more complicated in other ways since much of the content would vary between indicies, so I'm not sure if it is best to do as part of this change. But I'd defer to @AA-Turner on that.

I can also help implement this since I've reviewed the relevant code and what needs to be done, once we have consensus on the specifics of the implemntation.

[pradyunsg]

I wouldn't block implementing this on waiting on concensus on the naming + URL -- they can both be changed with a find-replace. :)

[pradyunsg]

IIUC, everyone is OK with the URL being peps.python.org/<header-name>/<header-value-lower-case> (only @AA-Turner wanted something different, and has since implemented this approach in #2579). If that's not the case, please holler!

If anyone has strong opinions on what name to pick amongst the ones proposed so far, for the PEP header name, now is a good time to state them. Specifically, if anyone feels strongly against Topic, please say so! :)

[ncoghlan]

I haven't checked the draft implementations, but I'd suggest that peps.python.org/<headername>/ be an auto-generated list of the track pages.

I'd still advocate for "Track" as the header name, specifically because people are less likely to assume they know what it means and instead look it up. There will only be a limited number of specifically defined tracks, and it's intended primarily as a subcategorisation of "Standards Track" PEPs (although we'd presumably allow using it for other PEP types).

"Topic" feels weird to me, as "What's the topic of PEP NNN?" already has an answer (the title field), and I don't consider it immediately obvious that "Topic" is short for "Primary Topic Area".

"Category" would probably be OK, but seems a bit easier to confuse with the existing "Type" field than "Track" would be.

[CAM-Gerlach]

I'd still advocate for "Track" as the header name, specifically because people are less likely to assume they know what it means and instead look it up. There will only be a limited number of specifically defined tracks, and it's intended primarily as a subcategorisation of "Standards Track" PEPs (although we'd presumably allow using it for other PEP types).

I think there's an important conceptual distinction (and decision) to be made here. A few specifically defined, special-purpose tracks (e.g. Packaging, Typing, etc) covering specialized domains and having a defined meaning and some degree of implied procedural weight sounded closer to the initial idea that was discussed on the Discourse thread, which was why "Track" initially made sense to me.

However, it seemed the discussion on this issue evolved more toward wanting something closer to a general-purpose, broader and more purely editorial "Category" label to organize PEPs by topic area; under that model, all PEPs could conceivably have a category, and potentially even belong to more than one.

As a practical matter, both serve the initial motivating purpose (a mechanism to label and display packaging PEPs separately) essentially equally, and realistically it wouldn't make sense to add both, but so I think we should just decide what we want this label to mean—a special-purpose, procedurally-meaningful "track", or a general-purpose, lightweight "category"?

Personally, I don't have a strong opinion on this, and would rather hear and help implement what the community would find more useful.

[ncoghlan]

Note that we know that Track would be useful, as it covers the Packaging use case: a procedurally distinct subset of PEPs covered by a standing delegation from the SC.

We don't have a clear use case for the generalisation to a broader "Category", "Categories", "Topics", or "Tags" notation. However, I also don't think having mutually exclusive procedural tracks defined would preclude adding non-exclusive topic area tagging later if someone came up with a compelling rationale for adding them.

[JelleZijlstra]

We don't have a clear use case for the generalisation to a broader "Category", "Categories", "Topics", or "Tags" notation.

For what it's worth, I have two other sub-indices (whatever they end up called) that I'd like to see:

• Typing PEPs
• Release schedule PEPs

I plan on creating those once the packaging page is done.

[pradyunsg]

So... the state of affairs right now seems to be:

• Add support for topic indices #2579 is basically ready for a merge now!
• Several PEPs: Add Topic: Packaging #2636 was merged, with Topic as the name for the header!
• The URL has concensus as well -- https://peps.python.org/topic/packaging

Once #2579 is landed, let's close this issue? I think it'll be cleaner to move the discussion for adding additional topics into dedicated issues+PRs for each topic, to avoid cross-talk and confusion.

The only broad thing that I can think of is that we might want to consider whether multi-topic PEPs would be a thing -- which I suggest deferring until it becomes a concern that we actually need to solve (i.e. we add a second topic or, even later, until we identify a PEP that belongs in two topics). :)