Proposal: Support DELETED trips in GTFS-RT

[mads14]

“DELETED” Enum Proposal

Background

Equipment and staffing challenges faced by transit providers during the COVID-19 pandemic resulted in, among many things, more frequent trip cancellations. The GTFS-RT spec defines an official way to convey to consuming applications that a trip has been canceled by a service provider, via the trip schedule_relationship of CANCELED. Many producers and consumers utilize this functionality today.

The current definition of CANCELED is “A trip that existed in the schedule but was removed.” This definition does not provide a way to distinguish between two very different types of removals:

Scenario 1: Trips where a transit provider would like to actively communicate a cancellation to riders
Scenario 2: Trips where a transit provider would like to entirely remove information about the corresponding trip from consuming applications, so the trip is not shown as canceled to riders, e.g. when one or more trips are entirely being replaced by other trips.

By current definition, consumers must make assumptions over what a producer means when they utilize CANCELED. Producers have no way to indicate the type of passenger communication desired. For scenario 2 above, this leads to cases where riders see that trips have been canceled (for example, by an app or trip planner showing a scheduled time crossed out), which is misleading when one or more trips have been replaced by different trips. It can be especially distracting if this is happening for many trips on a route.

Proposal

This proposal adds a new trip schedule_relationship of DELETED, which can be used to convey that a transit provider intends for a trip to entirely disappear from public-facing applications. To minimize impact on current implementations and maximize simplicity, the DELETED enumeration would be defined with the same definition as CANCELED but with additional precision: “A trip that existed in the schedule but was removed and is not intended to be seen by users.”

This addition to the spec immediately removes ambiguity over how to implement CANCELED for consumers. For producers, it allows service providers to communicate removals of service in a way that reflects rider understanding of the service - i.e., frequency-based, show-up-and-go service removals might be understood differently from less frequent services, or services where users know and understand the service at the trip level.

Going forward, this proposal also greatly clarifies how to display trips that are impacted by changes proposed in GTFS-ServiceChanges v3.1.

Authors notes

Submitted on behalf of a third-party: Maryland Department of Transportation
The following is a proposal drafted by a few folks at MDOT in Maryland that we (Swiftly) think it makes sense to introduce now.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[skinkie]

DELETED suggests realtime alternations of the scheduled timetable, as in a database that is updated, but the way you describe it is a CANCELLED without notification, and that is something that should deserve its own flag.

[mpaine-act]

Are there any issues with state transitions to consider between trip descriptors? For example, going from DELETED to SCHEDULED without an intermediate ADDED trip descriptor step?

should deserve its own flag

Does ADDED trip descriptor with an enabled "without notification" flag make sense?

[mads14]

@skinkie
Are you suggesting that these "DELETED" trips should still be marked as CANCELED, and that we should have a separate boolean on the trip like omit_from_rider_apps? If so, I think this could be a reasonable alternative. Could you elaborate on why you prefer this approach?

@mpaine-act

Are there any issues with state transitions to consider between trip descriptors? For example, going from DELETED to SCHEDULED without an intermediate ADDED trip descriptor step?

I don't foresee issues with state transitions. I don't think a DELETED trip (a trip in the original GTFS schedule) would ever "become" an ADDED or DUPLICATED trip. While a common use case might involve deleting several trips and adding new replacements, that is not strictly a requirement. This DELETED schedule relation is only intended to indicate whether rider-facing apps should explicitly show these canceled trips to riders. As stated in the proposal above:

• A trip that existed in the schedule but was removed and the cancelation should be communicated to riders is CANCELED
• A trip that existed in the schedule was removed and the cancelation should not be explicitly communicated to riders is DELETED. Note that if these trips are replaced by added trips, those added trips will be included in the trip-updates feed so passengers will be aware of the new/updated trip info.

Let me know if that helps clarify.

[colemccarren]

Chiming in here - thanks @mads14 for starting the proposal discussion and thank you @skinkie & @mpaine-act for providing your thoughts so far.

As mentioned above, folks at MDOT MTA (Baltimore, USA) have been contemplating a proposal like this for quite a while (beginning with Issue #315 back in April, when we were curious if a proposal like this could lend itself to better communicating rail disruption communications).

Today, we find consumer and producer implementations of CANCELED continue to vary. For example, some passenger-facing applications completely remove from view information about CANCELED trips. Others employ a "strike-through" over a trip's scheduled arrival time, or place the word "Cancelled" next to that time in red, etc.

As the proposal notes, from an agency or producer perspective, both approaches are extremely useful (actively communicating a removed trip, versus omitting all information about the removed trip) from a real-time passenger communication standpoint. But, the utility in either of these approaches varies depending on the context of transit service impacted (commuter services or services that a rider understands service at the trip level, versus "show up and go" services).

I find this similar to the approach taken in Proposal #221 in that it addresses the fact that multiple consumers and producers have different interpretations over a schedule_relationship. But, by nature of how CANCELED is used today, this proposal doesn't need to deprecate anything - it clarifies the definition of one enum with the addition of another.

Instead of pursuing a custom extension approach here, our team believes this proposal could accomplish a few different things in a relatively simple way, for consumers and producers alike. We're eager to keep hearing everyone's thoughts.

[westontrillium]

This proposal sounds interesting. Some more use cases and perhaps some screenshots would be useful to better understand how you envision DELETED* to be used in practice.

*or, as otherwise suggested, a separate boolean indicating that the cancelation be omitted from rider apps

[skinkie]

@skinkie Are you suggesting that these "DELETED" trips should still be marked as CANCELED, and that we should have a separate boolean on the trip like omit_from_rider_apps? If so, I think this could be a reasonable alternative. Could you elaborate on why you prefer this approach?

From my point of view DELETED would make sense as a database action, where DELETED and ADDED would find itself as a nice pair of operations that could just update the timetable by replacing 'virtually' the same trips, but with different identifiers. An example in The Netherlands would be that the original train is deleted, but the exact replacement is added. The rider wouldn't have a clue such switch took place.

omit_from_rider_apps or ShowCancelledTrip=false, as we have it in the Dutch realtime format is something that from management perspective really tells you: the trip has been cancelled, but shouldn't be shown on a display (for whatever reason). It could be due to high frequency trips, that have such short intervals that cancellations wouldn't make sense. Or the exact opposite: it would only show cancellations (so many) and never the next running trip.

[mads14]

@westontrillium thanks for the comment. Below I will outline the initial use case that Swiftly has in mind. I don't have screenshots of how this would be handled in various app, but I've typically seen deleted trips are communicated in real-time apps with grayed out text and often a strike-through, indicating that they are not served. For a DELETED trip, I'm imagining that there would be no info about this trip communicated to passengers at all.

The first use case that we (Swiftly) are planning to use DELETED for is when there is a major rail disruption, like an accident on the tracks, that causes trains to run on a truncated route on one or both sides of the disruption. In this case, the agency is likely deviating significantly from the original GTFS schedule but is still running service on large portions of the route.

We plan to handle this in GTFS-rt by canceling the original GTFS trips for the hours while the rail disruption is ongoing and will use added trips (duplicating trips in the original GTFS) with skipped stops to reflect the service that will actually be running on the truncated routes. While we can achieve this with the existing GTFS-rt spec, we think that explicitly showing all of the crossed-out canceled trips in a realtime app would distract from the more-important realtime predictions. Therefore, we think a deleted flag that indicates to consumers that these canceled trips should not be surfaced is appropriate.

I'll also mention that when a disruption like this is ongoing we would expect a rider alrert would also be made to give more context to riders.

@skinkie for the use case I mention here - there is not a 1:1 match between a deleted and an added trip, rather several trips will be deleted and several new trips will be added, but there will likely not be the same number of trips added as deleted.

[lauramatson]

This is a great proposal and fills a useful need in getting clear information to riders about what is happening. Either approach -- adding a new value to the enum or adding a new field -- seems fine. Glad to see this moving forward.

[SteveGladstone]

Thanks again @mads14 for starting the discussion! As others have said, the point here is the give a universal (or as close to universal as possible) methodology for producers and consumers to achieve the function of cancelling a trip but not explicitly displaying that cancelling in GUI applications in order to achieve spec compliance with the end goal being to minimize confusion and uncertainty in the realm of customer experience.

A picture is worth a thousand words, so consider this scenario. Winter weather fast approaching and for us in Maryland, that means ice on our Light Rail catenary lines, which in turn means we'll have short turns and split systems. As a rider, my expectation is that I open my real-time information screen of choice to see where my train is and when it will arrive.

• Current - scheduled GTFS trips get CANCELED; newer trips are ADDED or DUPLICATED; both appear in the user interface

• Future - scheduled GTFS trips get DELETED; newer trips are ADDED or DUPLICATED; only the running trips appear in the user interface... and there may be more ADDED or DUPLICATED trips than originally scheduled due to the lack of distance to cover in each direction

As you can see in the first screenshot, this is very confusing. Is the 8:25PM train cut or is it running a minute ahead of schedule? Is the 8:43PM train cut or not? Is that 9:00PM train cut or running 3 minutes late? How does this instill faith in the transit data being given?

In the second screenshot, all the DELETED trips aren't viewable. Everything else shows real-time data the rider would expect. There is no confusion here over what is and is not running. The additional trips compared to what might normally be scheduled (first screenshot) is an added bonus.

From a technical implementation perspective, I personally find DELETED better than a separate field like omit_from_rider_apps because, as a coder, the less fields to check for dependency the better. DELETED tells me all I need to know; CANCELED tells me I need to check for omit_from_rider_apps and then do something based on that. Trivial? Sure. Necessary? Not with this proposal. I'll take any code efficiency I can get since I don't write efficient code to begin with :D

Hopefully this helps! As one of the guys at the Maryland Transit Administration, I support this proposal :)

[sam-hickey-ibigroup]

We (IBI) agree there is a need for this functionality. A few comments and questions as we think about how this would be implemented.

(1) To remove potential ambiguity about how consumers should (or in this case should not) display information about a DELETED trip, we suggest changing “A trip that existed in the schedule but was removed and is not intended to be seen by users.” to “A trip that existed in the schedule but was removed and must not be shown to users.”

(2) The proposed change here is in the trip’s ScheduleRelationship only. Stop times also have a ScheduleRelationship that is defined in a different enum (see https://github.com/google/transit/blob/master/gtfs-realtime/proto/gtfs-realtime.proto#L223). If a trip has schedule_relationship = DELETED, what would consumers expect for the stop time’s schedule_relationship? SKIPPED is one option, but this is an existing enum value that may be interpreted by some applications to mean that an individual stop time should be shown as grayed out or crossed out. We suggest clearly requiring that a trip that has schedule_relationship = DELETED cannot have any stop_time_update that have schedule_relationship = SKIPPED. Instead, we would propose either (a) a new stop time schedule_relationship value of DELETED or (b) requiring that a trip that has schedule_relationship = DELETED does not have any stop_time_update.

(3) Including this change as part of the ScheduleRelationship enum seems like a more natural place for it but we’re not against a new boolean field like omit_from_rider_apps.

(4) Based on our understanding of the spec change process, this change should start as experimental before it is officially fully added to the spec. The following note has been added in the proto file for other experimental fields/values: // NOTE: This field is still experimental, and subject to change. It may be formally adopted in the future.

[SteveGladstone]

We suggest clearly requiring that a trip that has schedule_relationship = DELETED cannot have any stop_time_update that have schedule_relationship = SKIPPED. Instead, we would propose either (a) a new stop time schedule_relationship value of DELETED or (b) requiring that a trip that has schedule_relationship = DELETED does not have any stop_time_update.

I see what you're saying here, though part of me worries that adding a second enum to StopTimeUpdate schedule_relationship might make this too much from a GTFS-RT producer perspective. If those are the only two options, I would prefer that a trip with a schedule_relationship of DELETED have no StopTimeUpdates.

That does beg the question, though, of order of operations on the schedule_relationship side. I would think (hope?) that a trip's schedule_relationship trumps any StopTimeUpdate info from a consumer's perspective when necessary, such as with CANCELED or, here, a DELETED trip. I know on Swiftly's side any trip that's CANCELED has an empty StopTimeUpdate array, but our MARC Train GTFS-RT provider sets the schedule_relationship on StopTimeUpdates for CANCELED trips to be SKIPPED. That already works in GTFS-RT consumer apps like Transit, Google, Apple, etc.

Given the ambiguity around StopTimeUpdates and CANCELED trips and that being outside the scope of this proposal, would it be fair to say that the trip-level schedule_relationship is the main driver as a GTFS-RT consumer so that GTFS-RT producers have minimal impact to existing feed creation workflows?

[mads14]

@SteveGladstone thanks so much for the screenshots and comments! That's extremely helpful.

@sam-hickey-ibigroup thanks for your notes!

1. I like your wording suggestion and will incorporate that now.
2. I appreciate the concern, but I think I agree with @SteveGladstone on this one that the trip-level schedule_relationship is the main driver.
   As Steve noted there is already ambiguity on whether to add SKIPPED stop_time updates or to not include stop_time_updates for CANCELED trips and we also think that the trip-level status should trump the stop-level status. Swiftly will omit stop-time updates for these DELETED trips, but I don't know that we need to enforce that others producers do. It feels like introducing a stop_time_update of DELETED is unnecessary at this time.
3. I am happy to keep it as a schedule_relationship value unless there is strong objection from consumers.
4. Good catch. I'll add this now.

[gcamp]

At Transit, we agree that the proposal is need and we also prefer the DELETED option (even if both work).

Before a vote, reference.md needs to be updated with the new field.

[mads14]

@gcamp thanks for the comment! I'm updating the reference.md now.

[leonardehrenfried]

I didn't really understand really what the use case was until I saw @SteveGladstone's explanation and screenshots. Not sure if there is room in the spec for a long form discussion but I would love to see it referenced/reflected somewhere.

[bdferris-v2]

Generally ok with this addition and also prefer the enum option.

Per @SteveGladstone 's comments, I do think the the Description for TripUpdate.stop_time_update could use an edit. Specifically, it calls out expectations around stop time updates for CANCELED and DUPLICATED. I know there is some ambiguity in the spec here, but I think the text should be updated to include DELETED as well?

If you wanted to be particularly pedantic, there is text in the "best practices" doc suggesting that CANCELLED trips should include a corresponding service alert entry. That could be amended to mention that DELETED trips do NOT need a service alert, but maybe overkill?

And for my own benefit, if a trip needed to be "undeleted", it's sufficient to just update the TripUpdate schedule_relationship to SCHEDULED, yeah?

[mads14]

Thanks @bdferris-v2 for your comments.

Per @SteveGladstone 's comments, I do think the the Description for TripUpdate.stop_time_update could use an edit. Specifically, it calls out expectations around stop time updates for CANCELED and DUPLICATED. I know there is some ambiguity in the spec here, but I think the text should be updated to include DELETED as well?

Makes sense. I will add this call out now.

If you wanted to be particularly pedantic, there is text in the "best practices" doc suggesting that CANCELLED trips should include a corresponding service alert entry. That could be amended to mention that DELETED trips do NOT need a service alert, but maybe overkill?

Can you point me to the text you're looking at? I actually see in service-alerts.md that "delays and cancellations of individual trips should usually be communicated using Trip updates".

And for my own benefit, if a trip needed to be "undeleted", it's sufficient to just update the TripUpdate schedule_relationship to SCHEDULED, yeah?

Yes! Simply changing the schedule_relationship to SCHEDULED should un-delete it.

[mads14]

@leonardehrenfried
I totally agree that a picture is worth 1000 words in this case. I'm wondering how we can make it more obvious in the spec why this is important.

Right now in the spec I've added the following description.

A trip that existed in the schedule but was removed and must not be shown to users.
DELETED should be used instead of CANCELED to indicate that a transit provider would like to entirely remove information about the corresponding trip from consuming applications, so the trip is not shown as cancelled to riders, e.g. a trip that is entirely being replaced by another trip.

Maybe we could add:

This designation becomes particularly important if several trips are cancelled and replaced with substitute service. If consumers were to show explicit information about the cancellations it would distract from the more important real-time predictions.

Thoughts or suggestions?

[bdferris-v2]

Regarding my comment about "best practices", I was actually thinking of the docs on gtfs.org, which are maintained in a separate repo. Probably doesn't need to be updated quite yet.

As for this PR, I noticed you added the "experimental" notice to the proto file. Can you add a similar notice in the reference file ala "Caution: this field is still experimental...", similar to other examples in the file?

[sam-hickey-ibigroup]

@mads14 Yes, the following is helpful and would be a good addition to make.

This designation becomes particularly important if several trips are cancelled and replaced with substitute service. If consumers were to show explicit information about the cancellations it would distract from the more important real-time predictions.

Regarding DELETED trips in stop_time_update: agreed there is ambiguity in the spec about whether stop_time_update must/should/could be provided for CANCELED trips. The change made in this proposal to mention DELETED in the stop_time_update description looks good, though it would be nice to also add the following new text that is in bold to remove any potential ambiguity:

Updates to StopTimes for the trip (both future, i.e., predictions, and in some cases, past ones, i.e., those that already happened). The updates must be sorted by stop_sequence, and apply for all the following stops of the trip up to the next specified stop_time_update. At least one stop_time_update must be provided for the trip unless the trip.schedule_relationship is CANCELED, DELETED, or DUPLICATED - if the trip is canceled or deleted, no stop_time_updates need to be provided. When trip.schedule_relationship is DELETED, this value takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip.

[mads14]

@sam-hickey-ibigroup thanks for the suggestion! I would be fine to add the note about

When trip.schedule_relationship is DELETED, this value takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip."

however, it feels a bit funny to add this explicit call out for DELETED where there is not one for CANCELED. I would think that for trip.schedule_relationship=CANCELED would also takes precedence over stoptime_updates, but I'm not positive that all would agree. Would folks be ok with me saying the following or is the behavior around CANCELED precedence disputed?

When trip.schedule_relationship is CANCELED or DELETED, the trip.schedule_relationship takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip."

[IvanVolosyuk]

We as a consumer actually silently ignore STUs for a CANCELED update, so I'm ok for adding the clarification.

…

On Sat, 5 Nov 2022, 01:21 Maddie, ***@***.***> wrote: @sam-hickey-ibigroup <https://github.com/sam-hickey-ibigroup> thanks for the suggestion! I would be fine to add the note about When trip.schedule_relationship is DELETED, this value takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip." however, it feels a bit funny to add this explicit call out for DELETED where there is not one for CANCELED. I would think that for trip.schedule_relationship=CANCELED would also takes precedence over stoptime_updates, but I'm not positive that all would agree. Would folks be ok with me saying the following or is the behavior around CANCELED precedence disputed? When trip.schedule_relationship is *CANCELED or* DELETED, *the trip.schedule_relationship* takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip." — Reply to this email directly, view it on GitHub <#352 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HMDKXLR2RQUFVRQN3LWGULWRANCNFSM6AAAAAARDOVF5A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mads14]

This pull request has been open for more than one week, so I am calling for a vote. Note that this is for the adoption of a new trip.schedule_relationship enum DELETED to be added to gtfs-rt as an experimental field.

Please vote with a +1 (for) or -1 (against) in the comments. Voting ends on 2022-11-16 at 23:59:59 UTC.

Thanks,
Maddie

[gcamp]

+1 Transit

[SteveGladstone]

+1 Maryland Transit Administration

[skinkie]

+1 OpenGeo

[sam-hickey-ibigroup]

+1 IBI

[harringtonp]

+1 yourstop.info

[jfabi]

+1 on behalf of the MBTA for adding this as an experimental field.

[GabrielaCodreanu]

+1 Ito World.

[mads14]

Thanks all for participating! The vote ended on Wednesday, November 16 at 23:59:59 UTC with a unanimous consensus of 7 'yes' votes and 0 opposed.

As per the specification amendment process, this proposal is now accepted!

[mads14]

I am not able to merge this myself as I am a first time contributor to this repo.

It is also my understanding that Google will merge the approved PRs, is that accurate?

9. If the proposal is accepted:

• Google is committed to merging the voted upon version of the pull request (provided that the contributors have signed the CLA), and performing the pull request within 5 business days.
• Google is committed to timely updating https://github.com/google/gtfs-realtime-bindings repository. Commits to gtfs-realtime-bindigs that are a result of a proposal, should reference the pull request of the proposal.
• Translations must not be included into the original pull request. Google is responsible for eventually updating relevant translations into supported languages, but pure translation pull requests from the community are welcome and will be accepted as soon as all editorial comments are addressed.

[isabelle-dr]

I can merge this PR. Thank you for this awesome contribution! 🎉