Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scalable push query should terminate if the underlying persistent query is terminated #7981

Open
nateab opened this issue Aug 10, 2021 · 3 comments
Labels
bug query-engine Issues owned by the ksqlDB Query Engine team scalable-push

Comments

@nateab
Copy link
Member

nateab commented Aug 10, 2021

Describe the bug
When you terminate a persistent query, any scalable push query that was based on it should also be terminated since there will be no additional updates to receive, since scalable push queries don't read from the underlying kafka topic unlike regular push queries.

To Reproduce
Steps to reproduce the behavior, include:

  1. The version of KSQL.
    0.20

  2. Sample source data.

  3. Any SQL statements you ran

CREATE STREAM RATINGS (RATING_ID BIGINT, USER_ID INTEGER, STARS INTEGER, ROUTE_ID BIGINT KEY, RATING_TIME BIGINT, CHANNEL STRING, MESSAGE STRING) WITH (KAFKA_TOPIC='ratings', VALUE_FORMAT='JSON', partitions=1);

CREATE STREAM RATINGS_STREAM AS SELECT CAST((RANDOM() * 1000) AS INTEGER) KEY, * FROM RATINGS RATINGS;

SELECT * FROM RATINGS_STREAM EMIT CHANGES;
TERMINATE CSAS_RATINGS_STREAM_3;

Expected behavior
The persistent query is terminated, and the push query running on the persistent query is also terminated.
Actual behaviour
A clear and concise description of what actually happens, including:

  1. CLI output
  2. Error messages
  3. KSQL logs

The persistent query does terminate, but the scalable push query continues to run even though it will not report any changes.

Additional context
Add any other context about the problem here.

@stevenpyzhang stevenpyzhang added the query-engine Issues owned by the ksqlDB Query Engine team label Aug 13, 2021
@PeterLindner
Copy link

I disagree with the assumption that no additional updates will be recieved when the persistent query is terminated.
At any point in time another persistent query, which inserts into the stream, could be started or a application outside of ksqlDB could produce into the underlying kafka topic.
I firmly believe that consuming from a stream / topic should be decoupled from producing into it.

@nateab nateab changed the title Push query should terminate if the underlying persistent query is terminated Scalable push query should terminate if the underlying persistent query is terminated Aug 16, 2021
@nateab
Copy link
Member Author

nateab commented Aug 16, 2021

Yes these are fair points, I misunderstood when I created the issue, but this is specifically for scalable push queries. It would be good in the future to also decouple consuming and producing but the current implementation of scalable push queries does not yet allow for this.

@agavra
Copy link
Contributor

agavra commented Aug 17, 2021

might be related to #3967

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug query-engine Issues owned by the ksqlDB Query Engine team scalable-push
Projects
None yet
Development

No branches or pull requests

4 participants