kedro.framework.context.context.KedroContextError with Kedro 0.17.0

[zoumdatascience]

Description

Short description of the problem here.
I kept facing the ModuleNotFoundError then came across this post about the similar issue.
Further after implementing the recommended solution (snipet below), I got this error:

_add_src_to_path(Path.cwd()/"src", Path.cwd())
context = load_context(path_to_project)
output = context.run()

Context

How has this bug affected you? What were you trying to accomplish?
I was trying to deploy the kedro application on Docker using flask API. But the error occured while implementing the Flask API

Steps to Reproduce

1. [Create the flask python file]
   I created the python file containing my flask scripts with the useful imports.
   Everything works fine while using the default route to test the Flask API
   Everything breaks when I call the load_context(path_to_project) (snipet below)

@app.route('/run')
def run():
_add_src_to_path(Path.cwd()/"src", Path.cwd())
context = load_context(path_to_project)
output = context.run()
return output

2. [Call the run route]
   Got the kedro.framework.context.context.KedroContextError with the following detail:
   kedro.framework.context.context.KedroContextError: Failed to find the pipeline named 'default'. It needs to be generated and returned by the '_get_pipelines' function.

Expected Result

I should receive a JSON outpout showing: {"Model has a coefficient R^2 of 0.456."}

Actual Result

I face the error explained in step 2. [Call the run route]

 File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/kedro/framework/context/context.py", line 253, in _get_pipeline
    return pipelines[name]
KeyError: '__default__'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/zoumanakeita/Desktop/Personal/Learn_Kedro/iris-kedro/flask-app.py", line 16, in run
    output = context.run()
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/kedro/framework/context/context.py", line 561, in run
    pipeline = self._get_pipeline(name=pipeline_name)
  File "/Users/zoumanakeita/opt/anaconda3/envs/awesome-kedro-project-env/lib/python3.7/site-packages/kedro/framework/context/context.py", line 259, in _get_pipeline
    ) from exc
kedro.framework.context.context.KedroContextError: Failed to find the pipeline named '__default__'. It needs to be generated and returned by the '_get_pipelines' function.

Your Environment

Include as many relevant details about the environment in which you experienced the bug:

• Kedro version used (pip show kedro or kedro -V): kedro, version 0.17.0
• Python version used (python -V): Python 3.7.9
• Operating system and version: MacOS Catanina Version 10.15.4
• Environment: virtual environment used to run the program
• Error status: happens when running the Flask Application

[merelcht]

Hi @zoumdatascience, the error you're getting kedro.framework.context.context.KedroContextError: Failed to find the pipeline named '__default__'. It needs to be generated and returned by the '_get_pipelines' function. indicates that you are trying to run a pipeline that can't be found.

Have you registered the pipeline in hooks.py?

See below as example:

def register_pipelines(self) -> Dict[str, Pipeline]:
    """Register the project's pipeline.

    Returns:
        A mapping from a pipeline name to a ``Pipeline`` object.

    """
    de_pipeline = de.create_pipeline()
    ds_pipeline = ds.create_pipeline()

    return {
        "de": de_pipeline,
        "ds": ds_pipeline,
        "__default__": de_pipeline + ds_pipeline,
    }

[zoumdatascience]

Hi @MerelTheisenQB,
Thanks for your feedback, and here was my code before getting the error. My register_pipelines() methods is similar to yours:

    @hook_impl
    def register_pipelines(self) -> Dict[str, Pipeline]:
        """Register the project's pipeline.

        Returns:
            A mapping from a pipeline name to a ``Pipeline`` object.

        """
        data_engineering_pipeline = de.create_pipeline()
        data_science_pipeline = ds.create_pipeline()

        return {
            "de": data_engineering_pipeline,
            "ds": data_science_pipeline,
            "__default__": data_engineering_pipeline + data_science_pipeline,
        }

[merelcht]

Hi @zoumdatascience , that looks good thanks for posting that.

Could you just clarify what exactly the issue is you're having? I read your description above, but I'm not sure I follow what's going on. I've run these lines:

_add_src_to_path(Path.cwd()/"src", Path.cwd())
context = load_context(path_to_project)
output = context.run()

and they're fine, so I'm guessing your issues come from Flask? Can you

1. Confirm whether the code above works fine if you use it outside of the Flask context? E.g. just in the terminal.
2. Double check that Path.cwd() points to the right location when you're running it from Flask? _add_src_to_path(Path.cwd()/"src", Path.cwd()) Assumes your source directory is called "src" and Path.cwd() is path to the root of the project.

[zoumdatascience]

Hi @MerelTheisenQB,

Thank you for your feedback, the code that I was not pointing to the correct path of the project, so the step by step debugging (as you suggested) helped me find the issue. So my final code is below:

@app.route('/run')
def run():
    project_path = Path.cwd()
    metadata = _get_project_metadata(project_path)
    _add_src_to_path(metadata.source_dir, project_path)
    session = KedroSession.create(metadata.package_name, project_path)
    context = session.load_context()
    
    output = context.run()
    return output

if __name__ == '__main__':
    app.run(debug=True)

Have a great week!

[merelcht]

I'm glad you found the solution! Can I close the issue now?

[zoumdatascience]

Problem solved

[jacobweiss2305]

Hello,

Probably a noob questions, but what is:
metadata = _get_project_metadata(project_path)
_add_src_to_path(metadata.source_dir, project_path)

Thanks!

[merelcht]

Hi @jacobweiss2305, this code adds the project src directory to PYTHONPATH to make sure python can find the project as a module and run it.

[jacobweiss2305]

Right, but is that a function built by kedro, is it some open source function, or did the user build it? I am able to recreate error so would like to apply this fix. Thank you for the quick response btw!

[merelcht]

It's built by kedro, you can use it by importing it like this from kedro.framework.startup import _add_src_to_path 🙂