Update of primaryKey and foreignKeys when field is renamed

[fjuniorr]

Currently, when a field is renamed with steps.field_update shouldn't other properties that rely on field.name be updated as well? The examples that came to mind are primaryKey and foreignKeys (if inside a package).

Otherwise frictionless transform could take a valid resource and generate an invalid resource as output. Here's one example:

from frictionless import Resource, Pipeline, steps, Schema

source = Resource("https://raw.githubusercontent.com/frictionlessdata/framework/main/data/transform.csv")
source.schema = Schema.from_descriptor({
    "fields": [
        {"name": "id", "type": "integer"},
        {"name": "name", "type": "string"},
        {"name": "population", "type": "integer"},
    ],
    "primaryKey": ["id"]
})

source.validate()

pipeline = Pipeline(
    steps=[
        steps.field_update(name="id", descriptor={"name": "pkey"}),
    ],
)
target = source.transform(pipeline)
target.validate()

{'valid': False,
 'stats': {'tasks': 1, 'errors': 1, 'warnings': 0, 'seconds': 0.001},
 'warnings': [],
 'errors': [],
 'tasks': [{'name': 'transform',
            'type': 'table',
            'valid': False,
            'place': '<memory>',
            'labels': [],
            'stats': {'errors': 1, 'warnings': 0, 'seconds': 0.001},
            'warnings': [],
            'errors': [{'type': 'schema-error',
                        'title': 'Schema Error',
                        'description': 'Provided schema is not valid.',
                        'message': 'Schema is not valid: primary key '
                                   '"[\'id\']" does not match the fields '
                                   '"[\'pkey\', \'name\', \'population\']"',
                        'tags': [],
                        'note': 'primary key "[\'id\']" does not match the '
                                'fields "[\'pkey\', \'name\', '
                                '\'population\']"'}]}]}

[fjuniorr]

Otherwise frictionless transform could take a valid resource and generate an invalid resource as output.

Which in a way goes against the transform principle that metadata matters:

There are plenty of great ETL-frameworks written in Python and other languages. We use one of them (PETL) under the hood (described in more detail later). The core difference between Frictionless and others is that we treat metadata as a first-class citizen. This means that you don't lose type and other important information during the pipeline evaluation.

[shashigharti]

Thanks! @fjuniorr for reporting.

[fjuniorr]

Thanks @shashigharti and @roll!

Here a couple more examples that I would hope would work regarding foreignKeys.

Update a resource.name that is referenced in a foreign key

from frictionless import transform, steps, Package
from pprint import pprint

dp = Package('https://raw.githubusercontent.com/splor-mg/datapackage-reprex/main/reprex/20230725T075521/datapackage.json')

print(f"{dp.get_resource('fact').validate().valid=}")

transform(dp, steps=[
    steps.resource_transform(name='dim', steps=[
        steps.resource_update(name='dim', descriptor={'name': 'dm'})
    ])
])

pprint(dp.get_resource('fact').validate().flatten(['title', 'message']))

which yields

dp.get_resource('fact').validate().valid=True
[['Resource Error',
  'The data resource has an error: failed to handle a foreign key for resource '
  '"fact" as resource "dim" does not exist']]

Update a field.name that is referenced in a foreign key

from frictionless import transform, steps, Package
from pprint import pprint

dp = Package('https://raw.githubusercontent.com/splor-mg/datapackage-reprex/main/reprex/20230725T075521/datapackage.json')

print(f"{dp.get_resource('fact').validate().valid=}")

transform(dp, steps=[
    steps.resource_transform(name='dim', steps=[
        steps.field_update(name='uo', descriptor={'name': 'uo_cod'})
    ])
])

pprint(dp.get_resource('fact').validate().flatten(['title', 'message']))

which yields

dp.get_resource('fact').validate().valid=True
[['ForeignKey Error',
  'Row at position "2" violates the foreign key: for "uo_cod": values "1501" '
  'not found in the lookup table "dim" as "uo"'],
 ['ForeignKey Error',
  'Row at position "3" violates the foreign key: for "uo_cod": values "1251" '
  'not found in the lookup table "dim" as "uo"'],
 ['ForeignKey Error',
  'Row at position "4" violates the foreign key: for "uo_cod": values "1251" '
  'not found in the lookup table "dim" as "uo"']]

In this case I also can't export a valid descriptor with dp.to_yaml('datapackage.yaml') because I get

$ frictionless validate datapackage.yaml
─────────────────────────────────────────────────────────────── Dataset ────────────────────────────────────────────────────────────────
                 dataset                  
┏━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━┓
┃ name ┃ type  ┃ path          ┃ status  ┃
┡━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━┩
│ fact │ table │ data/fact.txt │ INVALID │
│ dim  │ table │ <memory>      │ INVALID │
└──────┴───────┴───────────────┴─────────┘
──────────────────────────────────────────────────────────────── Tables ────────────────────────────────────────────────────────────────
                                               fact                                               
┏━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Row  ┃ Field ┃ Type         ┃ Message                                                          ┃
┡━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ None │ None  │ source-error │ The data source has not supported or has inconsistent contents:  │
└──────┴───────┴──────────────┴──────────────────────────────────────────────────────────────────┘
                                               dim                                                
┏━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Row  ┃ Field ┃ Type         ┃ Message                                                          ┃
┡━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ None │ None  │ source-error │ The data source has not supported or has inconsistent contents:  │
└──────┴───────┴──────────────┴──────────────────────────────────────────────────────────────────┘

Do these make sense?

[roll]

Thanks @fjuniorr!

@shashigharti can you pleas take a look?