Extend with custom keyword to support nullable based on Open API 3 Specification

[ghost]

Hello,

we use internaly Swagger aka Open API. The schema differs from the JSON Schema defintion.
Eg. the type can only be one type instead of an array of types line
"type":["string","null"]

To support nullable values you must define
"type":"string", "nullable: true

I tried to add a new keyword nullable
ajv.addKeyword('nullable', { type: 'boolean', macro: function (schema, parentSchema) { return { type: !!schema ? [parentSchema.type, "null"] : parentSchema.type }; }});
but I can't delete the original error from the type validator.

Also I tried to override the type keyword, but this isn't possible.

How can I extend the existing type checking to prevent errors, when the value is null, based on the new nullable keyword?

I understand, that adding new keyword make the validation stronger. But how can I weaken the existing validation?

Thx

[epoberezkin]

You need to create inline keyword, it's the only way at the moment. I considered exposing validation errors to all keywords but decided against it.

Check out ajv-keywords and ajv-errors packages for examples of inline keywords. There are [edge] cases when type can be validated after custom keywords that will be addressed soon (see #485).

By the way, "type" in keyword definition determines data types that the keyword applies to, in case when you want it applied to all types (as in case with "nullable") you don't need to use "type". The way you've defined it it will be applied to boolean data only, it won't even be called for strings.

Macro keywords return the schema that will be used instead of them, there is no way at the moment (and there is no easy way to do it) to define macro keyword that would replace the parent schema.

[ghost]

Hi,

all the following stuff is based on

var inlineNullableTemplate = doT.compile("\
  {{ \
    if (it.schema.nullable) { \
      it.schema.type = Array.isArray(it.schema.type) ? it.schema.type : [it.schema.type]; \
      it.schema.type.push('null'); \
    } \
    var $data = 'data' + (it.dataLevel || '') \
      , $nullable = it.schema.nullable; \
  }} \
  {{=$nullable}} || ({{=$data}} !== null) \
");

var ajv = new Ajv({allErrors: true});
ajv.addKeyword('nullable', {
  inline: inlineNullableTemplate,
  errors: true,
  metaSchema: { type: 'boolean' }
});

var schema = {"properties": {"foo": { "type": "string", "nullable": true } } };
var validate = ajv.compile(schema);
validate({"foo": null});

I've studied your sourcecode, the compilation process and the compiled validators also.
But it seems there is no chance to extend or avoid the type checking or something else to ignore the error
Invalid: data.foo should be string
if my nullable flag ist true.

What I understand:

1. With all 4 Keyword types (inline, compile, ...) a validation code will be generated before the typechecking starts.

...
//myvalidationcode
var valid1;
valid1 = true || (data1 !== null);
//default custom keyword error handler
if (!valid1) {
...
}
//the default type checking
if (typeof data1 !== "string") {
...
}
var valid1 = errors === errs_1;
...

2. There is no chance to effect the default type validation nor to manipulate the schema within a new keyword, so the schema type can't be changed / extended (or something else on the schema).
   I've tried this via the
   it.schema.type
   reference but I had to find out, that (only) the types has a special handling in your code.
   var $typeSchema = it.schema.type,
$typeIsArray = Array.isArray($typeSchema);
   So I can't manipulate this in my code.

3. I understand, that the main purpose for addKeyword is, to extend the validations, to harden it. But in my case it must be weaker than before.

Currently my only solution is to transform the schema from
{"type":"string", "nullable":true"}
to
{"type":["string","null"]}
before the validation process start, but I will loose the correct error messages, based on "nullable".

Maybe I misunderstood Improve type validation #485, but changing the order of validation or something like this, doesn't solve my requirement.
Either I validate first and can't interfer the type validation after or I validate after the type and can't manipulate the error state.

Did you have another idea?

Thank you for your response.

Christian

[epoberezkin]

In most cases nullable will be validated after type checking and after #485 it will be in all cases. So you can manipulate error state by changing errors and vErrors variables to remove type errors in case data was null and it is ok. Just using true as validation result is not enough, you need to analyse and correct errors collected so far and their count.

[epoberezkin]

That's what ajv-errors is doing

[epoberezkin]

Changing schema as you do in the sample would achieve nothing, you should not be doing it during compilation - as you have said type is tied into validation process and it's been already used, so at best it would simply not work.

[khusamov]

@nexinto-gmbh Dear, but did you succeed in making a plugin to process the parameter "nullable"?

[ghost]

Hi and sorry for answering so late. I stopped developing on this code 5 month ago but I think I have a solution. Based on Swagger Extentions in Swagger API 2 I named the field x-nullable to have a minimum gap to the nullable option in OAI 3. It should also be compatible to OAI 3 nullable. {{ var $data = 'data' + (it.dataLevel || '') , $valid = 'valid' + (it.dataLevel || '') , $errs = 'errs__' + (it.dataLevel || '') , $isValid = it.util['checkDataType'](it.schema.type, $data, false) , $isNullable = !!it.schema.nullable || !!it.schema.x-nullable , $isOAI2 = it.schema.x-nullable !== undefined && it.schema.nullable === undefined , $keyword = $isOAI2 ? 'x-nullable' : 'nullable'; }} if({{=$isValid}}) { {{=$valid}} = true; } else if({{=$isNullable}} && {{=$data}} === null) { var i{{=it.level}} = vErrors.findIndex(function(error) { return err.keyword === 'type' && err.dataPath === (dataPath || '') + {{=it.errorPath}}; }); if(i{{=it.level}} > -1) { vErrors.splice(i{{=it.level}}, 1); errors--; } {{=$valid}} = true; } else if({{=$data}} === null){ vErrors.push({ keyword: '{{=$keyword}}', params: { type: 'boolean' }, message: 'should be true' }); errors++; {{=$valid}} = false; } else { {{=$valid}} = false; } import Ajv from 'ajv' import nullable from './dotjs/nullable' const ajv = new Ajv({ removeAdditional: true, allErrors: true }) ajv.addKeyword('nullable', { inline: nullable, errors: true, statements: true, metaSchema: { type: 'boolean' } }); ajv.addKeyword('x-nullable', { inline: nullable, errors: true, statements: true, metaSchema: { type: 'boolean' } }); export default ajv Do you need this for your own OAI 3 implementation? Von: Khusamov Sukhrob [mailto:notifications@github.com] Gesendet: Freitag, 8. September 2017 05:24 An: epoberezkin/ajv <ajv@noreply.github.com> Cc: Christian Kulbe <Christian.Kulbe@nexinto.com>; Mention <mention@noreply.github.com> Betreff: Re: [epoberezkin/ajv] Extend with custom keyword to support nullable based on Open API 3 Specification (#486) @nexinto-gmbh<https://github.com/nexinto-gmbh> Dear, but did you succeed in making a plugin to process the parameter "nullable"? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#486 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASzHHaRUaRlU6AnwbuusRCvN2GMVUk0mks5sgLNQgaJpZM4NX8r0>.

[khusamov]

Thanks for answer!

Do you need this for your own OAI 3 implementation?

Yes, it is.

Did you write the keywords "discriminator" and "readOnly" handlers?

[ghost]

Yes and no. We need some types and validation which doesn’t exists in 2 but be a part of 3 like nullable. If you have a working OAI 3 implementation in the near feature, I will use this instead of my modification. THX Von: Khusamov Sukhrob [mailto:notifications@github.com] Gesendet: Mittwoch, 13. September 2017 04:53 An: epoberezkin/ajv <ajv@noreply.github.com> Cc: Christian Kulbe <Christian.Kulbe@nexinto.com>; Mention <mention@noreply.github.com> Betreff: Re: [epoberezkin/ajv] Extend with custom keyword to support nullable based on Open API 3 Specification (#486) Thanks for answer! Do you need this for your own OAI 3 implementation? Yes, it is. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#486 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASzHHSB7ujRc3z8DyObV6deApXckO1yAks5sh0OKgaJpZM4NX8r0>.

[khusamov]

I'm now picking a validator that will add the keywords nullable and discriminator. But so far, unsuccessfully.

[epoberezkin]

@khusamov thank you.
These keyword are not part of JSON schema specification - that's why they are not supported in any of the validators.
I am surprised that Open API spec still chooses to maintain their own flavour of JSON schema rather than sticking to the standard, where you can use "type": ["string", "null"], even though it slowly converges to JSON schema spec /rant.
It's possible to implement nullable (and probably discriminator too, didn't look into it) as a custom keyword for ajv, as explained above. If you or anybody else decides to implement them, they can be added to ajv-keywords package. See ajv-validator/ajv-keywords#32

I'm closing the further discussion here, please submit any questions/comments to ajv-keywords issue.

[TageNielsen]

There is an invalid unit test added as part of this commit

      it('"nullable" keyword must be "true" if present', function() {
        should.throw(function() {
          ajv.compile({nullable: false});
        });
      });

nullable: false is a valid value for schemaNullable. false is the default value, though it may be explicitly defined.

[epoberezkin]

https://swagger.io/docs/specification/data-models/data-types/ - this page implied it. But happy to change - you can submit a PR