Provide a deep schema validation in the preprocessing step
So far we fully rely on pandas csv parser to do its job.
-
Define a pydantic schema to be assessed. -
Check that nullable / empty array fields are consistenlty parsed as null / False fields thorughout the pipeline.
Edited by Enrico UBALDI