Cleaning a Feed You Did Not Write
One reported assessment problem is simply a data file with duplicates and corrupted records in it, and instructions to produce a clean one. Candidates who treat it as an exercise in string handling do badly. It is an exercise in deciding what to do with input that is wrong, which is a design question with no single right answer and several clearly wrong ones.
The four failures, in the order they arrive
Duplicates. The same logical record twice. Sometimes byte-identical, which is easy, and sometimes the same event with a different arrival timestamp, which is not. The question to answer first is what makes two records the same, and the answer is a key drawn from the fields the sender controls, never from the fields your own pipeline stamped on.
The rest of this lesson is for subscribers
Unlock every lesson in Programming for Quantitative Developers, and every other premium course.
Subscribe to continueTest your knowledge
Keep reading Programming for Quantitative Developers
25 lessons in this course, and every other premium course, on one subscription.
- Every lesson in every course, with the worked examples and interactive simulators
- Graded questions on every lesson, with explanations for the wrong answers as well as the right one
- The trainers, timed assessments and brainteaser library that go with them