by Daimei Fu
As a great wizard once said, a prophecy is valid if and only if the supporting data is valid. Data cleaning, being the starter-branch of the ultimate divination magic — data analysis — is usually ignored by muggles and underestimated by junior wizards. However, wise wizards should treat data cleaning carefully to prevent a spell backlash when casting data analysis magic without clean data — otherwise, the consequencess could be CATASTROPHIC.
To demonstrate the importance of data cleaning, this article discusses a disaster which happened in an anonymous wizard school.
Mystery of a mass disappearance, a big explosion, or a never-detected academic fraud.
No one would’ve imagined that everything started with a simple task:
Find the mean height in each faculty house.
— the Headmaster, Dumblydoor
Table of Contents
The Humblepuffs — The Limit of Humanity
The Humblepuffs, being very humble, used a democratic approach to complete the task. They gathered into the grand hall, divided into groups, and started measuring the average with calculators. The huge piles of paperwork didn’t seem to bother them. Everyone was happy to be part of such a wonderful project.
According to legend, the Humblepuffs are still calculating at this very moment, and the calculations will go on until they finally agree with each other’s result. No one but the Humblepuffs are allowed in the grand hall, so no one can ever know when the Humblepuffs will finally come out with their result.
The Greyfingers — Wise Leaders Hire Experts
As leaders, the Greyfingers house selected a pure-blood 1st-year talented wizard to solve the whole problem. The chosen one, the proud pure-blood wizard who hates calculators due to family tradition, devoted all his energy to create a powerful spell.
mean(Height)
Sadly, the wizard died because the creation of this spell drained his life. The Greyfingers followed his last wish and gave this spectacular spell to all other houses in the wizard school.
Then, the Greyfingers hired a team of experienced experts in the field of experience to execute the spell for them. Everything went well; the money was well spent. The Greyfingers even held a huge party in order to celebrate their victorious attempt.
The Ravingcows — The Limit of the Wand
The Ravingcows, being the smartest cows in the wizard world, had plenty of experiences with the Headmaster’s annoying questions. A century ago, the Ravingcows started building a huge database containing every Ravingcow House student’s detailed information. This database included the students’ height, weight, study fields, hobbies, past experiences, and so on. The database, as the top-secret file in Ravingcows house, was both valuable and dangerous… but it was huge!
size(Data)= 2^31
Even with the best material and the best wand maker, the maximum capacity a wand can handle is no larger than 2^31-1.
— Unknown Scholar
To minimize possible consequences, the Ravingcows chose the perfect day and place for casting the spell. When the time came, the Ravingcows cast the spell carefully, staring at the spell intently. And they waited. And waited. And waited.
There are not many records left for the Ravingcows case, as everything was burnt into ash on that day. A student’s diary was found on the ground. Apparently its owner didn’t stop writing in the diary until the last second of their life.
The spell’s sparkling light became brighter and brighter, growing to almost the same size as the sun. The calculation went on, however, and didn’t seem to stop after draining the last energy from the wizard’s wand.
“It’s going to explode!” A student shouted.
“That, my child, is the limit of our wand.”
— unknown
The Stayingthins — Cautions of Casting Spells
The Stayingthins also had a huge database containing detailed information about the students. However, after seeing the tragedy that occurred in Ravingcows House, the Stayingthins asked their house-elves to collect only a small sample with only the height data for current students from the huge database.
A Stayingthin student was then asked to cast the ultimate spell for the house:
The 1st Trial — Syntax Error
She cast the spell, but she was unfortunately hit back by a spell with one of the worst side effects – “Not Amused”.
mean(Height)
[1] NA
The 2nd Trial — The Unrealistic “Goblin”
After seeing the injury, a second student, shivering, quickly took a look into the data, and saw something that shouldn’t be there.
Height = [..., 6), 59.3, 69.1, ...]
He then removed the invalid element “6)” and re-cast the spell.
mean(Height)
[1] 50
The second student’s joy didn’t last long, as, upon reporting the results, he was faced with a simple retort:
“So, our entire house is 50 inches (< 1.3m) tall, on average?”
— the one who shall be named, Voidmortgage
The 3rd Trial — The Survivor
The third student tried to test the spell’s validity. But even the greatest wizard couldn’t find anything wrong with it. The spell was perfect, as it should always be.
It didn’t take long for the student to realize that Voidmortgage was solely after salvaging the Stayingthins’ reputation. The student considered altering the results or the reality, but they still needed to figure out why the outcome was nothing like Voidmortgage’s expectation.
With a simple spell, the truth revealed itself:
plot(Height)
Obviously, there must have been some mistakes made by the messenger owls… why were there entries with no height?
But upon verification, none of these entries were in fact 0; rather, they read 0.65 instead of 65, and so on. The student quickly wiped out those “incorrect” measurements and replace them with the correct measures before respectfully delivering the final result.
mean(Height)
[1] 64
Everyone was satisfied. The headmaster received the result, the Stayingthins’ reputation was preserved (on average, still taller than 160cm, phew!), and the 3rd student was granted the title of “the one who lived”.
…and they all lived happily ever after.
The Winning House
The Headmaster announced the victory of Greyfingers and Stayingthins. For the additional contribution from the Greyfingers’ wizard who invented the spell, the Greyfingers won the House Cup for the academic year.
The wizarding school, however, added a new school rule shortly after:
In order to minimize casualties, no student is allowed to cast any spell before completing the data preparation/cleaning process.
Endnote:
Dear readers and wizards (and muggles who intend to become wizards), we also strongly recommend that you follow this rule, as we would prefer our clever wizards not to suffer a most uncomfortable death, or what’s worse, singed hair.
Dearly yours,
Daily Prompts
