I have attached a rough guideline for the midterm. Handle the features one column at a time. If you attempt to deal with all of the features at once, you'll notice that the errors you get aren't useful for determining their origin. Following are a few things worth mentioning.
1. Someone noticed that there are duplicate rows in a data table. Consider doing something about this.
2. Someone mentioned that if you handle the train and test separately, you get different number of features. This is entirely possible when a category does not appear in a particular set. There are 2 ways to handle this, following is one idea
Create a Master table which takes the train and test set and combines them. There is only one way to do this.
Before binding you'll need to create an index variable so that the Master table can eventually be separated back into the train and test.
Do the feature manipulation on the Master table.
Separate the Master table back into train and test.
This also avoids needing to do any feature manipulation twice. Melt and Dcast may prove useful.
3. You may decide that a particular feature needs to be removed entirely. This is fine. However you need to add comments for justification. Comments in general is something I'll be checking in this assignment. In general, I want you to use all features.