Monday, May 8, 2017

Make your test data comply with training data

When using scikit-learn libraries with pandas, you would often get errors if the test data does not have columns which match the model created from the training data set. You may have deleted columns that you did not need from the training data when creating the model or you may have constructed new columns based on existing data variables (for example, creating 'Age' from 'Date' or combining the effects of multiple variables). In most cases, this is simple to achieve by wringing the test data through the same function as the training data. However, when you are using One-Hot-Encoding of categorical data, the columns created as a result of this coding in the training set and the test set may not match for the simple reason that some values for categorical data may only be present in the training data set and others may be present only in the test data set. I have included here a python function that I wrote and works really well to match the test data with the training data:

def fixdata(data, train_data):
    # first step is to delete any columns in data that are not present in train_data
    columns_to_delete = data.columns.difference(train_data.columns)
    data = data.drop(columns_to_delete, 1)

# second step is to add columns which are in training data and set their values to zero
    columns_to_add = train_data.columns.difference(data.columns)
    df_add = pd.DataFrame(columns=columns_to_add, index=data.index)
    df_add.fillna(0, inplace=True)

return pd.concat([data, df_add], axis=1)

view raw

About Tukeys

Tukeys provides consulting and software services to companies in healthcare and communications that are dealing with technical or scaling challenges. Please reach out to me on LinkedIn.

An interesting quote from John Tukey, who was an American mathematician (1915-2000) -

An approximate answer to the right problem is worth a good deal more than an exact answer to an approximate problem.

Today maths and statistics form the basis of how software is developed and used around us.

About Me

Medhavi: I have over 20 years of experience in healthcare, communications and AI.

I have a passion for defining innovative applications, creating product/technology strategy and solving challenging problems. My product mindset is helpful to any company to find patterns, reduce costs, and gain efficiencies.

I was the VP of Engineering at Vibrent Health, where I led the web, mobile, API and data products that supported one of the largest NIH programs - All of Us Research Program. Our system enrolled almost a million users and collected data from EHRs, surveys and wearables. Before that, I worked at hCentive, a pioneer in health exchanges.

As the CTO of 3CLogic, I helped create the first PC and web-only contact center solution that implemented PBX and CRM functionality as an affordable SaaS solution. As the Chief Architect of NexTone, I was instrumental in the development of SIP and SIP based products. SIP and associated protocols form the basis of modern WebRTC APIs.

If you are a growth company or invest in growing companies that are facing technical or scaling challenges, I can be a tremendous help. Contact me on LinkedIn.

Featured Post

Random Forest Analysis of U.S. National Epidemiological Survey on Alcohol and Related Conditions (NESARC) data

In 2001/2002, the National Institute on Alcohol Abuse and Alcoholism (NIAAA) conducted the National Epidemiologic Survey on Alcohol and Re...