Exploratory Data Analysis is an important aspect of any data science project. It forms the initial steps before moving into the Machine learning aspects.
In this tutorial we will be exploring the drug review dataset using python in an elaborate way. In doing EDA (exploratory data analysis) it is recommended to keep in mind the basic questions you want to find answers to using your dataset. This will direct you on the various analysis to use and how deep to explore the given data for more insight. In our case we will be breaking our questions into questions on the following
- Drugs
- Reviews
- Ratings
- Conditions
- Combinations
We will be using the dataset from UCI machine learning repository which already have some basic info about what we will be doing.
By the end of this tutorial you will learn about
- The various libraries to use for EDA
- Descriptive analytics
- How to do value counts
- How to generate some plots for more insights
- How to classify drugs based on their suffixes
- How to do sentiment analysis on drug reviews
- How to find and identify genuine review
- Time series analysis on drug review and rating
- Distribution Analysis
- and More
You can get the entire code on Github here.
Let us start.
Data Science EDA Project From Scratch with Python
- Tools & Libraries
- EDA: Pandas
- Viz: Seaborn,Matplotlib
- NLP:spaCy,TextBlob,NeatText
- ML: sklearn,xgboost,pycaret
DataSource
Attributes
- drugName (categorical): name of drug
- condition (categorical): name of condition
- review (text): patient review
- rating (numerical): 10 star patient rating
- date (date): date of review entry
- usefulCount (numerical): number of users who found review useful
Questions
- Types of questions we can ask?(Drugs,Review,Rating,Conditions,Time,Genuiness,etc)
- What is the most popular drug?
- What are the groups/classification of drugs used?
- Which Drug has the best review?
- How many drugs do we have?
- The number of drugs per condition
- Number of patients that searched on a particular drug
- How genuine is the review? (Using sentiment analysis)
- How many reviews are positive,negative,neutral?
- Correlation between rating and review and users who found the review useful
- Can you predict the rating using the review?
- Distribution of rating
- Amount of review made per year and per month
- Which condition has the most review on drugs
In [2]:
# Load EDA Pkgs
import pandas as pd
import numpy as np
In [3]:
# Load Data Viz
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
In [4]:
# Load Sentiment Pkgs
from textblob import TextBlob
Question on Drugs
- How many drugs do we have?
- What is the most popular drug?
- What are the groups/classification of drugs used?
- Which Drug has the best review?
- The number of drugs per condition
- Number of patients that searched on a particular drug
In [27]:
# Load Dataset
df = pd.read_csv("drugsCom_raw/drugsComTrain_raw.tsv",sep='\t')
In [28]:
# Preview Dataset
df.head()
Out[28]:
In [29]:
# Columns
df.columns
Out[29]:
In [30]:
# Missing Values
df.isnull().sum()
Out[30]:
Narrative
- Most of the missing values are in the condition column
- This implies that most people don’t know their condition by name or privacy
Question on Drugs
- How many drugs do we have?
In [32]:
# How many drugs do we have?
df['drugName'].unique().tolist()
Out[32]:
In [33]:
# How many drugs do we have?
len(df['drugName'].unique().tolist())
Out[33]:
In [34]:
# What is the most popular drug?
df['drugName'].value_counts()
Out[34]:
In [35]:
# What is the most popular drug?
# Top 20 Drugs (Most Popular)
df['drugName'].value_counts().nlargest(20)
Out[35]:
In [36]:
# Top 20 Drugs (Most Popular)
plt.figure(figsize=(20,10))
df['drugName'].value_counts().nlargest(20).plot(kind='bar')
plt.title("Top 20 Most popular drugs based on counts")
plt.show()
Narrative
- Most of the commonest drugs are hormonal drugs
In [37]:
# Least 20 Drugs (Most Popular)
df['drugName'].value_counts().nsmallest(20)
Out[37]:
In [38]:
df['drugName'].value_counts().nsmallest(20).plot(kind='bar')
Out[38]:
In [ ]:
### What are the groups/classification of drugs used?
+ suffix or endings
In [39]:
drug_suffix = {"azole":"antifungal (except metronidazole)",
"caine":"anesthetic",
"cillin":"antibiotic(penicillins)",
"mycin":"antibiotic",
"micin":"antibiotic",
"cycline":"antibiotic",
"oxacin":"antibiotic",
"ceph":"antibiotic(cephalosporins)",
"cef":"antibiotic (cephalosporins)",
"dine":"h2 blockers (anti-ulcers)",
"done":"opiod analgesics",
"ide":"oral hypoglycemics",
"lam":"anti-anxiety",
"pam":"anti-anxiety",
"mide":"diuretics",
"zide":"diuretics",
"nium":"neuromuscular blocking agents",
"olol":"beta blockers",
"tidine":"h2 antagonist",
"tropin":"pituitary hormone",
"zosin":"alpha blocker",
"ase":"thrombolytics",
"plase":"thrombolytics",
"azepam":"anti-anziety(benzodiazepine)",
"azine":"antipyschotics (phenothiazine)",
"barbital":"barbiturate",
"dipine":"calcium channel blocker",
"lol":"beta blocker",
"zolam":"cns depressants",
"pril":"ace inhibitor",
"artan":"arb blocker",
"statins":"lipid-lowering drugs",
"parin":"anticoagulants",
"sone":"corticosteroid (prednisone)"}
In [40]:
def classify_drug(drugname):
for i in drug_suffix.keys():
if drugname.endswith(i):
print(True)
print(drug_suffix[i])
In [41]:
classify_drug('Valsartan')
In [43]:
classify_drug('losartan')
In [44]:
def classify_drug(drugname):
for i in drug_suffix.keys():
if drugname.endswith(i):
return drug_suffix[i]
In [45]:
classify_drug('valsartan')
Out[45]:
In [46]:
df['drug_class'] = df['drugName'].apply(classify_drug)
In [47]:
df[['drugName','drug_class']]
Out[47]:
In [48]:
# How many Groups of Drugs By Class
df['drug_class'].unique().tolist()
Out[48]:
In [50]:
# How many Groups of Drugs By Class
len(df['drug_class'].unique().tolist())
Out[50]:
In [51]:
# Which of class of drug is the most commonest
df['drug_class'].value_counts()
Out[51]:
In [52]:
# Which of class of drug is the most commonest
plt.figure(figsize=(20,10))
df['drug_class'].value_counts().plot(kind='bar')
plt.title("Distribution of Drugs By Class")
plt.show()
Narrative
- The most commonest class/group of drugs used is
- Antifungal
- Opiod Analgesics(Pain Killers)
- Oral Hypoglycemics (DM)
- Antibiotic
In [69]:
# Distribution of Drugs Per Drug Group based on size
drug_groups = df.groupby('drug_class').size()
In [70]:
type(drug_groups)
Out[70]:
In [71]:
# Convert to DF
# Method 1
drug_groups.to_frame()
Out[71]:
In [73]:
# Convert to DF
# Method 2
drug_groups_df = pd.DataFrame({'drug_class':drug_groups.index,'counts':drug_groups.values})
In [75]:
# Seaborn Plot
plt.figure(figsize=(20,10))
g = sns.barplot(data=drug_groups_df,x='drug_class',y='counts')
plt.show()
In [76]:
# Seaborn Plot
plt.figure(figsize=(20,10))
g = sns.barplot(data=drug_groups_df,x='drug_class',y='counts')
g.set_xticklabels(drug_groups_df['drug_class'].values,rotation=30)
plt.show()
In [77]:
# Seaborn Plot
plt.figure(figsize=(20,10))
g = sns.barplot(data=drug_groups_df,x='drug_class',y='counts')
plt.xticks(rotation=30)
plt.show()
In [ ]:
### Question on Conditions
+ How many conditions are there?
+ Which conditions are the most common?
+ Distribution of conditions and rating
In [54]:
# Number of Conditions
df['condition'].unique()
Out[54]:
In [55]:
len(df['condition'].unique().tolist())
Out[55]:
Narrative
- We have 885 different conditions
In [56]:
#### Distribution of Conditions
df['condition'].value_counts()
Out[56]:
In [57]:
#### Most commonest Conditions
df['condition'].value_counts().nlargest(20)
Out[57]:
In [58]:
#### Most commonest Conditions
df['condition'].value_counts().nlargest(20).plot(kind='bar',figsize=(20,10))
Out[58]:
Narrative
- The most commonest condition is Birth Control,followed by Depression and Pain and Anxiety
- Makes sense as compared to the drug distribution
In [60]:
df['condition'].value_counts().nsmallest(20)
Out[60]:
In [59]:
#### Least commonest Conditions
df['condition'].value_counts().nsmallest(20).plot(kind='bar',figsize=(20,10))
Out[59]:
Questions on Drugs and Conditions
- How many drugs per condition
In [63]:
# How many Drugs per condition (Top 20)
df.groupby('condition')['drugName'].nunique().nlargest(20)
Out[63]:
In [66]:
# How many Drugs per condition (Top 20)
plt.figure(figsize=(15,10))
df.groupby('condition')['drugName'].nunique().nlargest(20).plot(kind='bar')
plt.title("Number of Drugs Per Condition")
plt.grid()
plt.show()
Narrative
- Pain,Birth Control and HBP have the highest number of different/unique drugs for their condition
In [ ]:
#### Questions on Rating
+ Distribution of rating
+ Average Rating Per Count
In [78]:
df['rating']
Out[78]:
In [79]:
# Distrubtion of Rating By Size
df.groupby('rating').size()
Out[79]:
In [80]:
# Distrubtion of Rating By Size
df.groupby('rating').size().plot(kind='bar')
Out[80]:
In [81]:
# # Distrubtion of Rating By Size Using Histogram
plt.figure(figsize=(20,10))
df['rating'].hist()
plt.title("Distrubtion of Rating By Size Using Histogram")
plt.show()
Narative
- Most people rated at the extremes
In [83]:
# Average Rating of Drugs
avg_rating = (df['rating'].groupby(df['drugName']).mean())
In [84]:
avg_rating
Out[84]:
In [86]:
# Average Rating For All Drugs
plt.figure(figsize=(20,10))
avg_rating.hist()
plt.title("Distrubtion of Average Rating For All Drugs")
plt.show()
In [92]:
# Average Rating of Drugs By Class
avg_rating_per_drug_class = (df['rating'].groupby(df['drug_class']).mean())
In [93]:
avg_rating_per_drug_class
Out[93]:
In [94]:
# Average Rating For All Drugs
plt.figure(figsize=(20,10))
avg_rating_per_drug_class.hist()
plt.title("Distrubtion of Average Rating For Drug Classes")
plt.show()
In [96]:
# Which Group of Drugs have the higest mean/average rating
avg_rating_per_drug_class.nlargest(20)
Out[96]:
In [97]:
# Which Drugs have the higest mean/average rating
avg_rating.nlargest(20)
Out[97]:
In [98]:
df.columns
Out[98]:
In [ ]:
### Question on Review
+ How genuine is the review? (Using sentiment analysis)
+ How many reviews are positive,negative,neutral?
+ Correlation between rating and review and users who found the review useful
+ Distribution of rating
+ Amount of review made per year and per month
+ Which condition has the most review on drugs
+ Can you predict the rating using the review?
In [99]:
# How genuine is the review? (Using sentiment analysis)
from textblob import TextBlob
In [100]:
df['review']
Out[100]:
In [101]:
def get_sentiment(text):
blob = TextBlob(text)
return blob.polarity
def get_sentiment_label(text):
blob = TextBlob(text)
if blob.polarity > 0:
result = 'positive'
elif blob.polarity < 0:
result = 'negative'
else:
result = 'neutral'
return result
In [102]:
# text fxn
get_sentiment("I love apples")
Out[102]:
In [104]:
# text fxn
get_sentiment_label("I love apples")
Out[104]:
In [105]:
# Sentiment Score for Review
df['sentiment'] = df['review'].apply(get_sentiment)
In [106]:
# Sentiment Labels for Review
df['sentiment_label'] = df['review'].apply(get_sentiment_label)
In [107]:
df[['review','sentiment','sentiment_label']]
Out[107]:
In [109]:
# How many positive and negative and neutral reviews?
df['sentiment_label'].value_counts()
Out[109]:
In [110]:
# How many positive and negative and neutral reviews?
df['sentiment_label'].value_counts().plot(kind='bar')
Out[110]:
In [111]:
#### Correlation Between Our sentiment and rating
sns.lineplot(data=df,x='rating',y='sentiment')
plt.show()
Narrative
- The rating increases with increase in sentiment
In [112]:
# Correlation btween rating and sentiment
sns.lineplot(data=df,x='rating',y='sentiment',hue='sentiment_label')
Out[112]:
In [ ]:
# How many reviews are genuine as compared to the rating
+ genuine good rating =positive + rating 10-6
+ genuine bad rating = negative + rating 4-1
In [119]:
# Genuine Good Rating Per Review
good_review = df[(df['rating'] >= 6) & (df['sentiment_label'] == 'positive')]
In [117]:
# Genuine Bad Rating Per Review
bad_review = df[(df['rating'] <= 4) & (df['sentiment_label'] == 'negative')]
In [120]:
good_review.head()
Out[120]:
In [122]:
good_review.iloc[0]['review']
Out[122]:
In [ ]:
In [ ]:
#### Questions on UsefulCount
+ number of users who found review useful
+ Top UsefulCount By Drugs/Class
+ Best drugs based usefulcount
In [124]:
df.groupby('drugName')['usefulCount'].value_counts()
Out[124]:
In [126]:
# Top Drugs Per UsefulCount
df.groupby('drugName')['usefulCount'].nunique().nlargest(20)
Out[126]:
In [127]:
# Top Drugs Per UsefulCount
df.groupby('drugName')['usefulCount'].nunique().nlargest(20).plot(kind='bar')
Out[127]:
In [128]:
# Top Drugs Class Per UsefulCount
df.groupby('drug_class')['usefulCount'].nunique().nlargest(20)
Out[128]:
In [129]:
# Top Drugs Class Per UsefulCount
df.groupby('drug_class')['usefulCount'].nunique().nlargest(20).plot(kind='bar')
plt.title("Top Drug Class Per Usefulcount")
plt.show()
In [130]:
# Top Drugs Class Per UsefulCount
df.groupby('drug_class')['usefulCount'].nunique().nsmallest(20).plot(kind='bar')
plt.title("Least Drug Class Per Usefulcount")
plt.show()
In [131]:
### Correlation between Rating and Usefulcount
sns.lineplot(data=df,x='rating',y='usefulCount')
Out[131]:
Narrative
- As the rating goes up the usefulcount goes up
In [133]:
#### Question on Date
df.columns
Out[133]:
In [134]:
# Rating Per Year
df.groupby('date')['rating'].size()
Out[134]:
In [135]:
# Averaging Rating Per Day of A Year
df.groupby('date')['rating'].mean()
Out[135]:
In [138]:
# Average Rating Per Day of Every Year
df.groupby('date')['rating'].mean().plot(figsize=(20,10))
plt.title("Average Rating Per Day of Every Year")
plt.show()
In [139]:
# Average Useful Per Day of Every Year
df.groupby('date')['usefulCount'].mean().plot(figsize=(20,10))
plt.title("Average UsefulCount Per Day of Every Year")
plt.show()
In [140]:
# Average Sentiment Per Day of Every Year
df.groupby('date')['sentiment'].mean().plot(figsize=(20,10))
plt.title("Average sentiment Per Day of Every Year")
plt.show()
In [144]:
# Amount of Review Per Day of Every Year
df.groupby('date')['review'].size().plot(figsize=(20,10))
plt.title("Amount of Review Per Day of Every Year")
plt.show()
In [145]:
# Amount of Review Per Day of Every Year
df.groupby('date')['review'].size().plot(kind='bar',figsize=(20,10))
plt.title("Amount of Review Per Day of Every Year")
plt.show()
In [ ]:
In [150]:
#### Using DatetimeIndex
grouped_date = df.groupby('date').agg({'rating':np.mean,'usefulCount':np.sum,'review':np.size})
In [151]:
grouped_date
Out[151]:
In [154]:
grouped_date.index
Out[154]:
In [155]:
grouped_date['date'] = grouped_date.index
In [157]:
grouped_date['date'] = pd.DatetimeIndex(grouped_date['date'])
In [158]:
grouped_date.dtypes
Out[158]:
In [159]:
grouped_date = grouped_date.set_index('date')
In [161]:
# Select A Particular Date Range
grouped_date['2008'].plot()
Out[161]:
In [164]:
# AMount of Review Fr 2008
grouped_date['2008']['review'].plot()
plt.title("Amount of Review For 2008")
plt.show()
In [166]:
# AMount of Review Fr 2008
grouped_date['2008':'2009']['review'].plot()
plt.title("Amount of Review For 2008-2009")
plt.show()
In [167]:
# Distribution of Rating Over Time
grouped_date['2008':'2009']['rating'].plot()
plt.title("Distribution of Rating Over Time")
plt.show()
In [169]:
# Distribution of Rating Over Time
grouped_date['2008':'2012']['rating'].plot(figsize=(20,10))
plt.title("Distribution of Rating Over Time")
plt.show()
In [172]:
grouped_date['2008-04'].plot()
Out[172]:
In [173]:
# Distribution of Rating Over A Month
grouped_date['2008-4':'2008-5']['rating'].plot()
plt.title("Distribution of Rating Over Time")
plt.show()
In [174]:
# Save Dataset
df.to_csv("drug_review_dataset_with_sentiment.csv",index=False)
You can also check out the video tutorial on YouTube or below
Thanks for Your Time
Jesus Saves
By Jesse E.Agbe(JCharis)