Caste and Concrete: Deciphering India's Demographics through PMAY-G Data Analysis

As a Data Scientist, part of my role involves delving into data to unearth insights and formulate recommendations. In this article, I've taken a close look at the caste demographics data released by the government of India (https://data.gov.in/catalog/pradhan-mantri-awaas-yojana-gramin). After thorough analysis, I've come up with my own insights and suggestions. However, it's important to note that these insights should be further reviewed by policy experts.

With India's vast population of 1.4 billion, understanding the intricacies of its demographics for policy-making is indeed a colossal task. In my work, I feel like I've only scooped a drop from the ocean of data available, but I'm hopeful that even this small contribution can make a meaningful impact in the grand scheme of things.

Link to my code:

• Github
• Google Colab

import pandas as pd
import warnings

# Ignore all warnings
warnings.filterwarnings('ignore')

data = pd.read_csv('/content/OGDSECCAwaasPlusData_18092023.csv', error_bad_lines=False)
data.head()        

# Dropping rows with missing values in any of the social group columns
data_cleaned = data.dropna(subset=['Minority', 'Others', 'SC', 'ST'])

# Convert social group columns to numeric, if not already
data_cleaned['SC'] = pd.to_numeric(data_cleaned['SC'], errors='coerce')
data_cleaned['ST'] = pd.to_numeric(data_cleaned['ST'], errors='coerce')
data_cleaned['Minority'] = pd.to_numeric(data_cleaned['Minority'], errors='coerce')
data_cleaned['Others'] = pd.to_numeric(data_cleaned['Others'], errors='coerce')        

Population Distribution by State

# Population Distribution by State
# We'll calculate the total population for each state

state_population_totals = data_cleaned.groupby('state_name')['Total'].sum().reset_index()

# Sorting states by total population
sorted_state_population = state_population_totals.sort_values(by='Total', ascending=False)        

Most populous states

# Identifying the top 5 most populous states
top_5_populous_states = sorted_state_population.head(5)
top_5_populous_states        

Least populous states

My Insights:

• I noticed that the most populous states in India are spread across different geographical regions, showing a diverse distribution of the population.
• On the other hand, the least populous states, especially Puducherry and Telangana with their zero population figures, make me think there might be inconsistencies or missing data.

My Recommendations:

• In highly populated states such as West Bengal and Madhya Pradesh, I believe it's crucial to focus on scalable infrastructure and resource management strategies to support the large number of people.
• For states that show very low or zero populations, I recommend a thorough investigation into the data. It's likely that these figures point to missing or incomplete data.

Social Group Distribution Across States

# Social Group Distribution Across States
# We'll calculate the proportion of each social group (SC, ST, Minority, Others) in each state

# Convert social group columns to numeric, if not already
data_cleaned['SC'] = pd.to_numeric(data_cleaned['SC'], errors='coerce')
data_cleaned['ST'] = pd.to_numeric(data_cleaned['ST'], errors='coerce')
data_cleaned['Minority'] = pd.to_numeric(data_cleaned['Minority'], errors='coerce')
data_cleaned['Others'] = pd.to_numeric(data_cleaned['Others'], errors='coerce')

# Group by state and calculate total for each social group
state_social_group_totals = data_cleaned.groupby('state_name')[['SC', 'ST', 'Minority', 'Others']].sum()

# Calculating proportion of each social group in each state
state_social_group_totals['Total'] = state_social_group_totals.sum(axis=1)
state_social_group_totals['SC_proportion'] = (state_social_group_totals['SC'] / state_social_group_totals['Total']) * 100
state_social_group_totals['ST_proportion'] = (state_social_group_totals['ST'] / state_social_group_totals['Total']) * 100
state_social_group_totals['Minority_proportion'] = (state_social_group_totals['Minority'] / state_social_group_totals['Total']) * 100
state_social_group_totals['Others_proportion'] = (state_social_group_totals['Others'] / state_social_group_totals['Total']) * 100        

Top 5 states with highest SC population

# Sorting states based on SC proportions
sorted_by_sc = state_social_group_totals.sort_values(by='SC_proportion', ascending=False).head(5)
sorted_by_sc[['SC_proportion']]        

Top 5 states with highest ST population

# Sorting states based on ST proportions
sorted_by_st = state_social_group_totals.sort_values(by='ST_proportion', ascending=False).head(5)
sorted_by_st[['ST_proportion']]        

My Insights:

• Punjab's High SC Population: Punjab has a notably higher SC population compared to other states, a unique demographic aspect.
• ST Populations in the Northeast and UTs: The ST population is predominantly in the northeastern states and Union Territories like Lakshadweep and Ladakh.
• High ST Percentages in Some States: Some states, like Lakshadweep with a 100% ST population, show exceptionally high percentages, which might indicate unique demographics or a need for data verification.

My Recommendations:

• Understanding Demographics for Policy-Making: Recognizing these demographic distributions is crucial for creating targeted social and economic policies in states with significant SC or ST populations.
• Investigating Unusual Proportions: Further investigation is needed in states with unusual demographic proportions, like Lakshadweep, to ensure data accuracy and understand the demographic dynamics.

Demographic Composition at the Panchayat Level (Rural Demographics)

# Analyzing demographic composition at the Panchayat level

# Grouping data by state and panchayat and calculating the mean for social groups
panchayat_demographics = data_cleaned.groupby(['state_name', 'Panchayat_Code']).agg({
    'Minority': 'mean',
    'Others': 'mean',
    'SC': 'mean',
    'ST': 'mean',
    'Total': 'mean'
}).reset_index()

# Calculating state-wise average demographics at the panchayat level
state_avg_panchayat_demographics = panchayat_demographics.groupby('state_name').mean()

# Displaying the demographics for a few states for brevity
state_avg_panchayat_demographics.head()        

Insights:

• There is considerable variation in the composition of social groups at the panchayat level among different states.
• States like Assam and Bihar show higher numbers for Minority and Others groups, while ST is significantly higher in Arunachal Pradesh. The Total Population average at the panchayat level varies widely, indicating differences in panchayat sizes and population densities across states.

Recommendations:

• Policies and resources allocation at the rural level could be tailored based on these demographic characteristics, especially in states with higher concentrations of certain social groups.
• Further investigation into states with large minority populations at the panchayat level (like Assam and Bihar) could be crucial for targeted developmental programs.

Geographical Distribution of Social Groups within Selected States

I analyzed the mean population of SC, ST, Minority, and Others groups by district in a few selected states: West Bengal, Bihar, Maharashtra, Tamil Nadu, and Uttar Pradesh.

# Geographical Distribution of Social Groups within Selected States
# For this analysis, we'll select a few states and look at the distribution of social groups by district

# Selecting a few states for analysis
selected_states = ['WEST BENGAL', 'BIHAR', 'MAHARASHTRA', 'TAMIL NADU', 'UTTAR PRADESH']

# Filtering data for the selected states
selected_states_data = data_cleaned[data_cleaned['state_name'].isin(selected_states)]

# Grouping by state and district and calculating the mean for social groups
district_social_group_distribution = selected_states_data.groupby(['state_name', 'district_name']).agg({
    'Minority': 'mean',
    'Others': 'mean',
    'SC': 'mean',
    'ST': 'mean'
}).reset_index()

# Displaying the social group distribution for districts in these states
district_social_group_distribution.head()        

Insights:

• There's a noticeable variation in the distribution of social groups among different districts within the same state.
• For instance, in Bihar, districts like Araria have a high minority population compared to others like Arwal. The variation in SC and ST populations also indicates diverse demographic compositions across districts.

Recommendations:

• Regional policies and development programs should consider these demographic variations.
• For example, districts with higher minority populations might require different approaches compared to those with higher SC or ST populations. Further detailed analysis, including other states and districts, can help in understanding regional demographic dynamics, which is essential for tailored socio-economic planning and resource allocation.

Comparison of Districts within Selected States

I've compared the districts within selected states (West Bengal, Bihar, Maharashtra, Tamil Nadu, and Uttar Pradesh) to identify those with notably high or low populations for each social group (Minority, Others, SC, ST). Here's a summary for a few districts in Bihar:

# Comparison of Districts within Selected States

# Calculating the total population for each social group by district in each state
district_totals = selected_states_data.groupby(['state_name', 'district_name']).sum().reset_index()

# We'll look for districts with notably high or low populations for each social group within each state
# Calculating the mean and standard deviation for each social group in each state
state_district_stats = district_totals.groupby('state_name').agg({
    'Minority': ['mean', 'std'],
    'Others': ['mean', 'std'],
    'SC': ['mean', 'std'],
    'ST': ['mean', 'std']
})

# Merging these stats back into the district totals
merged_data = district_totals.merge(state_district_stats, on='state_name', suffixes=('', '_state_stats'))

# Calculating the z-score for each social group in each district
for group in ['Minority', 'Others', 'SC', 'ST']:
    merged_data[f'{group}_z_score'] = (merged_data[group] - merged_data[(group, 'mean')]) / merged_data[(group, 'std')]

# Filtering districts with high or low populations (z-score > |2|)
unusual_districts = merged_data[(merged_data['Minority_z_score'].abs() > 2) | 
                                (merged_data['Others_z_score'].abs() > 2) | 
                                (merged_data['SC_z_score'].abs() > 2) | 
                                (merged_data['ST_z_score'].abs() > 2)]

# Selecting relevant columns for display
unusual_districts_display = unusual_districts[['state_name', 'district_name', 'Minority', 'Others', 'SC', 'ST', 
                                               'Minority_z_score', 'Others_z_score', 'SC_z_score', 'ST_z_score']]

# Displaying a few entries for brevity
unusual_districts_display.head()        

Insights:

• Districts like Araria and Gaya in Bihar show a significantly high population of Minority and SC groups, respectively.
• Pashchim Champaran has a notably high ST population, while Purbi Champaran stands out for its high 'Others' population.

Recommendations:

• Districts with unusually high populations of certain social groups might require specialized policies and programs tailored to their unique demographic compositions.
• The disparities in social group populations across districts within the same state highlight the need for decentralized planning and resource allocation to address local needs effectively.
• For the next analysis, let's explore the block-level demographics to uncover more localized trends. This can offer insights into the distribution of populations in smaller administrative units within the states.

Block-Level Demographic Analysis in Bihar

I conducted a block-level demographic analysis in Bihar, focusing on the mean population of social groups (Minority, Others, SC, ST) in each block. Here are the findings for a few blocks in the district of Araria as an example:

# Block-Level Analysis

# Selecting a state for a more detailed block-level analysis
# For demonstration, let's choose 'Bihar'
bihar_data = data_cleaned[data_cleaned['state_name'] == 'BIHAR']

# Grouping by block and calculating the mean for social groups
block_level_demographics = bihar_data.groupby(['district_name', 'block_name']).agg({
    'Minority': 'mean',
    'Others': 'mean',
    'SC': 'mean',
    'ST': 'mean',
    'Total': 'mean'
}).reset_index()

# Displaying the demographics for a few blocks for brevity
block_level_demographics.head()        

Insights:

• There is significant variation in the distribution of social groups even at the block level within the same district.
• Some blocks, like Araria and Jokihat, have notably high minority populations, whereas others like Bhargama have higher SC populations.

Recommendations:

• Block-level insights are crucial for micro-level planning and implementing community-specific interventions.
• Understanding these variations can aid in designing more effective local governance strategies and allocating resources where they are most needed.

Analysis of Panchayat-Level Variance Across States

I analyzed the variance in population sizes and social group compositions at the panchayat level. This analysis helps us understand the diversity and disparities in demographic distributions within rural areas of different states. Here are the top states with the highest variance in total population at the panchayat level:

# Analysis of Panchayat-Level Variance

# We'll calculate the variance in population sizes and social group compositions at the panchayat level
# This will help us understand the diversity within rural areas in different states

# Grouping data by state and panchayat
panchayat_variance_analysis = data_cleaned.groupby(['state_name', 'Panchayat_Code']).agg({
    'Minority': 'var',
    'Others': 'var',
    'SC': 'var',
    'ST': 'var',
    'Total': 'var'
}).reset_index()

# Calculating state-wise average variance
state_avg_panchayat_variance = panchayat_variance_analysis.groupby('state_name').mean()

# Sorting states based on variance in total population
sorted_states_by_population_variance = state_avg_panchayat_variance.sort_values(by='Total', ascending=False)

# Displaying the top states with the highest variance in total population
sorted_states_by_population_variance.head()        

Insights:

• West Bengal shows a significantly high variance in total population at the panchayat level, indicating a wide disparity in population sizes across different rural areas within the state.
• Other states like Assam, Jharkhand, and Bihar also exhibit substantial variances, suggesting diverse population distributions in their rural areas.

Recommendations:

• States with high variances in population distributions, especially at the panchayat level, may require more nuanced and localized approaches to development and resource allocation.
• Understanding these variances is crucial for effective rural development strategies, as it highlights the need for policies and programs that are adaptable to the specific needs and characteristics of each panchayat.
• This comprehensive analysis, encompassing state, district, block, and panchayat levels, offers a multi-layered understanding of the demographic landscape in India. It underscores the importance of tailored approaches to policy-making and development initiatives, respecting the unique demographic profiles at each administrative level.