How to Extract S&P 500 Data and Build a Stock Screener in Streamlit with EODHD API

Introduction

In the fast-paced world of investing, having access to real-time and historical market data is essential. Whether you’re an investor looking for undervalued stocks or a developer building a financial application, accessing accurate S&P 500 data efficiently can give you a competitive edge.

Manually tracking the performance of 500 stocks is impractical. However, with the power of web scraping and API integration, we can automate this process and create a real-time stock screener. Imagine having a dashboard that allows you to search for any S&P 500 company and retrieve its historical stock prices and key financials instantly.

This guide will show you how to build a web app using Streamlit that extracts and displays data for all 500 stocks in the S&P 500 index. We’ll use web scraping to fetch stock symbols, integrate EODHD API to retrieve historical and fundamental data, and build an interactive search function so users can easily filter and analyze companies.

Early in my investment journey, I tried different platforms such as TradingView and Alpaca, but I found that they often required complex integrations or had limited free data access. After extensive research, I chose EODHD API (check it out here) for its ease of use, reliability, and comprehensive dataset.

As Peter Lynch once said, “Know what you own, and know why you own it.” Having the right tools makes this possible.

By the end of this tutorial, you will have a fully functional stock screener that fetches and visualizes data for any S&P 500 stock. Let’s get started!

Step 1: Required Libraries

Before running the script, install the necessary Python libraries by executing the following command:

pip install streamlit requests pandas plotly beautifulsoup4 openpyxl        

• streamlit: Used for building the interactive web app.
• requests: For making HTTP requests to fetch data from APIs.
• pandas: For handling and processing tabular data.
• plotly: For creating interactive stock price charts.
• beautifulsoup4: For web scraping the list of S&P 500 stocks from Wikipedia.
• openpyxl: Required for exporting data to an Excel file.

Step 2: About EODHD API and Endpoints Used

EODHD API provides high-quality financial data, including real-time, historical, and fundamental stock data. In this project, we use the following endpoints:

1. Fundamental Data API: Retrieves detailed company information such as sector, market capitalization, P/E ratio, revenue, and other key financial metrics.
2. Historical Data API: Fetches stock price history, including open, high, low, close, and volume data.

Example JSON Response from EODHD API

Here’s an example response from the Historical Data API:

[
    {
        "date": "1980-03-17",
        "open": 124.9008,
        "high": 126.9856,
        "low": 124.044,
        "close": 124.9008,
        "adjusted_close": 0.3814,
        "volume": 1422012
    },
]        

This data allows us to visualize market trends and make informed investment decisions.

Step 3: Web Scraping S&P 500 Stock Symbols

To retrieve data for all S&P 500 companies, we first need their stock symbols. We’ll use web scraping to extract them from Wikipedia.

import requests
import pandas as pd
from bs4 import BeautifulSoup

def get_sp500_symbols():
    url = "https://en.wikipedia.org/wiki/List_of_S%26P_500_companies"
    response = requests.get(url)
    if response.status_code == 200:
        soup = BeautifulSoup(response.text, 'html.parser')
        table = soup.find("table", {"class": "wikitable"})
        df = pd.read_html(str(table))[0]
        return df["Symbol"].tolist()
    return []        

This function scrapes Wikipedia and extracts the stock symbols of S&P 500 companies into a list.

Step 4: Building the Stock Screener in Streamlit

Now, let’s integrate our S&P 500 stock symbols into a Streamlit application that fetches and visualizes stock data.

import streamlit as st
import requests
import pandas as pd
import plotly.graph_objects as go

API_KEY = "67b2f38ee71ed8.40514694"
BASE_URL = "https://eodhd.com/api"

# Fetch stock price data
def get_stock_data(symbol, period):
    url = f"{BASE_URL}/eod/{symbol}.US?api_token={API_KEY}&period={period}&order=d&fmt=json"
    response = requests.get(url)
    return response.json() if response.status_code == 200 else []

# Fetch fundamental data
def get_fundamental_data(symbol):
    url = f"{BASE_URL}/fundamentals/{symbol}.US?api_token={API_KEY}&fmt=json"
    response = requests.get(url)
    return response.json() if response.status_code == 200 else {}

st.title("S&P 500 Stock Screener")

sp500_symbols = get_sp500_symbols()
ticker_search = st.text_input("Search for a stock symbol", "AAPL")
filtered_stocks = [s for s in sp500_symbols if ticker_search.upper() in s]
selected_stock = st.selectbox("Select a stock", filtered_stocks)
time_periods = {"Daily": "d", "Weekly": "w", "Monthly": "m"}
selected_period = st.selectbox("Select time scale", list(time_periods.keys()))

data = get_stock_data(selected_stock, time_periods[selected_period])

if data:
    df = pd.DataFrame(data)
    df["date"] = pd.to_datetime(df["date"])
    st.subheader("Stock Price Table")
    st.dataframe(df[["date", "open", "high", "low", "close", "volume"]])

    if st.button("Export to Excel"):
        df.to_excel("stock_data.xlsx", index=False)
        st.success("Data exported successfully! Check stock_data.xlsx.")

    st.subheader(f"Candlestick Chart ({selected_period})")
    fig = go.Figure()
    fig.add_trace(go.Candlestick(x=df["date"], open=df["open"], high=df["high"], low=df["low"], close=df["close"], name=selected_stock))
    fig.update_layout(xaxis_rangeslider_visible=False)
    st.plotly_chart(fig)        

Explaining the Implemented Code and Its Functionalities

The stock screener application built with Streamlit consists of multiple functionalities aimed at providing investors with real-time stock data and historical insights. Below is a breakdown of the main components, functionalities, and potential improvements for this project.

1. Functionalities of the Stock Screener Application

This Streamlit web app enables users to:

• Search for any S&P 500 stock using a dynamic search box.
• Fetch historical stock price data (open, high, low, close, volume) from EODHD API.
• Display stock price data in a table format.

• Generate interactive candlestick charts using Plotly.

• Retrieve fundamental financial data (P/E ratio, market cap, revenue, etc.).

• Export stock price data to an Excel file with a single click.

2. Breaking Down the Code

Fetching S&P 500 Stock Symbols with Web Scraping

The function get_sp500_symbols() scrapes the Wikipedia page listing S&P 500 companies and extracts their ticker symbols. This ensures that our screener always contains up-to-date stock listings.

Fetching Historical Stock Price Data

def get_stock_data(symbol, period):
    url = f"{BASE_URL}/eod/{symbol}.US?api_token={API_KEY}&period={period}&order=d&fmt=json"
    response = requests.get(url)
    return response.json() if response.status_code == 200 else []        

• This function calls EODHD’s Historical Data API and retrieves daily, weekly, or monthly price data for the selected stock.
• The JSON response contains stock open, high, low, close prices, and trading volume.

Fetching Fundamental Stock Data

def get_fundamental_data(symbol):
    url = f"{BASE_URL}/fundamentals/{symbol}.US?api_token={API_KEY}&fmt=json"
    response = requests.get(url)
    return response.json() if response.status_code == 200 else {}        

• is function queries the Fundamental Data API to extract company-specific details, including:
• Market capitalization
• P/E ratio
• EPS (Earnings Per Share)
• Revenue and other financial metrics

User Interface with Streamlit

• The st.text_input() allows users to search for stock tickers dynamically.
• The st.selectbox() let users choose between Daily, Weekly, and Monthly price views.
• A st.button() allows exporting the retrieved stock data to an Excel file.

Generating Candlestick Charts

fig = go.Figure()
fig.add_trace(go.Candlestick(x=df["date"], open=df["open"], high=df["high"], low=df["low"], close=df["close"], name=selected_stock))
fig.update_layout(xaxis_rangeslider_visible=False)
st.plotly_chart(fig)        

• The Candlestick chart provides an interactive price visualization.
• Users can explore trends and patterns directly from the web app.

3. How the Application Can Be Improved

While this stock screener is already powerful, three key improvements could make it even better:

1. Add Technical Indicators:

• Implement moving averages, RSI, and MACD indicators to enhance investment decision-making.
• Use the TA-Lib or pandas_ta library.

2. Enhance Data Filtering & Sorting:

• Allow users to sort stocks by market cap, P/E ratio, or price changes.
• Provide preset stock screen filters for undervalued and growth stocks.

3. Real-time Stock Price Updates:

• Integrate WebSockets or set a refresh button to update stock data dynamically.
• Display live price changes from the API.

4. Why I Chose EODHD Over TradingView & Alpaca

Initially, I explored different platforms like TradingView and Alpaca, but each had limitations:

• TradingView offers beautiful charts but lacks easy data export and API flexibility.
• Alpaca is great for trading execution, but requires account setup and lacks fundamental data.
• EODHD API was the most versatile, offering:

1. Historical and fundamental data in a single API.

2. No need for brokerage accounts.

3. Easy integration with Python (Get access here).

Conclusion

By combining web scraping, API integration, and Streamlit, we created a powerful S&P 500 Stock Screener. This tool allows investors and developers to search for any company, retrieve historical price data, and access key financials instantly.

For high-quality financial data, check out EODHD API with 10% of discount and start building your next investment tool today! ??

• 10% discount in EODHD using the following link
• Sign up for EODHD API today
• Subscribe to the Data Pulse Newsletter https://www.dhirubhai.net/newsletters/datapulse-python-finance-7208914833608478720