Building a YouTube Transcript Summarizer with Streamlit and IBM WatsonX

In this article, we'll create a YouTube Transcript Summarizer using Streamlit and IBM 's IBM watsonx Mistral AI Model. This application will allow users to input a YouTube URL, extract the video transcript, and generate a summary using the WatsonX Mistral AI model.

Github Link : https://github.com/arulbenjaminchandru/02-Youtube-Podcast-Summarizer/

Prerequisites

- Python installed on your system

- Basic understanding of Python programming

- IBM WatsonX API credentials

Step 0: Get watsonX API credentials

We need an API Key and project ID to access the models present in watsonX. To get that, please follow below instructions,

• Get a WML API key from https://cloud.ibm.com/iam/apikeys
• Create a watsonX project in https://dataplatform.cloud.ibm.com/wx/home and associate it with WML service on cloud (Watson Machine Learning). Also get the project id of this project.

//** Comment Below if you need detailed instruction on how to do this **//        

Step 1: Set Up Your Environment

First, ensure you have the necessary libraries installed. You can install them using pip:

pip install streamlit re youtube_transcript_api pysbd python-dotenv ibm-watsonx        

Next, create a .env file in your project directory and add your WatsonX API key and project ID:

WATSONX_API_KEY=your_watsonx_api_key

PROJECT_ID=your_project_id        

Step 2: Import Required Libraries

Begin by importing the necessary libraries:

import streamlit as st

import re

import os

from youtube_transcript_api import YouTubeTranscriptApi, TranscriptsDisabled

import pysbd

import dotenv

from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes

from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams

from ibm_watsonx_ai.foundation_models import ModelInference        

Step 3: Load Environment Variables

Load your environment variables from the .env file:

dotenv.load_dotenv()        

Step 4: Configure WatsonX Model

Create a function to configure the WatsonX Mistral AI Model:

def configure_watsonx():

    credentials = {

        "url": "https://us-south.ml.cloud.ibm.com",

        "apikey": os.getenv("WATSONX_API_KEY")

    }

    project_id = os.getenv("PROJECT_ID")

    print([model.name for model in ModelTypes])

    model_id = "mistralai/mixtral-8x7b-instruct-v01"

    parameters = {

        GenParams.DECODING_METHOD: "greedy",

        GenParams.MIN_NEW_TOKENS: 1,

        GenParams.MAX_NEW_TOKENS: 150

    }

    model = ModelInference(

        model_id=model_id, 

        params=parameters, 

        credentials=credentials,

        project_id=project_id

    )

    

    return model        

Step 5: Define Helper Functions

Create functions to extract the YouTube video ID, get the video transcript, chunk large text, and summarize the text:

seg = pysbd.Segmenter(language='en', clean=True)

def extract_youtube_video_id(url: str) -> str:

    found = re.search(r"(?:youtu\.be\/|watch\?v=)([\w-]+)", url)

    if found:

        return found.group(1)

    return None

def get_video_transcript(video_id: str) -> str | None:

    try:

        transcript = YouTubeTranscriptApi.get_transcript(video_id)

    except TranscriptsDisabled:

        return None

    return ' '.join([line["text"] for line in transcript])

def chunk_large_text(text_list: list, max_size: int) -> list[str]:

    txts = []

    para = ''

    for s in text_list:

        s_len = len(s)

        if para and len(para) + s_len > max_size:

            txts.append(para)

            para = ''

        if s_len <= max_size:

            para += s + '\n'

        else:

            if para:

                txts.append(para)

                para = ''

            cut = s_len // max_size

            chunk = s_len // (cut + 1)

            i = 0

            while i < s_len:

                if s_len - i <= chunk:

                    txts.append('… ' + s[i:] + ' …')

                    break

                clip_i = s.find(' ', i + chunk)

                txts.append('… ' + s[i:clip_i] + ' …')

                i = clip_i + 1

    if para:

        txts.append(para)

    return txts

def summarize_large_text(text_list: list, max_size: int) -> str:

    summaries = ""

    txts = chunk_large_text(text_list, max_size)

    summaries = summaries.join(txts)

    return summaries        

Step 6: Create the Streamlit UI

Set up the Streamlit interface for user input and display:

st.title('YouTube Transcript Summarizer using watsonx Mistral 7B')

url = st.text_input('Enter YouTube URL')

submit = st.button('Submit')        

Step 7: Handle User Input and Display Results

When the user submits a YouTube URL, extract the video ID, get the transcript, summarize it, and display the results:

if submit and url:

    video_id = extract_youtube_video_id(url)

    if video_id:

        transcript = get_video_transcript(video_id)

        if transcript:

            text_list = seg.segment(transcript)

            summary = summarize_large_text(text_list, max_size=2048)

            st.subheader("Extracted Text from Video")

            st.write(summary)

            if summary:

                model = configure_watsonx()

                prompt = "Summarize the following content within 150 tokens : "

                summary = prompt + "\n" + summary

                response = model.generate(summary)

                st.subheader('Summary')

                st.write(response.get('results')[0]['generated_text'])

        else:

            st.write("No transcript found for this video.")

    else:

        st.write("Invalid YouTube URL.")        

Conclusion

You've now built a YouTube Transcript Summarizer using Streamlit and IBM WatsonX Mistral AI Model. This application extracts the transcript from a given YouTube video URL, chunks the transcript into manageable pieces, and generates a concise summary. Feel free to expand this project further by adding more features or improving the summarization process.