In this blog, we will build a virtual AI news reader who will read out news with accurate lip syncing.
Introduction - The Rise of AI News Anchors
Odisha TV, a private news channel based in Odisha, recently launched the first regional AI news anchor called ‘Lisa’. She is an AI-generated avatar clad in traditional Odia attire, and presents news in both Odia and English across the network's television and digital platforms. This development follows the debut of 'Fedha', an AI-generated news presenter introduced by Kuwait News, affiliated with the Kuwait Times.
The rise of AI news anchors like Lisa and Fedha highlights the growing importance and potential of artificial intelligence in the media industry. These AI presenters offer several advantages, such as the ability to deliver news consistently and efficiently without the need for breaks or time off. Moreover, they can be programmed to present news in multiple languages, making information more accessible to a wider audience.
Workflow
The workflow of building this application, in sequential order, is as follows:
- We fetch the latest top news using thenewsapi.com.
- We chunk the news articles into smaller documents and store them into a vector database.
- User inputs query.
- Information relevant to the query, along with the original query, is sent to the LLM (Llama 3 on Ollama) for generating a response.
- Response is converted to audio using TTS.
- This audio is then lip synced onto a standard video of a news reader using Wav2Lip.
The Code
Since we’ll be using many different types of AI technologies, we need a high-performance GPU for our task. E2E Networks provides a fleet of such GPUs geared for building our AI application. You can check out the offerings at https://myaccount.e2enetworks.com/.
Once you have spun a GPU node, the first step is to install the required libraries.
TTS
langchain
ollama
torch
First set up the text splitter, embeddings model, and prompt template for the RAG pipeline.
from langchain_text_splitters import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
# Set a really small chunk size, just to show.
chunk_size=512,
chunk_overlap=20,
length_function=len,
is_separator_regex=False,
)
from langchain.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(
model_name="BAAI/bge-large-en"
)
from langchain_core.prompts import PromptTemplate
# Define the ChatPromptTemplate for user interaction
template = """Answer the following question from the context
context = {context}
question = {question}
"""
prompt = PromptTemplate(input_variables=["context","question"], template= template)
Now, we’ll write a function that sends an API request to thenewsapi to receive the latest trending stories from India. Make sure to get your free API key by registering on this website.
import requests
from datetime import datetime, timedelta
def get_top_news():
# Get the current time and subtract one day
one_day_ago = datetime.now() - timedelta(days=1)
# Format the date and time
published_after_time = one_day_ago.strftime("%Y-%m-%dT%H:%M:%S")
# Define the endpoint URL
url = "https://api.thenewsapi.com/v1/news/top"
# Set the query parameters
params = {
'api_token': NEWS_API,
'locale': 'in',
'limit': 3,
'category': 'sports',
'published_after': published_after_time
}
# Send the GET request
response = requests.get(url, params=params)
# Check if the request was successful
if response.status_code == 200:
# Return the JSON response
return response.json()
else:
# Return the error message
return response.text
The response looks something like this:
{'meta': {'found': 1183, 'returned': 3, 'limit': 3, 'page': 1},
'data': [{'uuid': '40fc3f6e-4671-435f-9f3c-933500c61bb7',
'title': 'Al Nassr vs Al Ittihad Live Streaming: How To Watch Cristiano Ronaldo Play Live',
'description': 'Al Nassr vs Al Ittihad Saudi Pro League 2023-24 will be played today, Monday, 27 May. Know how to watch the live streaming of the football match in India. Check...',
'keywords': 'Al Nassr, Al Ittihad, Al Nassr vs Al Ittihad date, Al Nassr vs Al Ittihad time, Al Nassr vs Al Ittihad live streaming, Al Nassr vs Al Ittihad live telecast in India, Al Nassr vs Al Ittihad Saudi Pro League 2023-24, Al Nassr vs Al Ittihad Saudi Pro League, Al Nassr vs Al Ittihad Saudi Pro League 2024, Saudi Pro League 2023-24',
'snippet': 'Al Nassr is gearing up to face Al Ittihad in the final Saudi Pro League 2023-24 match on Monday, 27 May. The Al Nassr vs Al Ittihad match will be conducted at t...',
'image_url': 'https://images.thequint.com/thequint%2F2024-05%2Fe4d46606-a954-43f5-bf42-c7619c56fc3c%2F7e480a46f76c54b8a07de537b1b1121a.jpg',
'language': 'en',
'published_at': '2024-05-27T11:06:24.000000Z',
'source': 'thequint.com',
'categories': ['general'],
'relevance_score': None,
'locale': 'in'},
{'uuid': '8d73ddd9-f40a-409c-850a-86a53fd88cbd',
'title': "Iran's acting President addresses new Parliament after helicopter crash killing President, others",
'description': 'Iran’s acting President Mohammad Mokhber addressed the country’s new parliament in his first public speech since last week’s helicopter crash that killed ...',
'keywords': 'Iran, Iran parliament, Iran Raisi, Iran President, Iran acting President, Mohammad Mokhber',
'snippet': "Iran's acting President Mohammad Mokhber addressed the country's new parliament on May 27 in his first public speech since last week's helicopter crash that kil...",
'image_url': 'https://th-i.thgim.com/public/incoming/12j8jk/article68221259.ece/alternates/LANDSCAPE_1200/APTOPIX_Iran_Politcis_37563.jpg',
'language': 'en',
'published_at': '2024-05-27T11:03:49.000000Z',
'source': 'thehindu.com',
'categories': ['general', 'politics'],
...
'published_at': '2024-05-27T11:03:05.000000Z',
'source': 'thehindu.com',
'categories': ['general', 'politics'],
'relevance_score': None,
'locale': 'in'}]}
The above response contains URLs to the news articles. In order to get the complete news, we have to scrape the articles. We can do so using the function below:
import requests
from bs4 import BeautifulSoup
from langchain.docstore.document import Document
from langchain_community.vectorstores import FAISS
def scrape_news_data():
news_data = get_top_news()
scraped_data = []
for article in news_data['data']:
url = article['url']
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
# Extract the title of the article
title = soup.find('title').text
# Extract the text of the article
article_text = ''
for paragraph in soup.find_all('p'):
article_text += paragraph.text + '\n'
# Create a document to store the scraped data
scraped_article = Document(page_content= article_text, metadata= {'title': title})
scraped_data.append(scraped_article)
vectorstore = FAISS.from_documents(text_splitter.split_documents(scraped_data),embeddings)
global retriever
retriever = vectorstore.as_retriever(search_kwargs= {'k':10})
return 'News Collected Successfully'
Next, we write a helper function for a RAG application. This function generates the context from the vector store given a query.
After this, we write a function that uses Llama 3 from Ollama to generate a response to the user query.
from langchain_community.llms import Ollama
def respond_to_query(query):
context = get_context(query, retriever)
llm = Ollama(model="llama3")
return llm.invoke(prompt.format(question=query, context= context))
Make sure you’ve installed Ollama on your system, launched an Ollama server, and pulled Llama 3. You can follow the instructions here.
Then, we’ll create a function that takes text as input and uses TTS to generate the corresponding audio clip.
import subprocess
import TTS
def run_tts_command(text):
# Define the command as a list of arguments
command = [
'tts', # Command executable
'--text', text, # Text for TTS
'--model_name', 'tts_models/multilingual/multi-dataset/xtts_v2', # Model name
'--vocoder_name', 'vocoder_models/universal/libri-tts/wavegrad', # Vocoder name
'--out_path', '/home/vardh/ai-news-avatar/Wav2Lip/output.wav', # Output path
'--speaker_idx', 'Brenda Stern', # Speaker index
'--language_idx', 'en' # Language index
]
# Run the command
try:
result = subprocess.run(command, check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print("TTS generation successful, output saved to 'output.wav'")
return '/home/vardh/ai-news-avatar/Wav2Lip/output.wav' # Return the standard output if needed
except subprocess.CalledProcessError as e:
print(f"An error occurred: {e.stderr.decode()}")
return None
Finally, we come to the lip syncing part. First, clone the Wav2Lip repository:
git clone https://github.com/Rudrabha/Wav2Lip.git
Then download the model weights as shown in Readme, and place them in the checkpoints folder.
Another weight (missing in Readme) can be downloaded from here: https://drive.google.com/drive/folders/1oZRSG0ZegbVkVwUd8wUIQx8W7yfZ_ki1. Name it as mobilenet.pth and place it in the checkpoints directory.
Then we’ll write a function that generates a lip-synced video from the previous audio clip. It returns the final path of the generated video. The parameter face represents the input video.
import os
import subprocess
def run_wav2lip_command():
# Navigate to the Wav2Lip directory
wav2lip_dir = '/home/vardh/ai-news-avatar/Wav2Lip'
os.chdir(wav2lip_dir)
# Construct the command
command = [
'python', 'inference.py',
'--checkpoint_path', 'checkpoints/wav2lip.pth',
'--face', 'face.mp4',
'--audio', 'output.wav'
]
# Run the command
result = subprocess.run(command)
return '/home/vardh/ai-news-avatar/Wav2Lip/results/result_voice.mp4'
Gradio code for the UI:
import gradio as gr
with gr.Blocks() as demo:
with gr.Row():
btn = gr.Button("Fetch Latest News")
response = gr.Text()
with gr.Row():
query = gr.Textbox(label= "Ask me about the news")
news_text = gr.Textbox(label= "Response")
with gr.Row():
news_audio = gr.Audio(label= 'Audio Response', type= 'filepath')
news_video = gr.Video(label= 'Lip Synced Video')
btn.click(fn= scrape_news_data, inputs= None, outputs= response)
news_query = query.submit(fn= respond_to_query, inputs= query, outputs= news_text)
audio_query = news_query.then(fn= run_tts_command, inputs= news_text , outputs= news_audio)
audio_query.then(fn= run_wav2lip_command, inputs= None, outputs= news_video)
demo.launch(server_name='0.0.0.0')
Results
Here’s a short video demonstrating the quality of the lip-syncing.
Final Words
This blog provides a comprehensive guide to building an AI news reader that can fetch the latest news, generate responses to user queries, convert the responses to audio, and create a lip-synced video of a virtual news anchor presenting the news.
By leveraging advanced technologies like Llama 3 for text generation, TTS for voice synthesis, and Wav2Lip for lip syncing, one can easily generate a news reader avatar for custom use cases.
