Unique Building Blocks of DSPy
DSPy offers unique features for simplifying the process of building an advanced RAG pipeline.
First it offers Signatures, which are a declarative/programmatic method of structuring prompts. This helps avoid long and tedious verbalization in creating prompts.
Secondly, there are Modules, which abstract a prompting technique on top of Signatures. One of the popular modules is dspy.ChainofThought, which injects a rationale (e.g., “think step by step to come to a conclusion”) before the outcome of a prompt. This module makes multiple calls to the language model (using the prescribed Signature) and gradually builds reasoning until it is ready to answer the output of the Signature using the input questions and the reasoning it has developed. This approach significantly improves the quality of the final answer.
Finally, we have Optimizers that can tune your entire DSPy program based on a Metric. This program can be a single Module or a set of Modules bundled together to perform a complicated task. DSPy Metrics can define exactly which aspect of the program we want to optimize. For example, we can define a Metric that checks for the length of the final response generated by the language model. This way, we can optimize our final program to only output responses of a certain desired length.
The above features make DSPy a powerful alternative to traditional prompt engineering techniques. Crafting multiple hand-written prompts can often be tedious and clunky, and unpredictable in terms of outcome, especially when changing the underlying Large Language Model. This is because each language model adheres to its own specific inner template that it performs best on.
DSPy overcomes these challenges by automating the laborious work of prompt engineering through the use of Signatures, Modules, and Optimizers.
Multi-Hop Optimized RAG
In a multiple-hop RAG pipeline, the original question is broken down into multiple queries over several steps. In each step, the language model forms a query and retrieves context based on it. This iterative process of collecting context by breaking down the original question into smaller queries allows for a more enhanced retrieval method. This way, the language model has access to a factually richer context for generating the final response.
To further optimize the pipeline, we can introduce a few metrics. We can ensure that none of the sub-queries are rambling by limiting their length to fewer than 100 characters. Additionally, we verify that no two sub-queries are similar to each other. This way, we make sure we are retrieving unique contexts in each step.
In this blog, we’ll demonstrate how to build a multi-hop optimized RAG using DSPy.
Build with E2E
This blog is part of the “Build with E2E series” which showcases various emerging technologies for building AI applications.
E2E Networks is an NSE-listed AI-first hyperscaler that provides advanced cloud computing and GPU infrastructure to developers, startups, enterprises, and research organizations.
E2E Cloud’s TIR AI-ML platform is a powerful AI development platform built around Jupyter Notebooks. TIR includes pre-configured environments for popular frameworks like PyTorch, TensorFlow, Stable Diffusion, RAPIDS, and Transformers. It also features an inference platform for deploying models directly on the E2E Cloud platform.
Check out more about the platform at https://myaccount.e2enetworks.com/.
Let’s Code
Begin by installing the dependencies.
pip install dspy-ai ollama qdrant-client fastembed
We’ll be using an external dataset of Mobile Phone Reviews from Kaggle. This dataset contains cell phone brands and their reviews. The aim is to design a RAG pipeline that can list various phone names by analyzing their reviews against the specific questions posed by the user. Let’s format the data before inserting it into the vector database.
import pandas as pd
# Read the CSV file into a DataFrame
df = pd.read_csv('Amazon_Unlocked_Mobile.csv')
# Initialize an empty list to store the formatted strings
formatted_list = []
# Iterate over each row in the DataFrame
for index, row in df.iterrows():
# Format the string as required
formatted_string = f'Phone Name: {row["Product Name"]}\nReview: {row["Reviews"]}'
# Append the formatted string to the list
formatted_list.append(formatted_string)
Some examples of how the data looks:
random.sample(formatted_list,3)
['Phone Name: Apple iPhone 3G Black, 16GB\nReview: My son loves it, I cant even touch it. LOL',
'Phone Name: HTC Desire 610 8GB Unlocked GSM 4G LTE Quad-Core Android 4.4 Smartphone - Black (No Warranty)\nReview: The phone is very good , takes very sharp pictures but the screen is not bright',
'Phone Name: Apple iPhone 6, Space Gray, 128 GB (Sprint)\nReview: I am very satisfied with the purchase, i got my iPhone 6 on time and even received a screen protectant with a charger. Thank you so much for the iPhone 6, it was worth the wait.']
Let’s take 2000 rows of the dataset at random and insert them into the Qdrant Vector Database.
import random
from qdrant_client import QdrantClient
# Initialize the client
client = QdrantClient(":memory:")
import random
def add_documents_in_batches(client, collection_name, formatted_list, batch_size=1000):
"""
Adds documents to the collection in batches and prints a message after each batch.
Parameters:
client: The client object to interact with the database.
collection_name (str): The name of the collection.
formatted_list (list): The list of documents to add.
batch_size (int): The number of documents to add in each batch. Default is 1000.
"""
# Shuffle and sample the list
sampled_list = random.sample(formatted_list, 2000)
# Iterate over the sampled list in chunks of batch_size
for i in range(0, len(sampled_list), batch_size):
batch = sampled_list[i:i + batch_size]
batch_ids = list(range(i + 1, i + 1 + len(batch)))
# Add the batch to the collection
client.add(
collection_name=collection_name,
documents=batch,
ids=batch_ids
)
# Print a message indicating the batch has been added
print(f"Batch {i // batch_size + 1} added with {len(batch)} documents.")
# Example usage
# Assuming `client` is your database client and `formatted_list` contains your documents
add_documents_in_batches(client, "phone_collection", formatted_list, batch_size=1000)
We’ll configure DSPy’s retriever and language model so that it can use them to make calls.
from dspy.retrieve.qdrant_rm import QdrantRM
import dspy
qdrant_retriever_model = QdrantRM("phone_collection", client)
ollama_model = dspy.OllamaLocal(model="llama3",model_type='text',
max_tokens=350,
temperature=0.1,
top_p=0.8, frequency_penalty=1.17, top_k=40)
dspy.settings.configure(lm= ollama_model, rm=qdrant_retriever_model)
Make sure you have Ollama installed on your system. You can do so by running the following in your terminal.
curl -fsSL https://ollama.com/install.sh | sh
Launch the server and pull the LLM llama3.
ollama serve
ollama pull llama
Now, we’ll write two DSPy Signatures. One to generate the sub-queries, and another to generate the final answer.
class GenerateSearchQuery(dspy.Signature):
"""Write a simple search query that will help answer a complex question."""
context = dspy.InputField(desc="may contain relevant facts")
question = dspy.InputField()
query = dspy.OutputField()
class GenerateAnswer(dspy.Signature):
"""Generate a numbered list of phone names"""
context = dspy.InputField(desc="may contain relevant facts")
question = dspy.InputField()
answer = dspy.OutputField(format=list)
The docstring is important as it will contain the instruction line that you want to include in your prompt. Similarly, the names of the variables should be self-explanatory and meaningful, as they will also be passed on to the prompt.
Then we write a simplified Baleen class that defines the multi-hop logic of our RAG pipeline. The __init__ method of the class is used to declare and initialize all the modules that are to be used in the program. In the forward method, the modules are run using the inputs of the pipeline. The final output is returned as a dspy.Prediction object.
It is important to follow this structure while writing DSPy programs, as they will be used during the optimization process.
from dsp.utils import deduplicate
class SimplifiedBaleen(dspy.Module):
def __init__(self, passages_per_hop=3, max_hops=2):
super().__init__()
self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]
self.retrieve = dspy.Retrieve(k=passages_per_hop)
self.generate_answer = dspy.ChainOfThought(GenerateAnswer)
self.max_hops = max_hops
def forward(self, question):
context = []
for hop in range(self.max_hops):
query = self.generate_query[hop](context=context, question=question).query
passages = self.retrieve(query).passages
context = deduplicate(context + passages)
pred = self.generate_answer(context=context, question=question)
return dspy.Prediction(context=context, answer=pred.answer)
Let’s create a dummy dataset for our trainset.
trainset_list = [
{
"Question": "Which phones have the best camera quality and battery life based on recent reviews and specifications?",
"Answer": "Here's a list of phones that meet your criteria:\n\n1. Samsung Galaxy S21 Ultra\n2. Google Pixel 6 Pro\n3. Apple iPhone 13 Pro Max\n4. OnePlus 9 Pro\n5. Xiaomi Mi 11 Ultra\n\nNotes: These phones were picked based on their high ratings for camera quality and long-lasting battery life, as reported by recent reviews and detailed specifications."
},
{
"Question": "What are the top-rated phones with the best display and performance in the market right now?",
"Answer": "Here's a list of phones that meet your criteria:\n\n1. Samsung Galaxy S22\n2. Apple iPhone 14 Pro\n3. OnePlus 10 Pro\n\nNotes: These phones were selected because they have received excellent reviews for their display clarity and performance speed, making them ideal for users seeking high-quality visuals and efficient processing."
},
{
"Question": "Can you recommend phones that have the best user interface and build quality according to recent user reviews?",
"Answer": "Here's a list of phones that meet your criteria:\n\n1. Nokia 8.3 5G\n2. Sony Xperia 1 III\n\nNotes: These phones were chosen due to their outstanding user interface design and robust build quality, which have been highly praised in recent user reviews and expert evaluations."
}
]
trainset = [dspy.Example(question=item["Question"], answer=item["Answer"]).with_inputs('question') for item in trainset_list]
Next, we’ll write our Metric to ensure that the hops are of limited length and are not repetitive. DSPy Metrics return bool values and, therefore, are strict measures that the optimizer will make the pipeline adhere to.
def validate_answer_and_hops(example, pred, trace=None):
# if not validate(pred.answer == example.answer): return False
hops = [example.question] + [outputs.query for *_, outputs in trace if 'query' in outputs]
if max([len(h) for h in hops]) > 100: return False
if any(dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8) for idx in range(2, len(hops))): return False
return True
Note that we are not making any use of the trainset here, since we are only optimizing for the hops. If we also want to optimize for the final response, we can write a function that validates the predicted answer (pred.answer) given an example question to be similar to the example answer. (Logic in comments.)
With that set, let’s optimize the pipeline.
from dspy.teleprompt import BootstrapFewShot
teleprompter = BootstrapFewShot(metric=validate_answer_and_hops, )
compiled_baleen = teleprompter.compile(student=SimplifiedBaleen(), trainset= trainset)
The optimization process tweaks each of the DSPy modules involved in the program. This includes the language model weights of these modules, their instructions, and demonstrations of input and output behavior.
Now, let’s try out a few tests:
compiled_baleen("Among the latest smartphones, which ones have received the best ratings for battery life in expert reviews and also offers the highest screen resolution according to its technical specifications?")
"Here is the numbered list of phone names that meet your criteria:
1. **BLU Energy X Plus**
2. **BLU VIVO 5R Refresh**
3. **I9220(N9000)**
4. **BLU PURE XL**
These phones are known for their long-lasting batteries and high-quality displays with good screen resolutions.
Here's the list in more detail:
1. **BLU Energy X Plus**: With its massive 4000 mAh super battery, this phone is designed to keep going all day.
2. **BLU VIVO 5R Refresh**: This phone has a large 3200 mAh battery and a high-resolution display with a screen resolution of up to FHD (1080p).
3. **I9220(N9000)**: Although it's an older model, this phone still packs a punch with its dual batteries and decent screen resolution.
4. **BLU PURE XL**: This phone has a large 3500 mAh battery and a high-resolution display with a screen resolution of up to FHD (1080p).
Please note that the ratings for these phones may vary depending on the source and methodology used in expert reviews."
uncompiled_baleen("Which smartphones are highly rated for its low-light camera performance also have a great front camera")
Based on the provided context and reviews, I've generated a numbered list of phone names that meet the criteria:
1. BLU Studio Selfie (Grey) - Unlocked Smartphone - GSM
2. BLU Studio Selfie (Gold) - Unlocked Smartphone - GSM
These two phones have great reviews and are likely to perform well in low-light conditions, considering the positive comments about their cameras.
Note: The other phone names mentioned do not meet the criteria as they have mixed or poor camera performance according to their reviews.
If we want to see the LM calls that are happening under the hood:
ollama_model.inspect_history(n=4)
Write a simple search query that will help answer a complex question.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the query}. We ...
Query: ${query}
---
Context: N/A
Question: Which smartphones are highly rated for its low-light camera performance also has a great front camera
Reasoning: Let's think step by step in order to Here is the search query:
Context: N/A
Question: Which smartphones are highly rated for its low-light camera performance also has a great front camera?
Reasoning: Let's think step by step in order to produce the query. We can start by searching for "best smartphone cameras" and then add filters such as "low light performance" and "front facing camera". This will help us find smartphones that are highly rated for their camera capabilities, including low-light performance.
Query: ("best smartphone cameras") AND ("low light performance" OR "night mode") AND ("great front camera")
This query uses a combination of keywords to search for smartphones with high-rated cameras that perform well in low-light conditions and have great front-facing cameras. The quotes around the phrases ensure that we find exact matches, rather than individual words.
Write a simple search query that will help answer a complex question.
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the query}. We ...
Query: ${query}
---
Context:
[1] «Phone Name: BLU Studio 7.0 II -Unlocked Smartphone - US GSM - Grey
Review: Phone has a poor camera»
[2] «Phone Name: BLU Studio C Super Camera -Unlocked Smartphone - US GSM- Green
Review: good»
[3] «Phone Name: BLU S480U Unlocked Studio 7.0 II Smartphone with 5MP Main Camera (Gold)
Review: Great phone just bad camera and storage room»
Question: Which smartphones are highly rated for its low-light camera performance also has a great front camera
Reasoning: Let's think step by step in order to Here is the search query that can help answer the complex question:
---
Context:
[1] «Phone Name: BLU Studio 7.0 II -Unlocked Smartphone - US GSM - Grey
Review: Phone has a poor camera»
[2] «Phone Name: BLU Studio C Super Camera -Unlocked Smartphone - US GSM- Green
Review: good»
[3] «Phone Name: BLU S480U Unlocked Studio 7.0 II Smartphone with 5MP Main Camera (Gold)
Review: Great phone just bad camera and storage room»
Question: Which smartphones are highly rated for its low-light camera performance also has a great front camera
Reasoning: Let's think step by step in order to ${produce the query}. We need to find phones that have good reviews for their cameras, specifically low-light performance. Additionally, we want phones with high ratings for their front-facing cameras.
Query: "best smartphones with good low light camera and great selfie camera"
This search query uses relevant keywords from the context and question to help answer the complex question about which smartphones are highly rated for their low-light camera performance and have a great front-facing camera.
Generate a numbered list of phone names
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ['$', '{', 'a', 'n', 's', 'w', 'e', 'r', '}']
---
Context:
[1] «Phone Name: BLU Studio 7.0 II -Unlocked Smartphone - US GSM - Grey
Review: Phone has a poor camera»
[2] «Phone Name: BLU Studio C Super Camera -Unlocked Smartphone - US GSM- Green
Review: good»
[3] «Phone Name: BLU S480U Unlocked Studio 7.0 II Smartphone with 5MP Main Camera (Gold)
Review: Great phone just bad camera and storage room»
[4] «Phone Name: BLU Studio Selfie - Smartphone - GSM Unlocked - Grey
Review: I love it thanks»
[5] «Phone Name: BLU Studio Selfie - Smartphone - GSM Unlocked - Gold
Review: Great»
[6] «Phone Name: BLU S480U Unlocked Studio 7.0 II Smartphone with 5MP Main Camera (Gold)
Review: It works great»
Question: Which smartphones are highly rated for its low-light camera performance also has a great front camera
Reasoning: Let's think step by step in order to Based on the provided context and reviews, I've generated a numbered list of phone names that meet the criteria:
1. BLU Studio Selfie (Grey) - Unlocked Smartphone - GSM
* Review: "I love it thanks" implies good performance for its camera.
2. BLU Studio Selfie (Gold) - Unlocked Smartphone - GSM
* Review: "Great" suggests a well-performing front camera.
These two phones have great reviews and are likely to perform well in low-light conditions, considering the positive comments about their cameras.
Generate a numbered list of phone names
---
Follow the following format.
Context: may contain relevant facts
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ['$', '{', 'a', 'n', 's', 'w', 'e', 'r', '}']
---
Context:
[1] «Phone Name: BLU Studio 7.0 II -Unlocked Smartphone - US GSM - Grey
Review: Phone has a poor camera»
[2] «Phone Name: BLU Studio C Super Camera -Unlocked Smartphone - US GSM- Green
Review: good»
[3] «Phone Name: BLU S480U Unlocked Studio 7.0 II Smartphone with 5MP Main Camera (Gold)
Review: Great phone just bad camera and storage room»
[4] «Phone Name: BLU Studio Selfie - Smartphone - GSM Unlocked - Grey
Review: I love it thanks»
[5] «Phone Name: BLU Studio Selfie - Smartphone - GSM Unlocked - Gold
Review: Great»
[6] «Phone Name: BLU S480U Unlocked Studio 7.0 II Smartphone with 5MP Main Camera (Gold)
Review: It works great»
Question: Which smartphones are highly rated for its low-light camera performance also has a great front camera
Reasoning: Let's think step by step in order to Based on the provided context and reviews, I've generated a numbered list of phone names that meet the criteria: 1. BLU Studio Selfie (Grey) - Unlocked Smartphone - GSM * Review: "I love it thanks" implies good performance for its camera. 2. BLU Studio Selfie (Gold) - Unlocked Smartphone - GSM * Review: "Great" suggests a well-performing front camera. These two phones have great reviews and are likely to perform well in low-light conditions, considering the positive comments about their cameras.
Answer: [] Based on the provided context and reviews, I've generated a numbered list of phone names that meet the criteria:
1. BLU Studio Selfie (Grey) - Unlocked Smartphone - GSM
2. BLU Studio Selfie (Gold) - Unlocked Smartphone - GSM
These two phones have great reviews and are likely to perform well in low-light conditions, considering the positive comments about their cameras.
Note: The other phone names mentioned do not meet the criteria as they have mixed or poor camera performance according to their reviews.
Final Words
DSPy provides an entirely new paradigm for building AI applications by offering a programmatic approach to prompting instead of manual crafting. Any multistage AI agentic system is extremely sensitive to prompt quality and design. By offloading the tedious prompt engineering work to DSPy, we can easily conceive complicated multistage agents and RAG pipelines for our specific use cases.
By taking inspiration from this blog, you can deploy your DSPy-powered AI stack on E2E Cloud. Head over to E2E Networks to spin up a cloud GPU node, or use the readymade TIR AI platform!
