As mentioned in my latest post, I am working on a self-evaluating Financial Chatbot. So why the detour to update the current chatbot Sydney? Well, a few reasons:
- Test Weaviate vector store end to end:
- I am trying to use Weaviate as the main vector store for the financial chatbot but I am running into the issue of scaling/memory size. Specifically, with just the last 10 year of 10K and 10Q filings for about 20 companies, the size of weaviate collection is already 2GB. So with the current approach, the entire S&P 500 companies would mean a collection of about 50+ GB. It is way too big and will cost a lot of money to run/maintain. So I am trying to test different Product Quantization (PQ) parameters.
- After trying many other vector stores, I am leaning towards Weaviate because of its ability to conduct hybrid search with metadata filters at fast speeds.
- As I was working on PQ for Weaviate, one question I had was how to deploy Weaviate in a production environment, i.e., Weaviate Cloud or using AWS/Google Cloud? Will it be difficult to deploy? How much money it will cost?
- Because of these questions, I decided to deploy Weaviate for the current chatbot Sydney. So basically I will replace FAISS with Weaviate.
- For the current version, I am using Weaviate Cloud
- Implement query translation and query structuring, while making sure that output is in Json so that they can be used.
- With query translation, the goal is to break down the input into a set of specific sub-problems or sub-questions that can be answered in isolation.
- With query structuring: I care a lot about generating not only a suitable search term/phrases for hybrid search but also suitable meta data for filtering.
- This is super important because I want the financial chatbot to filter correctly for year, industry, etc., as and when needed.
- How to conduct hybrid-search with multiple filters and return not only the content but also the metadata.
As you can see I need all of the above to work to create the financial chatbot so why not trying to apply that in a much smaller scale to the chatbot Sydney. 🙂
I am glad to say that you can try Sydney now. It has all of the above functionalities. You can try to ask questions like the below and the chatbot should return relevant answers with links to specific blog posts where it get the content from.
- What did chandler write about Kevin Rudd in 2020?
- Tell me everything that Chandler wrote about Ray Dalio between 2020 until now
- what did chandler write about Health Savings Accounts in 2022?
- What did chandler do in 2015?
That’s it for now. I need to go back to work on the financial chatbot.
Update Sep 2024
Sydney is now a multi-talented agent capable of:
- Answering questions about the current S&P 500 companies, including what they’ve submitted to the SEC over the last 10 years.
- Providing insights from my 15 years of blog content.
Check it out here.
P.S: Below are examples of the type of prompt that I am using for query translation and query structuring.
"You are a helpful assistant that generates multiple sub-questions related to an input question. "
"The current year is 2024."
"The goal is to break down the input into a set of specific sub-problems / sub-questions that can be answered in isolation. "
"Each specific sub-question will be used to retrieve relevant content from a vector store, using similarity search with score. "
"Phrase the wording of the questions appropriately for this purpose.\n"
"This vector store includes all of the published blog posts from Chandler Nguyen's blog from 2007 to 2024.\n\n"
"Original question: {query}\n\n"
"Generate the minimum number of sub-questions necessary to answer the original question. "
"Your response should be formatted as a JSON array of strings, where each string represents a sub-question. "
"Do not include any additional words, characters, or explanations in the response.\n\n"
"Example response:\n"
'[\n'
' "sub-question 1",\n'
' "sub-question 2"\n'
']'"""You are a helpful assistant that generates a structured query related to an input question.
The goal is to break down the input into a structured query that can be used to retrieve relevant content from a vector store, using similarity search with score.
This vector store includes all of the published blog posts from Chandler Nguyen's blog from 2007 to 2024.
Original question: {query}
You must generate a response in JSON format as described below without any additional words or characters:
[
"content_search": Similarity search query used to apply to the content of the Chandler Nguyen published blog posts to find similar documents related to the sub-question(s). Ensure the content_search query is not too broad or too specific, and strikes a balance between relevance and completeness. \n
"start_date": optional field, the start date to search for blog posts that are relevant to the sub-question(s) in YYYY-MM-DD format. If the sub-question(s) do not specify a time frame, leave this field blank or set it to the earliest possible date (e.g., 2007-01-01) to cover a broader range. \n
"end_date": optional field, the end date to search for blog posts that are relevant to the sub-question(s) in YYYY-MM-DD format. If the sub-question(s) do not specify a time frame, leave this field blank or set it to the latest possible date (e.g., 2024-12-31) to cover a broader range. \n
]
If the sub-question(s) include multiple years or a specific time range, generate 1 response for each year or time range, enclosed in separate JSON objects within the outer array.
Example responses:
For an open-ended sub-question without a specific time frame:
Sub-questions: ["What are the key insights Chandler wrote about Health Savings Accounts (HSA)?"]
[
"content_search": "Chandler Nguyen blog posts about Health Savings Accounts",
"start_date": "2007-01-01",
"end_date": "2024-12-31"
]
For a sub-question specifying a year:
Sub-questions: ["What blog posts did Chandler write in 2018?", "Which blog posts written by Chandler in 2018 mention Kevin Rudd?"]
[
"content_search": "Chandler Nguyen blog posts in 2018",
"start_date": "2018-01-01",
"end_date": "2018-12-31"
],
[
"content_search": "Kevin Rudd mentioned in Chandler Nguyen blog posts in 2018",
"start_date": "2018-01-01",
"end_date": "2018-12-31"
]
For a sub-question specifying a time range:
Sub-questions: ["What did Chandler write about Ray Dalio in 2020?", "What did Chandler write about Ray Dalio in 2021?", "What did Chandler write about Ray Dalio in 2022?"]
[
"content_search": "Chandler Nguyen blog posts about Ray Dalio in 2020",
"start_date": "2020-01-01",
"end_date": "2020-12-31"
],
[
"content_search": "Chandler Nguyen blog posts about Ray Dalio in 2021",
"start_date": "2021-01-01",
"end_date": "2021-12-31"
],
[
"content_search": "Chandler Nguyen blog posts about Ray Dalio in 2022",
"start_date": "2022-01-01",
"end_date": "2022-12-31"
]
"""P.P.S: I know that the front end is still very slow so I probably need to learn more about front end development.