PyAirbyte Demo This demo uses the PyAirbyte libary to read records from Github, converts those records to documents, which can then be passed to LangChain for RAG.
Prerequisites: A Github personal access token. For details on configuring authetication credentials, refer to the Github source connector documentation . OpenAI API Key. You can create one by signing up on https://openai.com/ and going to the "Keys" tab on left sidebar. Install PyAirbyte and other dependencies
!apt-get install -qq python3.10 -venv
%pip install --quiet airbyte langchain langchain_openai langchainhub chromadbLoad the Source Data using PyAirbyte import airbyte as ab
read_result = ab.get_source(
"source-github" ,
config={
"repositories" : ["airbytehq/pyAirbyte" ],
"credentials" : {
"personal_access_token" : ab.get_secret("GITHUB_PERSONAL_ACCESS_TOKEN" )
}
},
streams=["issues" ],
).read()Read a single record from stream to examine the fields first_record = next ((record for record in read_result["issues" ]))
display(list(first_record.keys()))
display(first_record)Use PyAirbyte to_documents() method on a dataset This demo uses a new to_documents() method, which accepts record property names which point to specific aspects of the document.
When we set render_metadata=True, then metadata properties are also published to the markdown file. This option is helpful for small-ish documents, when passing the entire document to the LLM. It will be less helpful on long documents which are planned to be split into smaller chuns.
Note: We use rich to print the documents as markdown, although that's not strictly necessary.
import textwrap
from rich.console import Console
docs = list (read_result["issues" ].to_documents(
title_property="title" ,
content_properties=["body" ],
metadata_properties=["state" , "url" , "number" ],
render_metadata=True ,
))
console = Console()
console.print (str (docs[10 ]))Use Langchain to build a RAG pipeline. Here, we just show a generic method of splitting docs. This and the following steps are copied from a generic LangChain tutorial.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=512 , chunk_overlap=30 )
chunked_docs = splitter.split_documents(docs)
print (f"Created {len (chunked_docs)} document chunks." )Now we can publish the chunks to a vector store. Ensure you have added your OPENAI_API_KEY to the secrects tab on left.
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
import os
os.environ['OPENAI_API_KEY' ] = ab.get_secret("OPENAI_API_KEY" )
vectorstore = Chroma.from_documents(documents=chunked_docs, embedding=OpenAIEmbeddings())
print ("Chunks successfully stored in vectorstore." )Set up a RAG application using LangChain.
from langchain_openai import ChatOpenAI
from langchain import hub
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
retriever = vectorstore.as_retriever()
prompt = hub.pull("rlm/rag-prompt" )
llm = ChatOpenAI(model_name="gpt-3.5-turbo" , temperature=0 )
def format_docs (docs ):
return "\n\n" .join(doc.page_content for doc in docs)
rag_chain = (
{"context" : retriever | format_docs, "question" : RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
print ("Langchain RAG pipeline set up successfully." )Ask a question.
console.print (rag_chain.invoke ("Show me all documentation related issues, along with issue number, each on a new line ."))
GitHub
Airbyte