The langchain-airbyte package integrates LangChain with Airbyte.
It has a very powerful function AirbyteLoader which can be used to load data as document into langchain from any Airbyte source!
This notebook demonstrates the usage of langchain_airbyte to load data from an Airbyte source (Github Repository) , store the data into a vector database, and perform a basic QnA on that data using FAISS and OpenAI embeddings.
Prerequisite 1) OpenAI API Key :
Create an OpenAI Account : Sign up for an account on OpenAI .Generate an API Key : Go to the API section and generate a new API key. For detailed instructions, refer to the OpenAI documentation .2) Github Personal Access Token :
Create a Github Account : Sign up for an account on Github .Generate an API Key : Cick on your profile icon->Settings->Developer Settings and generate a new API key. For detailed instructions, refer to the Github documentation .Installing Dependencies Lets start by installing all the required dependencies!
!apt -get install -qq python3.10-venv
!pip3.10 install -qU langchain-airbyte faiss-cpu langchain-community langchain-openai Load Data using AirbyteLoader Now we use AirbyteLoader to fetch data from the source source-github.here
For more information regarding this package refer
The last step of converting data to documents ensures that the raw data (GitHub commits) is converted into a standardized format that includes both the main content and any associated metadata.
from langchain_airbyte import AirbyteLoader
from langchain.schema import Document
loader = AirbyteLoader(
source="source-github" ,
stream="commits" ,
config={
"credentials" : {
"personal_access_token" : "your_personal_access_token"
},
"repositories" : ["your_username/repository_name" ]
}
)
docs = loader.load()
docs = [Document(page_content=record.page_content, metadata=record.metadata) for record in docs]Split Documents into Chunks and Store these Chunks in Vector Store using FAISS Large documents are split into smaller chunks to make them easier to handle. This also helps in improving the efficiency of the retrieval process, as smaller chunks can be more relevant to specific queries.
The chunks of documents are transformed into vectors using an embedding model (OpenAI embeddings).
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=512 , chunk_overlap=30 )
chunked_docs = splitter.split_documents(docs)
print (f"Created {len (chunked_docs)} document chunks." )
from langchain_openai import OpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import FAISS
import os
os.environ['OPENAI_API_KEY' ] = "YOUR_OPENAI_API_KEY"
if not chunked_docs:
raise ValueError("No valid documents to store in the vector store." )
embeddings = OpenAIEmbeddings(openai_api_key=os.getenv("OPENAI_API_KEY" ))
vector_store = FAISS.from_texts([doc.page_content for doc in chunked_docs], embeddings)
print ("Chunks successfully stored in vectorstore." )Perform QnA on Stored Data Finally we perform the Question And Answer here.
When a query is made, the vector store retrieves relevant document chunks based on their vector similarity to the query. The language model (OpenAI) then generates answers based on the retrieved chunks.
from langchain.chains.question_answering import load_qa_chain
llm = OpenAI( openai_api_key= os.getenv( "OPENAI_API_KEY" ) )
qa_chain = load_qa_chain( llm= llm, chain_type= "stuff" )
query = "What are the latest commits in the repository?"
inputs = { "question" : query , "input_documents" : chunked_docs}
answer = qa_chain.invoke( inputs)
print( "QnA Result:" , answer) 
Airbyte