Qdrant has quickly become a go-to vector database for managing embeddings and powering AI applications, thanks to its performance, scalability, and ease of use. In the previous article, we explored the fundamental concepts behind Qdrant, highlighting why it’s an essential tool for handling high-dimensional data.

In this follow-up, we’ll move from theory to practice. This hands-on guide will walk you through the process of installing Qdrant using Docker, whether on a local machine or remote server. We’ll also cover basic database operations, giving you the tools to start managing and querying embeddings with Qdrant in your own projects.

Installation and Setup

1. Set Up the Qdrant Database Server

The most direct way to start using Qdrant is by using the Docker image to install the software.

Pull the Qdrant image from Docker Hub:

$ docker pull qdrant/qdrant

Create a directory location to use as storage for the Qdrant database:

$ mkdir /opt/databases/qdrant/storage

Run the docker image:

$ docker run --publish 6333:6333 --volume /opt/databases/qdrant/storage/:/qdrant/storage qdrant/qdrant

In the above command, the --publish option maps the port number 6333 on the host to port 6333 on the container. The --volume option maps the database directory (which you created earlier) on the host to the default storage path for the containerized Qdrant database.

Qdrant is now running and you are ready to connect to the database server at the URL http://localhost:6333. For trying things out on the localhost or a test server, the above setup is sufficient. 

To run Qdrant in production, a few more steps are necessary (this article does not discuss these in detail):

• Add or Update the YAML configuration file to restrict database access to users who have an API Key.
• Configure the firewall to allow connections to the Docker port.
• Set up a reverse proxy, such as Nginx, to access Qdrant via a human-readable URL instead of an IP address. This also gives you the freedom to move the database to a different server without affecting the client-side code.

2. Connect to the Qdrant Database Server

There are two broad approaches to connect to Qdrant:

• Using a programming language, such as Python, Rust, Java, etc. This approach is useful to connect to the database server from a backend server, like a webserver.
• Over the HTTP protocol, using a command-line tool like cURL or a GUI like Postman. This approach is useful if you connect to the database from a frontend client, like a web-browser or a mobile app.

3. Set up the Database Client

As with most other databases, to connect to the database engine, you need a client. Install the client:

$ pip install qdrant-client

The next steps are in Python, run them from a Python (3.10+) terminal or a locally running Jupyter notebook.

Import the QdrantClient object:

>>> from qdrant_client import QdrantClient

Initialize this client object to connect to the URL on which the Qdrant database engine is listening:

>>> client = QdrantClient(url="http://localhost:6333")

If you are connecting to a remote instance of Qdrant, replace the URL in the above command. This client object is all you need to start using Qdrant.

The next sections show how to perform some basic operations in Qdrant.

Create a New Collection

In a relational database, you create a table before inserting rows. In Qdrant, you create a collection before adding points to it. The following steps explain how to create a new collection.

1. Import the Prerequisite Packages

>>> from qdrant_client.models import Distance, VectorParams

• The Distance package contains different metrics for calculating the distance between vectors. You need to associate a specific distance metric with each collection. 
• VectorParams is used to package in a single object different variables used to parametrize the vectors. You need to specify parameters such as number of vector-dimensions, while creating the collection. 

2. Declare a Few Variables to Store the Parameters of the New Collection

>>> collection_name = 'test'

This is the name of the collection, analogous to table-name in a relational database. Just as table names need to be unique, you cannot have two collections with the same name.

>>> dimensions = 2

This is the number of dimensions of the vectors you want to store in this collection. For instance, embeddings based on the OpenAI text-embedding-ada-002 model have 1536 dimensions. To test things manually, as in this example, use vectors with just 2 dimensions. Do not mix vectors of different dimensions in the same collection.

>>> metric = Distance.COSINE

The metric is the function used to calculate the distance between vectors. For applications related to LLMs, dot product and Cosine similarity are commonly used. In general, choose the same metric as was used while training the model.

3. Create the New Collection

Using the parameters explained above, create a new collection:

>>> client.create_collection(
        collection_name = collection_name,
        vectors_config = VectorParams(
            size = dimensions, 
            distance = metric
        ),
)

Get Information About a Collection

Use the get_collection command to check the specifications of a collection:

>>> client.get_collection(collection_name=collection_name)

This shows information like the number of points, number of indexed vectors, the distance metric, and more. For example, to check the number of points, do the following:

>>> client.get_collection(collection_name=collection_name).points_count

On-disk Storage

By default, Qdrant stores the entire collection, i.e., all the vectors, in memory. This is good for fast access and computations. However, for large collections, the available RAM can be insufficient. In such cases, add the on_disk parameter to the VectorParams configuration object above, and set its value to True. So, the configuration looks like this:

vectors_config = VectorParams(
    size=dimensions, 
    distance=metric, 
    on_disk = True
),

The on_disk setting enables the use of Memmaps. A Memmap is a data structure that uses a virtual address space based on page files and caching. This allows Qdrant to take advantage of the available memory while maintaining most of the data on disk. The Qdrant documentation on Storage discusses these concepts in detail.

Delete and Re-create Collections

In many cases, such as while testing things, you re-run the same script repeatedly, with minor modifications. Such scripts often include commands to create a test collection to experiment with. Trying to create a collection with the same name as an existing collection is an error. 

Before (re-)creating a collection, check if there is already a collection with the given name. If there is, either use a different name or first delete the old collection and then recreate a new collection with the same name.

• To check whether there is a collection with a given name, use the client.collection_exists function. 
• To delete a collection, use the client.delete_collection function.

The code snippet below checks if there is already a collection with the name assigned to the variable collection_name. If there is, it deletes it.

>>> if client.collection_exists(collection_name = collection_name): 
    client.delete_collection(collection_name = collection_name)
    print("deleted collection with the name ", collection_name)
else:
    if not client.collection_exists(collection_name = collection_name): 
        print("there is no collection with the name ", collection_name)

After deleting the old collection, recreate a new collection, as shown in the previous section.

Note: There is also a wrapper function called client.recreate_collection. This is (scheduled to be) deprecated.

Fictional Data Example 

Consider a hypothetical situation where you store vectors representing animals. 

• The vectors are two dimensional. 
• The payload of each point contains:some text
  • The name of the animal 
  • Whether the animal is a wild animal. 

In practice, dimensions of embedding vectors do not have human-understandable meanings. However, in this example, to make the queries and results meaningful, suppose that:

• The X axis of the two-dimensional vectors represents whether the animal is trainable (a higher X value means the animal is more trainable). 
• The Y axis represents whether the animal is endangered (higher Y value means the animal is more endangered). 

In pseudo-code, points representing this data look like the example below: 

(
    id = <ID>
    vector = <[X,Y]> # 2-D vector
    payload = (
        "creature": <animal_name>, 
        "is_wild": <YES / NO>
    )
)

The data and queries shown in the following sections are based on the example scenario described above. 

1. Insert Data (Points)

Use the upsert method to add a few example points to the test collection created earlier. Recall that the collection test is defined to have vectors with 2 dimensions. Note also that Qdrant automatically normalizes inserted vectors.

The method PointStruct is used to declare a point as a single object - including its ID, payload, and the vector. Import this method:

>>> from qdrant_client.models import PointStruct

To insert a single point, use upsert directly:

>>> client.upsert(
        collection_name=collection_name,
        points=[
            PointStruct(
                id=1,
                payload={
                    "creature": "dog",
                    "is_wild": "no",
                },
                vector=[0.9, 0.1],
            ),
        ],
    )

This command inserts a point representing the animal “dog”. The payload indicates it is not a wild animal. The vector indicates that the animal is highly trainable (X-axis) and not endangered (Y-axis). 

To insert more than one point, it is more efficient to use the batching feature. To do this, pass a list of points to the points parameter in the upsert command. The following example inserts two more points - for “wolf” and for “cat”. 

>>> client.upsert(
        collection_name=collection_name,
        points=[
            PointStruct(
                id=2,
                payload={
                    "creature": "wolf",
                    "is_wild": "yes",
                },
                vector=[0.1, 0.7],
            ),
            PointStruct(
                id=3,
                payload={
                    "creature": "cat",
                    "is_wild": "no",
                },
                vector=[0.3, 0.3],
            ),

        ],
    )

The above two examples of adding points are record-oriented. Each point is written separately as a complete unit. There is also a column-oriented format, where point-elements, like vectors and payloads are grouped together. If you are loading points into Qdrant from an external source, such as a HuggingFace dataset or a dataset on disk, the column format is easier to work with. 

The example below uses the column format to create new points representing “cow” and “elephant”. 

>>> client.upload_collection(
        collection_name=collection_name,
        ids=[4, 5],
        payload=[
            {"creature": "elephant", "is_wild": "yes" },
            {"creature": "cow", "is_wild": "no" },
        ],
        vectors=[
            [0.7, 0.8],
            [0.3, 0.2],
        ],
    )

Note that upload_collection also lets you upload points without explicitly specifying their IDs. Qdrant automatically generates and assigns each point an ID. However, you can still provide an ID while inserting new points.

2. Query the Database

Scroll Through a Collection

The scroll command is analogous to doing a SELECT * in SQL. It allows you to get all the stored points. For example:

>>> client.scroll(collection_name = collection_name, with_vectors=True)

You can also use Payload Filters (discussed later) together with the scroll command. This allows to show only those points whose payloads contain specific key value pairs. 

Retrieve Points

If you know the IDs of the points you want, you can use the retrieve command to fetch those points directly. The desired IDs should be supplied in the form of a list. For example: 

>>> client.retrieve(
    collection_name=collection_name,
    ids=[0,2],
    with_vectors = True
)

You can also use the Python range function to automatically generate a list to get all the points whose IDs are in that range. For example: 

>>> client.retrieve(
    collection_name=collection_name,
    ids=list(range(0,10)),
    with_vectors = True
)

3. Similarity Search

Searching refers to finding vectors that are close (based on the collection's distance metric) to the given vector. To do a search, specify the name of the collection, the given vector, and the number of search results. The limit parameter specifies how many points (at most) to return. 

The snippet below shows how to do a simple similarity search. In this example query, the target vector is [0.9, 0.9]. 

>>> target_vector = [0.9, 0.9],
>>> client.search(
        collection_name=collection_name, 
        query_vector = target_vector, 
        limit=3
    )

This returns a list of the points whose vectors are most similar to the given vector. It also shows their respective similarity scores. The list contains points in descending order of similarity - points with the highest similarity score (to the given point) are ordered first. 

You can extract the payload and other details of the similar vectors. For example, to see the payload and similarity score of the topmost similar vector:

>>> r = client.search(
        collection_name=collection_name, 
        query_vector = target_vector, 
        limit=3
    )


>>> print(r[0].score)
>>> print(r[0].payload)

Take a look at an individual point in the returned list:

>>> r[2]

The structure of the output resembles the example below: 

ScoredPoint(id=4, version=2, score=0.9977851, payload={'creature': 'elephant', 'is_wild': 'yes'}, vector=None, shard_key=None)

Notice that the results do not include the vectors themselves. This is to save space. Because in most cases, the vector itself is only used internally for computations (such as similarity score). To include the vectors, use the with_vectors = True parameter in the search query.

>>> client.search(
        collection_name=collection_name,
        query_vector = target_vector,
        limit=3,
        with_payload = True,
        with_vectors = True
    )

For sake of completeness, the above query also includes the parameter with_payload = True. Check that the result (of the above command) now includes the vectors for each point. Notice also that the values of the vectors are normalized.

4. Payload Filters

It is possible to restrict the search or query results based on the metadata in the payload field. Payload Filters allow you to declare that results should be restricted only to points containing specific key value pairs. You can use Payload Filters with similarity search as well as with Scroll.

Similarity Search with Payload Filters

Suppose, in the previous (similarity search) query, you want to limit your search to wild animals only. To do this, add a query filter parameter which restricts the "is_wild' key in the payload to a value of "yes". 

The snippet below shows how to do this:

>>> from qdrant_client.models import Filter, FieldCondition, MatchValue

>>> client.search(
        collection_name=collection_name, 
        query_vector = target_vector, 
        limit=3,
        with_payload = True,
        with_vectors = True,


        query_filter = Filter(
            must = [FieldCondition(
                key = "is_wild",
                match = MatchValue(value = "yes")
            )]
        )
    )

‍

Using Payload Filters needs the following additional methods:

• Filter - This is used to create a Payload Filter, which consists of the criteria below. 
• FieldCondition - This declares the condition based on which the filtering happens
• MatchValue - This specifies the value that the key-value pair of the payload should match.

The results of the similarity search are now further filtered based on the criteria that is_wild in the payload should be "yes". 

Scroll with Payload Filters

You can also use Payload Filters with other commands, like scroll. To do this, add the Filter object (as shown above) to the scroll_filter parameter of the scroll command. The example below shows how to do this: 

>>> client.scroll(
	collection_name = collection_name, 
	with_vectors=True,
    
	scroll_filter = Filter(
            must = [FieldCondition(
                key = "is_wild",
                match = MatchValue(value = "yes")
            )]
        )
)

Conclusion

In this article, we covered how to install Qdrant locally using Docker and perform basic operations with example vectors. These foundational steps will help you start using Qdrant for managing embeddings in AI applications.

For those looking to integrate Qdrant into their AI workflows, Airbyte offers powerful data integration solutions that work seamlessly with Qdrant. Airbyte’s tools simplify the process of connecting Qdrant to other data sources, enabling you to build end-to-end retrieval-augmented generation (RAG) workflows and further enhance your AI capabilities.

To dive deeper into practical implementations, check out these valuable resources:

• Quickstart for End-to-End RAG using GitLab, PyAirbyte, and Qdrant
• Airbyte's Partnership with Qdrant
• Qdrant's Example on RAG Customer Support with Cohere, Airbyte, and AWS

Stay tuned to learn how to fully leverage Qdrant and Airbyte to supercharge your AI projects!