Unlocking the world of Sneakers with SoleSavy

About Diego Redondo

Diego Redondo is the senior director of technology at Solesavy, based in Vancouver, Canada. Before joining SoleSavy in 2021, he worked at IBM, TransX, and Rise People, leading IT teams.

Diego shared with us how Solesavy isn't just an online sneakers marketplace but a community of genuinely passionate sneakerheads. His company relies on the power of Slack to engage with the community — members utilize Slack channels to share tips on how to buy, trade, and find coveted sneakers. Looking for a solution that could consolidate all the data flowing through Slack and help analyze the business, he and his team discovered Airbyte.

About SoleSavy

Founded in 2018, SoleSavy operates as an exclusive marketplace, allowing sneaker enthusiasts to purchase trendy sneakers and accessories that are difficult to find. With a bold vision of “Building the sneaker community of tomorrow” we have an open, vibrant community of thousands of members on Slack. Members can connect, get expert advice, and access tools to stay on top of every important sneaker release.

What was the problem?

Difficulty in measuring growth metrics

As our Slack community grew to over 11,000 members, our team needed an effective way to extract growth-and-churn metrics from its ten different Slack communities. However, Slack did not offer a focused measure of community growth across various channels. So to derive critical insights from our data, we had to build data pipelines that pull information from several community databases and bring them into a centralized space.

“We needed metrics on these Slack communities and had to be able to cross-reference Community A with Community B. We have north of 10 communities that we want to measure growth. We must have this information in a centralized space where we can see all accounts at once and measure our corresponding growth.”

Our old data architecture failed to support our growth because:

Inability to process large amounts of data 

Our team used an ELK stack as our primary search and analytics engine. However, it was quickly evident that this did not adequately meet our business needs as the community grew. In addition, with over 100 MB of raw unstructured data added to each Slack community daily, it became increasingly challenging to analyze the data. 

“I used ELK stacks in the past for several things. However, it didn’t meet the needs for this use case. We had to process almost a gigabyte of data a day. After a month of data collection, that was a very slow query for Metabase to graph, making the solution impractical.”

Failure to extract critical information from multiple databases 

Slack communities were previously linked to databases. Therefore, on-demand data analysis was possible by quickly identifying, filtering, and graphing the data. However, over time the size and volume of these databases increased, making it harder to perform cross-database joins and correlate data stored within multiple databases. In addition, we couldn’t extract critical information from the large amounts of collected data due to the lack of SQL language support in ElasticSearch.

How did we discover Airbyte? 

Since cloud infrastructure and Docker containers are critical to our architecture, we needed an easy platform to integrate and deploy. Our team had tried several options but in vain. Meltano was difficult to deploy at scale, and it couldn’t be integrated seamlessly with Slack. Other platforms were slow to set up and failed to provide a quick and efficient way to analyze user data. Finally, after much searching, we turned to Airbyte for its ease of use and scalability, hoping to meet our future growth needs as our community expands.

How was the problem solved?

Automatic data extraction to a single database 

Data from multiple sources is tapped back into Postgres and then extracted using a single database instance in our new architecture. As a result, we can now automatically discover all the available Slack entities across users or channels and only capture a subset of the data needed for analytics. 

“I just wanted to use an easy-to-implement tool to start collecting data from Slack right away. Using Airbyte was incredibly simple and just took a few clicks. Once you put in the credentials, it connected to the database, did the thinking, and the task was soon complete. So after trying out Airbyte, we decided that this was the tool we had to use.”

Easier consolidation and analytics 

With all the data in a consolidated database, rich insights can be extracted and quickly reported. All we need to do is point Metabase, the company’s analytics dashboard tool, to the newly consolidated database. As a result, I no longer worry about slow dashboard queries and focus on other infrastructure needs. 

How do we feel about using Airbyte?

With Airbyte, my team and I could re-platform our data architecture and simplify our data aggregation, analytics, and reporting processes. As a result, it was possible to collect crucial data right away without any downtime or additional disruption to our network of users. In addition, as our community grows, we are confident in Airbyte’s capabilities to help our team consolidate and analyze data across multiple databases. 

“SoleSavy has a very young engineering team, and we want to be part of open source. Being involved benefits both our team and the community.”