Will Rust kill Python for Data Engineers? If you only came here to know this, my answer is no. Betteridge's Law strikes again!
But then again, you have to ask: was Python made for Data Engineering in the first place?
Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering. Let's explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running.
What is Rust? Former Mozilla employee Graydon Hoare initially created Rust as a personal project. The first stable release, Rust 1.0 was released on May 15, 2015. Rust is a multi-paradigm programming language functional styles, supporting generic programming and metaprogramming statically and dynamically.
The goal of Rust is to be a good programming language for creating highly concurrent, safe, and performant systems . What Is Unique about Rust? Rust solves pain points from other programming languages with minimal downsides. With Rust being a compiled programming language, strong type and system checks are enforced during compile-time—meaning pre-runtime! Unlike Python 's interpreted way, most errors only surface during the coding phase. It can be frustrating to fight every single mistake before being able to test or run a quick script, but at the same time, a prominent feature like the compiler is much faster at finding bugs than me. Additionally, the Rust community puts a lot of effort into making the error messages super informative.
An example of how Rust surfaces an error during development and suggests changes 📝 There are much more specifics that differentiate Rust from other programming languages. Concepts such as Ownership for memory safety, Reference Borrowing , and many more, but this article is not meant to be a deep dive into Rust, but rather map it to the field of data engineering.
Why Rust for Data Engineers? When you write any code, the goal is that the code doesn't break during the weekend or at night when you sleep. Rust will show you errors and improvements while coding and fails as much as possible at compile-time, which is less costly than later in production at run-time.
The go-to language for data engineers is Python, which isn't the most robust or safe language, as many engineers working with data will agree. Rust's developer experience goes much further than just offering a language specification and a compiler; many aspects of creating and maintaining production-quality software are treated as first-class citizens in the Rust ecosystem
What Rust Does Well Python is dynamically typed (with only recent support for type hints ) and requires writing extensive tests to catch these costly errors. But that takes a lot of time, and you must foresee all potential errors to write a test for it.
Rust is the opposite; it forces you to define types (or does it implicitly with type inference ) and enforces them. This does not obsolete testing of course, but for example, the rust compiler will analyze e.g. borrow checking , and does things to your code that other compilers don't do—check out rust-analyzer for bringing them into your IDE of choice. This makes it very good for data engineers as we have many moving parts such as incoming data sets that we do not control. Defining expectations with data types and having vigorous checks at coding and compile time will prevent many errors.
Less relevant for data engineers, but super helpful: speed . Rust, as a compiled language, is super fast at run-time. To many, Rust is primarily an alternative to other systems programming languages, like C or C++. But you don't need a systems use case to use a systems language, as both Vercel and Crowdstrike are noticing.
Another one is integrations . With data pipelines being the glue code in most cases, connecting otherwise foreign systems, Rust almost runs platform agnostic. Rust makes it easy to integrate and communicate with other languages through a so-called foreign function interface (FFI) zero-cost abstraction where function calls between Rust and C have identical performance to C function calls. Rust can be called easily from C, Python, Ruby, and vice-versa. Find more on Rust Once, Run Everywhere .
A less technical but still important element is to love or have fun , enjoying your programming language. Rust is a more complex language to learn, but it was the most loved technology for seven years (2022 , 2021 , 2020 , 2019 , 2018 , 2017 , 2016 ) in a row on the Stack Overflow survey:
Loved vs. Dreaded and most Wanted Programming Language on StackOverflow Survey 2022 Besides the love, it's also rising in awareness of different trends such as Google Trend , one from 2019 Ranking on GitHub , or the StackOverflow below:
StackOverflow Trends ℹ️ Why Rust is Popular ?
