The world of data analytics is constantly evolving, with new tools and technologies emerging every day. One popular new tool is DuckDB, open-source software touted as a faster, more efficient alternative to traditional databases. So, what exactly is DuckDB, and why is it gaining so much attention in the data…

Learn to build a production-grade ML pipeline with Flyte, Banana, and Hugging Face — Before you train an ML model requires you to procure, prepare and analyze the data you’ll use to build and train it. When there’s additional data, you may need to repeat the steps multiple times — a process known as “retraining.” To ensure reproducibility, it may be necessary to version…

A look at the differences between ETL and machine learning tasks — Apache Airflow is an open source platform that can be used to author, monitor, and schedule data pipelines. It is used by companies like Airbnb, Lyft, and Twitter and has been the go-to tool in the data engineering ecosystem. With an increased necessity for orchestration of data pipelines, Airflow witnessed…

So your company is building jaw-dropping machine learning (ML) models that are performant and outputting the best results. The next task is to promote the models to production. It seems easy. After all, the most complex phase — building ML models — yielded a fruitful result. And shouldn’t it be…

To build a production-ready ML application and ensure its stability in the long run, we need to take care of a long checklist of requirements which include the ease with which the models could be iterated, reproducibility, infrastructure, automation, resources, memory, and so on. On top of that, we need…

All production-friendly — ML isn’t all about training a K-means classifier on Iris data in a Jupyter notebook. You might want to train tons of data using a complex ML model in the presence of high-end infrastructure. Now, this cannot be done with a single tool. We need a collection of them! With…

Understand how Horovod and Flyte, along with Spark and MPIOperator, simplify building robust distributed data pipelines — This blog is with reference to a talk titled “Efficient Data Parallel Distributed Training with Flyte, Spark & Horovod”, presented by Katrina Rogan and Ketan Umare at OSPOCon 2021, Seattle. To get the ball rolling, let’s understand what the title implies.

And handle feature-engineered data effectively in an ML pipeline — This blog is with reference to a talk titled “Self-serve Feature Engineering Platform Using Flyte and Feast,” which was presented by Ketan Umare and Felix Wang at OSPOCon 2021, Seattle. Feature engineering is one of the greatest challenges in applied machine learning. It is the process of transforming raw data…

Open-source is growing fast, so are its contributors. At Flyte, we have had meaningful contributions from many contributors who have helped us improve Flyte’s codebase and made us think through unseen perspectives. We have always wanted to have more such contributions to expand our horizons. We have been thinking about…