So your company is building jaw-dropping machine learning (ML) models that are performant and outputting the best results. The next task is to promote the models to production. It seems easy. After all, the most complex phase — building ML models — yielded a fruitful result. And shouldn’t it be…

To build a production-ready ML application and ensure its stability in the long run, we need to take care of a long checklist of requirements which include the ease with which the models could be iterated, reproducibility, infrastructure, automation, resources, memory, and so on. On top of that, we need…

All production-friendly — ML isn’t all about training a K-means classifier on Iris data in a Jupyter notebook. You might want to train tons of data using a complex ML model in the presence of high-end infrastructure. Now, this cannot be done with a single tool. We need a collection of them! With…

Understand how Horovod and Flyte, along with Spark and MPIOperator, simplify building robust distributed data pipelines — This blog is with reference to a talk titled “Efficient Data Parallel Distributed Training with Flyte, Spark & Horovod”, presented by Katrina Rogan and Ketan Umare at OSPOCon 2021, Seattle. To get the ball rolling, let’s understand what the title implies.

And handle feature-engineered data effectively in an ML pipeline — This blog is with reference to a talk titled “Self-serve Feature Engineering Platform Using Flyte and Feast,” which was presented by Ketan Umare and Felix Wang at OSPOCon 2021, Seattle. Feature engineering is one of the greatest challenges in applied machine learning. It is the process of transforming raw data…

Open-source is growing fast, so are its contributors. At Flyte, we have had meaningful contributions from many contributors who have helped us improve Flyte’s codebase and made us think through unseen perspectives. We have always wanted to have more such contributions to expand our horizons. …

Understand how Flyte offers Recoverability and Reproducibility — We all love “undoing” when things don’t work as expected. Imagine the pain we would have to experience without such an option; we can’t just revert to or pull from what we have been working on before. From the perspective of machine learning or data pipelines, going back in time…

Understand multiprocessing in no more than 6 minutes — Multiprocessing is quintessential when a long-running process has to be speeded up or multiple processes have to execute parallelly. Executing a process on a single core confines its capability, which could otherwise spread its tentacles across multiple cores. …

Highly performant means to shard and process large datasets in parallel — MapReduce is a prominent terminology in the Big Data vocabulary owing to the ease of handling large datasets with the “map” and “reduce” operations. A conventional map-reduce job consists of two phases: one, it performs filtering and sorting of the data chunks, and the other, it collects and processes the…