Making Agile work for data science

Data scientists and engineers don’t always play well together. Data scientists will plan out a solution, carefully build models, test them in notebooks, then throw that solution over the wall to engineering. Implementing that solution can take months.

Historically, the data science team has been purely science-driven. Work on methodologies, prove out something that they wanted to achieve, and then hand it over to the engineering organization. That could take many months.

Over the past three to five years, they’ve been moving their engineering and data science operations onto the cloud as part of an overall Agile transformation and a move from being sales-led to being product-led. With most of their solutions migrated over, they decided that along with modernizing their infrastructure, they wanted to modernize their legacy systems, add new functions and scientific techniques, and take advantage of new technologies to scale and meet the demand coming their way. 

While all of the rituals and the rigor of Agile didn't always facilitate the more open-ended nature of the data science work at 84.51°, having both data science and engineering operating in a similar tech stack has been a breath of fresh air. Working cross-functionally has shortened the implementation delay. At the same time, being closer to the engineering side of the house has given the data science team a better sense of how to fit their work into the pipeline. 

Getting everyone on the same tech stack had a side effect. Between the increasing complexity of the projects, geographic diversity of the folks on these projects, a rise in remote work, and continued growth, locating experts became harder. But with everyone working in the same tech, more people could answer questions and become SMEs. 

Of course, we’d be remiss if we didn’t tell you that 84.51° was asking and answering questions on Stack Overflow for Teams. It was helpful when Chris and Michael no longer had to call on the SMEs they knew by name but could suddenly draw more experts out of the woodwork by asking a question. Check out this episode for insights on data science, agile, and building a great knowledge base for a large, increasingly distributed engineering org.