Agile/Scrum is now everywhere. But what about AI research, data science and machine learning? Should that follow the same process?

Update: Follow the discussion on LinkedIn.

AI researchers discussing algorithms.

More and more companies are looking for AI-first approaches to innovate and drive their business, and as they do the number of scientists in their teams grow. Often they discover a friction between how the scientists want to work and how software developers want to work. Some start to make exceptions. And while most teams can still work effectively if there are 1-2 exceptions to whom some of the processes do not apply to, but as that number grows, the dynamics of research need to be integrated into the ways of working of the bigger cross-functional group in order to be productive and effective.

I experienced exactly these kinds of problems a few years ago when I joined KONUX. I had led mixed software and data/AI teams for a number of companies, and I thought that I understood agile. I came fully embracing Scrum, and had to learn that it is not a holy book but relies on a set of assumptions, some of which are not working really well for research. I believe the exercise of bringing more agile practices to research was in the end a fruitful one, but me and the team both had to adapt. All sides had to give way to find the middle ground between completely free but occasionally unfocussed research, and a strictly controlled but less creative environment.

Initially when doing exploratory data analysis, or EDA, you don’t know what you are going to find or when you will find it – and yet this is a crucial first step to achieve anything with data. “Curiosity-driven research” as Geoffrey Hinton calls it, is by nature impossible to plan and predict. You iterate from one day to the next formulating and refining hypothesis about the data as you go. In the context of a sprint planning, with the goal of keeping the requirements during 1-2 weeks fixed, what task are you going to define for this? “Explore dataset X” is not measurably completed and a detailed description will invariably change.

There’s more: Not only can the task be difficult to describe, but the low time horizon of only a single sprint can lead to bad decisions. Building advanced machine learning models and their datasets is an activity typically measured in months not weeks, and so without a commitment that the team would be allowed to work on a given problem for a longer time period, they often chose to explore only simple methods, out-of-the-box algorithms and low-hanging approaches. Next sprint: Same situation, no progress possible towards bigger steps.

In my mind, there are situations where forcing research to comply with standard agile patterns will lead to unhappy researchers and poor business outcomes.

Another way to say this would be that I think the high-level ideas of agile make sense for many applications beyond software, but the implementation of those ideas in specific agile artifacts like roles and sprint planning meetings depends on the application.

Two AI researchers pair-programming together.

But there is something else that you can do. You can think back to at what agile was about from the beginning. People over process, get everyone who is needed to fulfill some customer need on the same table, and then give them the ability to find their own solutions. Make them accountable to solving the business problem, not the speed of closing tickets.

In my mind, that is the way to go. Empowered teams that are responsible for driving their own success. If you cannot predict the details of the next steps then stop trying, and instead create clarity around the business goal and set yourself a time horizon for how long you want to give it. Define a general sprint goal what you would like to explore and start with that sprint goal in mind. If you have no way to predict the terrain, already a compass will get you far.

This will allow teams to find whatever processes work best without management or meeting overhead. It will allow them to deliver their best. But it requires an engaged, committed and responsible team. When it works, it can be an exhilarating experience.

It means going back to the high-level principles that many years ago started the agile movement: People over process, trust, and embracing change.

So in the end, the solution to the friction of research with agile is, perhaps counter-intuitively, more agile.