The year ahead in DevOps and agile: bring on the automation, bring on the business involvement

DevOps has an automation problem, while agile has an identity problem. Both face organizational problems. These two, intertwined methodologies — or philosophies, if you will — will make the difference in the march toward digital transformation in the year ahead, but organizations still have a great deal of finessing to make things work. The goal is to have everyone on the same page, moving in the same direction, delivering quality software quickly. In this next of a series on the year before us, I canvassed industry leaders about the prospects for DevOps and agile,   

The challenge going forward with DevOps is there are still too many manual processes. “Usually what’s missing is complete automation,” says Eric Newcomer, CTO of WSO2.  “A CI/CD pipeline tool provides the capability to fully automate the application build and deploy process. But because many organizations separate these functions — build and deploy — the build process often gets automated but the deployment doesn’t.” 

Automation is key across the board, agrees Kief Morris, principal cloud technologist at ThoughtWorks. “We are looking at low-code tools with AI assistance baked in as a real game changer for cloud-native computing. Secure API management and governance are other very important tools in the toolbox to help make sense of the proliferation of microservice-based development models and ensure they are providing consistent business value rather than engendering chaos.”

The slower-than-desired pace of automaton stems from “organizations prohibiting developers from accessing production environments, probably because developers made changes in production previously that caused production problems,” says Newcomer. “It’s hard to change that kind of policy, especially when incidents have occurred. Another reason is simple institutional inertia – processes and procedures are difficult to change once fully baked into daily practice, especially when it’s someone’s specific job to perform these manual deployment steps.”

DevOps and agile progress needs to be well-measured and documented. “People have different definitions of DevOps and agile,” says Lei Zhang, head of Bloomberg’s Developer Experience group. Zhang’s team turned to the measurements established within Google’s DevOps Research and Assessment guidelines — lead time, deploy frequency, time to restore, and change fail percentage, and focus on the combination. “We think the effort is cohesive, while the results have huge varieties. Common contributors to such varieties include complex dependencies due to the nature of the business, legacy but crucial software artifacts, compliance requirements, and low-level infrastructure limitations.”

The challenge with DevOps and agile, Zhang continues, “a lot of decisions are made more locally and incrementally, while organizational scalability, alignments and insights become an afterthought. For instance, as more and more metrics are gathered and used effectively in the local context, the ability to utilize these for more organizational and strategic decision making remains an area for huge improvement.”

Over the coming year, we’re going to see more efforts to get agile right. Agile, which provides a sense of that all-important business collaboration as digital development moves forward, has been hampered by misunderstanding and miscommunication. While Newcomer reports seeing many instances of agile being successfully employed, there are still too many cases of the antithesis of agile, the waterfall approach, still in practice — while being labeled as “agile.”  There are many companies that “say they are doing agile, but often it’s a modified waterfall,” he says. “I find this to be true especially when the dev team is focused on maintaining a legacy app. When teams have a chance to start from scratch with new technologies and new dev practices, it’s a lot easier to achieve the benefits of agile.”

DevOps also suffers from a similar identity problem. “Just like you see with agile, DevOps means different things to different people, so different people do different things and call it DevOps,” says Morris. “People often focus on tools and the superficial forms, rather than on the principles and on outcomes. So you see DevOps teams that run Jenkins servers and maybe write Ansible, but you don’t always see developers involved in operational aspects of the code they write, and you don’t often see everyone across different roles including testing and governance collaborating effectively on building the right things into software.”   

There’s even a tendency for organizations to attempt to simply rebrand existing process as “DevOps,” without changing the fundamentals underneath. “It surprises me when I meet with leaders in other organizations and find out that their DevOps team members are really just rebranded ops members,” says David Torgerson, senior director of engineering for Lucid Software. “DevOps is often implemented by embedding ops team members in engineering teams, with the hopes of increasing the speed of work, creating camaraderie between teams and eliminating the us-versus-them mentality. Unfortunately, this strategy doesn’t scale well and can often exacerbate issues of miscommunication as ops team members start to feel siloed.”

DevOps is more than embedding ops members in dev teams, Torgerson says. “Their focus tends to stay on maintaining the production system instead of focusing on improving software development lifecycle, continuous delivery, and high-quality output. By focusing on the goal of shorting the software development lifecycle, implementing continuous delivery, and increasing quality of output, the way that traditional ops and dev teams interact changes, and that is when the true value of DevOps emerges.”

At the same time, agile often gets boxed away, rather than baked into corporate culture. “I see a sort of disconcerting trend to view agile as more of a program or project management practice change, rather than a dev practice change,” Newcomer says. 

“A healthy agile business technology organization extends beyond the IT group,” Nicolas Chabanoles, CTO at Bonitasoft, agrees. “Bring in the business lines. Agile technology initiatives include business lines, especially as process managers continue to get more deeply involved in automation, so digital process automation technologies that can be used for close collaboration across organizations will positively impact agile projects. For example, automation technologies should include wide options to allow visual programming for citizen developers to create user interfaces, define business rules and conditions, and coding capabilities for developers like SDKs, templates, archetypes, and extension points.”

This full-throttle engagement of the business is crucial, especially in terms of getting people in the corner offices involved. “Most agile initiatives are missing executive sponsorship that is both visible and durable,” says Bryan Stallings, chief evangelist for Lucid. “Most agile initiatives lack the ability to quantify the impact of the initiative in comparison to its costs. As the attention of the sponsor turns elsewhere, the program starts to look like a pile of expenses without a quantifiable return. Invariably, business pressures mount and those involved in enabling the change are eliminated or returned to a focus on business as usual.” In ideal situations, top executives — especially the CEO — should communicate their commitment to the effort, and take personal accountability, he says.