How to overcome Top DevOps challenges in the time of Covid-19

As we are getting used to the new normal forced upon us by COVID-19, we decided to analyze its impact on companies and how that has changed priorities for devops team. Firstly, companies have to live with uncertainty which means cash conservation is important and so is quick pivot. Secondly, companies are going remote which needs collaboration, delegation and security first approach. Though it may appear that these are integral parts of devops, there is an increasing sense in the community of how woefully short devops tools are of their promises.

Launching new applications and supporting infra takes ages and is error prone

Companies can no longer assume business as usual. There is considerable impact on their bottom line and top line. Even multi billion dollar companies have filed for bankruptcy. Companies which were raising hundreds of millions of dollars of VC money are raising a few millions and figuring out how to sustain current valuations. For many, even business models have become unviable and they have pivoted to delivering essentials, masks or sanitizers to keep their nose above water.

We expect all great startups to be capable of launching new products quickly, sadly that’s not the case. Devops tools today are built to make predictable changes like deploying code changes. When it comes to setting up new microservices and supporting infrastructure or anything outside their predictable pattern, manual is the way to go for most of them. Manual is not only time consuming but is also prone to error.

Solution: Make flexible tools which can be easily extended for custom requirements

Follow principles of infrastructure as a code. Every change should be repeatable, testable and should be applied through gitops. Invest time in identifying the requirements of stakeholders and build right abstractions in the product to support different use cases so that infrastructure and microservice provisioning can be customized even by developers.

Cost of running application goes out of control

Another fallout of uncertainty is the need to reduce costs so that companies can weather out COVID-19. Till now, companies’ focus was on growth and infrastructure cost was a small price to pay for exponential growth. But now, every single penny has to go through a magnifying glass. Currently for most of the organizations cost analysis is not integrated in devops. It’s a manual process which happens whenever companies feel financial stress ( Check out our article on reducing cloud infrastructure cost).

Solution: Integrate continuous infrastructure cost rationalization with devops tool chain

Now that companies are staring at an uncertain ecosystem, they would want cost optimization to happen on a continuous basis. This can happen only through tools which continuously monitor resource utilization, take corrective actions, automatically including suggesting changes in configuration or changing configuration automatically. Cost and resource optimization system should extend current deployment systems to minimize impact on the user and to ensure that changes are DR compliant.

It is taking longer to release critical features and debug issues in wfh scenario

With wide adoption of cloud infrastructure, it is natural to assume that companies are remote ready. But even for companies distributed across the globe, the majority of the times people working together are co-located.

I am sure you have heard and lived stories of war rooms. People from different teams coming together in a room, burning proverbial midnight oil to stabilise major releases. Pinging commands on chat for devops team to execute.

We can still create tickets for deployment, ping commands on chat but collaboration becomes much more difficult when you can’t walk over and talk to your team members. Screen sharing between multiple team members over video conferencing is not the ideal tool for collaboration for releases.

Only when a person with the ownership of impact of action takes the action, it is possible to have effective collaboration in remote settings. Other important capability is for all participants to have the same single pane view of the cause effect chain.

To understand it better let’s take an example, one of the team members has worked on optimizing database queries of an application. To deploy it to production, you will either open a ticket with database queries to be executed along with the success criteria or push them into git. Devops team will then execute them (maybe you are sitting next to them) and share results with you. They will also make changes in the deployment configuration and inform you about the changes and results. Then you will start deployment. Although highly inefficient, it still worked in pre-COVID world.

Solution: Build devops products which empower technology team through responsible delegation

What we need today, is a capability to delegate access to the application owners. Different team members should be able to execute steps they own because they are the subject matter experts of that step. Everyone involved should have the same single pane view of cause effect so that they can debug issues quickly without sharing logs/names/artifacts over chat.

Inadequate access controls have made infrastructure vulnerable to compromise

Distributed teams also mean that access to critical infrastructure is happening from home. Even though VPN mitigates the risk, it doesn’t remove it completely. Most of the companies don’t have complete visibility of what is running in their infrastructure, there have been cases where their infrastructure was being used for bitcoin mining or database is for sale in the dark web.

Solution: Log and audit every mutable action

Each user should be given access through personally identifiable information and every mutable action taken by them should be logged. Audit logs should be query-able so that you know who did what and when. Apart from user access**, continuous auditing of workloads should be done** to ensure that only applications with whitelisted configurations are running under allowed users.

Conclusion

As unprecedented circumstances unraveled across the whole world, employees are getting used to the new reality of working from home, organizations are forced to re-engineer IT processes. Devops have to play a crucial role during these challenging times. New expectations will force devops to rethink the whole premise of devops tools chain. Different tools connected through scripts and labour intensive methods can no longer meet the needs of the business.

Need of the hour is a DIY devops platform that is easy to understand and use in a collaborative way which gives complete visibility around cost and performance while ensuring right security.

Interested in learning more about how these challenges are solved by Devtron, Please reach out to our team.