The science of debugging

The perception is that software development is developers creating features by writing code. The reality is different. As developers, we actually spend most of our time finding and fixing bugs… Feature development is just what we do in those little breaks between debugging.
For something that occupies most of our time, you’d think most be very good at it.. But often not the case. The sad thing is that we often don’t use humanities best tool for acquiring knowledge: The scientific method. When people say science, thoughts often go to beakers, lab coats and weird hair. Where actually, science is just a series of steps used to acquire knowledge. Each step is critical in the process.
The steps are as follows (shortened for relevance):
  1. Ask a question –  Obviously, we can’t find an answer if we don’t have a question
  2. Do background research – someone may have already answered our question or had a similar question.
  3. Construct a hypothesis – A hypothesis is a potential answer. The hypothesis must be unpacked into a series of predictions.
  4. Test the predictions with an experiment – Critically, we test to prove them WRONG. The sooner it’s proved wrong, the sooner we stop wasting time on a flawed hypothesis.
  5. Analyse your data and draw a conclusion – Spend time on the results. Nothing wastes more time than misunderstood results.
  6. Communicate your results – Communicating our results allows others to validate the methodology and test the results
  7. ..aaaaannnd repeat
Back to debugging…
When we have a bug, we have a question: “what is causing this sh*t to break!”. The scientific method is the most effective way to this knowledge if applied as a simple heuristic. First step is background research.. Google the error code, method and other relevant information. This is done right up front. We shouldn’t waste time on a problem that someone has already solved.
Based on our findings (or lack of), we construct a hypothesis – a falsifiable hypothesis which has testable predictions. If our hypothesis is un-falsifiable, it is too abstract and should be decomposed into other hypothesis. Importantly, all assumptions are just camouflaged hypothesis. If our hypothesis is both unfalsifiable and can’t be decomposed, it’s an answer that isn’t worth wasting further energy on as there’s no knowledge to be gained. For example: as nice as it feels to vent about it being Windows fault, that’s not a particularly useful hypothesis.
Next, we do an experiment i.e. prod the application in a way that tests the hypothesis and resulting predictions. Analyse our results of the test.. Don’t argue with the results but let the results lead us. If it didn’t behave as expected, we’ve just learned something valuable about our assumptions.
Lastly communicate our results. Most aha moments come from a simple chat to a fellow developer who challenges our biases.
It’s science bitches!

Azure as part of a development process

To enable any proper development process, multiple deployment environments are essential. Historically, the biggest problem I have encountered here are the issues related to creating and maintaining and sharing these environments. Although many of these problems can be addressed through a private cloud, our dev team has been finding Azure to be a great tool in the development lifecycle. I’ll be sharing some of the benefits, tips and lessons learnt with Azure.

The pains of non-cloud, developer environments

Mark, works for a typical dev house. In his dev team, when a new project gets kicked off, the team’s first step is to organize the required servers for all the environments (typically dev, QA, UAT and production). Jill in IT will then be requested to create these environments. Often, this results in a long drawn out process with Jill asking questions such as “What specs are required for the servers?”. This question is ridiculous at this point because even the customer isn’t quite sure what they need yet. So Mark makes an educated guess, and multiplies it by 2 to just be safe. He then informs an unimpressed Jill that the servers will also need access from your external developers and customers, and the process complicates further. After the environments are up, Jill’s burden continues with the maintenance and support of these environments, including the continuing support of Mark’s previous project’s servers. Worse yet… after all this energy and cost is expended at the beginning, Mark’s POC shows that the customer will be better off with Excel and the project is scrapped.. just in time for Jill to have the first server ready.

Benefits of Azure

In response to similar pains, recently, our development environments have all been created in Azure. In addition to fixing many obvious pains, we also had the following benefits:

  • Typically we have 3 environments (dev, QA, UAT) up within an hour
  • The costs are spread through the project, rather than all upfront
  • It’s very easy to tear down and re-create servers to quickly resolve big issues e.g. corrupt servers
  • It’s simple to enable access for the customer and offsite developers
  • We were able to quickly and cheaply simulate various server configurations. These could then be used to predict production environment requirements
  • Table storage is great for logging
  • Remote monitoring of the environments becomes easier

Using Azure with the demo environment also helps our customers better understand Azure and the benefits of a cloud hosted solution. This often leads to situations where the customer is swayed into using Azure for production, getting over the reluctance some customers have with the cloud.

Our approach

We are still busy tweaking our process with the Azure development environments. Therefore, we still have a couple of issues to iron out e.g. we currently aren’t using our build server in this process and deployments aren’t automated but are scripted.

Our process is currently as follows.. After the initial kick-off meeting with the developers, all 3 basic environments are created in Azure. We are now ready for the first iteration. Throughout each day, the shared developer server is updated by each developer – this process needs improvement, and our planned re-introduction of the build server will certainly help. Successful builds are packaged and deployed by the developer, assigned the “Deployment manager” role, to the QA environment. This happens twice per week. After successfully testing the deployment in the QA environment, the build is deployed to the UAT environment which will be used in the next iteration customer demo. Our iterations are typically 2 weeks. In addition to the above, we also create ad-hoc QA environments to simulate multi-server production configuration scenarios.

The above approach has led to very productive use of developer time, easy adaption of the infrastructure based on agile requirements and stable builds for the UAT environment.

Gotchas, tips and tricks

It’s critical to understand that Azure isn’t exactly the same as your usual in-house environment. Some architectures work much better in Azure than others and having your servers in Europe or the USA will have some impact. Luckily, understanding the issues can mostly mitigate them.

For starters, apps that are chatty with the DB are going to have performance issues. Therefore, avoid architectures that encourage this. Our South African bandwidth is sometimes an issue but wasn’t as disruptive as we expected with the exception of uploading large customer data sets which is both time consuming and costly.

If the production solution won’t be hosted in Azure, the security options can get complicated. Try to keep with security configurations that will work in both Azure as well as the customer’s environment. Check out for Azure security guidance.

As our entire developer environment is now moving over to Azure, it’s critical that your Azure account is managed correctly. Early on, we had an incident where our account was suspended resulting in a very stressed team for a few hours. A well-defined process for turning off inactive servers is a must, as mismanagement of the account can get expensive and wasteful.

In summary

The usual steep learning curve and pain associated with introducing radical changes and new tech wasn’t there with Azure. Azure is very intuitive and it all just works. Using it for our internal dev environment is a great way to solve internal IT problems and skill up the team for the coming wave as enterprise customers begin adopting this platform.

For more detailed/technical information, contact Intervate Information Systems & Architecture consulting at