Devops Mastery Brian Wagner, Jason Didonato

A lot of people new to automation are very nervous about letting a script or program like Chef change a production system. That fear can really slow the advance of a DevOps movement. But ignoring those people's fears is not the answer. So here are some tips and tricks to get everyone, including yourself, to be more confident about writing world changing scripts.

Whenever possible create virtual environments. The use of virtual machines or cloud servers gives you considerably more options and abilities than bare metal hardware. Simply the ability to snapshot a server and revert back to it in minutes is a game changing concept. It let's you run any automation from beginning to end virtually without fear. I say virtually only because you occasionally run scripts that affect systems that cannot easily be virtualized. This is now more the exception than the rule. If you take a snapshot of the server, run the automation, find a mistake/error/bug you simply revert the server back to the starting point, fix the problem and try again. The only thing you have lost is the time it takes to run the script. This might be a lot of time but still less than having to rebuild the environment between each test run. If you have the luxury of multiple environments to use you can and should follow the same procedure used in the development of the script for future deployments. So you just take the script that worked in Dev and run it in Test after taking a snapshot there. Then rinse and repeat what you did all the way to production.

Test it over and over and over again. It sounds crazy but I now run the completely new scripts I am ready to bless for migration to the next environment multiple times in each environment. This gives me the confidence that they really are working. I want to know I didn't do something weird that made the script work when I fixed the bug. Once the script runs cleanly multiple times I am all ready to go and I move it to the next tier.

Ask someone to do a peer review of the script. This is just a good practice in general. When possible I suggest walking the person through the script and letting them ask questions the first time through. This has often turned up bugs and other issues all on it's own. Having to explain something uses a different part of your brain and opens your mind up to see the problems or possible problems.

Ask someone else to run the scripts for you. I normally do this in a Dev/Test environment. I do this for two reasons 1) Validate my documentation 2) validate the procedure I am doing. Often my teammates have asked questions that point out things I forgot to account for or code in. They also tend to be less stressed than if they are deploying to a production environment. Doing this earlier is better. Repeating it when you have to make a change makes you a better programer/scripter. Your teammate finding that one thing that would have driven you nuts during a deploy is, in a word, priceless.

Communicate what it is you are trying to automate. Be as specific as possible in the communications. Making everyone involved aware of what it is you are doing goes a long way towards building their trust. It is the most basic way to be inclusive in your process. Do your best not to make it sound like you are asking for permission to do the automation. You are doing this to make the environment better and no one should ever question that. This doesn't mean that you shouldn't be open to feedback. You should actually be ready for it because it will happen. While everyone in any given IT organization may not be expert communicators you need to take all feedback without feeling persecuted. Remember this is just as scary for them as it is for you at the beginning. So they may need time to adjust and gain confidence also. The worst thing you can do is start or continue a flame war about your automation process. Respond respectfully and with clarifications about thing

Bob from Minneapolis sent us some great feedback and a couple of great questions. In this post we are going to tackle communicating what scripts are available and what they do. I spend a lot of my time as a consultant writing up documentation. At most sites I am pretty sure the work is lost before I get out the door on my last day. So how did I try to handle this before as a Tech Lead?

To be completely honest it always seems to be hit or miss. I have never found a single solution that works for all purposes and with all types of people. Let's face it we as WetWare are the most difficult to communicate with. Hardware wants power and bits to process. Software wants data and other inputs. But humans, a.k.a WetWare, all want something different. There are all kinds of factors from Gender to skill level to happiness with their current job that can affect and require different types of communications. Since we can't solve all of these problems, and at least one is unsolvable, we need to focus on what we can.

Let's start with the one thing I know doesn't work and that is network file shares. It's been my experience that network file shares full of Word or Text documents are where good documentation goes to die. There is no automatic version tracking, you can never find a structure everyone will agree on, and unless you ingest it into a database somehow it's basically unsearchable. Since no one can ever find the document they are looking for or be sure it's the most current it quickly becomes a thing we do because Management says we have to not because we find it at all useful.

Wiki's can work well and solve at least a few of these issues but they still aren't a cure all. They tend to be search-able and track versions. They still have the structure issue though. If Google has taught us nothing it is that we don't have to care how things are stored in the great database that is the Internet as long as it can be searched we can find anything. With that being said the structure is probably less of an issue with a Wiki because they are generally speaking searchable. So it's always going to stay on my list. There are so many options from your own version of the Wikipedia to all javascript single page solutions. It's still not perfect but it works better than a file share and generally looks better too.

Version Control systems like Git believe it or not when combined with something like gitlabs, gitblit, or github(public or private) can also work as a suitable solution. It keeps the documentation with the code and is search-able. Since the code and documentation are kept together it works a lot better for scripts and programs than it does for something like the procedure to shutdown your whole data center. It doesn't mean it can't or shouldn't use it just that it may not work great for all use cases. So again we are close but still not a home run.

Knowledge-base/Content Management Systems/anything like it all work to varying degrees. This concept is really more implementation and software dependent. For the most part the difference between these and a Wiki is generally the flexibility to document changes to the documents and who made them. These systems tend to be very rigid and generally require defining a taxonomy to make them work efficiently. There is nothing wrong with defining a taxonomy but it normally provides very little reward for the time and frustration put into them to get everyone on the same page. They generally just become difficult to manage over time and then grow less useful as people stop putting the effort in.

What seems to work in a lot of environments is a blending of the Wiki and Version Control(GIT) concepts. Yes your documentation ends up somewhat scattered but if everyone knows where to look for which kind of document then it works pretty well. I generally suggest putting things like procedure and policy items go in a Wiki so they are accessible to everyone. Then for things

Everyone loves a new toy. New tools help us do our jobs better, faster and more accurately. Which is great when you understand exactly what you want the tool to do for you. How often does that really happen though?

As an Enterprise Architect I am asked on a regular basis to help companies deploy tools. The problem I run into the most is they don't completely understand what the tools can do for them. They know it will fix one problem but may not realize that it could be fixing others. Then there is the situation where the competing tool could have solved even more problems and easier. So why don't my large enterprise clients go through a process that prevents this? Every company is different, but sometimes it's a time factor. Other times it's not understanding the problem they need to solve. Each company is unique on what their specific issue is. Usually the core issue is that the selection process didn't have enough parameters or ignored parameters altogether.

In a lot of situations and evaluations it's a problem of not communicating with other members of the evaluation team or with other teams. When people love a tool and have their heart set on using it, very often it's hard to get them to hear criticism about it. To avoid those situations they exclude people or groups that may have a different opinion or a different tool they are passionate about. People may also be trying to avoid paralysis through analysis. No matter what the reason the result will be the same. You will end up with a tool that you either never use all of the features or doesn't have enough features. This will require you to get a second tool when an alternative tool that covers both problems exists.

The best way I have found to avoid this is too do selections in three phases. The first is to solicit the wish list of features from both the tools target users and the operations teams supporting it. Also ask them for suggestions of tools they like or hates using in the past. The second is to research the tools then start looking for feature lists. Evaluate each against the wishlist and not each other. Try to narrow the set down to two or three at most. The final step is to do a Proof of Concept with your top choices. In the Proof of Concept Phase Install them, set it up in a development environment, and actually make it do something. Ask the Users and support teams if each tool is doing what you think you need it to. Once you have things working then demo each environment. This will show how well, if at all, they are answering the items on the wish list for a small subset of the tools users and operations staff.

If the Demo's go well then your choice should be clear. You will have done what you can to make people happy. Just don't expect everyone to love the choice. you can never make everyone happy.

If you are really going to master DevOps you need to master communications. At least the ability to clearly motivate and pull people into the discussion and the goal you are trying to achieve. Spending an extra fifteen or twenty minutes writing an E-Mail can save you days and sometimes weeks of discussions, arguments and apologies. I have a lot of tips about how to be more effective in communicating ideas but today I am going to give you my top six.

What is the goal of your communications?
A goal is essential to forming any effective communications. If you don't really know what you want to achieve with the message how will anyone else. Some common goals for me when I was an operations lead were getting help with a Memory Leak, let everyone know about upcoming patches to the middle ware, or getting someone to help figure out why a build or deploy is failing. Once you know what the goal is you can easily expand on it and form a complete thought and translate it into a communications.

What is your motivation for writing the communications?
Sounds weird I know. Communicating why you are trying to solve the problem often goes a long way towards getting people's sympathy or at least an understanding about why you care about the problem. If you try to hide this or omit it you run the risk of pushing people away. That will also draw people in. In those cases where you are trying to help someone else it will also draw them into the communications and compel them to read on. So state it clearly but don't go and on about it. I normally try to keep my motivations locked up in one sentence. To much discussion can come off as bragging, whining, or self-righteousness depending on what is being said.

What are you trying to motivate others to do?
This one sounds super obvious but it is really more about how you say it then what is being asked or said. This is really the sales pitch part of any communications. I know that for a lot of people this is the most difficult part. We all normally hate being sold things so the idea of selling things to others can be difficult. The thing that really good sales people know is that the best way to sell is not to be a pushy sales person. Instead focus on the benefits of the person you are -selling to- asking for help from. Everyone wants to look good doing what they do best. Making people feel important by asking them for help in areas you aren't the best in is also helpful. A little humility will go a long way towards winning people over to your way of working and thinking.

Stay away from certain exclusive words and use more inclusive words.
I only speak English so I am not sure if this part holds true in all languages. In English though words like I, me, my, us, Department/Group Names, them and you tend to imply an exclusion of the reader or you as the writer. This exclusion while a minor thing really can put people off and make them restive to your ideas for various reasons. Some people will think you are blaming them, others may think you are dumping work on them, and still others just may not know you so they will use these words as an excuse to ignore you. Instead focus on using words or phrases that imply inclusion like we, our, as a team, etc. Depending on the context these words will have different levels of affect. They will always be positive it's just that some situations are just naturally inclusive so the effect of this is just less visible.

Leave the door open to Dialog.
Often times the communications I receive are worded as statements of fact and not statements open for discussion. They seem to be written by people who have already decided what the best approach is and they want me to rubber stamp it. It's my job as a Solutions Architect and previously as a Team Lead to resist this. If you are talking about my area of expertise and haven't asked me for input you might even piss me off. So while stating your plan may be acceptable make sure

What follows are the criteria I use to determine what to script or automate first, second, and so on. It's another of the "it depends" questions that never have an easy answer. I will try to help you make some rational decisions but ultimately it's your world to live and you need to decide.

The thing I see everyone want to do first is automate and that takes the longest to do. That task takes the better part of a week to complete because you are constantly getting pulled in a hundred different directions. This is not the first thing you need to automate or should automate. Not that the duration of the task isn't important but the number of times you have to do it, is more important. I may only spend 2-3 minutes compressing some logs on a development web server. If I am doing it on 10-20 various servers every week it's not only annoying but it's sucking a lot more time because of the context switching it's causing. There is also the cost to your company in the time spent by the users of the system trying to troubleshoot the full disk.

So how do I figure out what's sucking up all my time? Do I track everything I do all the time? Heck no I don't. I use a simple spreadsheet to track my work for 3-4 weeks at a time. I repeat this every 3 months or so. Then look at what I spent my time on. I track smaller items with just tick marks and an average time to complete the task. I don't worry about what time of day it is. If you want to note the time it takes you for longer tasks it will give you better data but not necessarily better results for this purpose. Remember the time you are spending on individual tasks isn't as important as the number of times you have to repeat it.

Now let's flash forward to 3 weeks from now after you have some data collected and answer the following questions.

What are the top 10 issues you resolve on a regular basis?
Are any of the tasks on the list affecting mission critical parts of your organization? Do you think you should add them to the list? You will likely want to add them and pay close attention to the next item on these tasks.
How many steps are there to automate in the top 20 tasks you do to resolve these issues? (Exact numbers are not needed, just a really good guess.)

Your top ten list plus the items affecting Mission critical parts of the organization should be your starting point. To further narrow it down start with the items with the least number of steps. When doing this though make sure none of the steps are "Install the OS" or "Install Oracle". Those task steps are not really one step but multi-step tasks on their own. Now that you have your list ask someone else for input. Be sure to provide them with the complete list. I have often described myself and co-workers as being on a mission when they start automating. This is great, shows focus and determination. It also can in some cases cause mission blindness. You want so badly to automate that really hard thing that you miss the quick wins lying at your feet. Asking someone else to look at your lists will hopefully help you keep from being blind.

Take your own time if you need too complete the list. As much as the company will benefit from your automation efforts, your work life balance will be the most improved. So spending a little of your time up front will give you more of your back time later.

If you can reduce a series of tasks you do daily that takes 30 minutes into a single script that runs in 30 seconds over the course of a month you will have 9 more hours to do new scripts. You will also gain from the lack of typos the script will make on Monday mornings. If you can take that same script and have it run as a scheduled tasks (think cron or at commands), then you can get back the whole 10 hours to put back into scripting the next thing.

Remember the goal of automation is to free you from the mundane tasks so that you can work on the more complex and pro

Automated Deployment or Automated Build tools are all complicated to give a true evaluation. Before you even try to evaluate the tools, working through the complete process manually is normally the best place to start. You need to understand how the deployment process works before you can figure out what tool will meet your needs. Things to take note of is the tests you will need to run, the operating systems involved, and how you plan to do the deployments to production. You are trying to achieve consistency across all your environments so deploying a tool designed for web applications may not have the features you need to deploy smart phone apps. This isn't to say that most of the tools can't do both just that you need to probe deeply to make sure all your requirements are met.

Below is my top list of things to look at when choosing a tool of this class. As with all the other articles on tools I have written this list probably isn't complete. You will need to assure that you are meeting all of the "it depends" requirements for your environment. Taking a disciplined approach in choosing the tool can save you months of time trying to configure it for your companies needs.

*Do you need to host the tool yourself or can you use one of the many PaaS options available?

Tools like TravisCI are wonderful tools which will let you scale your costs of the tool and infrastructure as your company grows. However, if you are an established company with a lot of existing applications it may be cost prohibitive. So be sure to look at the cost of the solution before committing to the cloud.

How are you going to test your applications or infrastructure as part of this process?

Does the tool support all of your testing tools? Or do they support it? Some testing tools are better than others at inter-operating with this class of tools than others. In some cases, PaaS options most notably, may not be an option because you can't connect to one or more pieces of infrastructure in your test environment.

* Does the tool have hooks into your version control system that will adequately support your needs?

This is less and less of a problem with most systems but it should be validated before a final decision is made. If you use SVN and the tool only supports GIT it's going to knock it out completely. Also where does the code have to be stored? If it only supports GitHub and Bitbucket but you store your code in a local gitlab server it is not going to work for you.

* How much and what kinds of work flow do you need?

Some of these tools are only designed to do the most basic of work flows like push the code to the next environment. Others can do what would be considered full orchestration like building complete systems from scratch. From a DevOps perspective not being able to do full orchestration is an issue. We want the system to never need human interaction. So only doing things half way isn't helping us to reach the goal. Luckily most of the systems while they may not directly support full orchestration can call on other tools like Chef via Build/Deploy scripts to accomplish the task. At the same time if you are only going to be developing desktop and smart phone apps a manual process may be your only option. Not all distribution mechanisms have API's and ways to automatically deploy them. While these are becoming more rare every day it's something to keep in mind.

What kinds of reporting do you need it to do?

This is normally a rather broad set of requirements. Generally they will create a web report of the progress of the current builds, last successful build, and last failed builds. What is contained in those reports can vary far more than you might expect. So be sure to pay attention to what the sample reports show you. Be sure to discuss this with everyone to make sure that all of the teams needs are being met.

How much does the tool do out of the box versus ne