Network Management and Operations - Tools of the Trade

Cisco has most recently addressed this in their Service Provider Operations certification track, however there has always been a certain degree of an "ops" perspective seeded throughout most Professional certifications. My approach is more about the tools, methodologies, and tasks that one utilizes on a daily basis to successfully maintain an enterprise network.
Before we embark down the path of being truly successful in managing our enterprise, let us examine why they call it "Operations":
Quoted from http://dictionary.reference.com/browse/operation
op·er·a·tion [op-uh-rey-shuhn] noun
1.an act or instance, process, or manner of functioning or operating.
2.the state of being operative (usually preceded by in or into ): a rule no longer in operation.
3.the power to act; efficacy, influence, or force.
4.the exertion of force, power, or influence; agency: the operation of alcohol on the mind.
5.a process of a practical or mechanical nature in some form of work or production: a delicate operation in watchmaking.
For us, bullet points 1 and 5 are most relevant. As a fellow VIP - Scott Morris - has mentioned more than once, you can often break down any given task into a subset of smaller, simpler tasks. Network Management is the epitomy of this if you really dive into the details; this is why "operations" is the key word given to most groups that execute the Network Management responsibility. It is a series of processes and acts that collectively comprise a full suite of capabilities to help you maintain your IT infrastructure, in this case specifically the network. As most folks that have been in the industry for a bit know by now, networks grow - whether organically or by design. With that growth comes the need to scale your operations to maintain efficiency and reign in costs. Several tools and processes come to mind which allow us to do just that, which we'll discuss in this blog series. Several of these topics will be expanded on in successive blog posts independantly. However, the holistic goal here for now is to show people what tools are out there from a conceptual perspective, why they are important, and what they can do for you individually.

Network Monitoring

One of the primary tools that will enable us to run our networks is a capable network monitoring system. This facilitates near real-time visibility into the status and health of our network. Tools such as SolarWinds Orion, HP OpenView, NetCool, SMARTS, all give the network team the ability to see what is happening based on SNMP and Up/Down tracking of devices. Often called "alerts", when the notification comes through that any given metric has surpassed a threashold, it allows the Network Team to react to it. Most times this involves a ticket being created to track this event. I'll circle back around to incident management down the road, but that is the system you would ideally have in place to facilitate these "tickets".

When you first roll out a monitoring tool, especially if this is the initial introduction of a tool like this in that environment, you may choose to start only with up/down monitoring enabled. This allows the IT staff to really come to terms with dealing with alerts, having the network tell them what is going on, how to use the software, etc. Up/Down alerts are a good way to break staff into this kind of growth of responsibilities/capabilities. Over time you can introduce link status, errors, utilization, et al.

Here are a few favorite alarms that I've seen companies track:

• Up/Down Status
• CPU Utilization
• WAN Link utilization
• WAN Link health (errors, drops, etc)
• Critical LAN link status/health

There are myriad more alerts that most systems employ, but those are ones you typically see at any given shop, earning them a spot on the list of what I call universal favorites.

Configuration Management

With a capable configuration management tool you can automate many tasks that may otherwise tie up valuable man-hours. Suppose you need to update an on-call number within SNMP for EVERY DEVICE in the network. That could be 30k devices! If you have 30, maybe logging in and changing that one variable is feasible. However, for 30k, that could literally take weeks. With configuration management, you are looking at writing a script to update the configuration, and then selecting the scope of devices to run the script against, and viola - done! Just be sure your script works prior to blasting it out to 30k devices......

Another example is deploying devices - you can have your staff deploy a switch with the meat of the config, VLANs, VTP, uplinks, etc. They get it up and running - and then you pull it into the management domain and deploy your management template. This can include SNMP, AAA, security ACLs, etc. All centrally managed - which reduces the chance of error.

These tools can often be useful as well to execute custom poll scripts to devices. This can help you tool reports to specifically target a special case that exists on your network, or target specific information you need without having to poll through an entire "show run" or "show tech". This is especially useful the larger your network gets.

Biggest benefits you often gain out of configuration management systems:

• Historical configuration backup
  • Easy way to find last known good config during outage
• Mass change function
  • Intelligent scripting can cut time on large-scale simple changes
• Config reporting
• Ability to quickly poll a stored data set for patters/configs w/o impacting production network

Incident Management

This is the fabled "ticketing system", which tracks incidents via records, also known as trouble tickets, event tickets, work orders, task orders, etc. There are as many names for it as there are versions out there. Remedy is one prevalent platform, as is Heat. I've worked on several internally developed platforms that usually outperform both, but that is because they were built from scratch for exactly those environments. Tough to do from a template.

The Incident Management System(IMS) is typically seen as the chronological life of the network from an operations perspective. You can track chronic issues at sites, you can track trends, you can track man-hours spent on projects, you can track the utilization of your personnel, etc. Often times you can use these metrics to justify a project/expenditure : "we currently work 3000 unique tickets a week, with this upgrade we could cut that to 500, freeing up X man hours". On the flip side, the IMS can also serve as the record for changes made on the network for break-fix situations.

Common things that IMS tickets are used to track are as follows:

• Timeline for incident
• What troubleshooting was done
• What was found to be the exact problem
• What actions were taken to resolve
• What was root cause of problem

The need to document all of the above cannot be overstated. To be able to sit down and say "we have 12 tickets with this root cause a week, we need to investigate why this is occurring", is invaluable. If you can resolve the root cause moving forward, you have then just avoided those future issues. While this kind of analysis typically benefits larger scale organizations, the thought-process and methodology can benefit any size shop. This is the kind of optimization that can really save a company money under the IT budget, and everyone loves doing that.

Change Management Controls

This is many people's worst enemy - change management! The idea is to keep a historical record of all the changes that go on in the lifecycle of the network. The benefit of having this kind of looking glass into the past is multifaceted; metric tracking, root cause analysis, accountability to stakeholders(more on that later....), and perhaps above all - providing visibility into the stability of the network.

Part of the difficulty many organizations face with change management is fully integrating the business facet into the IT world. Not only does this require the IT group accepting the fact that the business has the power to approve/decline changes, it also requires the business to understand the strategic and tactical nature of how their IT systems support and/or drive their business vertical. Without going into specifics, if a business is in the process of making you money, you want the network to ASSIST in that process, not be the cause for financial loss. Robust, well developed, and fully integrated change management policies paired with an easy to use tool to track this is critical for companies to develop stringent control over the lifecycle of the network.

When a business unit fully realizes the control and peace of mind that can result from this kind of framework, they often buy into it and get involved. Balancing business versus IT needs can often be precarious at best, a well forumalted decision matrix can help ease those tensions. When the change control process is followed dogmatically by all of the parties involved, two huge benefits are realized. The vertical can now hold IT accountable for outages they cause - which makes for a more calculated approach to dealing with network changes. On the other hand, the IT group can then say "we made no changes", and the business vertical should have a reasonable level of trust that this is true by looking into the change management system. Checks and balances should always exist, and I've seen large scale shops build in scripting tools to track EVERY keystroke of an engineer and log it to a third party within the company for reconciliation purposes. While this is an extreme case - it goes to show you how far this kind of concept can be taken to balance the need for action and the requirement to follow policy.

More to Follow.....

With that background, in future blogs I will go on to show you how you can tie these tools together using policies and processes. Each of these tools alone provide great value in and of themselves, but they truly shine and provide an exponential ROI when your internal practices leverage them properly. Before I can show you that, though, they would need to be up and running in your environment, no? So, I will give you a few walk-throughs on basic deployment of these tools within your environment. Considerations that need to be addressed, how to pick the best product, the pros and cons of buying Commercial Off The Shelf products versus developing some of them in-house, and so on.

Once we can get them up and running, we are going to discuss integrating your business model as an IT shop around them, how to work with your customers - whether internal or external - and re-tool your relationship with them based on these new capabilities. In addition I'll try and show you can leverage them to provide SLA agreements with internal customers, what you can do to use these tools to bring truth in advertising to other groups within your organization, and a few other neat features that you'll find.

I hope you've enjoyed this blog, if you have any questions or requests, please leave a comment! I will try to respond as best I can, if you don't get a timely response PM me and point me towards the thread.

Thanks everyone for reading!

REF:
Network Monitoring Tools

Network Specialist: Wireless Network Engineer

When looking at any specialization, the deeper you go, the more you realize how much you still have to learn. As I dig deeper into the Wireless area of networking, I have discovered just how much details are involved in being a Wireless Specialist! Although I work in many different Networking areas, wireless is one where I spend a considerable amount of time and an area I am getting deeper into.

First, a wireless specialist has got to have a solid understanding of the physical layer they are working with. That's right, I am talking about RF! All that boring talk in your physics class comes to life in wireless. Things like EIRP, dBm and bandwidth are foundational principles that will promise you failure in the wireless realm if you do not understand them. Understanding the electromagnetic spectrum is a wonderful start. This base RF knowledge is a requirement in performing one of the most important tasks that a Wireless Engineer can have, performing a site survey!

Now performing a site survey in and of itself can be a monumental task and takes a lot of time. There are several things to keep in mind just when preparing to do a survey.

1) Understand what the customer wants. This can be one of the hardest things especially if the customer isn't exactly sure what it is they want other than they want their wireless network to just work.

2) Understand what restrictions the FCC or any other regulatory body may impose on your RF environment.

3) Know the facility you are surveying. There may various procedures, policies and even restrictions that you need to keep in mind. For example, you may need a security clearance to even get inside the building before you can start a survey. There may be OSHA and other fire and safety requirements that must be followed.

4) Plan out what type of survey you are doing. Are you surveying for a Data network, Voice, Location or some combination? Each type will have different RF and AP requirements.

Once you have a plan, start your survey. Typically when I do surveys, I will go over a building map and trace the map with colored pencils what kind of walls, doors or windows there are and then import that map into WCS or some other predictive survey tool. Then, I enter such information to give me a semi accurate predictive survey. It’s a lot of work, but worth it. Next, I take a spectrum analyzer and do a walkthrough of the building just to see what potential interference issues I may come up against. You may be surprised at what you find.

Next, is the Layer 2 survey. Here is where you really look at signal strength, plan out channel placements and survey the overall performance and coverage of the AP you are surveying. I will try to place a few temporary APs in locations that I determined with my predictive survey tool. Then I would take a laptop and perform the Layer 2 survey by slowly walking through the building, gathering Wi-Fi stats from my WNIC. Now there are several ways to approach this. I started out just using the Advanced Stats on my Intel Pro WNIC. It gave be the basic information, but really didn't give things like data rates, transmission errors and those real details that happen in a live wireless network. During some studies, I was introduced to Airmagnet and anyone who is serious about doing a quality survey will want it or a tool like it.

Once your survey is complete, I have already walked the building at least 3 times if not more. First to get the details for my map, second, to perform a Layer 1 survey with a spectrum analyzer and then another with a Layer 2 survey.

Now it’s time to deploy your APs! Where are they being placed, how are they being mounted, how are you powering them up? There are many many considerations when deploying APs. I will typically mount APs in the ceiling when possible and will prefer to use Power over Ethernet to power up the devices and using APs with internal antennas. Now there will be special cases where you will want to use APs with external antennas.

During the survey and installation process a big issue is channel allocation. For the 2.4 GHz band, there are only 3 non over lapping channels. You need to make sure that your APs are using these channels and are all being separated from each other. There is nothing worse than causing your own interference problems by placing 2 APs on the same channel right next to each other! Technologies such as RRM are a big help in this regard.

Whew! That is a lot of work and we are just getting started! Now this is just the survey and installation of a Wireless network. There are many other design aspects, such as what kind of security to use. Ah, security. You can't leave your wireless network without it, unless you want a pounding headache that not even morphine can cure! There are 2 aspects to security in Wireless. The first security aspect is authentication. You want to know who is on your network and permit who is and is not allowed to use it. Authentication is the mechanism to do this. There are various forms of authentication. The most popular are Open, which is basically no authentication, mac address authentication, although it is not scalable and very easy to spoof. Then we have EAP (Extensible Authentication Protocol). With EAP, there are various flavors and the Wireless Specialist will be familiar with those flavors and be ready to implement whatever their customer requires. The most popular EAP methods being:

EAP-TLS - which is certificate based
PEAP - uses user credentials passed through a secure tunnel
EAP-FAST - similar to EAP-TLS but does not use certificates, it uses a Pac file instead.

The second part of security is encryption. Typically in today’s networks, TKIP or AES encryption are used. If no encryption is used, it really isn't too hard to sniff traffic out of the air and see what is going on in the wireless world around you. Because I am paranoid, I will typically use a VPN when connected to a public Wi-Fi hot spot, just to provide encryption.

When choosing a security type, there are many considerations. The main consideration is "do your clients support it"? It won't do you good to choose EAP-FAST if your clients don't support it. PEAP and EAP-TLS, WEP and WPA-PSK are the typical methods that most clients support.

Wireless security doesn't stop there. Remember that the wireless network is an extension of the wired LAN and so you need to not only secure the clients and APs via the air, but you need to take measures to secure your APs on the wired side as well. One of the most deadly threats are rogue APs. You know, the APs that are not a part of your network but are seen by your network, either over the air or on the wire. Things like rogue detection, ACLs, MFP, vlans & firewalls, RADIUS servers, weather its ACS, IAS or some other kind of RADIUS server and the Wireless LAN Controller are all used to help protect your wireless network.

Once the WLAN is installed and secured and users are using it all is well, right? Maybe, hopefully, but there may be times where there are problems and you will need to troubleshoot what is going on.

Now, troubleshooting wireless is a little different than your standard wired network. Why? Unless you’re Superman, you can't see the physical layer. Part of troubleshooting a wired network is checking your physical layer. Well in wireless you need to use tools to help you detect and mitigate interference. One tactic is to use the 5 GHz band. Since most interference sources reside in the 2.4 GHz band, I try to use the 5 GHz band whenever I can. I also suggest to clients to try to get wireless devices that are dual band. There are a surprising amount of devices that claim that they are 802.11n, but only support the 2.4 GHz band. A great troubleshooting tool that helps with interference is Cisco's CleanAir technology that is found in their newer APs. These APs have a spectrum chip in them that help identify the interference sources, such as Bluetooth, analog cameras, microwaves and so on.

Also, look at the load of your APs. Remember, APs are half duplex like hubs and are a shared medium. The more clients on your AP, the slower it will get.

What else does a wireless specialist do? Well what is the point of having a wireless network? Usually it’s for mobility reasons. There is little point in having a wireless network if you don't need to roam around and be somewhat mobile. Things like roaming come into play, weather its setting up mobility groups on a controller or using WDS on an autonomous solution. Being mobile is what makes wireless so cool. A wireless specialist is going to make sure that mobility is one of the basic functions of your WLAN and that it works well.

As we live in the days of Unified networks, it won't be too uncommon where Specialists of various areas will work together to create the network as a whole. A Wireless Specialist may work with a Security specialist regarding wireless security. The Wireless Specialist may work with a Routing & Switching Specialist to tie in the WLAN to the wired back bone. The Wireless specialist may work with a Voice expert when a customer requests Voice over WLAN services.

A wireless specialist will also work with outdoor wireless such as MESH and wireless bridging. Although there seems to be many general areas of a wireless specialist and there are, the wireless specialist works with RF as the physical layer to deliver the same applications that we are used to using on the workstation that is plugged into the wall. As wireless continues to explode, you will continue to see a demand for the wireless specialist out in the field. All of today’s coolest gadgets are all wireless devices and they are not going away any time soon!

Ref:
Wireless Specilaist

Just what DO you want to do with your life???

Throughout the IT industry, people have different roles. Some people are very deep into one particular technology or niche. Other people are very broad across multiple technologies.

There is always interest in the "I wonder what they do?" question. It likely stems from some version of the grass always being greener on the other side, but we figured we would take some time and write a little bit about what it means to be a Network Specialist.

This is often useful information in determining what things may be interesting to you as you start, change or otherwise morph your career! (e.g. the ever-important question of "What do you want to do with your life?"!!!

In a previous blog, we already had some exposure to the Network Generalist. So now the questions abound... What's different? What's interesting? What's annoying? What do you actually do with your day? So really following in-line with Cisco's method of certification tracks (and not necessarily in any order), we have:
Routing & Switching
Security
Wireless
Service Provider Operations
Design
Unified Communications

Between each of the VIPs, we will attempt to hit on each of these areas in order to share our views and a little about our jobs/lives with you along the way! We certainly appreciate commentary and hope that it sparks some great additions and discussions from others who are in similar situations!
But all in all, the idea is about sharing experiences. And helping to give perspective to newcomers who are trying to decide what they want to do with their lives!
Keep in mind that these perspectives are our own and certainly aren't going to contain every possible scenario! But they will give you some idea about both the interesting and entertaining aspects of these technical fields and the roles that go along with them!
We have also pondered the idea of adding some management (from team lead to director/manager to CIO roles) into the mix as well. We will certainly enjoy any feedback that we receive along the way and will react accordingly!

All of us hope that you enjoy the series, and get ready to ask whatever interesting questions you can think of!
---------------
Ref:cisco.com