Blog

In response to the large number of requests received to publish a follow-up on my last post, I am posting this in response.

With my last post I hit on the multiple issues that represent real world roadblocks to seeing Unified Communications (voice, video and collaboration) being successfully rolled out within the Department of Defense (DOD) enterprise. The major issues surrounded the lack of acceptable base/local network infrastructure needed to successfully deliver real time services, as well as the lack of a single information assurance solution that is actively supported across the department. Adding to that is the lack of a service delivery model that is based on a single technology platform that can meet the many integrated UC requirements for the millions of DOD users located around the world.

So the question that has been proposed is “Where do we go from here”?

Step 1: Policy Requirement(s).

The DOD Chief Information Officer (CIO) has a responsibility to publish designed to provide guidance and instruction to the department. The last piece of policy published by the DOD CIO office was the DODI 8100.04 (Unified Capabilities) signed out on December 9, 2010. The objective of this instruction was to “require” the department’s commands, components and agencies to transition from legacy voice technology to new voice and video over Internet Protocol (IP) in adherence to the Defense Information Systems Agency (DISA) Unified Capabilities Requirements (UCR) technical specification and the DOD CIO published UC Master Plan. One of the underpinnings was the continued requirement for technology to be tested and certified by the Joint Test & Interoperability Command (JITC) and placed on an Approved Products List (APL) where acquisition commands within the DOD could go to ensure products being proposed by bidders had been approved for integration into the DOD IP networks. There was a provision within the DOD UC Master Plan for all components and agencies to submit to DISA a comprehensive Implementation Plan demonstrating how each was going to accomplish this transition, and a timeline for when it would begin and when it would be completed. This is where the wheels fell of the cart… DISA did not receive the required plans in the time frame required. In fact, over 24 months passed before they had responses from all the services, and at that time much of the technologies captured in the plans were no longer on the APL, rendering many of the plans worthless to DISA.

Due to the pace of advances in UC technology, for the DOD to be successful, there must be a single voice that is projected to both industry as well as internally that captures the specifics of a UC technical architecture, design and defined roles and responsibilities. The Joint Information Environment (JIE) work has an opportunity to host such an initiative, but for it to be successful all combatant commands, components and agencies would have to participate with both operators and acquisition authorities in a single working group with the mission of creating a single Solution Architecture and Implementation Plan. Such a plan would be required to be signed out by the DOD CIO as the replacement for the current UC Reference Architecture and Master Plan. Parts of the new plan would be needed to update the existing Instruction to sustain its relevancy.

Step 2: Acquisition Planning

One of the key phrases repeated often by the DOD CIO is that there is “no new money”. In fact, because of the past eight fiscal years being funded mostly with continuing resolutions, he is absolutely correct. By law, no new projects can be funded with continuing resolution funds. In short, the money allowed to be spent is designed to only “kick the can” on the existing solutions, limiting spending to sustainment and technology refresh efforts. This becomes an interesting dance for acquisition commands, as many of the technologies in operation are no longer on the APL and in many cases are no longer supported by the vendors. The end result is many parts of the DOD do not have active maintenance agreements with industry putting the overall health of the DOD Information Technology at risk, as well as our national security.

Another acquisition challenge is dealing with the many “rice bowl” issues within the DOD. Many Program of Records (PORs) have successfully fenced their funding, and their scopes are so narrowly defined that allowing for any new technology to be sponsored through the POR is all but impossible. Separating operations dollars from procurement dollars is also challenging, as any money that can be used to purchase and install technology cannot be used to fund maintenance or operational support. To often there has been technology purchased without any maintenance or training tail, leaving operators with a new responsibility and no means to support it. The end result is the technology is never used, and taxpayer dollars are wasted.

In order for the DOD to successfully accomplish an acquisition needed to deliver a successful UC enterprise, it is imperative that acquisition rolls and responsibilities be established within each component and agency. There truly needs to be a single voice within each component and agency that has single acquisition authority for that individual component or agency. Without such authority, what ultimately gets purchased and fielded will become disjointed and accomplishing a single integrated enterprise will be all but impossible.

Step 3: Network & Cyber Operation

Too often it is easy to focus on the acquisition part of the equation without regard to the how the delivered solution will be managed, operated and protected. Today there are many “pockets” of voice and video operational management and in some cases… none at all. As the department transitions to a single integrated platform, how the end-to-end management of the services will be accomplished has to be vetted and agreed to by all responsible stack-holders within the department. While the promise of JIE is supposed to be addressing this topic, most all focus has been only on data networking and applications, and the unique requirements for real time service protocols have gone wanting.
Lastly from a cyber perspective, there will need to be new tools and procedures for protecting real time service protocols within the enterprise, as their uniqueness left unattended will result in voice and video communications being compromised… and no, just adding a Session Border Controller (SBC) is not going to solve the problem.

As many vendors of legacy TDM telephony technology end their support for classic End Office and PBX solutions, customers are faced with the challenge of deciding what their transition strategy is going to be and how much is it going to cost. With flagship TDM companies like Nortel and Lucent no longer in the voice telephone switch business, many customers are left with 99.999% reliable telephony systems (i.e. Nortel SL-100, CS2100, CS1000M; and Lucent 5ESS) that are still performing, but are now without any manufacturer support. There are the few businesses that once performed as integrators of these technologies who have swept up some of the unemployed talent and are offering break-fix maintenance support at prices typically found in monopoly markets, which ultimately is becoming a forcing function for many to abandon the reliable TDM telephony systems for less expensive and less reliable Voice over IP (VoIP) options. But is that really the only option available? The answer is no.

When one looks under the hood of a TDM telephone switch, one will find three key components: 1) Processors; 2) Switch fabric; and 3) Line & Trunk Interfaces. The processors are typically proprietary processors developed by the telephone switch manufacturer that run proprietary operating systems and software applications unique to that given telephone switch. The switch fabric employs Time Division Multiplexing (TDM) technology to route calls between line and trunk ports. The line and trunk interfaces are physical ports that allow for the termination of Primary Rate Interface (PRI) T1 trunk interfaces, DS0 (64 Kbps) trunk interfaces, and analog, digital and BRI ISDN line interfaces for terminating handsets. Considering the support for the operating systems and applications responsible for running the telephone switch is now no longer available, the real cost of maintaining legacy TDM voice solutions is in replacing hardware components critical to keeping the telephone switch operational. The gamble owners of these TDM telephony systems must consider is how long the operating systems and associated software will continue to run without corruption. In truth, considering these systems are closed computer environments with no external connectivity to untrusted data networks, the answer lies with who has physical access to the telephone switch, as that is the only real threat to corrupting the software. The only other potential risk is the age of the processors and hard-drives themselves, as eventually there will be a failure, and then replacing the software with the installation of a new hard-drive (if not backed up) will be the downfall of the telephone switch.

For those who don’t have the stomach for risk, this represents a crisis, and that typically is followed by the push to replace the TDM telephone switch with a new and shiny VoIP solution. Vendors from all around will be lined up to sell you on their particular technology, with many “over selling” their solutions reliability and capabilities. What they won’t tell you is all the hidden costs associated with software licenses, end-point replacement costs, and network dependencies necessary to achieve service reliability anywhere close to what you’re used to with the old TDM telephone switch. And why would they? They are only interested in making the sale. It is your job to become quickly savvy on this new technology, and to keep the vendor’s honest.

So let’s breakdown the actual real cost drivers of deploying VoIP, and then explore some logical options to bring down that cost.

When you look at most all VoIP solutions, you have two key components: 1) SIP Session Managers; and 2) Application Media Servers. They may be wrapped into vendor specific names, and be represented as more than two components, but holistically, the functionality of an open systems compliant solution will always boil down to SIP Session Managers and Application Media Servers. The role of these components is fairly simple. The SIP Session Manager replaces the old TDM fabric and interfaces with SIP signaling and negotiates the set-up and teardown of SIP media sessions (which can be both voice and VTC media). The Application Media Servers is the toolbox of goodies that delivers telephony features (i.e. voicemail, audio conferencing, caller ID, call forwarding, etc.) to users subscribed to the VoIP solution. Licensing takes many different forms depending on the vendor, and in many ways it can quickly become as confusing as trying to understand how airlines price airfares! In many cases, there is not one license to consider, but many. You will pay for the number of concurrent SIP sessions the SIP Session Manager can support. The more concurrent sessions you want it to support, the more expensive the solution. What drives that number is similar to TDM when sizing the number of trunk lines you needed to ensure line side quality of service calculated by the probability of blocking. But then there is another set of licenses tied to the Application Media Server, because depending on how many “features” you want activated, and how many substantiations of that feature you want also contributes to the final cost.

The unspoken cost, which is often the highest, is the replacement cost of the legacy analog/digital/ISDN handsets. While the last five years have seen the costs drop significantly, the average costs of an office grade multiline SIP handset still will run anywhere between $150-$250. If your business supports, say 2,500 phones within an office building, the replacement costs would be $375,000-$625,000! Your CIO then may say “Let’s roll out soft phone clients instead”. The average cost for that is typically $50 a seat (includes client license and headset for the PC or Laptop), so then your endpoint cost would be $125,000. You may think, “OK, we can swing that”, but then you have to evaluate your Local Area Network (LAN). Is it sized to support the new voice and VTC traffic you are going to put on it? Does your access switches support Power over Ethernet (POE) so the desktop phones you decide to deploy can be centrally powered? Can your LAN support VLAN and QOS programming to ensure Voice packets are not competing with the other LAN data traffic? If you have to replace your LAN switching technology, you can easily be looking at another $330,000 for the support of 2,500 ports.

Interestingly, the actual cost of the VoIP servers and associated software and licenses may be attractively priced at somewhere around $68,000, which when compared to the maintenance cost of a legacy TDM switch will look very attractive; but when you add the end-point replacement cost, and any LAN upgrade costs needed, your prized $68,000 cost will balloon to as high as $773,000.

But all is not lost… there is another option. Instead of replacing all your legacy analog handsets (which for many represent anywhere from 80%-95% of their handset inventory), keep them in service and augment your VoIP servers with analog media gateways. Media Gateways have been around for a long time, and companies like Genband offer carrier grade high density analog media gateways (e.g. Genband G5) specifically designed to be installed in commercial carrier central offices and connected to the existing main distribution frame (MDF). Other manufactures, like Patton Electronics out of Gaithersburg, MD offer lower density options, and most brand name VoIP solution vendors like CISCO and Avaya also offer media gateway options (albeit typically more expensive than going pure VoIP). But for the savvy CIO, this is where doing a comprehensive Cost Benefit Analysis (CBA) is worth the time and investment, as one can explore the idea of replacing the unsupported TDM telephony processor and switching fabric with a VoIP solution, while keeping the existing analog phones and associated wiring in service. This will result in a much lighter load being placed on the existing LAN, and will significantly reduce the number of endpoint replacements you need to invest in. I once heard it said “Why replace a piece of plastic that is paid for with another piece of plastic that is going to cost you an average of $200?” To me that makes good financial sense!

When engineering a Unified Communications solution for any enterprise, the issue of security has to be baked in from the beginning. Many commercial users of Unified Communications are not aware that their telecommunications experience is often being carried over the public internet in the clear for any hacker to tap into. Spoofing SIP packets is not a complicated act, and any two-bit hacker can do it.

There are many security considerations, as well as security appliances now on the market that will mitigate the risk of compromised Unified Communications. First and foremost, implement TLS and SRTP encryption (employing AES keys) within your design and technology to ensure both signaling and media streams are encrypted between endpoints and soft switches. Secondly, consider including Session Border Controller (SBC) technology within your security boundary configuration.

Many engineers who implement Unified Communications solutions within existing network environments end up establishing port forwarding rules for ports 5060 and 5090 which are used by VoIP to pass SIP media and signaling. This is not an advised security practice, as it clearly established security vulnerability for the enterprise, as well as results in frequent scans by unwanted VOIP agent queries found on the internet. It can also introduce VoIP performance issues, as often when there is a re-invite SIP message sent, most firewalls will open a new port for the media which the endpoints won’t recognize, causing asynchronous audio experiences.

The correct engineering practice to mitigate both potential VoIP performance issues, as well as mitigate introducing new vulnerabilities to the enterprise security posture, is to make use of SBC technology within the UC solution design. SBC technology basically provides a “stateful” security appliance that opens and closes ports dynamically to meet SIP signaling and media requirements, maintaining secure connectivity between endpoints and soft switches. It is recommended they be implemented in parallel to traditional stateful data firewalls, and include implementing separate VLANs for data and voice/video traffic on the enterprise LAN; by doing so, you will be virtually separating data and any voice/video traffic allowing the data traffic to be directed through the data firewall, and the real time service traffic through the SBC.

It amazes me how DOD leadership continues to ignore the growing crisis stemming from aging telephony technology and infrastructure. No, it’s not as sexy as C4I technology and applications, and it doesn’t have a bayonet lug; but the need for highly reliable telecommunications in support of the warfighter is as important today as it was fifty years ago. I have heard some the most irrational comments about how communications can be supported strictly with desktop chat programs; or during a crisis like what the Army experienced at Ft. Hood, that soldiers can just use their cell phones to reach first responders. Unfortunately in a crisis, the current public telephone system and wireless networks cannot support the surge in use caused by a public crisis, nor are they designed to.

That is why the Defense Switched Network (DSN) supported Multi-Level Precedence and Preemption (MLPP). It wasn’t just about giving commanding officers the ability to Flash Override calls, but also to provide a level of call access control for all users. Back in the 1990’s, the NCS had the large public exchange companies integrate a call access control capability within the Government Emergency Telecommunications System (GETS) network. This provided Government officials with a predetermined PIN the ability to gain first rights to the public switched network in the times of high demand. With the legacy Time Division Multiplex (TDM) based telephony technology and the associated outside plant cable systems aging to the point that it now jeopardizes the technology’s ability to deliver reliable communications, it is time to invest in creating a replacement system that can provide the same level of reliable telecommunications.

The challenge is that DISA and the MILDEP communities, who have been attempting to establish an IP based Unified telecommunications solution leveraging Unified Communication technology like Voice over IP (VoIP) and desktop collaboration and Video Teleconferencing (VTC), have experienced nothing but budget reductions and being openly dismissed by the data network and Information Assurance communities as not being important.

UC is not the Red Headed Step-Child of the DoD, but rather it offers the potential for higher gained efficiencies in the basic need for human communications. Like the idea that wars can be won with air power alone, wars cannot be successfully executed with strictly data communications. It is time for Congress and the Pentagon to get serious about funding Unified Capabilities within the DOD budget before it is too late.