Project 25 Responds to GAO Criticisms

September 17, 2007

 

  

In April, the General Account Office (GAO) released a scathing report criticizing the Project 25 (P25) standards process, along with the Department of Homeland Security (DHS) Safecom program, which has a goal of furthering public-safety communications interoperability. In this interview with MissionCritical Communications, Craig Jorgensen, co-chair of the P25 steering committee responds to the GAO's report findings. The full GAO report can be found  here.

MCC: The GAO report states that the P25 standards have led to incompatibilities among products by different vendors. The report specifically cites tests conducted by the National Institute of Standards and Technology (NIST) during 2003 and 2005. Is that an accurate criticism, and what remedy steps have been taken if so?

Jorgensen: P25 is an open process that combines the expertise of both industry and the public-safety community. We expect and welcome both user and industry input, including what has been provided by the GAO.

It’s incorrect to believe the report accurately portrays either the process or the actions that have taken place in regard to this issue. The standards didn’t lead to incompatibilities, rather different interpretations of the standards by different manufacturers led to some minor problems. Because the P25 process is ongoing, an internal P25 process is in place to resolve these types of occurrences. The NIST tests, coupled with some prior documented experiences from a couple of federal P25 system installations, led to the initial body of work for a compatibility group established within P25. Unlike many other standards processes, P25 addresses compatibly organizationally and doesn’t rely on user community identification and fixes to evolve the technology.

The P25 steering committee quickly engaged the manufacturers that were involved in an ad-hoc group hosted by P25 to immediately address solutions to the initial issues. While the tests to which GAO referred showed some inconsistencies in test results, to my knowledge, none of the tests showed that a user of product “A” couldn’t talk to a user with product “B.” The tests did show subtle differences between how the manufacturers interpreted some of the standards. They also showed that the written test procedures didn’t fully describe the test methodology that needed to be used. Furthermore, the individuals testing the products didn’t fully understand the methodology they should have used, which in at least one case left them to erroneously believe a product had failed that test. Obviously, that was another indication we needed to update some of our conformance documents.

Although the inconsistencies didn’t interfere with interoperability, the P25 steering committee approved a motion to formally request NIST to work on our behalf with our partners in the Telecommunications Industry Association (TIA) to isolate any standard or test-procedure deficiencies, modify them, and upgrade them as necessary. Embodied in that request was an understanding that NIST/Institute for Telecommunications Sciences (ITS) would develop a series of compliance assessment guidelines and work with our partners in the TIA to ensure the conformance, performance, and interoperability standards we created could be validated in the formal testing process. The standards, test guidelines, and the actual testing that will take place as a result of these various documents will form the foundation of the compliance assessment program. The compliance guidelines and all the appropriate standards will also be used in each of the NIST-assessed labs, some of which will be independent laboratories and others will be internal to product manufacturers. Every effort will be made to protect public-safety consumers without increasing end user’s costs.

MCC: The report also notes that many options available on radios aren’t specified in the P25 standards, allowing vendors to incorporate unique or proprietary technology and causing interoperability problems. Is that a fair criticism?

Jorgensen: To foster continued evolution and enhancement of the technology, manufacturers are specifically allowed to develop features outside of the mandatory elements. Manufacturers, for example, may include protocols in the handsets that would allow them to work with an existing proprietary infrastructure from that manufacturer. The public-safety customer may have specifically asked for that. Manufacturers may also add new, yet unstandardized features to their handsets, similar to how a new cell phone may also take and send pictures. The ongoing P25 process continually evaluates the merits of such features and their potential for becoming optional or mandatory requirements based on their universal need. At that time, the relevant documents are updated through the TIA process to standardize the new additional features. It’s probably fair for an independent auditor, such as those from the GAO with limited knowledge of either the standards process or the technology embraced within the standards, to make those assumptions regarding options.

However, P25 standards are predicated on a comprehensive statement of requirements (SoR). The SoR includes a list of mandatory requirements that every P25-compliant product provider must provide. The standards ensure that all P25 manufacturers meet a core set of mandatory requirements and provide compliant equipment.

The SoR also includes a list of “standard options” that a manufacturer may choose to provide, but isn’t required to. If a product provider chooses to offer a standard option, that option must meet a specific standard. Some public-safety users prefer a similar proprietary option to a standard option. When that occurs, vendors are required to provide both the standard option and their proprietary option. While the use of proprietary options can lead to some feature interoperability issues, they are carefully monitored to ensure they don’t create barriers to basic communications interoperability. For example, if you sent a picture taken with your cell phone to someone who doesn’t have a photo feature on the receiving cell phone, the recipient couldn’t see the photo, yet the two parties could still talk to each other on their cell phones.

By allowing industry to offer added-value features, we foster future enhancements of the basic program. The P25 process continually evaluates the merits of such features and makes some of them mandatory, as users universally need them. At that time, the relevant documents are updated through the normal TIA process.

The standards intentionally leave room for product providers to offer what they consider to be added-value features and functions. Consumers who choose to implement added-value features and functions do so with the full knowledge that they are acquiring a product that meets P25 standards but also may include options that don’t, which may lock them into that provider. Those decisions are consumer decisions, not a failure of the standards or the standards process.

MCC: Can you summarize the status of the compliance project?

Jorgensen: Table 6 (Page 40) in the GAO report generally reflects our schedule for progressing toward the completion of our conformance, performance, and interoperability standards documents and the NIST compliance assessment guidelines. But the table doesn’t reflect sufficient detail to fairly and accurately portray the work taking place. For example, the GAO report mentions the work that is being done by NIST, but fails to mention that the first draft compliance assessment document wasn’t formally introduced until November 2006. Nor does the report mention or even provide a copy of the constructive material NIST offered to GAO in its comments in the report with regard to the issue.

The GAO report also fails to recognize that the P25 process is user driven. P25 isn’t a large federally funded P25 organization; we are a group of public safety and industry volunteers, a contracted director, and a limited staff. The output of this process is the creation of standards for a small market with limited funding. Therefore, in an open marketplace, absent of massive federal funding, we are dependent on either the resources the manufacturers and TIA members provide or the services offered by our public-safety volunteers. In the U.S. it’s extremely fortunate that these parties have dedicated hundreds of thousands of man hours and millions of dollars in corporate investments to the public-safety community and the citizens they serve.

The P25 process is the only serious standards effort in North America to improve interoperability, add spectrum efficiency and security, and develop new features needed by the public-safety community. For decades, this same community has been neglected by regulators and funding agencies. While the GAO tries to measure how long it has taken, before the P25 process began public-safety users experienced decades of buying proprietary systems that were less secure and spectrum efficient than current technology. While the process and work can always be improved upon, it’s better than anything anyone else is doing. While others talk and write about what needs to be done, P25 volunteers try to meet the demands and expectations of our federal, state, and local users and Congress who don’t always have the same collective priorities. The “flexibility” the GAO urged in its report only leads us back to proprietary technology, which is why DHS and Safecom didn’t accept it. While DHS and Safecom support policies that will lead to progress, GAO wants to maintain the status quo, or worse, reverse the progress we have made to date.

MCC: The GAO report says only limited aspects of the common air interface (CAI) have been defined fully enough to conduct interoperability tests. Is that true?

Jorgensen: The report takes out of context a legitimate concern expressed by NIST/ITS that P25 officials needed to enhance some existing standards language to better accommodate compliance testing. GAO doesn’t accept the reality that the P25 compliance group and the TIA standards process have already completed most of the work on the initial messages, and they continue to work on other critical standards and new features. In this effort, P25 officials are defining more specifically some messages and protocols, so it will be more difficult to have different interpretations on how to implement the messages. While the P25 steering committee and TIA staff recognize the importance of the concerns expressed by GAO, GAO has overstated the extent of potential problems with the CAI. However, any problems with the standards must be corrected as soon as practical.

According to estimates, more than 1 million P25 units have been sold worldwide, and the FCC adopted the CAI as the interoperability mode in 700 MHz. To my knowledge, no one has complained that the CAI doesn’t work. The report also doesn’t recognize that 15 of the nation’s existing 32 statewide networks are based on platforms built on some level of P25 standards, and 29 of those networks have already started to integrate P25 mobiles and portables into their system platforms to begin a graceful migration path.

MCC: Do you agree with the “Status of P25 Interfaces” table (Table 4, Page 35) in the report, which outlines the status of each interface?

Jorgensen: While the table is probably adequate as a high-level reference, it fails to contain enough specificity to accurately reflect the work that is taking place. GAO officials have been myopic by focusing on what they perceive are the critical standards without understanding what those standards really are, talking to the P25 steering committee or TIA, or understanding how they impact the public-safety community. Until two or three years ago, we had only one major system provider in the marketplace, yet GAO failed to recognize that because of the work that took place in the 1990s and from 2000 – 2002, we now have three competitive system providers.

In its report, GAO focuses on standards completed during the past two years without recognizing some of those standards weren’t needed until more than one supplier was available. GAO also refers to trunking as if it magically appeared in the marketplace without giving credit for the numerous standards documents completed before multiple vendors could provide trunked P25 systems.

As a part of a response to the GAO report, P25 officials hope to help GAO and others understand the differences between completed “standards” and completed “interfaces,” which are supported by a series of standards. Unfortunately, it’s an oversimplification to interchangeably use the terms. Since P25’s inception in 1993, 125 P25 technical documents have been published. From 1993 to 2005, the GAO reported that no standards were completed and 114 P25 documents were published. Of those 114 technical documents, 51 are actual standards documents and the rest are supporting documents of interest primarily to product providers.

While P25 officials understand the difficulty the GAO and others may have in separating standards from interfaces, the distinction is important. In reality, the “eight interfaces” that the GAO and others like to reference are predicated on a series of standards, and until a substantial number of those related standards are complete, the interface isn’t complete. That problem is further aggravated because product manufacturers with long histories of building such products require less written standards and technology information before they begin to build and market products than users think are required to ensure that adequate conformance, performance, and interoperability testing can be performed later. The committee must determine which tests are necessary to ensure conformance, performance, and interoperability and which tests are necessary to assist manufacturers in product development, which are often the same. Our challenge during the past few years has been to narrow disagreements between the vendors and the individuals and agencies trying to confirm their conformance with the standards so the products actually being built and put into service in the field can match the ability to independently test them for compliance.

The table concentrates on compliance and compliance testing, which is being addressed both on the standards level and in efforts with NIST, but the GAO staff doesn’t recognize that those efforts are taking place within the framework of the overall P25 process. This enhanced cooperative work began in 2005 and continues. Prior to the initial ITS tests, there were few indications of potential problems, and the identified problems were immediately addressed in the standards process.

Furthermore, the GAO report fails to identify that twice before 2005 the P25 steering committee attempted to recruit interested parties, including Department of Defense (DoD) agencies or contractors, to perform conformance, performance, and interoperability testing. In each attempt, it was agreed that the process and efforts could not be funded without outside resources and a significant level of internal support. Where others chose not to join the process and offer their services, NIST, ITS, and Safecom stepped up to help.

The P25 standards process is an open, consensus process, and either their companies or agencies fund most participants. It is, therefore, difficult to understand how GAO can so cavalierly criticize the P25 efforts when the federal government has been a minority player in the entire process. The resources offered by the general government pale in comparison to those dedicated by TIA. Consequently, the majority of this work is performed by industry and federal, state, and local government participants from agencies that recognize the problem and are willing to contribute to its solution. The P25 steering committee has aggressively addressed all potential issues relating to our standards as soon as we become aware of them, including those brought forward by federal partners and Congress.

To our knowledge, none of the authors of the GAO report have observed the P25/TIA process, attended a meeting, talked with the P25 steering committee, or met with TIA officials.

For a detailed summary of the status of various P25 interfaces, click here.

MCC: The report notes the high cost of P25 equipment, citing radios can cost up to $5,000. Would you agree, and if so, what issues could help reduce costs?

Jorgensen: The P25 steering committee has no direct way of comparing per-unit product costs. The costs of the products are often dependent on the type of radio purchased, the features it contains, and the number of radios being purchased with each order, and whether prices are for just radios, other infrastructure, or supporting equipment such as backup power. Digital P25 radios have been created from the ground up to replace analog products in the marketplace for more than 60 years. For example, the GAO report Table 5 lists a small quantity of radios purchased by Albany County, N.Y., at $1,328 per radio, assuming the total cost was for just radios — otherwise there would be a lower per-unit radio cost. If you compare that per-radio cost to the initial Motorola DYNA-Tac, the first handheld, analog cell phone introduced in 1983 at a price of $3,995 in 1983 dollars, P25 radio prices are favorable by comparison. Consumers currently pay more than $1,000 for a high-definition TV when they could spend only $100 for an analog set, except the FCC established new rules that force a migration to a new technology platform.

There is no detail in Table 5 of the GAO report about what costs or equipment were included. P25 standards have been adopted by 23 equipment suppliers, building one or more pieces of P25-compliant products or systems that they are selling worldwide. While the figures used in the GAO report listed in Table 5 are perhaps a fair snapshot of federal funds spent by each of these communities, it would be difficult to draw any conclusions without more information. Clearly, reports of alleged high product costs are a concern, but I think more information is needed before drawing GAO’s conclusions.

MCC: The report states that some agencies with P25 systems haven’t seen interoperability benefits because they have replaced only part of their infrastructure and/or purchased fewer radios than needed because of high costs. Is that an accurate criticism?

Jorgensen: There are probably millions of 10- to 30-year-old analog radios in service that should have been replaced years ago using local funding. Much of this equipment will now need to be replaced to achieve the FCC’s unfunded mandate that requires public safety to move to narrowband channels. While public safety has perhaps begrudgingly accepted its responsibility to take this action to protect its finite spectrum, the GAO is adopting or recommending positions counterproductive to that effort. The public-safety community, and those involved in P25 specifically, carefully considered the options of staying with a narrowband analog FM platform or moving to a new digital platform in the initial stages of the P25 process. In the end, the decision was obvious for numerous reasons.

Many agencies the GAO report references are making partial progress in achieving their interoperability and narrowbanding goals because they have some federal funding to assist them. In this context, simply replacing the largest number of radios can take precedence over a vision to migrate an entire system for greater interoperability and enhanced operations. These agencies have chosen to invest in P25 technology now to replace obsolete equipment as they continue to migrate to a future digital platform. Therefore, they have ensured they will be able to talk to both existing analog FM systems and any future P25 infrastructure. They may not yet be using all the P25 features or capabilities dependent on a compliant infrastructure, but they are making wise investments in their future by not buying analog-only technology.

A number of users are migrating to P25 over time and within the limited funding available. P25 standards were created to provide for a graceful migration from analog FM radios to digital radios within a defined migration plan. It was a conscious decision to create standards for a transparent migration path from analog FM to a new digital narrowband standardized technology that didn’t leave the agencies we serve with stranded investment. As the GAO has noted in its report, one of the primary requirements in solving interoperability problems is comprehensive and advance planning. The attributes of our standards and the products developed from those standards are tools used as components of a comprehensive plan.

Current analog radios achieve their limited interoperability because they are all based on a common modulation scheme (standard) that evolved during the past 60 years via TIA-603 and other older standards. As the public-safety industry moves toward a digital technology, the existing level of interoperability can be maintained only through a new common modulation scheme that supports it. The vision of P25 has been to provide that common scheme. But the P25 team has endeavored to go far beyond the traditional level of interoperability in defining standards that will allow for interoperability among disparate radios systems — conventional vs. trunking, FDMA vs. TDMA, and different frequency bands.

MCC: Can P25 move forward more quickly?

Jorgensen: Of course, there are always ways to improve P25 and Safecom programs. The problem is a limited level of resources and unlimited expectations. Both Safecom and the federal government could do more, but some positive things have already taken place.

1. In 2005, Safecom agreed to support NIST in providing technical resources to create the NIST-approved compliance guidelines.

2. NIST/Safecom provided preliminary funding needed by other governmental agencies to create a conformance testing and laboratory recognition process for P25-compliant products.

3. Safecom and DHS provided substantial resources to ITS in Boulder, Colo., to assist in advanced vocoder testing in public-safety environments.

4. DHS has supported some committee testing and research activities with federal resources.

5. NIST/ITS and Safecom have been active participants in the TIA/P25 process.

6. DHS has supported the active participation in the P25 process of the Federal Partnership for Interoperable Communications (FPIC).

7. DHS has supported the participation of two DHS employees in the P25 process.

8. Safecom has worked to consider some concerns with regard to the Safecom program and its methods and process. In fact, a number of P25 steering committee members who participate in Safecom have been aggressively requesting Safecom to create a national system architecture or interoperability design umbrella, a challenge that has now been accepted.

9. Safecom and the federal government need to listen more to users and talk less.

10. Congress, the federal government, and political leaders must recognize we are replacing a technology and, in some cases, products that that have been used for more than 40 years. While some non-public-safety service providers claim they have silver bullets to solve our problems, none have fully addressed them or shown that they can.

11. It will take 30 – 40 years to replace all the obsolete hardware currently in place, at the current rate of federal funding. Unfortunately, the last major funding for new or upgraded public-safety systems was in the 1970s from the Law Enforcement Assistance Agency (LEAA). Yet in a world where local and national public-safety agencies face many problems, our political leadership struggles to commit the planning and funding resources to bring those systems in line with needs.

12. During the past several years, NIST and Safecom have helped organize and host a series of public-safety interoperability summits that bring together stakeholders to discuss the issues that need to be addressed.

13. Comprehensive interoperability is predicated on the following:

• Correctly defining that problem.

• Obtaining multiagency acceptance of that problem.

• Creating and maintaining a broad-based, cooperative management structure and process to ensure all users’ needs are understood and addressed.

• Creating an agreeable plan to resolve that problem.

• Finding adequate funding to correct those problems.

• Understanding that the planning process is an evolutionary process that requires annual updates and revisions based on changing needs, technologies, and services currently available or that may become available.

• Implementing standardized systems and technology to ensure full, transparent interoperability.

• Understanding problems will not be resolved until adequate funding becomes available on a cooperative basis among all parties.

Many of the above points are not new. They were discussed in the earlier National Institute of Justice (NIJ) report in 2003, “Why Can’t We Talk?” issued by the National Task Force on Interoperability (NTFI) found here.

Recognizing that solutions to this national issue can only be achieved through cooperation among all levels of government, 18 national associations representing state and local elected and appointed officials and public-safety officials formed a task force to address this issue. Compare what the NTFI says (Pages 54 – 55) about P25 standards with the GAO report. In spite of this support, we don’t always agree, but we always work toward a consensus, which can take longer.

MCC: GAO sent a draft of the report to DHS/Safecom before its release. Did P25 standards officials also receive a draft copy of the report?

Jorgensen: Don Pfohl is a co-chair of the P25 steering committee. GAO interviewed Pfohl, but only in his capacity as a state of Oregon representative. He wasn't interviewed on behalf of P25 issues. I wasn't interviewed, and neither of us received an advance copy of the GAO report. To my knowledge, no one in an official capacity from TIA was included in the interview process either.

MCC: Is there anything else you'd like to add?

Jorgensen: The P25 steering committee fully understands and supports the need for GAO to conduct performance audits and provide its interpretation of the facts. More importantly, the legislative intent passed by Congress with regard to P25 standards as both productive and important because it reflected the needs as Congress interpreted them on a national basis, which wasn't necessarily reflective of users' priorities. Knowing Congress' specific concerns allowed us to adjust our priorities to coincide with theirs and focus on the ISSI, console interface, and FSI, which weren't our highest priorities. Our highest priority was to ensure a high degree of interoperability among portables, mobiles, and fixed base stations that would normally be used for mission-critical services when the networks or systems of systems were placed out of service because of natural or man-caused disasters. While we have strong opinions regarding the work performed by GAO, the office has provided some beneficial insight into both real and potential problems, as well as a clear indication of GAO’s perception of what we are doing. In that regard, we recognize perception is in the eyes of the beholder, and we have a challenge to ensure GAO’s next report is more reflective of the true nature of the circumstances and that we work in tandem on the same objectives with the same understanding. P25 officials hope to benefit from the spirit and the intent of the report, while also highlighting some of its inaccuracies. However, the fact that many P25 standards have developed since 1993, the technology they enable, and the P25-compliant manufactured products used throughout the world as a leading public-safety technology for mission-critical systems must be factored into any future GAO studies.

 

---