 |
Revisiting cost-versus-benefit analyses for continuous monitoring modalities (wired or wireless networks and standalone monitoring instruments) that facilitate the ability to comply with auditors' requests for proof of regulatory compliance is very timely. Moreover, the ever-increasing costs for APIs and the R&D efforts to create them are such that the economic costs of failure in the totality of monitoring systems are greater than ever before. All monitoring methods — whether wired, wireless or standalone instrumentation — need to be scrutinized for systemic weaknesses that allow human error to compromise product quality, system failure probabilities and overall costs of ownership.
This article discusses six approaches to monitoring critical environments such as pharmaceutical freezers, stability rooms and warehouses. Quality, facility and IT managers employ different methods for maintaining the quality of products and information, and the article will evaluate different monitoring methods and present the risks and costs of ownership for each.
Introduction: Modalities for Monitoring Critical Environments
There are many competing monitoring technologies and brands of systems purporting to provide for regulatory compliance. For purposes of this paper we will examine six modalities for temperature and humidity monitoring: 1) wired systems with UPS power backups; 2) wired systems with UPS power backups and use of PoE (Power over Ethernet); 3) wireless WiFi; 4) wireless mesh; 5) non-networked/standalone data loggers; and 6) chart recorders.
Briefly, chart recorders are the oldest technology — paper-based, powered either by AC or batteries. Standalone non-networked data loggers also use either AC or batteries, and they require manual downloading of data at regular intervals. Wired networking technology has been around for decades, and while this technology continues to evolve and remains the mainstay of most pharmaceutical operations, wireless has fast become an interesting alternative. Each method of communicating data has its advantages and disadvantages. When it comes to regulatory-compliant applications involving public health, however, the criteria for using one method over the other should be well understood. The following two charts provide an overview of risk factors and cost-of-ownership differences between the continuous monitoring modalities.
Figure 1. Risk Factors — Continuous Monitoring Modalities. This chart provides general guidelines only to risks associated with meeting GMP requirements. Source: Veriteq, a Vaisala Company. Click here for hi-res version (opens new tab/window).
Figure 2. Costs of Ownership Factors — Continuous Monitoring Modalities. The following chart provides general guidelines only for some of the more salient factors affecting costs of ownership related to the six monitoring options. Varying plant sizes and scale of operations affect the impacts of various cost factors. Source: Veriteq, a Vaisala Company. Click here for hi-res version (opens new tab/window).
|
Paper-Based Chart Recorders
In the last decade, nearly every leading pharmaceutical company that had relied on paper chart recorders has either replaced them with one network-based system or another, or is in the thinking stages of doing so. Chart recorders that cost as little as a few hundred dollars can still be found in the marketplace.
Most pharmaceutical quality managers consider this obsolete technology due to both the considerable costs of maintaining chart recorder-based monitoring systems, and the obvious risks of handling paper-based records and limited or no alarm notification. It does not go unnoticed that chart recorders rely on humans for daily or weekly checks to replace paper, check pens and write deviation reports. In any event, costly staff hours must be devoted to tracking which charts need to be changed when, and which batteries need to be changed, and in what intervals. AC-power based chart recorders without batteries offer no ability for continuous data records in the event of power outages.
The possibilities for human error are multifold. At the time of this writing there have been no known instances of regulators rejecting chart recorder based monitoring systems. However, regulatory agencies encourage the move away from manually intensive processes to more automation with the purpose of tightening up quality systems and making better use of quality resources.
Data Loggers
Unlike paper chart recorders, standalone data loggers are not as likely to break. They do, however, incur considerable labor costs for manually downloading data, especially in large plants where hundreds of data loggers are required to ensure environmental standards in both processing and storage areas.
These costs are magnified in the current environment where inspection-readiness can be an issue. (Remember the new 15-day period that the FDA requires for responding to observational deficiencies.) Operational costs for complying with regulators' requests for information and the interference on normal operations that audits can involve can be considerable. As with chart recorders, the capability to access accurate and complete records throughout the record retention period as required by FDA 21 CFR Part 11 and EU GMP Annex 11 may be compromised if it takes too long to the locate records or they are incomplete.
There are also multiple potential sources of human-error with standalone data logger-based systems. First, staff may neglect to download data before the storage capacity of the instrument is exceeded. Secondly, battery-powered standalone data logger systems also require ongoing monitoring of batteries, which creates an opening for lost data even when so-called battery alerts are in place in the event that no one was present to see the alert. Third, AC-powered data loggers without batteries may not provide gap-free records in the event of power outages.
If one thinks of technology investments as ways to automate routine tasks to eliminate the costs of labor and potential error any time there is human intervention, standalone data loggers do not generally pass muster. The reliance on human labor to download data, investigate deviations because they were not seen in time and maintain these files opens the door to regulatory objections and the staggering costs that ensue with production delays. With the exception of monitoring the contents of a few chambers, standalone recorders put undue risk on companies over other monitoring methods that automate more procedures and reduce reliance on human systems.
Wired Networks — With and Without PoE Capabilities
The pharmaceutical industry, like many others, has long relied on a wired infrastructure using Ethernet standards for making the connection to transmit and receive data. A hard-wired network allows communications to proceed securely and continuously with few possibilities to intercept or interrupt the flow of data.
Uninterruptable power supplies (UPS) ensure that servers are always available for data exchange. However, a potential problem with data continuity in monitoring controlled environments arises in the event of a power outage at the facility. The UPS maintains network uptime, but devices connected to the network may be without power, which could mean loss of critical data. Until recently, traditional wired networks lacked a cost-effective alternative to maintain data flow with these critical devices.
Power over Ethernet (PoE), originally implemented for voice-over-Internet (VoIP) technology, allows electrical power and data to travel on the same Ethernet cable. Since 2003, companies have been integrating data and power standards on the manufacturing floor with PoE (IEEE 802.af)-capable devices.
The advantages to deploying a PoE network are many:
- Saves the cost of running additional AC power, which usually requires a licensed electrician, aided by the low cost of network switches with built-in PoE power capability;
- Provides greater flexibility to locate devices around the plant because they can be installed wherever a LAN cable can be run;
- Increases data communication protection from power outage because the server's UPS provides backup to PoE connected devices;
- Uses less energy and can be managed from a central location; and most importantly,
- Protects critical data through the outage period.
With PoE, security, maintenance and access can all be managed within an existing IT framework because staff is trained on setting up and maintaining communications networks based on worldwide standards.
|
For many pharmaceutical plants, and especially those in older facilities where there are difficulties in running Ethernet cabling, wireless communications can be a convenient and cost effective method of connectivity. Ease of installation, reduction in cabling cost, and the ability to obtain measurements in otherwise inaccessible areas are among the major factors driving the adoption of wireless networking.
Unlike the wired 802.3 international standard, several wireless communications protocols have emerged, including the popular wireless version of Ethernet, commonly referred to as WiFi (802.11b and, more recently, 802.11g). Other network methodologies used for monitoring include a mesh structure based on the Zigbee (802.15.4) protocol.
WiFi is often the wireless system of choice because it uses the same IT infrastructure already in place in an organization. Wireless mesh (Zigbee) is a network architecture that uses access points or nodes to communicate with one another as well as with the host. It is designed to detect a degraded signal at one access point and reroute it to another nearby access point. Nodes have a low power requirement, which has the expectation of less drain on battery life, but at the same time low power inherently means less signal strength than WiFi. The low power requirement of Zigbee networks also means that there needs to be a sufficient number of nodes to maintain continuous data flow.
Whether WiFi or wireless mesh, the greatest potential downsides are the possibilities for network interruptions, such as those that occur when lift trucks move throughout a plant or when inventory is rearranged, equipment is moved out of range or other potential obstacles to transmission; and in turn the ability to ensure gap-free records. Fixed obstacles that could block the signal can be overcome using a sufficient number of wireless access devices. Intrusions from a fork lift or storage equipment, such as water-based gel packs or office modifications, may not be so well anticipated.
Figure 3. Topology of WiFi and Mesh Wireless Networks. WiFi devices connect directly to the company network and use WiFi access points to transmit data to a central host (server). Mesh devices connect to a gateway that can either host the data or forward to a central server. Source: Veriteq, a Vaisala Company.
The range for a wireless device is largely dependent on radio strength, which is also tied to the battery power. Installations using wireless monitoring technology have to accommodate signal range and barriers, which become important factors when continuous data are required. It can mean, for example, that more wireless devices are needed (and greater upfront cost) to ensure network transmission integrity in all situations.
With wireless mesh networks, signals are diverted to maintain data flow, but this increases the load on other nodes picking up the signal, having implications for reduced battery life in unpredictable ways. These systems need a vigilant source for detecting and alerting for low-battery issues well before data are lost.
With wireless systems, signals carrying critical data can also degrade from interfering sources such as other devices communicating in the same 2.4 GHz band (WiFi and Zigbee), and in today's pharmaceutical plants this especially includes security cameras, microwave ovens and Bluetooth devices. Wireless mesh also is a proprietary network that needs to be integrated with the different standards of the existing infrastructure, involving IT hours, which potentially impacts the costs of ownership.
The batteries used in wireless devices are specified with "up to" so many months or years of life. The "up to" condition is often stated for ideal or laboratory conditions. This is because there are no typical operating conditions where power consumption can be calculated. Devices draw down battery power with each transmission, which include measurement updates at a frequency established by the user, or events such as alarms or communication problems brought on by various circumstances, such as a blocked access device.
Battery drain is even less predictable in a mesh infrastructure. For example, when the signal between a temperature device and a node is blocked, another nearby node picks up the signal for transmission. It now adds the new transmissions to those from other measurement devices with which it was already communicating. The extent to which a wireless network requires battery replacements reintroduces human error potentials into what are assumed to be relatively error-proof automated systems.
Most connectivity methods have low risk of losing data when the time between real-time updates for data and alarms is long. For example, the initial requirements of a monitoring system may not have called for frequent data updates, but at some point someone may want to know if a chamber door was left open or to detect other behavior that led up to a temperature excursion. In these instances, a faster sample rate would be needed, requiring more transmissions and thus making more demands on the battery.
Reduced battery life is well and good if standard operating procedures can anticipate the need, staff has the time for required maintenance without fail, and the expense of more frequent service (labor hours, replacements) are not burdensome. Some devices provide low-battery indicators and alarms. However, the question is: What happens to data if batteries are not replaced in time? Moreover, it is highly probable that data will be lost in systems that use the same batteries to power both the wireless radio and data memory electronics. Battery life therefore is not a minor specification but in fact has great impact on the ability to ensure compliant gap-free records and minimizing impacts of human error on quality assurance systems.
|
Whether deploying a system of standalone devices or a wired or wireless network for monitoring critical environments, the need to have a continuous record of data and events is the same. There will be times when the facility experiences network and power interruptions, among other unexpected disruptions. Assuming the need for a continuous record of quality, the monitoring system should be capable of filling in a database when temperature, relative humidity, pressure and other data cannot be communicated in real time. This is nearly impossible and impractical with standalone monitoring instruments, making them obsolete technology.
In networked systems, recording data independently at the point of measurement is one key factor in protecting data. A system capable of identifying the time period of a communication interruption and bringing in data and events to fill the gap is essential to ensuring complete and accessible records. This assumes measurement devices have calibrated time-base clocks (specified with an accuracy over a temperature range) to ensure correct time/data recording. The capability to backfill data after power or network interruptions ensures a continuous record of data and events. Completeness of records also assumes there is an audit trail to capture all system events. Documented evidence of data and events during an outage will reduce quality management's involvement in having to review excursions and investigate deviations. Gap-free records save time during and after an audit, reduce unnecessary staff involvement and limit disruptions to ongoing production and shipments.
Data Security
There are two aspects of security that regulatory compliance (21 CFR Part 11) requires: protecting data from unauthorized access and preventing alteration to data. Secure data begin at the measurement device and end at a designated collection point, usually a network server. Secure access refers to specific levels of permission given to authorized users and other protocols for ensuring authenticity.
Devices communicate using protocols or common rules for data format and can either be open (public) or proprietary. An open protocol means just that: anyone who knows the rules can potentially access the files. A secure monitoring system ensures that the measurement device has a secure protocol in addition to other authentication and confidentiality features. This is a major factor in how and why communicating over wire is inherently more secure. In wired systems, devices are only accessible within the building. A wired network can potentially be compromised only by someone who has already gained access to the facility.
On the other hand, wireless communications are inherently less secure and require additional measures to maintain protections. Wireless devices also are manufactured with security features built in but may not be upgradable to the changing standards or security requirements of an organization. A capital investment made now may need replacement in two years time. Non-standard proprietary wireless networks require IT training and increase risk due to IT staff turnovers.
Conclusion: Quantify Risk and Cost
Whether you use standalone monitoring instruments or wired or wireless connectivity for your temperature and humidity measurement devices, it is important to understand the limitations of each methodology. For the most part, the significant labor costs involved in standalone monitoring devices, combined with the multiple ways in which such systems insert human error potential, make them less than desirable compared to more automated network-based monitoring technology.
Wireless communication has the advantages of being flexible to install and providing for monitoring of environments that either have limited access to running cable or where refrigerators, freezers or other monitored storage units are moved on a frequent basis. Wired networks have the advantage of speed, security and data redundancy. Generally speaking, if your goal is to reduce the risk of data loss to zero or nearly so, then wired systems are the best course. The lower upfront cost of wireless can disappear quickly if you have to write deviation reports from missing data, experience product loss or regulatory missteps. The good news is that connectivity technologies can be mixed — wired and wireless — solving the physical installation challenges that many facilities pose.
Of course, network connectivity methods alone do not maintain product quality. They have to be used in conjunction with the capabilities of the facility's monitoring system, with specific attention to automating records (data, events, audit trail), to ensure a continuous, gap-free history.
The details of how humans interact with systems — whether driving lift trucks, replacing batteries, downloading data, etc. — are important factors in determining if a particular continuous monitoring technology truly minimizes risks introduced by the inevitability of human error. You have to weigh the costs of potential problems versus how much your organization is willing to invest to protect your operations.
Risk really comes down to consequences or the implications of system failure. Analysis of risk is both qualitative and quantitative. Ideally, you should be able to answer the following questions:
- Can you afford to handle the expense of downtime? This can be in the form of missing data, equipment failure or human error.
- For an unplanned production stoppage, how long will it impact other research, production or shipments?
- What is the financial impact of losing product or research specimens?
- Can you define downtime cost either by time, lost production, blemish to reputation or other stakeholder pain?
- Can you identify single points of failure and ways to reduce them?
- Does it cost more to recover from equipment failure, product loss, internal and external reviews and other unplanned interruptions than it would cost to invest in continuous operation?
Sources:
- http://standards.ieee.org
- http://www.veriteq.com/chart-recorders
- Part 11 US CFRs, EU GMPs, ICH Guidelines
- http://en.wikipedia.org/wiki/Power_over_Ethernet
- http://en.wikipedia.org/wiki/Wireless_security
