By Heather Teale & Kris Gillen

Downtime preparedness is the item we all keep putting on the back burner … until it strikes.  You’re enjoying supper at home when you receive the page. Epic has gone down throughout your entire hospital.  You reflect upon your process for maintenance and limited end-user training and hope you’ve done enough to get through a downtime, short or extended.

Does this sound like your organization?  If so, you’re not alone.  Downtime preparedness is a difficult topic to garner buy-in and ongoing support.  It’s time-consuming and, in most organizations, there are many competing priorities.  In this two-part series, we will discuss the four key topics in maintaining a successful downtime preparedness strategy: planning, maintenance, preparation, and training.

Planning

Downtime preparation begins at the start of implementation and requires a regular review and update of processes.  An organization needs to define an overall strategy for downtime and determine the best-suited person for leading when Epic goes down.  We’ll begin by analyzing the different types of resources available during a downtime.

Most staff are familiar with Epic Shadow Read Only (SRO) environments.  This generally takes the form of a copy of the standard Epic Production environment in a, you guessed it, read-only format.  SRO is typically deployed during planned downtimes, but there’s no reason to take the icon away when Epic is running normally!  Time spent deploying the SRO environment to clinical workstations can take precious moments away from resolving an unplanned downtime scenario.  Our recommendation?  Leave the SRO environment available 24/7 for staff.

The next line of defense during a downtime is the BCA Web.  In the event Epic is not accessible via SRO, the BCA Web access brings the end user to the server location where downtime reports are housed by department on a scheduled basis.  BCA Web can be deployed to every workstation in a clinical environment.  Remember to enable LDAP configuration for the BCA Web with a daily download of user IDs and passwords.  After all, BCAs aren’t much good if the reports are stored, but no one has access to view them!

The third type of backup is the BCA downtime terminal, where reports are stored directly on the workstation and are updated on a regular basis – as often as every fifteen minutes for high volume units such as the emergency department.  Most organizations implement a minimum of one Business Continuity Access (BCA) downtime terminal per unit, often more if it’s a busy floor with critically ill patients.  In a worst-case scenario where communication with the servers has been completely cut off, the BCA downtime terminals may be the sole source of patient care information for clinicians.

The final fail-safe for downtime is a “master BCA” downtime terminal.  Unlike the BCA downtime terminals deployed on each unit, the master BCA can be housed in a secure location where IT staff routinely perform testing to ensure reports are up to date, access is verified on a regular basis, and test printouts can be generated routinely.  Downloading downtime reports for the entire system can be easily configured with the assistance of your Epic TS.  I’d like to say I’ve never had to use the master BCA before, but the truth is, it’s a great security blanket to have in place and I’ve relied upon it in the past for selected units.

Hopefully, you have most of the four downtime methods implemented at your organization.  What comes next is the hard part!  How do you verify your downtime workstations are up to date, downtime resources are in place, and end users know how to use the resources at their disposal?

Maintenance

With the 2017 release of Epic, monitoring tools were substantially enhanced.  Gone are the days of wondering if the downtime terminals are powered on and receiving reports.  With the setup of System Pulse Business Continuity reports, IT staff can now view if workstations are communicating with the server.  System pulse will validate how many reports are on the BCA downtime terminal, as well as the last time the workstation was accessed.  It does not, however, validate that reports are routinely updating.  When a terminal goes offline, or another pre-set aberration is experienced, alerts can be published to IT staff.

When issues are found, the creation of a decision matrix to simplify trouble-shooting will greatly enhance problem resolution for BCA devices.  For example, if a downtime terminal stopped communicating with the server, the help desk technician would deploy one of the desktop team members to check on the workstation.   Utilization of a standardized template with steps to take for investigation and subsequent escalation allows IT staff to understand their role in what can be a complicated and confusing process.

In addition to maintaining the workstations themselves, the downtime resources (forms, policies, instructions, printer cables) and corresponding downtime equipment can be challenging to manage.  Experience has shown varying approaches to downtime resource storage.  Some organizations choose to have a centralized storage location. An alternative can involve locating the downtime materials in a standardized location in each department.  Unfortunately, the downtime resources are often moved to new locations, are locked inside an office, are not restocked after a downtime, or supplies are removed or out of date. Assigning a department owner can mitigate this issue.

In addition to the maintenance of paper forms, special care and testing is necessary to print the downtime reports in an extended downtime.  Many facilities utilize printing through the server vs. local printing.  If your organization has recently replaced printers or you haven’t checked your printer setup in a while, it’s necessary to verify they are enabled to allow printing from a local connection.  If the printer has been relocated away from the downtime terminal, a cable of significant length may be needed to connect the workstation to the printer.  To perform a test in preparation for a downtime situation, users can remove wired or wireless connectivity from a workstation, then attempt to print reports from the BCA terminals.  Your organization will need to determine who is best suited to perform regular testing of downtime BCA terminals on the units.

Our recommendation is to assign a task for desktop services to test each BCA on a monthly or quarterly basis.

Lastly, train the unit coordinators or unit clerks to verify login functionality, equipment location, and validate reports are updating on a routine basis.  Unit coordinators are in an ideal position to help in a downtime.  Training them to be a key component of the system is a great fit!

One final suggestion to aid users in BCA downtime terminal identification: If your organization utilizes a specific group policy in the active directory domain for the BCA downtime workstations, using a red desktop background is a key element to aid IT staff in identifying the specific needs for workstation replacement and location.  When IT staff view workstations, the red background can be identified more quickly than a sticker on a monitor or a colored keyboard, particularly if the IT staff member isn’t present on site.

The solution?  Accountability through training and clinician ownership.

That’s all for Part I in our two-part series on Epic downtime preparedness. Stay tuned for Part II on Wednesday, January 30th. If you have any questions or would like to chat, feel free to send an email to Kris Gillen.