Tuesday, January 30, 2018

Full Text: State of Hawaii report on false missile alarm

Report can also be found here.


TO:                   Major General Arthur J. Logan
Director of Emergency Management (HI-EMA)

FROM:            Brigadier General [Ret] Bruce E. Oliveira
Investigating Officer

DATE:             January 29,2018


SUBJECT:              FALSE BALLISTIC MISSILE ALERT INVESTIGATION FOR JANUARY 13, 2018


ASSIGNMENT:


On January 15, 2018, Major General Arthur J. Logan, the Director of HI-EMA for the State of Hawaii, assigned me to investigate this incident to determine by a preponderance of the evidence the following:

a. What was the exact chain of events that led to the issuance of a state­ wide ballistic missile alert (BMA) message, initiated during the course of the HI­ EMA State Warning Point (SWP) change of shift at approximately 0805 hours on January 13, 2018?

b. What was the established protocol that HI-EMA SWP members were to follow during the change of shift BMA Checklist simulation and who established the BMA checklist?

c. How many times had HI-EMA SWP members previously practiced the BMA checklist that led to the issuance of the false alert message, and what, if any, training records exist?

d. Did the established BMA checklist during the simulated test include a response protocol in the event a false state-wide BMA message was issued, and if so, did HI-EMA SWP members follow BMA response protocol on January 13,
2018?


SYNOPSIS:


At approximately 8:06 am, the SWP mistakenly issued a BMA.  As the investigating officer and carefully considering the facts, I find a preponderance of evidence exists that insufficient management controls, poor computer software design, and human factors contributed to the real-world BMA and the delayed false BMA correction message that was issued over the WEA/EAS on January
13,2018.


DATE/TIME OF OCCURRENCE:


January 13, 2018


LOCATION OCCURRED:


The incident occurred at the State Warning Point of the Hawaii Emergency Management  Agency located within the State Emergency Operating Center in Diamond Head.



DEFINITIONS:



AAR BMA
cow
CEM
EAS EOC FEMA HAWAS HI-DoD HI-EMA HSEEP IPAWS PACOM PIO
SOP
STE SWP VTC WEA


After Action Review Ballistic Missile Alert Civil Danger Warning
Civil Emergency Message Emergency Alert System Emergency Operating Center
Federal Emergency Management Agency
Hawaii Warning System
Hawaii Department of Defense
Hawaii Emergency Management Agency
Homeland Security Exercise and Evaluation Program
Integrated Public Alert and Warning System
United States Pacific Command Public Information Officer Standard Operating Procedures Secure Telephone Equipment State Warning Point
Video Teleconference
Wireless Emergency Alert




TIMELINE OF EVENTS:



This is a chronological account of events that took place on the morning of Saturday, January 13, 2018 before, during, and after, the simulated SMA response drill.  The simulated SMA response drill is intended to verify that the SWP shift performed the required SMA response drill checklist tasks to standard using the closed loop alert over WEA/EAS.  Employee 1 erroneously initiated a real-world alert over the WEA/EAS.

0800 - In preparation for the SMA response drill, Employee 4 and Employee 2 discussed their plan to conduct a BMA drill at the end of their shift. Employee 4 prepared a recording of PACOM SMA notification script from the SMA Checklist manual.  Employee 4 discussed leaving the SWP area while Employee 2 told the dayshift about the previous shift change drill.

0803- The incoming day shift entered the SWP.  Employee  4 met with
Employee 5 and discussed the SMA drill.

0806 - Employee 4 initiated the SMA drill using a phone outside the SWP area calling into the SWP STE.  Employee 2 activated the STE speaker.  At this time, it was announced loud and clear, "EXERCISE, EXERCISE, EXERCISE," and then concluded with, "EXERCISE, EXERCISE, EXERCISE," which is a normal procedure for all drills including this BMA drill.  Both day and night shift began executing the SMA checklist.  Employee 2 recorded the time of the simulated ballistic missile impact.  Employee 3 started the Countdown timer. Employee  2 simulated activating the BMA wail tone siren and verbally indicated completion of that action.

0806 - Employee 1 logged into the AlertSense system and waited for Employee 3's announcement  of the simulated siren warning activation. Following the simulated siren announcement, Employee 1 erroneously activated the real-world Alert Code.

    There is a drop-down menu that includes:
o    Test missile alert (this sends message internally to agency)
o    Missile alert (this sends message to public)
    Missile alert was selected.
    The computer asks to confirm choice.
o    Employee 1 clicked yes.

0807 - SWP began receiving WEA/EAS message on their personal smartphones


0807 - Employee 13 called the SWP and received confirmation of the
"False Alert."

0808 - Employee 6 called the SWP and received confirmation of the
"False Alert."

0808 - The Adjutant General called the SWP and was notified of "False


Alert."


0809- Employee 6 notified Employee 11 of "False Alert" confirmation.

0809- Employee 4 used the HAWAS broadcast to the counties: "Attention


all stations- This is the State Warning Point- There is no ballistic missile threat to Hawaii- this is a drill - I repeat, this is a drill."  Employee 4 then conducted roll call to confirm each county's acknowledgement  of the False Alert.

0810- Honolulu Police Department called SWP and was notified of the
False Alert by HI-EMA.

0811 -State Siren Vendor called Telecom and was notified that it was a
False Alert.

0812- Employee 5 directed Employee 1 to send out the cancel message on AlertSense.  Employee 5 stated that Employee 1 just sat there and didn't respond. Employee 3 returned to the SWP from starting the time clock and saw Employee 4 and Employee 10 repeatedly get on the HAWAS letting Counties know that it was just a drill.  Employee 2 was on the phone with Employee 6 briefing them on the situation.  Employee 5 was alerting the command staff. Employee 1 was sitting and seemed confused.           Employee 3 took control of Employees 1's mouse and sent the cancel message.  At no point did Employee 1 assist in the process.

0813 - SWP issued a cancellation of the COW Message.  This prevented the initial event from being rebroadcasted to phones that may not have received it yet (i.e. If phone was not on at 0807, someone was out of range of cell coverage, passengers on a flight). But does not send out an all clear message.

0813- Employee 10 notified County Emergency Management  via
HAWAS.

0813- EMPLOYEE 6 posted "there is no threat to Hawaii" and that it is a

"False Alert" on personal Facebook page.

0814 - CDW message and "Cancel" received in SWP e-mail box.


0815 - SWP continued to receive calls from the public about EAS
message.

0815 - Employee 11 notified a KSSK engineer of the "False Alert."

0817- SWP received message from Employee 11 about WEA/EAS message.  Employee 11 sent a message to all HI-EMA Staff that there was no threat.

0818- HI DoD Notified Hawaii News Now.

0818 - HI-EMA Public relations Officer contacted Employee 6.  Employee
6 notified them that it was a "False Alert".  Suggested to start posting on social media.
0819- From sources within HI DoD, Hawaii News Now alerted the public that the SMA message was mistakenly sent. The following message, from the Hawaii News Now app, was pushed to mobile devices: "There is no current ballistic missile threat. The emergency alert warning has been sounded by mistake, according to Civil Defense."
0819- State Information Technology (IT) Contractor called Employee 6
and was notified of no threat.

0819- The HI-EMA's official Twitter account sent out a tweet that said there is "No missile threat to Hawaii." The message was posted at 0820 and is only distributed to those who follow @Hawaii_EMA on Twitter.

0820 - HI DoD notified KHON.

0822 - HI DoD notified KITV.

0822 - Employee 6 called Employee 11 and provided a status update and stated he will call County Emergency Management Agencies.

0823- HI-EMA posted to its official Facebook page. "No missile threat to
Hawaii. False Alert. We're currently investigating."

0824 - Employee 6 notified Hawaii County Civil Defense of False Alert.

0825 - Employee 6 notified Maui County Emergency Management of
False Alert.

0825- Olelo Television  called Employee 6 who notified Olelo that it was a
False Alert.

0826 - Employee 6 tried calling FEMA.  Did not get through.


0827 - Employee 6 called Employee 11 to discuss options.  Another COW
was not advisable. Decided on a CEM.

0830 - Employee 6 called FEMA for advice and FEMA agreed that criteria and guidelines of the event code to send a CEM were met.

0831 - Employee 6 informed Employee 11 that they were going to send the CEM.

0832 -State DAGS called Employee 6 and was notified of the False Alert.

0832 - Employee 6 remotely logged in to the WEAJEAS System via an encrypted connection, and a CEM message was drafted and sent.

0845 - CEM False Alert Message posted.  The following action was executed by the EAS:

1. EAS message over Local TV/Radio Audio Broadcast & Television Crawler Banner:  "False Alert. There is no missile threat to Hawaii."  "False Alert. There is no missile threat or danger to the State of Hawaii. Repeat. There is no missile threat or danger to the State of Hawaii. False Alert

2. WEA message sent to cell phones: "False Alert. There is no missile threat or danger to the State of Hawaii."


BMA CHECKLIST:


b. The HI-EMA established BMA checklist that SWP members followed during the change of shift test on 13 January 2018 was a 20 step SWP Ballistic Missile Quick Response Checklist.  Note: There is a sense of urgency to get the message out as quickly as possible.

INITIAL NOTIFICATION

Step 1. Receive notification message from PACOM over STE on speaker mode and record.

The exercise message is preceded and ended with "EXERCISE, EXERCISE, EXERCISE" during a practice drill.  The simulated PACOM message on 13 January 2018, was preceded and ended with "EXERCISE, EXERCISE, EXERCISE."  Employee 1 stated that he/she did not hear "EXERCISE, EXERCISE, EXERCISE," but claimed he/she heard "This is not a drill."


0 TO 5 MINUTE ACTION ITEMS

Step 2.  Sound attack warning siren for all counties from siren computer. [Simulated]

Step 3.  Activate Emergency Alert System [Actual]

Step 4.  Contact all counties over the HAWAS. [Simulated] Step 5.  Announce over the PAin EOC. [Simulated]
Step 6. Disconnect HECO power from the EOC. [Simulated] Step 7. Close and secure the EOC steel front doors. [Simulated]
AT THIS POINT THE SWP BMA DRILL ENDED

5 TO 10 MINUTE ACTION ITEMS Step 8.  Notify all leadership.
Step 9.  Contact FEMA

Step 10.  Turn on the VTC Event Brief in the Conference Room. Step 11. Activate the phone recording system.
Step 12.  Create an incident in WebEOC and enter in the event log. Step 13.  Verify all Checklist items have been completed. ONGOING ACTION ITEMS
Step 14. Standby to operate the STE and communicate with PACOM Step 15.  Monitor defensive and/or retaliatory response by the U.S. and its allies.


Step 16.  Do not respond to media.  [This is the Public Information Officer's (PIO) responsibility.]

Step 17.  Advise staff contacting SWP to shelter-in-place. DEACTIVATION  ACTION ITEMS
Step 18.  PACOM determines missile will not impact Hawaii

Step 19.  If missile impact time expires and PACOM has not updated the SWP.


Step 20.  Update leadership of any PACOM updates.




FINDINGS:



Based on the documents reviewed, information collected and interviews conducted during the investigation I find the following:

a.          SWP has the BMA checklist but does not have SMA procedures included into the SOP although it's a new initiative.

b.          The SWP successfully practiced the SMA Missile Checklist 26 times as determined by Telecom Branch records of SWP reporting on AlertSense.

c.          Notification for Step 1 of the SMA checklist is used for actual ballistic missile attack using phrases "Real world" and "This is not a drill."

d.          The established SMA checklist did not include a response protocol in the event of a false-ballistic missile message.  There was a cancel message which stopped further ballistic missile messages from displaying, but does not send a new message to the public.  Eventually a CEM was developed and sent. The established protocol has since been updated to include a False Alert message.

e.          The Preparedness Branch previously identified the need to refine and reformat the SWP BMA checklist to include a "Deactivation" section or "All Clear" EAS message.  Had the protocol been developed within the last 2 months, the delay in sending out an official notification over WEA/EAS that the alert was false would not have happened on January 13, 2018

f.           The Preparedness Branch worked with the Operations Branch to conduct three no-notice exercises of the BMA checklist during the month of December.  The design of these exercises was in compliance with Homeland Security Exercise and Evaluation Program (HSEEP) and included a written Exercise Plan and Exercise Evaluation Guide.  A formal AAR was conducted, but there was nothing in the AAR that a deactivation section in the checklist should
be further developed.

g.          All SWP shifts did not participate in the HSEEP evaluation and were conducting their own drills separately from the exercise program, without exercise control protocols.

h.          Many questions on policies and procedures regarding the SMA Checklist have previously been asked by SWP personnel and HI-EMA personnel outside of the SWP without answers resulting in confusion and frustration.



iHI-EMA has suspended all SWP BMA drills until investigation is complete.



j.           BMA Checklist is not detailed which allows shifts to interpret and differ in protocol significantly when executing the drill. Furthermore, the drill was conducted during a change of shift which added to confusion of who is in charge and which shift is executing checklist actions.     HI-EMA operations provided the following guidance to the SWP in December:

1. Each shift will review and discuss the BMA Checklist at the beginning of each shift to ensure there are no questions or concerns as to what the checklist is requiring.

2.  Each shift will at a minimum of twice a week take time to run the checklist with different SWP staff members.

3.  Shifts should look at differences in executing the checklist with
2, 3, 4 personnel and make recommendations to operations.

4.  SWP shifts will create an incident at least three times during the
week.

k.          The AlertSense program replaced an older program in mid- December 2017.  No technical training was provided to the SWP, only basic application.   Many SWP personnel felt this training was inadequate.

I.           All SWP employees with access to AlertSense are required to take, and did complete, FEMA course IS 247a prior to accessing AlertSense. This course provides basic information on the IPAWS. The goal of this course is to provide authorized public safety officials with: increased awareness of the benefits of using IPAWS for effective public warnings; skills to draft more appropriate, effective, and accessible warning messages; and best practices in the effective use of common alerting protocol to reach all members of their communities. There are no personnel training records which make it difficult to determine what training each SWP member has taken.  Employee 7 stated that Employee 1 completed the course.

m.        The Operations Branch does not have comprehensive documentation of training that can be used to determine exactly how many times the HI-EMA SWP members practiced the established protocol or which shift practiced.  In addition, there are documented inconsistencies  of how each BMA protocol was practiced prior to the issuance of the warning message on 13
January 2018.


n.          HI-EMA mistakenly believed that it had to consult with FEMA to issue an event code which contributed to the 38 minutes to issue an official notification that the threat was false. HI-EMA has corrected this concern and understands that it is the HI-EMA's authority to issue an official notification.

o.          SWP Supervisors do not consistently ensure employee performance levels meet expectations.

p.          Poor performance evaluations have been documented on select
SWP individuals.  This has created morale and competency issues.

q.          Employee 1 has been a source of concern for the same SWP staff for over 10 years.  Employee 1's poor performance has been counseled  and documented and the SWP members have stated that they are "not comfortable with Employee 1 as a supervisor, two-man team, or as a part of the SWP in general.  He is does not take initiative and has to be directed before he takes action.  He is unable to comprehend the situation at hand and has confused real life events and drills on at least two separate occasions."

r.           Employee 1 stated that he/she initiated the real-world COW. Employee 1 stated he/she did not hear the announcement, "EXERCISE, EXERCISE, EXERCISE" at the beginning and end of the simulated message. Employee 1 also stated, that he/she did not recognize the callers voice, and heard, "This is not a drill" over the speaker.  However, all other SWP members clearly heard, "EXERCISE, EXERCISE, EXERCISE" at the beginning and end of the drill.

s.           Employee 5 directed Employee 1 to send out the cancel message. Employee 5 stated that Employee 1 just sat there and didn't respond. Employee
3 returned to the SWP from starting the time clock and saw Employee 4 and Employee 10 repeatedly get on the HAW AS letting the Counties know that it was just a drill.  Employee 2 was on the phone with Employee 6 briefing them on the situation.  Employee 5 was alerting the command staff.  Employee 1 was sitting and seemed confused.   Employee 3 took control of Employee 1's mouse and sent the cancel message.  At no point did Employee 1 assist in the process of correcting the False Alert.

t.           Employee 1 has been terminated.

u.          HI-EMA has offered Mental Health counseling to SWP members to help recover from this traumatic event.



RECOMMENDATIONS:


I recommend the following actions:

a.          Install a computer process to rapidly issue alert cancellations on the
WEAIEAS System (completed January 13, 2018).

b.          Add an "ALL Clear" selection to the AlertSense drop down menu in Step 2 of the BMA Checklist and develop deactivation procedures  in the BMA checklist (completed 1/24/18).

c.        Add into the confirmation message what action is confirmed, i.e. "Are you sure you want to send a "Real World Ballistic Missile Alert."

d.         Add a function to confirm the specific selections on the drop-down menu with the supervisor's password required to enforce the two-person confirmation procedure.

e.          Revise Step 1 Notification of the BMA checklist for a practice drill eliminating the phrases used for actual ballistic missile attack "Real world" and "This is not a drill."  Replace with "this is a drill."  Continue the established
practice of preceding drill messages  and ending drill massages  with "EXERCISE, EXERCISE, EXERCISE."

f.           Revise BMA Checklist to detail actions to standardize best practices and eliminate confusion.  Include in the revision; repeat all directives and actions aloud and use a two-person rule to verify directives and actions.

g.          Eliminate conducting practice drills during change of shift to clarify command  and control.

h.          Establish a formal review/update  schedule to assess current SWP guidance, SOPs, documents and protocols.   Identify opportunities for enhancement.   Evaluate whether they are current, user friendly and consistent in design and content.

i.           Establish a SWP knowledge library where current protocols and checklist are housed and easily accessible to ensure all staff are working off the same versions of procedures.

j.           Add Risk Management into all events to develop controls and make risk decisions. As control measures are developed,  risks are re-evaluated until
the residual risk is at a level where the benefits outweigh the cost. Ensure that the appropriate decision authority then makes the decision.


k.          Ensure future SWP planning efforts include input from planners in the agency's Preparedness Branch.

I.           Review other state warning programs and protocols to determine best practices.

m.         Ensure future planning efforts are sufficiently completed and SWP
personnel are trained before initiating procedural changes

n.         Document and maintain training records. With any training program, there are associated training records. Records document who was trained, when they were trained, and what skills they have mastered. Training records provide documentation for agencies, information for personnel evaluations, and support for promotion or salary increases.

o.          Provide AlertSense technical training to SWP personnel. Revise
SOP to include BMA procedures

p.         Include simulation training and a practice simulation computer for drills. The mistakes participants make in a simulation are learning experiences, the outcome is not damaging to the organization, its equipment, or its personnel. If the SWP participates in simulations during a practice drill that are modeled after real-world issues, it can be free to make mistakes, learn from them, and apply the learning before going out to the real world.

q.          Conduct a needs assessment to determine knowledge, skills, and abilities that the SWP staff should possess and identify training, protocols or policies to address gaps.

r.           Fill the four SWP vacant positions to remove the added burden on SWP personnel.                         Dealing with current SWP staffing challenges and preparing for the challenges that lie ahead requires a two-pronged approach that focuses on:
1) attracting, hiring and retaining qualified SWP personnel, and 2) scheduling existing personnel to maximize efficiency.

s.        Evaluate whether the duties of SWP staff could be expanded to support other HI-EMA initiatives and keep staff engaged without jeopardizing their core mission.
t.        Ensure all training to SWP is formalized with specific learning goals, objectives, and performance metrics.


u.          Conduct a command climate survey.to  all HI-EMA members to determine morale and communication within the division.

v.          HI-EMA leadership  should be properly trained to ensure employee performance meets or exceeds expected levels.

w.         Based on my findings the Director of HI-EMA should take appropriate disciplinary  action regarding involved employees.





-···
:;;:- """--------
.· :.e-:-:--c:_....                   -       - - -_c:::::-  )         '----
Brigadier General [Ret] Bruce Oliveira
Investigating  Officer