TO: Major General Arthur J. Logan
Director of Emergency
Management (HI-EMA)
FROM: Brigadier General [Ret] Bruce E. Oliveira
Investigating Officer
DATE: January 29,2018
SUBJECT: FALSE BALLISTIC
MISSILE ALERT INVESTIGATION FOR JANUARY 13, 2018
ASSIGNMENT:
On January 15, 2018, Major General Arthur J. Logan, the Director of HI-EMA for the State of Hawaii, assigned
me to investigate this incident
to determine by a
preponderance of the evidence the following:
a. What was the exact chain of events
that led to the issuance
of a state wide ballistic
missile alert (BMA) message, initiated during the course of the HI EMA State Warning Point (SWP) change of shift at approximately 0805 hours on January 13, 2018?
b. What was the established protocol that HI-EMA SWP members
were to follow during the change of shift BMA Checklist simulation and who established
the BMA checklist?
c. How many times had HI-EMA SWP
members previously practiced
the BMA checklist that led to the issuance
of the false alert message,
and what, if any, training records exist?
d. Did the established BMA checklist
during the simulated
test include a response protocol in the event a false state-wide BMA message
was issued, and if so, did HI-EMA SWP members follow BMA response protocol
on January 13,
2018?
SYNOPSIS:
At approximately 8:06 am, the SWP mistakenly issued a BMA. As the investigating officer and carefully
considering the facts, I find a preponderance of evidence exists that insufficient management controls, poor computer software
design, and human factors contributed to the real-world BMA and the delayed false BMA correction message that was issued over the WEA/EAS
on January
13,2018.
DATE/TIME OF OCCURRENCE:
January 13, 2018
LOCATION OCCURRED:
The incident occurred
at the State Warning Point of the Hawaii
Emergency Management Agency located within the State Emergency Operating Center in Diamond Head.
DEFINITIONS:
AAR BMA
cow
CEM
EAS EOC FEMA HAWAS HI-DoD HI-EMA HSEEP IPAWS PACOM PIO
SOP
STE SWP VTC WEA
After Action Review Ballistic Missile Alert Civil
Danger Warning
Civil Emergency Message Emergency Alert System Emergency Operating Center
Federal Emergency Management Agency
Hawaii Warning System
Hawaii Department of Defense
Hawaii Emergency Management Agency
Homeland Security Exercise and Evaluation
Program
Integrated Public Alert and Warning System
United States Pacific Command Public
Information Officer Standard Operating Procedures Secure Telephone Equipment State Warning Point
Video Teleconference
Wireless Emergency Alert
TIMELINE OF EVENTS:
This is a chronological account of events that took place on the morning
of Saturday, January
13, 2018 before, during,
and after, the simulated
SMA response drill. The simulated SMA response drill is intended to verify that the SWP shift performed the required SMA response drill checklist tasks to standard using the closed loop alert over WEA/EAS.
Employee 1 erroneously initiated a real-world alert over the WEA/EAS.
0800 - In preparation for the SMA response drill,
Employee 4 and Employee 2 discussed
their plan to conduct a BMA drill at the end of their shift. Employee 4 prepared
a recording of PACOM SMA notification script
from the SMA Checklist manual. Employee 4 discussed leaving the SWP area while Employee 2 told the dayshift about the previous
shift change drill.
0803- The incoming
day shift entered the SWP.
Employee 4 met with
Employee 5 and discussed
the SMA drill.
0806 - Employee 4 initiated the SMA drill using a phone outside the SWP area calling into the SWP STE. Employee 2
activated the STE speaker. At this time,
it was announced loud and clear, "EXERCISE, EXERCISE, EXERCISE," and then concluded with, "EXERCISE, EXERCISE,
EXERCISE," which is a normal procedure
for all drills including
this BMA drill. Both day and night shift began executing
the SMA checklist. Employee 2 recorded
the time of the simulated ballistic missile impact. Employee 3 started the Countdown timer. Employee 2 simulated activating the BMA wail tone siren and verbally indicated
completion of that action.
0806 - Employee 1 logged into the AlertSense system and waited for Employee 3's announcement of the simulated siren warning activation. Following the simulated siren announcement, Employee 1 erroneously activated the real-world Alert Code.
• There is a drop-down menu that includes:
o Test missile alert (this sends message internally to agency)
o Missile alert (this sends message
to public)
• Missile alert was selected.
• The computer
asks to confirm choice.
o Employee 1 clicked
yes.
0807 - SWP began receiving WEA/EAS message on their personal
smartphones
0807 - Employee 13 called the SWP and received confirmation of the
"False Alert."
0808 - Employee 6 called the SWP and received confirmation of the
"False Alert."
0808 - The Adjutant General called the SWP and was notified
of "False
Alert."
0809- Employee 6 notified Employee
11 of "False Alert" confirmation.
0809- Employee 4 used the HAWAS broadcast to the counties:
"Attention
all stations- This is the State Warning Point- There is no ballistic
missile threat to Hawaii- this is a drill - I repeat, this is a drill." Employee 4 then conducted roll call to confirm each county's acknowledgement of the False Alert.
0810-
Honolulu Police Department called SWP and was notified
of the
False Alert by HI-EMA.
0811 -State Siren Vendor called Telecom and was notified
that it was a
False Alert.
0812- Employee 5 directed Employee 1 to send out the cancel message on AlertSense. Employee 5 stated that Employee 1 just sat there and didn't respond. Employee 3 returned
to the SWP from starting the time clock and saw Employee 4 and Employee 10 repeatedly get on the HAWAS letting Counties know that it was just a drill. Employee 2 was on the phone with Employee
6 briefing them on the situation. Employee 5 was alerting the command staff. Employee 1 was sitting and seemed confused. Employee 3 took control of Employees 1's mouse and sent the cancel message. At no point did Employee 1 assist in the process.
0813 - SWP issued a cancellation of the COW Message. This prevented the initial event from being rebroadcasted to phones that may not have received
it yet (i.e. If phone was not on at 0807, someone was out of range of cell coverage, passengers on a flight). But does not send out an all clear message.
0813- Employee 10 notified County Emergency Management via
HAWAS.
0813-
EMPLOYEE 6 posted "there is no threat to Hawaii" and that it is a
"False Alert" on personal Facebook page.
0814 -
CDW message and "Cancel" received in SWP e-mail box.
0815 - SWP continued to receive
calls from the public about EAS
message.
0815 - Employee 11 notified a KSSK engineer
of the "False Alert."
0817- SWP received message from Employee
11 about WEA/EAS message.
Employee 11 sent a message
to all HI-EMA Staff that there was no threat.
0818- HI DoD Notified Hawaii
News Now.
0818 - HI-EMA Public relations Officer contacted Employee 6. Employee
6
notified them that it was a "False Alert". Suggested to start posting on social media.
0819- From sources within
HI DoD, Hawaii News Now alerted the public that the SMA message
was mistakenly sent. The following message, from the Hawaii
News Now app,
was pushed to mobile devices:
"There is no current ballistic missile threat. The emergency alert warning has been sounded by mistake, according to Civil Defense."
0819- State Information Technology (IT) Contractor called
Employee 6
and was notified of no threat.
0819- The HI-EMA's official Twitter account sent out a tweet that said there is "No missile threat to Hawaii." The message was posted at 0820 and is only distributed to those who follow @Hawaii_EMA on Twitter.
0820 - HI DoD notified KHON.
0822 - HI DoD notified KITV.
0822 -
Employee 6 called Employee 11 and provided a status update
and stated he will call County Emergency Management Agencies.
0823- HI-EMA posted to its official Facebook page. "No missile threat to
Hawaii. False Alert. We're currently investigating."
0824 - Employee 6 notified Hawaii County Civil Defense of False Alert.
0825 - Employee 6 notified Maui County Emergency Management of
False Alert.
0825- Olelo Television called Employee 6 who notified Olelo that it was a
False Alert.
0826 - Employee 6 tried calling FEMA. Did not get through.
0827 - Employee 6
called Employee 11 to discuss options.
Another COW
was not advisable. Decided on a CEM.
0830 - Employee 6
called FEMA for advice and FEMA agreed that criteria and guidelines of the event code to send a CEM were met.
0831 - Employee 6 informed Employee 11 that they were going to send the CEM.
0832 -State DAGS called Employee
6
and was notified of the False Alert.
0832 - Employee 6 remotely logged in to the WEAJEAS System via an encrypted connection, and a CEM message was drafted and sent.
0845 - CEM False Alert Message posted. The following action was executed by the EAS:
1. EAS message over Local TV/Radio Audio Broadcast
& Television Crawler Banner: "False Alert. There is no missile threat to Hawaii." "False Alert. There is no missile threat or danger to the State of Hawaii. Repeat. There is no missile threat or danger to the State of Hawaii. False Alert
2. WEA message sent to cell phones: "False Alert. There
is no missile threat or danger to the State of Hawaii."
BMA CHECKLIST:
b. The HI-EMA established
BMA checklist that SWP members followed during the change of shift test on 13 January 2018 was a 20 step SWP Ballistic Missile Quick Response Checklist.
Note: There is a sense of urgency to get the message out as quickly as possible.
INITIAL NOTIFICATION
Step 1. Receive notification message from PACOM over STE
on speaker mode and record.
The exercise message is preceded and ended with "EXERCISE, EXERCISE, EXERCISE" during a practice drill. The
simulated PACOM message on 13 January 2018, was preceded and ended with "EXERCISE, EXERCISE, EXERCISE."
Employee 1 stated that he/she did not hear "EXERCISE, EXERCISE, EXERCISE," but claimed he/she heard "This is not a drill."
0 TO 5 MINUTE
ACTION ITEMS
Step 2. Sound attack warning siren for all counties from siren computer. [Simulated]
Step 3.
Activate Emergency Alert System [Actual]
Step 4.
Contact all counties over the HAWAS. [Simulated] Step 5. Announce over the PAin EOC. [Simulated]
Step 6. Disconnect HECO power from the EOC. [Simulated] Step
7. Close and secure the EOC steel front doors. [Simulated]
AT THIS POINT THE SWP BMA DRILL ENDED
5
TO 10 MINUTE ACTION ITEMS Step 8. Notify all leadership.
Step 9. Contact FEMA
Step 10. Turn on the VTC Event Brief in the Conference Room. Step 11. Activate the phone recording system.
Step 12. Create an incident in WebEOC and enter in the event log. Step
13. Verify all Checklist items have been completed. ONGOING ACTION ITEMS
Step 14. Standby to operate the STE and communicate with PACOM Step 15. Monitor defensive and/or
retaliatory response by the U.S. and its allies.
Step 16. Do not respond to media.
[This is the Public Information Officer's (PIO) responsibility.]
Step 17. Advise
staff contacting SWP to shelter-in-place. DEACTIVATION ACTION ITEMS
Step 18. PACOM determines missile will not impact Hawaii
Step 19. If missile
impact time expires and PACOM has not updated
the SWP.
Step 20. Update leadership of any PACOM updates.
FINDINGS:
Based on the documents reviewed, information collected and interviews conducted during the investigation I find the following:
a. SWP has the BMA checklist but does not have SMA procedures included into the SOP although it's a new initiative.
b. The SWP successfully practiced the SMA Missile Checklist 26 times as determined by Telecom Branch records of SWP reporting on AlertSense.
c. Notification for Step 1 of the SMA checklist is used for actual ballistic missile attack using phrases "Real world" and "This is not a drill."
d. The established SMA checklist did not include a response protocol in the event of a false-ballistic missile message. There was a cancel message which stopped
further ballistic missile messages from displaying,
but does not send a new message
to the public. Eventually a CEM was developed and sent. The established protocol has since been updated
to include a False Alert message.
e. The Preparedness Branch previously identified the need to refine and reformat the SWP BMA checklist to include a "Deactivation" section or "All Clear" EAS message.
Had the protocol been developed within the last 2 months, the delay in sending out an official
notification over WEA/EAS that the alert was
false would not have happened on January 13, 2018
f. The Preparedness Branch worked with the Operations Branch
to conduct three no-notice exercises of the BMA checklist during the month of December. The design of these exercises was in compliance with Homeland Security Exercise and Evaluation Program (HSEEP) and included a written Exercise Plan and Exercise Evaluation Guide. A formal AAR was conducted, but there was nothing in the AAR that a
deactivation section in the checklist should
be further developed.
g. All SWP shifts did not participate in the HSEEP evaluation and were conducting their own drills separately from the exercise program, without exercise control protocols.
h. Many questions
on policies and procedures regarding the SMA Checklist have previously been asked by SWP personnel
and HI-EMA personnel outside of the SWP without
answers resulting in confusion and frustration.
iHI-EMA has suspended all SWP BMA drills until investigation is complete.
j. BMA Checklist is not detailed
which allows shifts to interpret and differ in protocol significantly when executing the drill. Furthermore, the drill was conducted during a change of shift which added to confusion
of who is in charge and which
shift is executing checklist
actions. HI-EMA operations provided the following guidance to the SWP in December:
1. Each shift will review and discuss
the BMA Checklist at the beginning of each shift to ensure there are no questions or concerns as to what the checklist is requiring.
2. Each shift will at a
minimum of twice a
week take time to run the checklist with different
SWP staff members.
3.
Shifts should look at differences in executing the checklist with
2, 3, 4 personnel and make recommendations to operations.
4. SWP shifts will create an incident at least three times during
the
week.
k. The AlertSense program replaced an older program
in mid- December 2017. No technical training was provided to the SWP,
only basic application. Many SWP personnel felt this training
was inadequate.
I. All SWP employees with access to AlertSense are required to take, and did complete, FEMA course
IS 247a prior to accessing AlertSense. This course provides basic information on the IPAWS. The goal of this course is to provide authorized public
safety officials with: increased awareness of the benefits of using IPAWS for effective public warnings; skills to draft more appropriate, effective,
and accessible warning messages; and best practices in the effective
use of common alerting protocol
to reach all members of their communities. There are no personnel training
records which make it difficult to determine what training each SWP member has taken. Employee 7
stated that Employee 1 completed the course.
m. The Operations Branch does not have comprehensive
documentation of training that can be used to determine exactly how many times the HI-EMA
SWP members practiced the established protocol or which shift practiced.
In addition, there are documented inconsistencies of how each BMA protocol was practiced prior to the issuance
of the warning message on 13
January 2018.
n. HI-EMA mistakenly believed that it had to consult with FEMA to issue an event code which contributed to the 38 minutes to issue an official notification that the threat was false. HI-EMA has corrected
this concern and understands that it is the HI-EMA's authority to issue an official notification.
o. SWP Supervisors do not consistently ensure
employee performance levels meet expectations.
p. Poor performance evaluations have been documented on select
SWP individuals. This
has created morale and competency issues.
q. Employee 1 has been a
source of concern for the same SWP staff for over 10 years. Employee 1's poor performance has been counseled and documented and the SWP members have stated that they are "not comfortable with Employee 1 as a supervisor, two-man team, or as a part of the SWP in general.
He is does not take initiative and has to be directed
before he takes action. He is unable to comprehend the situation at hand and has confused real life events and drills on at least two separate occasions."
r. Employee 1 stated that he/she initiated
the real-world COW. Employee 1 stated he/she did not hear the announcement, "EXERCISE, EXERCISE, EXERCISE" at the beginning and end of the simulated message. Employee 1 also stated, that he/she did not recognize the callers voice, and heard,
"This is not a drill" over the speaker.
However, all other SWP members clearly heard, "EXERCISE, EXERCISE, EXERCISE" at the beginning and end of the drill.
s. Employee 5 directed Employee 1 to send out the cancel message. Employee 5 stated that Employee
1
just sat there and didn't respond.
Employee
3
returned to the SWP from starting the time clock and saw Employee 4 and Employee 10 repeatedly get on the HAW AS letting the Counties know that it was just a
drill. Employee 2 was on the phone with Employee 6 briefing
them on the situation. Employee 5
was alerting the command staff. Employee 1 was sitting and seemed confused. Employee 3 took control of Employee 1's mouse and sent the cancel message. At no point did Employee 1 assist in the process of correcting the False Alert.
t. Employee 1 has been terminated.
u. HI-EMA has offered Mental Health
counseling to SWP members
to help recover from this traumatic event.
RECOMMENDATIONS:
I recommend the following actions:
a. Install a computer process
to rapidly issue alert cancellations on the
WEAIEAS System (completed January 13, 2018).
b. Add an "ALL Clear" selection to the AlertSense drop down menu in Step 2 of the BMA Checklist and develop deactivation procedures in the BMA checklist (completed 1/24/18).
c. Add into the confirmation message what action is confirmed, i.e. "Are you sure you want to send a "Real World Ballistic Missile
Alert."
d. Add a function to confirm the specific selections on the drop-down
menu with the supervisor's password required to enforce the two-person confirmation procedure.
e. Revise Step 1 Notification of the BMA checklist for a practice
drill eliminating the phrases used for actual ballistic missile
attack "Real world" and "This is not a drill."
Replace with "this is a drill." Continue the established
practice of preceding drill messages
and ending drill massages with
"EXERCISE, EXERCISE, EXERCISE."
f. Revise BMA Checklist to detail actions
to standardize best practices and eliminate confusion. Include in the revision; repeat all directives
and actions aloud and use a two-person
rule to verify directives and actions.
g. Eliminate conducting practice drills
during change of shift to clarify command and control.
h. Establish a formal review/update schedule to assess current
SWP guidance, SOPs, documents and protocols.
Identify opportunities for enhancement. Evaluate whether they are current, user friendly and consistent in design and content.
i. Establish a SWP knowledge library where current
protocols and checklist are housed and easily accessible to ensure all staff are working off the same
versions of procedures.
j. Add Risk Management into all events to develop controls
and make risk decisions. As control measures
are developed, risks are re-evaluated until
the residual risk is at a level where the benefits
outweigh the cost. Ensure that the appropriate decision authority then makes the decision.
k. Ensure future SWP planning
efforts include input from planners in the agency's Preparedness Branch.
I. Review other state warning programs
and protocols to determine best practices.
m. Ensure
future planning efforts are sufficiently completed and SWP
personnel are trained
before initiating procedural changes
n. Document and maintain training
records. With any training program, there are associated training records. Records
document who was trained, when they were trained, and what skills they have mastered.
Training records provide documentation for agencies, information for personnel evaluations, and support for promotion or salary increases.
o. Provide AlertSense technical training to SWP personnel. Revise
SOP to include
BMA procedures
p. Include simulation training and a practice
simulation computer for drills. The mistakes participants make in a simulation are learning experiences, the outcome is not damaging
to the organization, its equipment,
or its personnel. If the SWP participates in simulations during a practice
drill that are modeled
after real-world issues,
it can be free to make mistakes, learn from them, and apply
the learning before
going out to the real world.
q. Conduct a needs assessment to determine knowledge,
skills, and abilities that the SWP staff should possess and identify training,
protocols or policies
to address gaps.
r. Fill the four SWP vacant positions to remove the added burden on SWP personnel. Dealing with current SWP staffing challenges and preparing for the challenges that lie ahead requires a two-pronged approach
that focuses on:
1) attracting, hiring and retaining qualified SWP personnel, and 2) scheduling existing personnel to maximize
efficiency.
s. Evaluate whether the duties of SWP staff could be expanded
to support other HI-EMA initiatives and keep staff engaged without jeopardizing their core mission.
t. Ensure all training to SWP is formalized
with specific learning
goals, objectives, and performance metrics.
u. Conduct a command climate
survey.to all HI-EMA members to determine morale and communication within the division.
v. HI-EMA leadership should
be properly trained
to ensure employee
performance meets or exceeds expected
levels.
w. Based on my findings
the Director of HI-EMA should take appropriate disciplinary action
regarding involved employees.
-···
:;;:- """--------
.· :.e-:-:--c:_.... - - -
-_c:::::- ) '----
Brigadier General [Ret] Bruce Oliveira
Investigating Officer
No comments:
Post a Comment