It has been exactly four weeks since Homeland Security, the National Guard and LA DoE scheduled an emergency phone conference with all Technology Directors in the state of Louisiana.
During this briefing, we were informed that 6 school districts and 2 government agencies were attacked by a ransomware known as RYUK. The immediate reaction was frightening as the governor of Louisiana demanded a state of emergency. We were told to shut down internet access and remove local admin rights until further notice.
Keep in mind, we were two weeks out from the start of school (smart timing on RYUK). We had to finish deployments for hundreds of chrome books, projector installations, finalize surveillance installs and manage several other projects in our department.
A day passed before we received a strategic game plan from Homeland Security that detailed several phases of security implementations. Phase 1, turn off all internet access. This can be hard to do when your trying to deploy devices, run updates and have 150 staff members coming back to campus…
I’ll explain the technologies and how everything works later in the blog.
We spent a week tightening up the ship, blocking internet access based on firewall rules, attempting to have offsite backups work, deploying devices, installing software… we were extremely reliant on the internet.
Services were breaking constantly, as expected when you turn off the internet (LOL, if I don’t laugh, I’m crying). My boss could see the stress on our department and offered full support to us while we navigated these high seas. I have to say, I have one of the most supportive bosses in the world (Shout out)!
She granted the additional resources necessary to tackle this oncoming storm.
Four weeks later, 600+ hours between two employees, we now have all systems patched, removed local admin, wiped and deployed. In addition, all members of our organization have been trained on identifying phishing attacks (for your reference). And the entire network is locked down according to recommendations made by Homeland Security.
Known threats to block
deny any any 220.127.116.11/32 deny any any 18.104.22.168/32 deny any any 22.214.171.124/32 deny any any 126.96.36.199/32 deny any any 188.8.131.52/32 deny any any 184.108.40.206/32 deny any any 220.127.116.11/32 deny any any 18.104.22.168/32 deny any any 22.214.171.124/32 deny any any 126.96.36.199/32 deny any any 445 deny any any 447 deny any any 449 deny any any 8082 deny any any 16993
They have identified RDP (3389) and Email (80/443) as the two primary vectors of initiation.
How we “turned off” the internet
Using the firewall “deny any any” and manually adding 40+ pages of “trusted” ip addresses was not an option for us. It was extremely time consuming and impractical. I often fat-fingered IP and port numbers. I broke everything. I wish Meraki allowed me to use a CLI for this type of task. Luckily, Meraki had a second option for us.
Meraki offers Content Filtering, which allows you to blacklist everything (*) and whitelist URL’s. I chose this option. Upon blacklisting the entire internet with (*), I was then able to whitelist common sites much more efficiently.
Anything that ends with .gov and .edu were whitelisted, but not completely. Aside from these, every other site had to be whitelisted. Aside from the constant adding, this process is very easy.
All traffic is triple filtered with the leading Cisco, Google, and Meraki products in the globe. With dual content filtering, IPS/IDS and AMP screening, our traffic has been relatively clean – to say the least.
When it comes to Meraki, we were also able to filter traffic by country. This allowed us to block traffic from random countries that we have no business communicating with/through.
We commissioned a new AI based product to help protect all of our servers, faculty and staff. Hoping that their spread of knowledge with the recent attacks will help prevent attacks on our network.
Advanced email filtering & quarantines
Google allows for us to enable advanced email filtering and quarantine. I’ve enabled all features to flag suspicious emails and I’ve personally trained every employee on proper email usage and what to look for in an email.
As of today, we are not in the clear, but we are in a much better state now than we were a month ago. We were given the chance to reflect on our current policies, enforce new procedures and tighten up security campus wide. Other organizations were not given the same opportunity as us.
For anyone out there battling this, please reach out if you need support. This is a beast to navigate and cyber crimes are not going away anytime soon.
Read about Protecting your network
Read about Emotet Malware
Read about TrickBot