Continued Conversation: Mikhaila on Burning Questions

“Sometimes you can tell that by speaking to someone that is ideologically possessed. They are not talking with their brain: they are talking through ‘what they’ve read’ or social media or ‘what their friends think’. I feel like if you talk to those people for long enough, and stay as honest as you can, you can kind of get through to what they actually think. But that’s part of the problem with twitter. It’s so reactive that you don’t know what anyone behind the tweets actually feels.”

Mikhaila Peterson

This chain of thought from Mikhaila really speaks to the utility of honesty. Behind the curtain, there is an individual, marvelous and unique. Someone worth the time. Someone worth getting to know. If we stand our ground , and be honest with ourselves and the people infront of us (online or not), we will find that we ~ can ~ come together and agree. There doesn’t have to be an illusion.

Her point about social media is also deeply profound: that we don’t ever get how anyone feels behind these platforms. I don’t think she’s saying these platforms are inherently bad, but that we should look to use them to maximize human flourishing. It isn’t enough to blindly start using Social Media and hope for the best. It’s about re-tooling your idea of the platform to fit its best use case. When you read a social media post by someone, it kind of masks the general feeling (that is usually there) from a normal conversation. In social media however: That feeling is then substituted by whatever is most convenient to think for the individual (or perhaps another case: a collectivist ideology).

She made another point earlier in the video that I find a reoccuring theme in 2020: about people fundamentally disagreeing with something, and then either letting it slide or viscerally disagreeing and avoiding discussion. I find this quote so apparent:

“If you see something that is wrong, and it makes you viscerally uncomfortably – you should talk about it”

Mikhaila Peterson

So freakin true. Talk it out. Mental health is looming in our country and we need to learn to TALK about all these things.

Her adventures with dieting is absolutely insane as well. Really puts a perspective on how we need to re-think dieting, mental health, and drugs.

Thanks for listening to this ‘first’ edition of “Continued Conversation” (Don’t worry I’m trying to find a new name). It’s my first jab at it, and I am not sure if I will keep doing it. Ultimately, I love the idea of extending conversations that are already out there to further thoughts and ideas into the public. Feel free to comment or hit me up. I’d love to hear what you have to say. Until next time~

Conversation. Conversation.

Conversation. Conversation. There is a lack of genuine conversation around the emerging topics, political or otherwise. It’s worrying – but not so much ‘worrying’ that we can’t rise to the challenge, but rather that we are afraid to speak out. The ‘mob’ is leering at the psyche of america, waiting to judge and ridicule the mere suggestion of an opinion, a belief – or even a fact.

We need to be able to communicate in some way where we aren’t at our throats – that every remark is seen as some attack or personal strife. The sense-making apparatus of our global community is faltering. We can’t make out what is real, and what is not real, and what is right, and what is not right. We are in a time where freedom of speech is at it’s most volatile, yet it is the very exact thing we need the most. Stating a fact is seen as ‘hostile’, and ultimately responded with shame and guilt-tactics. Some how saying the truth is seen as a form of ‘privilege’. We need to be able to differentiate the fact of a certain set of data, and the emotional entanglement of an individual. They are two completely separate ball parks. Facts don’t do anything for the malevolence someone has been through. Let’s be intellectually honest, but at the same time be able to recognize and empathize that which someone has been through. Be a friend first, an intellectual fact-checker second. Be the person that saves the soul. It starts with you, and the hand you offer.

We all want the same thing. The problem is that we are spread across the spectrum in separate groups with a diverse set of communication trees and sub-culture identities. If we all succumb to tribalism now – society, science, everything will be brought to a screeching hault. You as an individual are interlaced in society in many facets. You are not alone. You are not just your group. You are so much more important.

I want to challenge everyone to be a part of the movement for change. To recognize when a friend is needed, or when the truth is needed. To recognize that sometimes its better to just give praise and be agreeable, or that sometimes its better to disagree – and find a common ground of understanding. Be a champion of discourse.

The narrative of the media is draining in every aspect. It’s impending and absolutely toxic. Our leaders are cowardly and lack a spine. Our celebrities succumb to the order and will of others; on a whim. Our leaders are not willing to stand up for whats right. The only person that thinks he’s right, is an authoritarian, incoherent clown. We find ourselves in a very unique situation. So I challenge you, the individual, to make a difference. It starts with all of us participating in the revolution of honest discourse, and intellectual symmetry. It starts with a conversation.

Energy Consumption | Infrastructure

TLDR (efficiently expressed – rhetorically); Reducing infrastructure hardware with virtualization technology and picking energy efficient hardware that ultimately reduced our energy consumption over the last six years. Certainly beating the annual increase on commercial energy costs and giving us insights into the power of software and hardware technology innovations. 

This project is: My measurements are a ‘broad stroke’ of what our potential savings were. We want to see if we A) saved money on electricity to lower the environmental impact and B) see if my operational planning is lowering our foot print, and requiring less of a demand on internal resources ($$$$) and operating more efficiently (this is what I really care about).

Infrastructure Energy Consumption Analysis

Well… I think the graph speaks for itself. We saved a lot of money on electricity. How did we do it? Keep reading.

What this analysis could have been: 

It could have been a complete analysis that broke down each system by CPU (E3, E5, AMD, etc..), RAM (8, 16, 32, 64, 128), Storage (RAID configs based on SSD’s, HDD’s), and exact amount of (network) data pushed by the hour. Gratefully, I get paid more than $0.082 per hour, unfortunately, that means that the time to perform a fully detailed analysis may dramatically reduce the cost savings. Then again, it could be tucked away as a hidden variable…, jk.

Remember, efficiency.  “When products use more power to perform the same amount of work, they are by definition less efficient.”

Why it wasn’t that: 

Cost benefit analysis i.e, not worth the time for a similar outcome. This analysis is being done in hindsight with 20/20 vision. If I wanted to make a decision about future infrastructure changes (collocation, hybrid public cloud, on-prem datacenter expansions) with vendor purchase agreements over $100,000 for a single refresh, then the cost/benefit may be exponential enough to measure. We don’t spend that type of money on infrastructure.

What it is: 

I mean, knowing that the new hardware consumes less wattage than the prior hardware and comes with software that supports low idle usage during low load times; means that the power consumption will be lower – in theory. But remember, all you know are the stats given to you from the vendor. That doesn’t include the environment variables (seasonal electrical cost changes, systems usage changes, and random implementations or deprecation’s, nor system count changes). That sentence alone is exhaustive and makes me want to switch to cloud computing! For instance, our year over year costs per device is on an upward trend, but our overall costs for our environment is on a downward trend. How is that? 

year over year electricity cost per device

Originally, I wanted to see how much of a cost savings or increase we would see from my infrastructure decisions. It quickly became apparent that most of my savings was not because I picked super efficient and ‘green’ hardware – but I did. The primary reason for my cost savings was actually a software technology – virtualization. Yes. It has been around for a LONG TIME.

The primary savings was accomplished during the partial and full virtualization phase (2016-2017). Reducing the onsite datacenter footprint from 24 servers down to 3 primary servers. Unfortunately, some of the technologies deployed required additional power consumption, increased demand on average server usage and increased PoE demand on all switches as more devices become Powered over Ethernet. 

Eitherway, we dramatically reduced our energy consumption! Yay, us. Sorry, Entergy.

What I plan to do with this: 

Increase system efficiency, cost effectively – I thought I made that clear.

The primary bottleneck that is limiting our system throughput are the disks I/O speeds. With future analysis, we will be able to determine if SSD’s can provide us with an operational cost savings through the direct cost of electricity, infrastructure purchasing costs by consolidating one of the Hyper visors from three down to two and comparing those cost savings to varying models. I’m considering and testing costs in both a hybrid-cloud infrastructure (which adds systemic processes (lowers efficiency)) and complexes the design (lowers troubleshooting efficiency without proper training) and increases demand for professional development. All variables must be considered before making a decision on our next infrastructure initiative. 

How I plan to measure: 

  • Electrical costs can continue to be measured on an annual basis by kWh per device based on average load/usage and multiplied by the total number of ‘like’ devices in the network. 
  • Processes can be measured by taking the collective salary average and dividing it by the support hours required to maintain, monitor, and support a hybrid-cloud. 
  • In the same light, troubleshooting time can be averaged by the salary over ticket completion times for systems / infrastructure tasks. 
  • And finally, PD costs are explicit when utilizing subscription plans, boot-camps, and training materials. The hardest aspect to measure will be personal, off-the-clock training time dedicated to increasing our staff knowledge on cloud computing maintenance and troubleshooting. 

Resources:

Inspiration: https://codeascraft.com/2020/04/23/cloud-jewels-estimating-kwh-in-the-cloud/

Some research: http://www.webtorials.com/main/resource/papers/cisco/paper112/EthernetPowerStudy.pdf

Amazing vendors:

https://www.dell.com/ (PowerEdge is amazing!)

https://meraki.cisco.com/ (Built in power consumption metrics)

https://www.cisco.com/ (manual power consumption stats)

#show power inline

module available used remaining

(watts) (watts)

1 370.0  39 331




Disclosure:

  • Not affiliated with anyone / anything in this post directly.
  • Excuse grammatical issues, I’m not a writer.
  • All analysis was inspired by others with a personal directive to save the earth and increase efficiency.

2020

As always, I’m keeping this short.

2020 is here and I have focal point for the year. My optics are tuned and set on the follow list:

  • Clarity. In my mind and of others. All too often, I’m immediately responding assuming that I understand. Sometimes I don’t. It often comes off as rude or aggressive and it needs to change. This is a joint effort, but at least I will be the one to start the shift among my colleagues.
  • Project management. I enjoy organizing projects, drafting reports on progress and achieving varying levels of progress.
  • CCNP, Python, Ansible. The new CCNP is around the corner. It’s going to involve Networking, DevOps, and Automation. I’m getting it.
  • Action. There’s no waiting. Just acting. I’ll assess all opportunities that come my way, see if they fit with my 5 year and 10 year roadmap and execute on all opportunities that align.
  • Intentionally . Doing the best with full intentions of performing the best I can.

That’s it. I’m really just focusing on key skills, growing professionally as a manager and leader, and chiseling away at progress.

Cisco – The Future of Internet

On December 11th, Cisco announced the future in five categories.

  1. Silicon
  2. Optics
  3. Software
  4. Systems
  5. Architectures

1. Silicon – Referenced as the “engine to a car”, Silicon One is Cisco’s programmable silicon architecture – Q100. This transistor can handle large buffers, advanced programability and greater bandwidth!

Nerd Knob #1: 10 Tbps carrier-class capability

Finance: Drastically reduces the OpEx industry rate which sits at a 1:5 ratio

Read more here: https://blogs.cisco.com/sp/one-silicon-one-experience-multiple-roles

2. Optics – Slower interface speeds could easily cost a solution 10%. With new silicon photonics reaching 400G, the cost per bit can be driven down.

With the hardware becoming more diverse and software driven, we are now going to see an increase in cost on the speed.

Can you imagine? 400GbE connections. That’s an insane amount of data movement.

Read more here: https://blogs.cisco.com/sp/optics-fundamental-to-build-the-internet-for-the-future

3. Software – As Cisco references Silicon as the car, they reference Software as the steering wheel. Their Network Operating System (NOS) becomes an even more critical component in the future of the internet. With Cisco’s IOS XR7, come prioritization on operations. Their goal was to simplify and improve automation tasks with the overarching goal of “zero-touch”. With better efficiencies, comes more complexities. XR7 NOS allows teams to utilize the computer for insights and analytics.

Read more here: https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-xr-software/index.html

4. Systems – Continuing the reference – the car. With Cisco 8000 series routers being deployed, we can now bundle the hardware and software for limitless opportunities. Okay, maybe not limitless for long, but definitely a game changer for the immediate future.

Nerd Knob #2:

1 RU Router can support 10.8Tb/s bandwidth…

3 modular form-factor platforms delivering support from 115Tb/s > 260Tb/s

Full Fabric redundancy

Top of the line security – Hardware based “Trustworthiness” for tamper proof control and visibility controlled by Cisco Crosswork Cloud

Finance:

Reduced power consumption per Gb (4W) which is 1/4th the consumption

Read more here: https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-xr-software/index.html

5. Architectures – Everything listed above has been re-imagined with performance, trust and OpEx in mind. By keeping all of this on track, Cisco is reinventing how the internet operates with people and business in mind.

Read more here: https://blogs.cisco.com/news/future-of-the-internet-its-here


*This post is not endorsed by Cisco, nor is it a direct reflection of their beliefs and opinions.

Reflection & Vision

When you’ve reached a peaked plateau in any aspect of your life, it’s easy to feel lost, and fogged. Especially when you’re at a peak – but your internal furnace that drives for growth and learning beyond your current levels continues to beg the question; what’s next? It’s common for people my age to want more – and more constantly. It’s often referred as greed. And the definition may be accurate with that statement. I’ll debate it later.

It’s not a bad place to be and I realize this. I’ve had mentors, father figures and friends refer to this point in life as many different things. All of which have resulted in complacency, losing businesses, or failing on their part of the deal in partnerships. That’s where I was earlier this year. I was at my turning point. A point in time where I could choose to make a change. It would either result in returning to Port, maintaining my anchor or setting sail for the high seas.

My decision? Well, none of the above. I overthink everything. I decided to come back to basics. Develop a strategy that will ensure my success – long term, expand on my foundation, and then embark on my journey.

So, I’m in the middle of developing key skills that will last me another decade in this vastly dynamic world of technology. Being a network and systems specialist is only a fraction of who I really am and even of that percent that I am, I can still be greater.

Skill #1 Systems & Networking – I’ll spend time learning Cloud infrastructure from a technical stance. Mastering understanding from a top level managerial view down to the cleanest whitepapers written by the creators of these technologies. Wikipedia is not a proper source.

Skill #2 Management – I’ll spend time learning how to encourage natural incentive and effort for a common strategic goal.

Skill #3 Leadership – being prepared and present more frequently in meetings. I’m already good at this, but I’ll be even better. Providing more follow up and follow through on projects and goals. Creating an environment of care and compassion among my peers and colleagues.

Skill #4 Entrepreneurship – I’ll work on my partnerships. Creating strategic partnerships that truly benefit both sides with sustainability in mind.

Skill #5 Personal – removing limiting thoughts and hesitation from my mental reflex. Preparing to be a father and a better husband with every opportunity.


I think it’s important to understand why I chose skills rather than certifications for my goals. Having an understanding of your desired skill will remove the limiting factors of choosing a single cert and believing that it’s all you need. In reality, I must master my skills, which may equate to five certs, 400 hours of video lessons and another 10 years of experience. The later is much more appealing and hard to beat, especially when competing against a team of nerds, a processor and code for future jobs. It also empowers me to go out, gather the best data on the subjects and study them to a mastery level – then apply what I’ve learned to be truly great.

Week 1 – Unifi for Home

It’s week one in my new house. I have Unifi powering my network with the security gateway, ubiquiti switch and AP Pro. Lots of information to come, but so far, I’m loving the Unifi dashboard and features. Especially the specifics of their automatic topology!

Unifi Ubiquiti Home network set up

Update #1 12/2/2019

Unifi for Home use is amazing when you combine the Security Gateway with any of their switches. Getting the live dashboard and alerts with their “insights” dashboard is an amazing feature. 100% worth the cost ($139).

Personal Finance

I’m not going to write about personal finance 101 because it would end up being version 10,000,000,000,000 ^365…

We’ve seen enough. Everything has been repeated and humans are humans.

I am, however, going to share my methods, which includes an excel sheet and a budgeting application.


The Excel sheet – I use it “planning” and being pro-active with my budgeting efforts. What I mean by this is, I plan purchasing a home, funding a college account for my future child, purchasing a car, or planning future investments. It’s sort of like a quick simulation.

Setting the goal, here’s how I do it – the categories:


Let the simulation(s) begin!

Just start filling it out based on a monthly view and allow excel to “summarize” it for you. Do the same for your expenses and see how much is left in the (“Net(+/-)) column at the end of the month.


On to the App! This is what everyone wants to see.

So, yeah, it does a lot – but it keeps things simple. My favorite feature about the application, aside from being able to view all accounts at once (Student loans, home loans, credit/debit, and 403b accounts), I’m able to view my “net worth” because of the vast account capabilities.

The other cool feature, is the ability to categorize. Throughout the month, I set a dollar amount budget (based on my monthly excel budget) and I categorize all expenses throughout the month. By the end of the month, I look at the budget dashboard and see where I am with my budget categories. Most of the time, I’ve over spent on restaurants or groceries, or wood working projects. I promise, one of these days, I’ll treat wood working like a business and actually make money from it!

The unspoken and indirect feature that Personal Capital offers, is the ability to reflect. I was feeling a little negative toward my finances – feeling as if I hadn’t made any progress in the last year. Personal Capital maintains record of all transactions and net worth… when I reflected on the app, I saw that I actually paid down $20,000 worth of debt, saved for a mortgage down payment, paid off my car and enjoyed an expensive vacation for a week in Florida… It instantly boosted my mood and made me realize that I am making steady progress on my financial, material and social goals.


The second App – Robinhood. If you’re investing but don’t have more than $200,000 invested in a stock trading brokerage account, then I suggest that you invest for free – in Robinhood. Seriously, you can make money instantly with their free trades. Buy Apple today, sell it in a few days and enjoy your 1-3% profit margins. If you capitalize on this technique which leverages compounding growth and the market continues to do well, you’ll do well!

Free money, create an account and start trading!


References

Message me if you would like a template or a more complete run through of anything in this post.

Quality of Service (QoS) Introduction

QoS provides predictable management of network resources during times of congestion. When a router is overloaded the memory buffers on it hit maximum capacity. The router has no other choice than to drop traffic. Congestion happens when the memory buffer is filled up on a particular interface on a router. This usually happens when traffic is being pushed passed the line rate for said cable. A router has certain memory reserved for each interface and when that memory gets full, it will drop packets. QoS gives control onto what packets can be dropped. It can also limit traffic by either policing or shaping it. Policing is the act of watching for the bandwidth of a particular stream of packets, and dropping any packets that are excess of that. Shaping is the act of watching for bandwidth of a particular stream of packets, and when the excess limit is reached, it holds the packet in memory until that interface is less congested.

During times of congestion on the network you can expect to see things like Delay, Jitter, and Drops. Delay is simply the latency for one packet to get to it’s destination. Jitter is the results of packets being received but in various time lapses. Drop is simply that the traffic had to be dropped because of the congestion.

To understand QoS, It is best to understand different switching and hardware architectures and how all these different platforms handle packets: particularly how packet is stored in memory and how those memory relate to the forwarding process.

Network Equipment is very much like a computer

Us network engineers know how to configure network equipment, analyze packets and influence the forwarding decision of those packets. However, sometimes we don’t know how the switches/routers actually do it! ‘It’ as in how switches/routers take packets and put them onto other interfaces. What is going on behind the scenes?

Switches and routers are just like a computer. They have their storage. There memory. They have a CPU. The big difference is that most network equipment have a thing called ASICs. ASICs stand for Application-specific integrated circuit – and they are really good at doing one thing and one thing only (or sometimes a subset of very specific tasks). That one thing could be looking up a MAC Address in a MAC Address Table. Another example would be looking up the routing destination for a IP Packet. Since these ASICs were made for a specific task, they perform these lookups very very quickly. In contrast, CPUs on routers/switches are much slower in there lookup. If you were to compare the two – a human could not differentiate, as the lookup on both would be similar to human perception. However, it makes a huge difference when you are handling thousands upon thousands of packets to use ASICs to make forwarding decision rather than CPUs. While a standard PC uses RAM/Memory to store the operating system, and various applications – Network Equipment use them the same way, but with a twist: they use memory to store packets ingressing and egressing the device. A network device has processes just like a computer. It runs an OS of some type, and it has processes that need to be stored into memory. Packets ingressing or leaving a network device have to be stored somewhere. That is where memory is used. There are lots of different network devices as well as alot of different hardware architecture for them. But the key take away is that memory in network devices are used for two things:

  1. For it’s own OS/processes (routing protocol, SNMP, OS, etc.) – These use CPU Resources
  2. For packets traversing the device (Packet Lookup) – These use ASIC Resources

How routers deal with a packet

Below is a high-level chronological overview of how routers deal with packets:

  • 1. Packet Arrives on ingress interface and its  placed in memory called the RX-Ring.
  • 2. Packet is then queued in the memory buffer. This is where CPU (or ASIC) takes control of that portion of memory and re classifies the memory.
  • 3. Forwarding Decision is made (routing via IP/Switching based on MAC etc.)
  • 4. Packet placed on TX-Ring. The same memory is then reclassified as TX-Ring. The outbound interface of the packet then takes control of that portion of memory. 
  • 5. Packet transmitted out egress media.

Think of RX Ring and TX Ring as the dedicated memory for that specific interface. Every port has both a RX Ring and a TX Ring. These ‘Rings’ Are completely separate from queues and buffers* More on that later. QoS has no control over the RX Ring and TX Ring. QoS has control over handling of packets and congestion from the Queues and Buffers.

Packets could be physically moved from one memory chip to another. Depending on the memory architecture of the device, the packet could be physically moved from one memory chip to another -or- simply re-classified, but not moved.

Memory Architecture

There are two types of memory architectures for switches. Shared memory and distributed memory. Shared memory essentially is one big block of memory that is used for all interfaces. The packets coming in and out are renamed and looked up by ASIC linked to that memory. A device with distributed memory has dedicated ASIC/memory for each port/a group of ports. A common shared ring that connects all the ASICs memory together tie them to other ports. Devices that use distributed memory are usually large switched chassis that have multiple line cards. Each linecard has ASICs, but they use high speed ring/bus to interconnect them all together. Below is high-level order of how packets are handled with shared/distributed memory.

How devices deal with packets (shared memory)

  • 1. Packet arrives on ingress interface
  • 2. Interface/Module ASIC forwards packet into a common shared memory pool.
  • 3. Forwarding decision is made by forwarding ASICs
  • 4. Memory ownership of packet buffer transferred to egress interface
  • 5. Packet transmitted onto the egress media

How devices deal with packets (distributed memory)

  • 1. Packets arrive on ingress interface
  • 2. Interface/Module ASIC places packet into memory (specific for port/group of ports
  • 3. Forwarding decision is made by forwarding ASIC
  • 4. Packet transmitted onto shared ring/bus to all egress interfaces
  • 5. Appropriate egress interface queues and then schedule the packet

Buffers and Queues

A Buffer is physical memory used to store packets before and after a forwarding decision is made. On a router this memory can be allocated to interfaces as ingress/egress. In a shared memory architecture, certain parts of memory are dedicated as buffers. However, that same sahred memory is used for other CPU Proccesses.

A queue is different depending on the platform. On Routers, it is a logical part of the shared memory buffer. On switches, individual interfaces/linecards have their own memory which is used as interface queues. Think of queues as the logical section of the physical memory (buffer).

Configuration of buffers is not part of QoS. Buffer configuration would involve modifying the quantity of buffers allowed for particular sized packet. QoS configuration applies to queues. With QoS you’re not modifying the quanitity of buffers allocated or a particular sized packet. Instead, you are taking existing buffers that have already been defined as interface queues and modifying how packets are treated when inside those queues. 
During times of no congestion, QoS is not needed because packets are transmitted First In First Out (FIFO) up to the line-rate of said interface. During times of congestion what happens is the queue is filled up and trying to pass traffic higher than the line-rate of the interface. 

Integrated and Differentiated Services

Integrated Services is a QoS Model in which the entire packet from end to end is ensured certain minimum QoS. Initial RFCs published by IETF in mid 1990s: 1633, 2211, 2212. RSVP is used as the primary protocol to setup the path. Requires every node along path to heed its reservation and to keep per-flow state. This type of Service for QoS did not gain much traction because it was unfeasible to implement across multiple vendors and organizations.

Differentiated Services is designed to address challenges of Integrated Serivces. These are the following RFCs: RFC 2474, 2597, 2598, 3246, 4594. The DiffServ Model Describes various behaviors to be adopted by each compliant node (called Per-Hop Behaviors(PHB)). Each device has the capability to apply QoS the way they want with whatever method they choose fit. With Integrated Services it was guaranteed that each packet had end to end guarantee of QoS. With Differentiated Services, there is no guarantee and each device can or may not be configured with QoS.

Classification/Marking

Traffic first must be divided into “classes”. A Class of traffic will receive the same type of QoS treatment. It analyzes the packets to differentiate flows. Packets are marked so that analysis happens only a limited number of times, usually at the ingress edge of a network. Usually this starts as a business decision and the business needs for the network. The whole idea behind classification is to identify traffic in your network that is critical to operation and quality of your buisness. After identifying what traffic is important, you can create rules to match that traffic – and mark them for QoS. Most ISPs will police ingress traffic. Traffic that is non-conforming (higher then the CIR) will be either dropped or marked down. Customers obviously don’t want any type of traffic drops, so shaping done on the egress interface leading to your ISP is recommended. 

Queuing When egress traffic cannot immediately be transmitted (aka on the TX Ring), it is placed in an egress queue. A single egress interface may have multiple associated egress queues differentiated by priority. QoS features designed for queuing provide control over which classified traffic is placed into each of these queues. Queueing can also preemptively drop traffic from within queues to make room for higher priority traffic. 

Scheduling

Scheduling is defining what packets are put on the wire depending on their priority. On routers, QoS queuing features such as WFQ affect queuing and scheduling behaviors. On switches, queuing and scheduling can be separate features. Traffic shaping is a function of scheduling. 

Congestion Management

Congestion management features allow you to control congestion by determining the order in which packets are sent out an interface based on priorities assigned to those packets. Below is high-level overview of congestion management process:

  • Creation of queues
  • Assignment of packets to those queues based on the classification of the packet
  • Selectively dropping packets from within queues when those queues reach pre-defined thresholds
  • Scheduling of the packets in queue for transmission

Features for Congestion Management: WFQ, CBWFQ, PQ, LLQ, WRR, and SRR

Traffic Shaping Features of Congestion Avoidance: RED, WRED, WTD, and Policing

Modular QoS Command-Line (MQC)

MQC allows QoS features that apply classification, policing etc to be configured independently and then linked together as needed. Similar to Modular Policy Framework (MPF) in ASA. MQC utilizes class maps, policy maps, and service policies. 

  • Class-maps are used to identify and classify traffic that you want to identify for QoS. Class-maps can reference ACLs to classify traffic, for example.
  • Policy-maps define what you want to do to the traffic. Each policy map can reference multiple class-maps. When you enter more than one class-map, it is done in chronological kind of like an ACL. Policy-maps apply things like policing, shaping based on your class-maps that you created.
  • Service policy is used then to apply the policy-map to a particular interface in a particular direction. 

Defending Against RYUK

Computer code on a screen with a skull representing a computer virus / malware attack.

It has been exactly four weeks since Homeland Security, the National Guard and LA DoE scheduled an emergency phone conference with all Technology Directors in the state of Louisiana.

During this briefing, we were informed that 6 school districts and 2 government agencies were attacked by a ransomware known as RYUK. The immediate reaction was frightening as the governor of Louisiana demanded a state of emergency. We were told to shut down internet access and remove local admin rights until further notice.

Keep in mind, we were two weeks out from the start of school (smart timing on RYUK). We had to finish deployments for hundreds of chrome books, projector installations, finalize surveillance installs and manage several other projects in our department.

A day passed before we received a strategic game plan from Homeland Security that detailed several phases of security implementations. Phase 1, turn off all internet access. This can be hard to do when your trying to deploy devices, run updates and have 150 staff members coming back to campus…

I’ll explain the technologies and how everything works later in the blog.

We spent a week tightening up the ship, blocking internet access based on firewall rules, attempting to have offsite backups work, deploying devices, installing software… we were extremely reliant on the internet.

Services were breaking constantly, as expected when you turn off the internet (LOL, if I don’t laugh, I’m crying). My boss could see the stress on our department and offered full support to us while we navigated these high seas. I have to say, I have one of the most supportive bosses in the world (Shout out)!

She granted the additional resources necessary to tackle this oncoming storm.

Four weeks later, 600+ hours between two employees, we now have all systems patched, removed local admin, wiped and deployed. In addition, all members of our organization have been trained on identifying phishing attacks (for your reference). And the entire network is locked down according to recommendations made by Homeland Security.


The Technical


Known threats to block

deny any any 84.146.54.187/32
deny any any 75.147.173.236/32
deny any any 218.16.120.253/32
deny any any 170.238.117.187/32
deny any any 195.123.237.129/32
deny any any 194.5.250.123/32
deny any any 85.204.116.158/32
deny any any 31.184.254.18/32
deny any any 186.10.243.70/32
deny any any 104.20.209.21/32
deny any any 445
deny any any 447
deny any any 449
deny any any 8082
deny any any 16993

They have identified RDP (3389) and Email (80/443) as the two primary vectors of initiation.


How we “turned off” the internet

Using the firewall “deny any any” and manually adding 40+ pages of “trusted” ip addresses was not an option for us. It was extremely time consuming and impractical. I often fat-fingered IP and port numbers. I broke everything. I wish Meraki allowed me to use a CLI for this type of task. Luckily, Meraki had a second option for us.

Meraki offers Content Filtering, which allows you to blacklist everything (*) and whitelist URL’s. I chose this option. Upon blacklisting the entire internet with (*), I was then able to whitelist common sites much more efficiently.

Anything that ends with .gov and .edu were whitelisted, but not completely. Aside from these, every other site had to be whitelisted. Aside from the constant adding, this process is very easy.

All traffic is triple filtered with the leading Cisco, Google, and Meraki products in the globe. With dual content filtering, IPS/IDS and AMP screening, our traffic has been relatively clean – to say the least.

When it comes to Meraki, we were also able to filter traffic by country. This allowed us to block traffic from random countries that we have no business communicating with/through.

Anti-virus

We commissioned a new AI based product to help protect all of our servers, faculty and staff. Hoping that their spread of knowledge with the recent attacks will help prevent attacks on our network.

Advanced email filtering & quarantines

Google allows for us to enable advanced email filtering and quarantine. I’ve enabled all features to flag suspicious emails and I’ve personally trained every employee on proper email usage and what to look for in an email.


As of today, we are not in the clear, but we are in a much better state now than we were a month ago. We were given the chance to reflect on our current policies, enforce new procedures and tighten up security campus wide. Other organizations were not given the same opportunity as us.

For anyone out there battling this, please reach out if you need support. This is a beast to navigate and cyber crimes are not going away anytime soon.


References

Center for Internet Security (Homeland Security)

Read about Protecting your network

Read about Emotet Malware

Read about TrickBot