Conversation. Conversation.

Conversation. Conversation. There is a lack of genuine conversation around the emerging topics, political or otherwise. It’s worrying – but not so much ‘worrying’ that we can’t rise to the challenge, but rather that we are afraid to speak out. The ‘mob’ is leering at the psyche of america, waiting to judge and ridicule the mere suggestion of an opinion, a belief – or even a fact.

We need to be able to communicate in some way where we aren’t at our throats – that every remark is seen as some attack or personal strife. The sense-making apparatus of our global community is faltering. We can’t make out what is real, and what is not real, and what is right, and what is not right. We are in a time where freedom of speech is at it’s most volatile, yet it is the very exact thing we need the most. Stating a fact is seen as ‘hostile’, and ultimately responded with shame and guilt-tactics. Some how saying the truth is seen as a form of ‘privilege’. We need to be able to differentiate the fact of a certain set of data, and the emotional entanglement of an individual. They are two completely separate ball parks. Facts don’t do anything for the malevolence someone has been through. Let’s be intellectually honest, but at the same time be able to recognize and empathize that which someone has been through. Be a friend first, an intellectual fact-checker second. Be the person that saves the soul. It starts with you, and the hand you offer.

We all want the same thing. The problem is that we are spread across the spectrum in separate groups with a diverse set of communication trees and sub-culture identities. If we all succumb to tribalism now – society, science, everything will be brought to a screeching hault. You as an individual are interlaced in society in many facets. You are not alone. You are not just your group. You are so much more important.

I want to challenge everyone to be a part of the movement for change. To recognize when a friend is needed, or when the truth is needed. To recognize that sometimes its better to just give praise and be agreeable, or that sometimes its better to disagree – and find a common ground of understanding. Be a champion of discourse.

The narrative of the media is draining in every aspect. It’s impending and absolutely toxic. Our leaders are cowardly and lack a spine. Our celebrities succumb to the order and will of others; on a whim. Our leaders are not willing to stand up for whats right. The only person that thinks he’s right, is an authoritarian, incoherent clown. We find ourselves in a very unique situation. So I challenge you, the individual, to make a difference. It starts with all of us participating in the revolution of honest discourse, and intellectual symmetry. It starts with a conversation.

Energy Consumption | Infrastructure

TLDR (efficiently expressed – rhetorically); Reducing infrastructure hardware with virtualization technology and picking energy efficient hardware that ultimately reduced our energy consumption over the last six years. Certainly beating the annual increase on commercial energy costs and giving us insights into the power of software and hardware technology innovations. 

This project is: My measurements are a ‘broad stroke’ of what our potential savings were. We want to see if we A) saved money on electricity to lower the environmental impact and B) see if my operational planning is lowering our foot print, and requiring less of a demand on internal resources ($$$$) and operating more efficiently (this is what I really care about).

Infrastructure Energy Consumption Analysis

Well… I think the graph speaks for itself. We saved a lot of money on electricity. How did we do it? Keep reading.

What this analysis could have been: 

It could have been a complete analysis that broke down each system by CPU (E3, E5, AMD, etc..), RAM (8, 16, 32, 64, 128), Storage (RAID configs based on SSD’s, HDD’s), and exact amount of (network) data pushed by the hour. Gratefully, I get paid more than $0.082 per hour, unfortunately, that means that the time to perform a fully detailed analysis may dramatically reduce the cost savings. Then again, it could be tucked away as a hidden variable…, jk.

Remember, efficiency.  “When products use more power to perform the same amount of work, they are by definition less efficient.”

Why it wasn’t that: 

Cost benefit analysis i.e, not worth the time for a similar outcome. This analysis is being done in hindsight with 20/20 vision. If I wanted to make a decision about future infrastructure changes (collocation, hybrid public cloud, on-prem datacenter expansions) with vendor purchase agreements over $100,000 for a single refresh, then the cost/benefit may be exponential enough to measure. We don’t spend that type of money on infrastructure.

What it is: 

I mean, knowing that the new hardware consumes less wattage than the prior hardware and comes with software that supports low idle usage during low load times; means that the power consumption will be lower – in theory. But remember, all you know are the stats given to you from the vendor. That doesn’t include the environment variables (seasonal electrical cost changes, systems usage changes, and random implementations or deprecation’s, nor system count changes). That sentence alone is exhaustive and makes me want to switch to cloud computing! For instance, our year over year costs per device is on an upward trend, but our overall costs for our environment is on a downward trend. How is that? 

year over year electricity cost per device

Originally, I wanted to see how much of a cost savings or increase we would see from my infrastructure decisions. It quickly became apparent that most of my savings was not because I picked super efficient and ‘green’ hardware – but I did. The primary reason for my cost savings was actually a software technology – virtualization. Yes. It has been around for a LONG TIME.

The primary savings was accomplished during the partial and full virtualization phase (2016-2017). Reducing the onsite datacenter footprint from 24 servers down to 3 primary servers. Unfortunately, some of the technologies deployed required additional power consumption, increased demand on average server usage and increased PoE demand on all switches as more devices become Powered over Ethernet. 

Eitherway, we dramatically reduced our energy consumption! Yay, us. Sorry, Entergy.

What I plan to do with this: 

Increase system efficiency, cost effectively – I thought I made that clear.

The primary bottleneck that is limiting our system throughput are the disks I/O speeds. With future analysis, we will be able to determine if SSD’s can provide us with an operational cost savings through the direct cost of electricity, infrastructure purchasing costs by consolidating one of the Hyper visors from three down to two and comparing those cost savings to varying models. I’m considering and testing costs in both a hybrid-cloud infrastructure (which adds systemic processes (lowers efficiency)) and complexes the design (lowers troubleshooting efficiency without proper training) and increases demand for professional development. All variables must be considered before making a decision on our next infrastructure initiative. 

How I plan to measure: 

  • Electrical costs can continue to be measured on an annual basis by kWh per device based on average load/usage and multiplied by the total number of ‘like’ devices in the network. 
  • Processes can be measured by taking the collective salary average and dividing it by the support hours required to maintain, monitor, and support a hybrid-cloud. 
  • In the same light, troubleshooting time can be averaged by the salary over ticket completion times for systems / infrastructure tasks. 
  • And finally, PD costs are explicit when utilizing subscription plans, boot-camps, and training materials. The hardest aspect to measure will be personal, off-the-clock training time dedicated to increasing our staff knowledge on cloud computing maintenance and troubleshooting. 

Resources:

Inspiration: https://codeascraft.com/2020/04/23/cloud-jewels-estimating-kwh-in-the-cloud/

Some research: http://www.webtorials.com/main/resource/papers/cisco/paper112/EthernetPowerStudy.pdf

Amazing vendors:

https://www.dell.com/ (PowerEdge is amazing!)

https://meraki.cisco.com/ (Built in power consumption metrics)

https://www.cisco.com/ (manual power consumption stats)

#show power inline

module available used remaining

(watts) (watts)

1 370.0  39 331




Disclosure:

  • Not affiliated with anyone / anything in this post directly.
  • Excuse grammatical issues, I’m not a writer.
  • All analysis was inspired by others with a personal directive to save the earth and increase efficiency.

2020

As always, I’m keeping this short.

2020 is here and I have focal point for the year. My optics are tuned and set on the follow list:

  • Clarity. In my mind and of others. All too often, I’m immediately responding assuming that I understand. Sometimes I don’t. It often comes off as rude or aggressive and it needs to change. This is a joint effort, but at least I will be the one to start the shift among my colleagues.
  • Project management. I enjoy organizing projects, drafting reports on progress and achieving varying levels of progress.
  • CCNP, Python, Ansible. The new CCNP is around the corner. It’s going to involve Networking, DevOps, and Automation. I’m getting it.
  • Action. There’s no waiting. Just acting. I’ll assess all opportunities that come my way, see if they fit with my 5 year and 10 year roadmap and execute on all opportunities that align.
  • Intentionally . Doing the best with full intentions of performing the best I can.

That’s it. I’m really just focusing on key skills, growing professionally as a manager and leader, and chiseling away at progress.

Cisco – The Future of Internet

On December 11th, Cisco announced the future in five categories.

  1. Silicon
  2. Optics
  3. Software
  4. Systems
  5. Architectures

1. Silicon – Referenced as the “engine to a car”, Silicon One is Cisco’s programmable silicon architecture – Q100. This transistor can handle large buffers, advanced programability and greater bandwidth!

Nerd Knob #1: 10 Tbps carrier-class capability

Finance: Drastically reduces the OpEx industry rate which sits at a 1:5 ratio

Read more here: https://blogs.cisco.com/sp/one-silicon-one-experience-multiple-roles

2. Optics – Slower interface speeds could easily cost a solution 10%. With new silicon photonics reaching 400G, the cost per bit can be driven down.

With the hardware becoming more diverse and software driven, we are now going to see an increase in cost on the speed.

Can you imagine? 400GbE connections. That’s an insane amount of data movement.

Read more here: https://blogs.cisco.com/sp/optics-fundamental-to-build-the-internet-for-the-future

3. Software – As Cisco references Silicon as the car, they reference Software as the steering wheel. Their Network Operating System (NOS) becomes an even more critical component in the future of the internet. With Cisco’s IOS XR7, come prioritization on operations. Their goal was to simplify and improve automation tasks with the overarching goal of “zero-touch”. With better efficiencies, comes more complexities. XR7 NOS allows teams to utilize the computer for insights and analytics.

Read more here: https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-xr-software/index.html

4. Systems – Continuing the reference – the car. With Cisco 8000 series routers being deployed, we can now bundle the hardware and software for limitless opportunities. Okay, maybe not limitless for long, but definitely a game changer for the immediate future.

Nerd Knob #2:

1 RU Router can support 10.8Tb/s bandwidth…

3 modular form-factor platforms delivering support from 115Tb/s > 260Tb/s

Full Fabric redundancy

Top of the line security – Hardware based “Trustworthiness” for tamper proof control and visibility controlled by Cisco Crosswork Cloud

Finance:

Reduced power consumption per Gb (4W) which is 1/4th the consumption

Read more here: https://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-xr-software/index.html

5. Architectures – Everything listed above has been re-imagined with performance, trust and OpEx in mind. By keeping all of this on track, Cisco is reinventing how the internet operates with people and business in mind.

Read more here: https://blogs.cisco.com/news/future-of-the-internet-its-here


*This post is not endorsed by Cisco, nor is it a direct reflection of their beliefs and opinions.

Reflection & Vision

When you’ve reached a peaked plateau in any aspect of your life, it’s easy to feel lost, and fogged. Especially when you’re at a peak – but your internal furnace that drives for growth and learning beyond your current levels continues to beg the question; what’s next? It’s common for people my age to want more – and more constantly. It’s often referred as greed. And the definition may be accurate with that statement. I’ll debate it later.

It’s not a bad place to be and I realize this. I’ve had mentors, father figures and friends refer to this point in life as many different things. All of which have resulted in complacency, losing businesses, or failing on their part of the deal in partnerships. That’s where I was earlier this year. I was at my turning point. A point in time where I could choose to make a change. It would either result in returning to Port, maintaining my anchor or setting sail for the high seas.

My decision? Well, none of the above. I overthink everything. I decided to come back to basics. Develop a strategy that will ensure my success – long term, expand on my foundation, and then embark on my journey.

So, I’m in the middle of developing key skills that will last me another decade in this vastly dynamic world of technology. Being a network and systems specialist is only a fraction of who I really am and even of that percent that I am, I can still be greater.

Skill #1 Systems & Networking – I’ll spend time learning Cloud infrastructure from a technical stance. Mastering understanding from a top level managerial view down to the cleanest whitepapers written by the creators of these technologies. Wikipedia is not a proper source.

Skill #2 Management – I’ll spend time learning how to encourage natural incentive and effort for a common strategic goal.

Skill #3 Leadership – being prepared and present more frequently in meetings. I’m already good at this, but I’ll be even better. Providing more follow up and follow through on projects and goals. Creating an environment of care and compassion among my peers and colleagues.

Skill #4 Entrepreneurship – I’ll work on my partnerships. Creating strategic partnerships that truly benefit both sides with sustainability in mind.

Skill #5 Personal – removing limiting thoughts and hesitation from my mental reflex. Preparing to be a father and a better husband with every opportunity.


I think it’s important to understand why I chose skills rather than certifications for my goals. Having an understanding of your desired skill will remove the limiting factors of choosing a single cert and believing that it’s all you need. In reality, I must master my skills, which may equate to five certs, 400 hours of video lessons and another 10 years of experience. The later is much more appealing and hard to beat, especially when competing against a team of nerds, a processor and code for future jobs. It also empowers me to go out, gather the best data on the subjects and study them to a mastery level – then apply what I’ve learned to be truly great.

Week 1 – Unifi for Home

It’s week one in my new house. I have Unifi powering my network with the security gateway, ubiquiti switch and AP Pro. Lots of information to come, but so far, I’m loving the Unifi dashboard and features. Especially the specifics of their automatic topology!

Unifi Ubiquiti Home network set up

Update #1 12/2/2019

Unifi for Home use is amazing when you combine the Security Gateway with any of their switches. Getting the live dashboard and alerts with their “insights” dashboard is an amazing feature. 100% worth the cost ($139).

Personal Finance

I’m not going to write about personal finance 101 because it would end up being version 10,000,000,000,000 ^365…

We’ve seen enough. Everything has been repeated and humans are humans.

I am, however, going to share my methods, which includes an excel sheet and a budgeting application.


The Excel sheet – I use it “planning” and being pro-active with my budgeting efforts. What I mean by this is, I plan purchasing a home, funding a college account for my future child, purchasing a car, or planning future investments. It’s sort of like a quick simulation.

Setting the goal, here’s how I do it – the categories:


Let the simulation(s) begin!

Just start filling it out based on a monthly view and allow excel to “summarize” it for you. Do the same for your expenses and see how much is left in the (“Net(+/-)) column at the end of the month.


On to the App! This is what everyone wants to see.

So, yeah, it does a lot – but it keeps things simple. My favorite feature about the application, aside from being able to view all accounts at once (Student loans, home loans, credit/debit, and 403b accounts), I’m able to view my “net worth” because of the vast account capabilities.

The other cool feature, is the ability to categorize. Throughout the month, I set a dollar amount budget (based on my monthly excel budget) and I categorize all expenses throughout the month. By the end of the month, I look at the budget dashboard and see where I am with my budget categories. Most of the time, I’ve over spent on restaurants or groceries, or wood working projects. I promise, one of these days, I’ll treat wood working like a business and actually make money from it!

The unspoken and indirect feature that Personal Capital offers, is the ability to reflect. I was feeling a little negative toward my finances – feeling as if I hadn’t made any progress in the last year. Personal Capital maintains record of all transactions and net worth… when I reflected on the app, I saw that I actually paid down $20,000 worth of debt, saved for a mortgage down payment, paid off my car and enjoyed an expensive vacation for a week in Florida… It instantly boosted my mood and made me realize that I am making steady progress on my financial, material and social goals.


The second App – Robinhood. If you’re investing but don’t have more than $200,000 invested in a stock trading brokerage account, then I suggest that you invest for free – in Robinhood. Seriously, you can make money instantly with their free trades. Buy Apple today, sell it in a few days and enjoy your 1-3% profit margins. If you capitalize on this technique which leverages compounding growth and the market continues to do well, you’ll do well!

Free money, create an account and start trading!


References

Message me if you would like a template or a more complete run through of anything in this post.

Quality of Service (QoS) Introduction

QoS provides predictable management of network resources during times of congestion. When a router is overloaded the memory buffers on it hit maximum capacity. The router has no other choice than to drop traffic. Congestion happens when the memory buffer is filled up on a particular interface on a router. This usually happens when traffic is being pushed passed the line rate for said cable. A router has certain memory reserved for each interface and when that memory gets full, it will drop packets. QoS gives control onto what packets can be dropped. It can also limit traffic by either policing or shaping it. Policing is the act of watching for the bandwidth of a particular stream of packets, and dropping any packets that are excess of that. Shaping is the act of watching for bandwidth of a particular stream of packets, and when the excess limit is reached, it holds the packet in memory until that interface is less congested.

During times of congestion on the network you can expect to see things like Delay, Jitter, and Drops. Delay is simply the latency for one packet to get to it’s destination. Jitter is the results of packets being received but in various time lapses. Drop is simply that the traffic had to be dropped because of the congestion.

To understand QoS, It is best to understand different switching and hardware architectures and how all these different platforms handle packets: particularly how packet is stored in memory and how those memory relate to the forwarding process.

Network Equipment is very much like a computer

Us network engineers know how to configure network equipment, analyze packets and influence the forwarding decision of those packets. However, sometimes we don’t know how the switches/routers actually do it! ‘It’ as in how switches/routers take packets and put them onto other interfaces. What is going on behind the scenes?

Switches and routers are just like a computer. They have their storage. There memory. They have a CPU. The big difference is that most network equipment have a thing called ASICs. ASICs stand for Application-specific integrated circuit – and they are really good at doing one thing and one thing only (or sometimes a subset of very specific tasks). That one thing could be looking up a MAC Address in a MAC Address Table. Another example would be looking up the routing destination for a IP Packet. Since these ASICs were made for a specific task, they perform these lookups very very quickly. In contrast, CPUs on routers/switches are much slower in there lookup. If you were to compare the two – a human could not differentiate, as the lookup on both would be similar to human perception. However, it makes a huge difference when you are handling thousands upon thousands of packets to use ASICs to make forwarding decision rather than CPUs. While a standard PC uses RAM/Memory to store the operating system, and various applications – Network Equipment use them the same way, but with a twist: they use memory to store packets ingressing and egressing the device. A network device has processes just like a computer. It runs an OS of some type, and it has processes that need to be stored into memory. Packets ingressing or leaving a network device have to be stored somewhere. That is where memory is used. There are lots of different network devices as well as alot of different hardware architecture for them. But the key take away is that memory in network devices are used for two things:

  1. For it’s own OS/processes (routing protocol, SNMP, OS, etc.) – These use CPU Resources
  2. For packets traversing the device (Packet Lookup) – These use ASIC Resources

How routers deal with a packet

Below is a high-level chronological overview of how routers deal with packets:

  • 1. Packet Arrives on ingress interface and its  placed in memory called the RX-Ring.
  • 2. Packet is then queued in the memory buffer. This is where CPU (or ASIC) takes control of that portion of memory and re classifies the memory.
  • 3. Forwarding Decision is made (routing via IP/Switching based on MAC etc.)
  • 4. Packet placed on TX-Ring. The same memory is then reclassified as TX-Ring. The outbound interface of the packet then takes control of that portion of memory. 
  • 5. Packet transmitted out egress media.

Think of RX Ring and TX Ring as the dedicated memory for that specific interface. Every port has both a RX Ring and a TX Ring. These ‘Rings’ Are completely separate from queues and buffers* More on that later. QoS has no control over the RX Ring and TX Ring. QoS has control over handling of packets and congestion from the Queues and Buffers.

Packets could be physically moved from one memory chip to another. Depending on the memory architecture of the device, the packet could be physically moved from one memory chip to another -or- simply re-classified, but not moved.

Memory Architecture

There are two types of memory architectures for switches. Shared memory and distributed memory. Shared memory essentially is one big block of memory that is used for all interfaces. The packets coming in and out are renamed and looked up by ASIC linked to that memory. A device with distributed memory has dedicated ASIC/memory for each port/a group of ports. A common shared ring that connects all the ASICs memory together tie them to other ports. Devices that use distributed memory are usually large switched chassis that have multiple line cards. Each linecard has ASICs, but they use high speed ring/bus to interconnect them all together. Below is high-level order of how packets are handled with shared/distributed memory.

How devices deal with packets (shared memory)

  • 1. Packet arrives on ingress interface
  • 2. Interface/Module ASIC forwards packet into a common shared memory pool.
  • 3. Forwarding decision is made by forwarding ASICs
  • 4. Memory ownership of packet buffer transferred to egress interface
  • 5. Packet transmitted onto the egress media

How devices deal with packets (distributed memory)

  • 1. Packets arrive on ingress interface
  • 2. Interface/Module ASIC places packet into memory (specific for port/group of ports
  • 3. Forwarding decision is made by forwarding ASIC
  • 4. Packet transmitted onto shared ring/bus to all egress interfaces
  • 5. Appropriate egress interface queues and then schedule the packet

Buffers and Queues

A Buffer is physical memory used to store packets before and after a forwarding decision is made. On a router this memory can be allocated to interfaces as ingress/egress. In a shared memory architecture, certain parts of memory are dedicated as buffers. However, that same sahred memory is used for other CPU Proccesses.

A queue is different depending on the platform. On Routers, it is a logical part of the shared memory buffer. On switches, individual interfaces/linecards have their own memory which is used as interface queues. Think of queues as the logical section of the physical memory (buffer).

Configuration of buffers is not part of QoS. Buffer configuration would involve modifying the quantity of buffers allowed for particular sized packet. QoS configuration applies to queues. With QoS you’re not modifying the quanitity of buffers allocated or a particular sized packet. Instead, you are taking existing buffers that have already been defined as interface queues and modifying how packets are treated when inside those queues. 
During times of no congestion, QoS is not needed because packets are transmitted First In First Out (FIFO) up to the line-rate of said interface. During times of congestion what happens is the queue is filled up and trying to pass traffic higher than the line-rate of the interface. 

Integrated and Differentiated Services

Integrated Services is a QoS Model in which the entire packet from end to end is ensured certain minimum QoS. Initial RFCs published by IETF in mid 1990s: 1633, 2211, 2212. RSVP is used as the primary protocol to setup the path. Requires every node along path to heed its reservation and to keep per-flow state. This type of Service for QoS did not gain much traction because it was unfeasible to implement across multiple vendors and organizations.

Differentiated Services is designed to address challenges of Integrated Serivces. These are the following RFCs: RFC 2474, 2597, 2598, 3246, 4594. The DiffServ Model Describes various behaviors to be adopted by each compliant node (called Per-Hop Behaviors(PHB)). Each device has the capability to apply QoS the way they want with whatever method they choose fit. With Integrated Services it was guaranteed that each packet had end to end guarantee of QoS. With Differentiated Services, there is no guarantee and each device can or may not be configured with QoS.

Classification/Marking

Traffic first must be divided into “classes”. A Class of traffic will receive the same type of QoS treatment. It analyzes the packets to differentiate flows. Packets are marked so that analysis happens only a limited number of times, usually at the ingress edge of a network. Usually this starts as a business decision and the business needs for the network. The whole idea behind classification is to identify traffic in your network that is critical to operation and quality of your buisness. After identifying what traffic is important, you can create rules to match that traffic – and mark them for QoS. Most ISPs will police ingress traffic. Traffic that is non-conforming (higher then the CIR) will be either dropped or marked down. Customers obviously don’t want any type of traffic drops, so shaping done on the egress interface leading to your ISP is recommended. 

Queuing When egress traffic cannot immediately be transmitted (aka on the TX Ring), it is placed in an egress queue. A single egress interface may have multiple associated egress queues differentiated by priority. QoS features designed for queuing provide control over which classified traffic is placed into each of these queues. Queueing can also preemptively drop traffic from within queues to make room for higher priority traffic. 

Scheduling

Scheduling is defining what packets are put on the wire depending on their priority. On routers, QoS queuing features such as WFQ affect queuing and scheduling behaviors. On switches, queuing and scheduling can be separate features. Traffic shaping is a function of scheduling. 

Congestion Management

Congestion management features allow you to control congestion by determining the order in which packets are sent out an interface based on priorities assigned to those packets. Below is high-level overview of congestion management process:

  • Creation of queues
  • Assignment of packets to those queues based on the classification of the packet
  • Selectively dropping packets from within queues when those queues reach pre-defined thresholds
  • Scheduling of the packets in queue for transmission

Features for Congestion Management: WFQ, CBWFQ, PQ, LLQ, WRR, and SRR

Traffic Shaping Features of Congestion Avoidance: RED, WRED, WTD, and Policing

Modular QoS Command-Line (MQC)

MQC allows QoS features that apply classification, policing etc to be configured independently and then linked together as needed. Similar to Modular Policy Framework (MPF) in ASA. MQC utilizes class maps, policy maps, and service policies. 

  • Class-maps are used to identify and classify traffic that you want to identify for QoS. Class-maps can reference ACLs to classify traffic, for example.
  • Policy-maps define what you want to do to the traffic. Each policy map can reference multiple class-maps. When you enter more than one class-map, it is done in chronological kind of like an ACL. Policy-maps apply things like policing, shaping based on your class-maps that you created.
  • Service policy is used then to apply the policy-map to a particular interface in a particular direction. 

Defending Against RYUK

Computer code on a screen with a skull representing a computer virus / malware attack.

It has been exactly four weeks since Homeland Security, the National Guard and LA DoE scheduled an emergency phone conference with all Technology Directors in the state of Louisiana.

During this briefing, we were informed that 6 school districts and 2 government agencies were attacked by a ransomware known as RYUK. The immediate reaction was frightening as the governor of Louisiana demanded a state of emergency. We were told to shut down internet access and remove local admin rights until further notice.

Keep in mind, we were two weeks out from the start of school (smart timing on RYUK). We had to finish deployments for hundreds of chrome books, projector installations, finalize surveillance installs and manage several other projects in our department.

A day passed before we received a strategic game plan from Homeland Security that detailed several phases of security implementations. Phase 1, turn off all internet access. This can be hard to do when your trying to deploy devices, run updates and have 150 staff members coming back to campus…

I’ll explain the technologies and how everything works later in the blog.

We spent a week tightening up the ship, blocking internet access based on firewall rules, attempting to have offsite backups work, deploying devices, installing software… we were extremely reliant on the internet.

Services were breaking constantly, as expected when you turn off the internet (LOL, if I don’t laugh, I’m crying). My boss could see the stress on our department and offered full support to us while we navigated these high seas. I have to say, I have one of the most supportive bosses in the world (Shout out)!

She granted the additional resources necessary to tackle this oncoming storm.

Four weeks later, 600+ hours between two employees, we now have all systems patched, removed local admin, wiped and deployed. In addition, all members of our organization have been trained on identifying phishing attacks (for your reference). And the entire network is locked down according to recommendations made by Homeland Security.


The Technical


Known threats to block

deny any any 84.146.54.187/32
deny any any 75.147.173.236/32
deny any any 218.16.120.253/32
deny any any 170.238.117.187/32
deny any any 195.123.237.129/32
deny any any 194.5.250.123/32
deny any any 85.204.116.158/32
deny any any 31.184.254.18/32
deny any any 186.10.243.70/32
deny any any 104.20.209.21/32
deny any any 445
deny any any 447
deny any any 449
deny any any 8082
deny any any 16993

They have identified RDP (3389) and Email (80/443) as the two primary vectors of initiation.


How we “turned off” the internet

Using the firewall “deny any any” and manually adding 40+ pages of “trusted” ip addresses was not an option for us. It was extremely time consuming and impractical. I often fat-fingered IP and port numbers. I broke everything. I wish Meraki allowed me to use a CLI for this type of task. Luckily, Meraki had a second option for us.

Meraki offers Content Filtering, which allows you to blacklist everything (*) and whitelist URL’s. I chose this option. Upon blacklisting the entire internet with (*), I was then able to whitelist common sites much more efficiently.

Anything that ends with .gov and .edu were whitelisted, but not completely. Aside from these, every other site had to be whitelisted. Aside from the constant adding, this process is very easy.

All traffic is triple filtered with the leading Cisco, Google, and Meraki products in the globe. With dual content filtering, IPS/IDS and AMP screening, our traffic has been relatively clean – to say the least.

When it comes to Meraki, we were also able to filter traffic by country. This allowed us to block traffic from random countries that we have no business communicating with/through.

Anti-virus

We commissioned a new AI based product to help protect all of our servers, faculty and staff. Hoping that their spread of knowledge with the recent attacks will help prevent attacks on our network.

Advanced email filtering & quarantines

Google allows for us to enable advanced email filtering and quarantine. I’ve enabled all features to flag suspicious emails and I’ve personally trained every employee on proper email usage and what to look for in an email.


As of today, we are not in the clear, but we are in a much better state now than we were a month ago. We were given the chance to reflect on our current policies, enforce new procedures and tighten up security campus wide. Other organizations were not given the same opportunity as us.

For anyone out there battling this, please reach out if you need support. This is a beast to navigate and cyber crimes are not going away anytime soon.


References

Center for Internet Security (Homeland Security)

Read about Protecting your network

Read about Emotet Malware

Read about TrickBot

Border Gateway Protocol (BGP) Fundamentals

Introduction

BGP is the premier routing protocol that runs on the internet. It is used by many (if not all) Internet Providers across the globe. BGP is designed as a Exterior Gateway Protocol (EGP). BGP is actually the only EGP that is standardized across the internet. In earlier years, EGP (not to be confused with the category EGP as previously stated) was the first routing protocol developed to communicate network reachability between two Autonomous Systems. BGP was developed as an extension of EGP, improving upon it. BGP is defined in RFC 1771/4271. The other category of routing protocols is an Interior Gateway Protocol (IGP). This includes protocols such as OSPF, EIGRP, and RIP. IGPs are meant to be run within a single Autonomous System. However for BGP, it is meant to be run between two Autonomous Systems.

BGP has a best-path algorithm to determine the best route for a particular destination. A total of up to 14 checks for each route could be learned from BGP to determine what is the best path for a given prefix. In contrast, IGPs really only use AD and Cost to determine the best path for a given destination. In this way BGP is very flexible in influencing its route selection. BGP by default does not do any type of load balancing. BGP advertises prefixes/length – otherwise known as Network Layer Reachability Information (NLRI). The Term NLRI is used within the protocol to describe certain prefixes.

IGPs comparison with BGP

BGP needs to form a neighbor relationship just like IGPs. However, BGP neighbors must be configured statically. There is no way to dynamically learn of neighbor in BGP. BGP advertises prefixes just like other IGPs, and also advertises the next hop for those prefixes. Another interesting thing about BGP is that neighbors do not have to be directly connected with each other. Two routers running BGP can form a neighbor relationship across multiple subnets. All BGP communications with its neighbor use unicast TCP packets on port 179. This is a big difference with most IGPs because IGPs use multicast packets to dynamically learn of and advertise subnets. BGP advertise things called Path-Attributes for each prefix/length to its neighbors so that the routers can make a best-path selection. In comparison, IGPs have to advertise their metric/cost. BGP uses Path Vector Logic, that is similar to IGPs running Distance Vector. BGP emphasizes scalability in its design. It is not nearly as fast compared to IGPs. But it was not designed for that. BGP was designed for mass scale routing across the internet.

iBGP and eBGP

There are two types of neighbors in BGP: Internal BGP (iBGP) or External (eBGP) neighbors. When two neighbors are in the same Autonomous System they are considered iBGP neighbors, while if two neighbors are in different Autonomous Systems they are considered eBGP neighbors. BGP behaves differently in several ways depending if it is a iBGP neighbor or eBGP neighbor. In addition, the neighborship requirements are different for routers wanting to be iBGP/eBGP neighbors. When BGP sends prefix updates to its neighbor it updates the AS Path Attribute depending on what type neighbor it is sending the update to. When a router is sending a prefix to a iBGP neighbor, it does not update the AS Path Attribute because the Autonomous System number is the same between the two neighbors. However for eBGP it updates the AS Path Attribute because it is moving from one Autonomous System to another Autonomous System.

The AS_Path attribute in BGP essentially tells the router receiving a BGP update what Autonomous System the updates went to before getting received by said router. The reason eBGP updates the AS path attribute is because eBGP neighbors are not in the same AS, so they update it to reflect what AS it’s going to. When a BGP router is modifying the AS Path to send to another eBGP neighbor, it adds that AS path (aka the latest) in the front of the list (aka on the left). So if you see a route that says : x.x.x.x/24 23 4000 56 702, the last time that route got an update was through AS 23. The next AS ‘hop’ for the update is 4000 and so on.

Autonomous System

We have mentioned Autonomous Systems but haven’t given them much attention to them. So what is an Autonomous System? An Autonomous System is a single organizational unit that administers and controls the networks related to said entity. An example would be the IT organization for a e-commerce website. Every company has it’s own network that it administers, and thats what a Autonomous System is. In regards to configuration, an Autonomous System is simply a number in BGP. For the rest of the article Autonomous System/Autonomous System number will be abbreviated to AS/ASN. AS numbers were first identified as 16-bit intergers. However it was then extended to a 32-bit interger in RFC 4893. There are a few ways to write the number (hexadecimal, asplain, or asdot).

There are two kinds of AS Numbers: Public and Private

  • Public AS number can be advertised over the internet.
  • Private AS number are not advertised over the internet. Can only be internally used.

The ranges of Public and Private AS Numbers:

  • Public: 1-64495, 131072-4199999999
  • Private: 65512-65534, 4200000000-4294967294

All other numbers in the 0 to 4294967295 range are reserved.

BGP Neighborship

! Start BGP with configuring the ASN

#router bgp [ASN]

! Configure a statically defined neighbor, and specify the remote ASN that the neighbor has

#neighbor [ip address] remote-as [asn]

To complete a neighbor relationship this has to be configured on both sides of the link.

Requirements to form a BGP neighborship:

  • The local routers ASN must match the neighboring routers reference to the ASN with the neighbor remote-asn command
  • The peers IP Address must be reachable via Connected, static or IGP route.
  • The BGP Router IDs must not be the same between the two neighbors. BGP elects a router ID in similar fashion to other IGPs: 1. Use Setting from router-id command 2.Choose highest numeric IP on loopback interface 3. Choose the highest numeric IP Address on any non loopback interface.
  • If configured, MD5 authentication must pass. This can be configured via the neighbor [ip address] password [key] command.
  • Each router must be able to complete a TCP 3-way handshake with the BGP Peer.
  • The source IP address used to reach that peer must match the peers BGP neighbor command.

When using the neighbor remote-as command, the source address is going to be the interface of wherever that route is pointing to. For redundancy purposes you can change the source interface of the BGP packet to something like a loopback. Changing it to a loopback interface makes it more redundant because it does not rely on an interface to be up to form a neighbor relationship. You can also have two neighbor statements going to the same router, one going to one link and the other link going to another link (different IPs, so there will be two neighbor statements). This will consume double the memory and CPU utilization on each router because even though the router has neighborship with the same box, it will receive the routes on both links. 

When a rotuer is trying to form an eBGP neighbor relationship, by default all eBGP messages have a TTL of 1. You can disable this using the neighbor [ip address] ebgp-multihop command. This command changes the TTL from 1 to 255. To change the source interface of BGP packets use the neighbor [ip address] update-source [interface].

! Configure an eBGP neighbor for multihop (increases TTL)

#neighbor [ip address] ebgp-multihop 

! Force a router to use its source address for BGP packets to use the specified interface

#neighbor [ip address] update-source [interface]

! Verify 

#show ip bgp summary

iBGP vs eBGP Neighborship Differences

The only difference between iBGP and eBGP neighbors is that iBGP neighbors have the same ASN between the two routers connecting each other. eBGP neighbors have different ASN numbers connecting each other The other difference is that the TTL value for iBGP neighbors is 255 by default. With eBGP, the TTL by default was 1 and needed to be changed to higher number so that it can communicate with routers multiple hops away. The configuration between an iBGP and eBGP relationship is the same.

BGP Neighbor States

There are various states that BGP goes through when forming a neighbor relationship with another BGP router. These states are the following:

  • Idle – The BGP process is either administratively down or awaiting the next retry attempt. 
  • Connect – The BGP process is waiting for the TCP connection to be completed. During this state the BGP router is actively trying to start a TCP session with the other neighbor. The connect-retry timer is started during this stage. If the connect-retry timer hits 0, and the TCP session was never able to finish, then the neighbor state will move to Active.
  • Active – The TCP connection failed during the Connect state, the connect-retry timer is started again, only this time it is passively listening for incoming TCP connection. The connection-re-try timer is a timer that specifies how long the BGP neighbor will try to establish a TCP session, and once the timer is reached during the connect state, the BGP routers stop trying to actively make a TCP session. During the active state, the router passively listens for incoming TCP messages. However, this implementation is based on the router/manufacturer. Ultimately the Active State means that the TCP 3-way handshake failed.
  • Opensent – The TCP connection exists, and a BGP Open Messages has been sent to the peer but the matching Open Messages has not yet been received from the other router. 
  • Openconfirm – An Open message has been both sent to and received from the other router.
  • Established – All neighbors parameters match. The neighbor relationship works, and the peers can now exchange Update messages.

BGP Message Types

Every header of a BGP packet is the same. BGP messages are carried inside a TCP/IP header.  It contains marker, length and type field. Marker field contains authentication if configured. If not it is all 1s. Type field contains a number to identify if it is a open, update, keepalive or notification message. 

BGP uses four (4) types of emssages:

  • Open
  • Update
  • Keepalive
  • Notification

BGP Open Message

  • Used in neighborship establishment
  • BGP values and capabilities exchanged

BGP Update Message

  • Informs neighbors about withdrawn routes, changed routes, and new routes
  • Used to exchange PAs (Path Attributes) and the associated prefix-length (NLRI) that use those attributes

TLV stands for Type Length Value. The TLV value is a number that tells you what type of path attribute is following. NLRI stands for Network Layer Reachability.  Since Path Attributes, and Withdraw routes field can vary in size they are accompanied each by a length field to specify how big they are.

BGP Notification Message

  • Used to signal a BGP error
  • Typically results in reset of neighbor relationship

BGP Keepalive Message

  • Sent on a periodic basis to maintain the neighbor relationship. The lack of receipt of a keepalive message within the negotiated hold time causes BGP to bring down the neighbor connection.
  • Only contains the BGP Header

BGP Table & Path Attributes

BGP has a table that it stores and keeps all of its routes. It is called the BGP table. You can view the table by issuing show ip bgp. The output will list all the BGP learned routes (locally injected plus learned routes). This command will only show a high level view of the table and not the details of each entry. 

The output of show ip bgp displays a high level overview of all the routes learned via BGP. To the left of the Network Column there are various codes to help identify the route:

  • * – Means it is a valid route and can be installed in the routing table
  • > – The best route BGP has discovered for that specific prefix
  • r – Failure to put this prefix in the IP routing table (Better route already in routing table, Routing table is maxed (memory is full), VRF routing table limit succeeded)
  • i – Learned about this prefix from a iBGP neighbor

A next hop of 0.0.0.0 means that the local router advertises this either via network or redistribution command. The Path Column shows the AS path that the particular prefix was learned from. A ? means that the prefix was locally learned within the routers AS. 

! Verify BGP Learned Routes

#show ip bgp [prefix/subnet]

#show ip bgp neighbors [ip address] received-routes

#show ip bgp neighbors [ip address] routes

#show ip bgp neighbors [ip address] advertised-routes

#show ip bgp summary

BGP uses multiple path attributes to determine best path for a certain prefix. By default, if no BGP PAs have been explicitly set, BGP routers use the BGP AS_PATH (autonomous system path) PA when choosing the best route among many competing routes. The AS_PATH attribute is also used to prevent routing loops. If a router receives a BGP Update and the AS_PATH (or AS_SET) has an autonomous number that is the same as its own, it will drop it. AS_SEQ is a component of the AS_PATH attribute also. The AS_SEQ is simply the list of ASs a BGP prefix goes through in order. When route summarization is performed on routes coming from multiple ASs, then something called an AS_SET is used. AS_SET is simply all the ASes that are in that summarization. However, since it cannot decipher the order it just lists them out in brackets like so {6 8 2 5}.

Injecting Routes into BGP

There are three (3) ways to inject routes into BGP:

  • By using the BGP network command
  • By using redistribution
  • By using route summarization

The network command for BGP is different than IGPs. It does not “turn on” BGP on an interface, nor does it allow for dynamic neighborship of BGP on interface (BGP has to have static neighbors anyways). It also doesn’t allow hellos on the interface (BGP uses keepalives). The network command in BGP looks for the exact prefix/length matches in the IP routing table, and originates that prefix/length into the BGP table. It does not matter if it is a directly connected, static, or IGP route. Aslong as the route lives in the routing table and it is not a BGP route, the network command will take that route and convert it into BGP. 

! To inject a route into BGP, use the following command in BGP config mode. The mask is optional. If the mask is omitted then the router assumes a classful boundary. 

#network [subnet] mask [mask]

There is also the auto-summary command in BGP. The auto-summary command does not affect any network commands with the mask command included. The specific mask specified for the prefix will look into the routing table and advertise only that specific prefix/length. If the mask command is ommited, then the auto-summary command will advertise the classful route.

The classful route is added if:

  • The exact classful route is in the routing table

AND

  • Any subset routes of that classful network are in the routing table

The second way to inject routes into BGP is by using redistribution command in BGP router config mode. This essentially does the same thing as the network command however it has the option of injecting alot more at once.

! Configure redistribution in BGP router config mode

#redistribute [static|ospf|eigrp|rip|connected]

This command has many other options like implementing route-maps and metrics. However, that is out of the scope for this article.

The third way to add routes into BGP is by using summarization. This aggregates several smaller subnets into a larger subnet and advertised out as one prefix rather than multiple individual ones.

! Configure the prefix to be sent out as a BGP Update with accompanying length

#aggregate-address [prefix] [prefix-length] [summary-only]

If you do not specify the summary-only command then BGP will advertise the summarized routes and the specific routes. Specifying summary-only only advertises the summary routes to its neighbor. This command has to be accompanied by a matching network or redistribute command to successfully send the summary. Applying this command alone will not create the route even if it is in your routing table. 

BGP Advertising

BGP has two rules for advertising routes to its peers:

  • Only advertise the best route in any BGP Update (BGP will never send an update with two possible next hops)
  • Do not advertise iBGP learned routes to iBGP Peers

By default a router running BGP will only send networks it originates to its neighboring iBGP router. Once the neighboring router receives those networks, it will not send it on to other iBGP neighbors. The reason is to prevent routing loops. When routes are advertised to iBGP neighbors, the AS_PATH attribute remains the same (thus BGP identifies it as a loop). So by default iBGP neighbors don’t send non-locally generated routes to other iBGP neighbors. This behavior can be changed with configuration, however.

When BGP advertises a prefix to an eBGP neighbor, the next hop IP address is changed by the advertising router. However, when iBGP advertises a prefix to an iBGP neighbor, the next hop IP address is not changed (this behavior is configurable/can be changed). Routes learned from eBGP neighbors can pass through multiple iBGP neighbors. However, since they pass through iBGP neighbors the next hop does not change. This can cause issues because since the next-hop IP address is not changed, routers receiving it may or may not have IP reachability to the next hop IP address advertised. Everytime a BGP update is received on a BGP Router (iBGP or eBGP) BGP will look into its IP routing table and see if the next-hop IP address is reachable. If it is not it will not install that BGP route into the routing table. 

If a router running BGP receives an update from an iBGP neighbor, and the next hop IP address is not reachable then:

  • iBGP-learned routes will not be installed in IP Routing Table
  • iBGP-learned routes will not be advertised to any other BGP Peers
  • Viewable via the show ip bgp prefix/length command as inaccessible

There are a few ways to resolve this issue:

  • Advertise those IP addresses into the internal network (static route, IGP)
  • Use the neighbor next-hop-self command

The neighbor next-hop-self command changes the next-hop IP address to the source address of the neighbor statement you have with your iBGP neighbor. By default, as stated previously, when iBGP neighbors send updates the next-hop IP address is unchanged. This command forces it to change to the source address of the neighbor interface. 

! Configure a iBGP neighbor to send the next-hop IP address of it’s source interface of neighbor relationship in the update message 

#neighbor [IP] next-hop-self