Duplicate IP! – 5.27.2019

Well, I’m dealing with this on my night off. 42 of my Meraki access points are yelling and complaining like a bunch of kids shopping with their mommy during a hot summer day about not finding home.

Yeah, I mean, I’m upset too.

I drove 45 minutes to work (yep, I commute)… Upon arrival, I decided to get my priorities straight, so I started Spotify and played by favorite playlist (lots of hip-hop) of aggressive music.

I then started to TSHOOT by logging into Meraki > Wireless > Monitor > Access Points where I confirmed if any errors were still populating. They were.

I immediately decided that I needed to verify if I added/removed any devices from my network by matching up the dates from when the alerting started and my ticket queue. We decommissioned a few network devices, but we made zero network changes.

Phase II, I RDP’d into my DHCP and DNS server to validate the AP IP addresses. All checked out. I then reviewed DHCP for any “Bad Addresses”. I had 50+ “Bad Addresses”… Yeah, that’s an issue. They were all on the same VLAN (20) that Meraki was claiming DHCP failures on (5/5 transmit failures on VLAN 20).

Okay, so I deleted the “Bad Addresses” since nobody was on campus just to see if we had a stuck entry or caching issue. Most of the IP entries did not come back online. Great. Moving on.

Phase III, I panned over to my DNS server. Wow, okay, I have a lot of clean up that I need to do… PTR entries from 2016!! Okay, I’ll delete most of those entries (since I knew that they were not needed). Checked AP status, we’re almost there, I’m starting to see AP’s come online.

I then decide to go back to DHCP and refresh the lists to see if any entries have been updated. Welp, there she was… ap0016xx.domain.com with a VLAN 20 ip address… I don’t know about you, but I don’t put my access points on access vlans. AP’s belong on the network VLANs.

I take the device name and search Meraki, bing! It pops up immediately with a conflicting IP address! I trace the source port and disable the switchport. The AP goes offline. I refresh my Meraki dashboard and continue to delete the remaining “Bad Addresses” from my DHCP.

Success! All AP’s are online.

I then, physically, traced down the rogue AP in my environment and found that it was coming from our intern VLAN with a DHCP and Print server on it… The dated DNS records was giving our intern server an old Cisco AP name! Several things happened here that could have prevented this issue, however, it was a great reminder that we must stick to our “Maintenance” schedules and keep our network as clean as possible with regular updates and checks of all systems.


Resource Credits:

CyberSec & Fraud – 5.22.2019

I attended a Cyber Security and Fraud conference today with special agent Eric in the white collar crimes division in New Orleans. It was very insightful to learn about our immediate attacks and pressures.

To start, there are countries are countries with written agendas to target countries like ours (USA). With bankers tightening their security and protocols for handling their processes, its now forcing the hackers and criminals to move their operations directly to the client(s). It’s great that banking institutions are cracking down on security (passwords, encrypted communications, malware/adware/ransomware detectors and scanners and phishing simulations).

With the immediate threat coming from foreign internationals, security agencies like the CIA, FBI and local police need to move quickly to identify and target these criminals. $350,000,000 was criminally taken through the banking system (I believe only in Louisiana), with 76% of which being recovered. That’s a great number recovered! At the same time, there’s a lot of money that was not recovered!

The FBI set up https://www.ic3.gov/default.aspx for community members to report suspected criminal activity. The quicker you report, the quicker they can deploy their task forces to combat the criminals.


Don’t let this overwhelm you, there are preventative measures that you can take to help combat these issues. One, keep all systems patched and updated – especially Windows environments. Two, training is the second best effort to protecting your data. The majority of criminals are allowed access (Directly/Indirectly) through email spoofing and spearheading. This means that they breach your account or a vendor and then monitor the accounts for months. Once they feel that they are ready, they are then able to “mimic” your rhetoric and attack others in your contacts list.


Image credit: https://www.pymnts.com/news/security-and-risk/2018/scams-bec-government-sfc-fbi-washington-dc/

CCNP Ch.1 – 5.19.2019

Today, I’m starting my journey for the CCNP v2 R&S.

I’m learning about the different Routing connections (Building Access, Building Distribution, Campus backbones, etc..). I’m glad to know that my campus is actually set up like their suggestions, with the exception of two buildings.

Knowing more, I now see where I may add a building distribution switch to limit the number of fiber connections running back to the core, but also increase feasibility of troubleshooting fewer switches in the long run.


Topics that I need to remember or work on the most:

  • Routing Protocols
    • RIP – Distance-Vector
    • EIGRP (Advanced) Distance-Vector
    • OSPF – Link-State
    • IS-IS – Link-State
    • BGP – Path-Vector

All routing protocols are currently IGP (Interior Gateway Protocols) except for BGP, which is an EGP (Exterior Gateway Protocol).

The second topic that I need to focus on is Split-Horizon and Poison Reverse.

Split-Horizon is the feature that prevents a route learned on one interface from being advertised through the same interface again. (CH.1)

The Poison Reverse feature causes a route received on the same interface to be advertised back out, however, it uses a metric of “infinite“.

The third important technology emphasized in this chapter were the different network traffic types.

  • Unicast – One to One
  • Broadcast – One to Many
  • Multicast – One to Many, but specific
  • Anycast – IPv6 only, assigned to multiple devices for One to Nearest

Reference:

Official Cert Guide by Kevin Wallace, CCIE No. 7945 for CCNP ROUTE 300-101

Cloud My Lab – 5.19.2019

Okay, I have to say, I’m really enjoying “Cloud my Lab”. They finally got my instance (Pod) up and running about 72 hours after my payment processed.

To get into the server, all I had to do was RDP in using my Windows RDP client and the provided IP and user credentials. Once I was in, I had all of the images pre-loaded and GNS3 configured for my first project.

For $30, I have to say, it’s totally worth it! Sure, it’s a convenience fee, but their technical staff stand ready to help with any trouble that I have.

In addition, I don’t have to worry about finding the best ios files and go through the hassle of uploading them. Also, this environment can be operated from a Chromebook RDP window app… that’s pretty convenient! I’ll create more posts later as I build out my lab environments and test additional features.

Edit:

I found out today that my instance, “Pod”, only has 4GB of RAM while my subscription is currently set to “Tiny” which supports 8GB of RAM… I decided to upgrade to a “Small” instance because I noticed a little lag when I launched my text editor “Atom.io”, after doing so, I checked the CPU and RAM from the system properties and noticed that I was not getting the level of service that I paid for during my original subscription period. I’ve contacted their support and they are working diligently to resolve my issue.

I’m very excited to have the full 8vCPU’s and 16GB of RAM! I may even use this system for remote testing VPNs and ICMP from outside of my network.

Edit #2: This is what matters…

I was completely wrong about the configuration and setup over there at Cloud My Lab. After discovering that my host machine only had 4GB of RAM allocated, I contacted support to get it fixed. With the $30/mnth “Tiny” package, you should be getting 8GB RAM. Each time I started a text editor or web browwser, the CPU and RAM would spike! So I was a little frustrated.

After communicating with support, they explained that the GNS3 hosted instance gets the 8GB RAM remotely and that the Windows Host that you RDP into only gets the 4GB… This made a lot of sense once it was explained. It certainly explained the reason for the Windows Host maxing resources while the GNS3 Host continued to respond perfectly fine.

Ansible (Red Hat) – 5.18.2019

Today, I’m installing Ansible on my Mac…

For Mac, you need to install pip, which is done by following the instructions provided by Ansible, which didn’t work for me, so I found an easier command:

$ sudo easy_install pip

With that command, I was able to make it to the next phase… which didn’t work. At this point, I was very frustrated! After deciding to read all of the instructions on Ansible, I found a paragraph regarding MacOS 10.9 . Well, I was running 10.14. It obviously inherited its bitchiness from 10.9. So, I ran the command:

$ CFLAGS=-Qunused-arguments CPPFLAGS=-Qunused-arguments pip install --user ansible

All problems were solved. At this point, I’m considering a degree in Linux based systems so that I can actually understand this shit. But no, I’m going to continue my path toward the #CCIE and #CCAr!


Credit: Ansible for making an amazing product that’s openSource, Apple for making amazing hardware with very low specs, and Cisco for being amazing.

Carpal Tunnel & RSI – 5.17.2019

Well, today is one year too late. I really feel like I’ve been abusing myself by not taking care of my RSI (Repetitive Strain Injury), which has now formed into carpal tunnel.

Between my daily commute where I put pressure on the wrist, the 8+ hour work days where I hold my mouse improperly, and the additional time on the computer at home has taken a toll on my right arm and wrist.

Now of course, this is a self diagnosis. I have not spoken to a trained professional, except for my family friend who is a licensed chiropractor. She has guided me for several months, reducing the strain on my wrist nearly 80%! This is huge. However, it’s still unbearable. I’m currently scheduling doctor visits with our local “Hand Center of Louisiana” to get my wrist checked out.

I’m feeling nervous, but less than I was eight months ago. I decided to postpone this procedure because I thought that it would take several months of recovery. After speaking with a colleague , it’s now apparent that I will be able to recover within two hours, functional recovery in four days and complete recovery within three months. That’s not bad.

If you’re experiencing pains in your wrist please consider getting ergonomic equipment such as a standing desk, vertical mouse, and standing desk! If you do not want to stand then invest in a proper chair that can support your body type with the option of adjusting between 90 and 130 degrees to allow for proper blood flow.

Seriously, take care of yourself. Invest in proper equipment, good food, and study materials!

Edit: I visited a doctor to undergo extended rehab. We identified about eight (8) areas of concern that have all lead to the issues in my arm and wrist. Seriously, starting in my ankles, progressing through the hips and through the spine… Please, spend the time to take care of yourself with daily stretching and ergonomic checkups. Doing this can help you avoid serious irreversible damage to your body.

I oofed… – 5.10.2019

Today, I started the day off with an oof…

Picture this, Friday morning, I’m in the office early (7am) to start work on some configs before the campus started filling up with Admin, Faculty and students. I grab my coffee, sat down at your desk and logged my PC.

I was feeling confident, started studying my CCNP, felt like I knew my environment like it was the back of my hand, so I launched SuperPutty, configured my sessions (10.2.2.75) over SSHv2. I proceed to show my current interface status:

#show interface status
From here, I saw that there were several updates that needed to be done.

Performing the show interface status command showed that I had several things that needed to be updated. To list a few: out of date descriptions, port-channel groups on non-redundant switchports, dated hostname, and various open ports that should be disabled.

So, I address the quick and dirty: renamed switchport descriptions, disabled unused ports, changed down ports to an arbitrary VLAN # in the event that it did come back online, and fixed my “archive” link that automatically backed up my config to a local tftp server. I know, I should be using SFTP, but I don’t know how to! Teach me.

Moving forward, I wanted to tackle the port-channels. I log into both switches (Core and the adjacent switch) and proceed to remove the fiber uplinks from the assigned channel-groups. Well, I started with the core…

That was a mistake, it completely threw my switches into error state. It’s now 8AM, students are rolling in and teachers are starting to check students into our Student Information System (SIS).

Then, all phones go offline, my primary DC (domain controller), goes offline, printers aren’t working and several people are calling me to tell me that the internet is not working!

It’s because I blocked Google. ahahaha, Just kidding.

I quickly realized that the switchports originally configured are now in errdisable state! I oofed! First off, I should have waited until the end of the day to make these changes so that this sort of thing can be mitigated without causing a service outage. Second off, I needed to be in two places at once so that I could cycle both ports, prevent loops, and then verify that they were both up and running but I was alone. My colleague was still on his way in for the day.

After several minutes of frustration, running back and forth on an 18-acre campus, I finally did it. I cycled the ports, waited two minutes for the entire core to come online (I did a full restart) – we were finally online. I then had to test the DC to make sure it was happy, pinged the phone system and validated that the printers were operational again.

We were back online after 15 minutes of outage due to my stupidity. Reflecting on my ID-10-T error, I realized that I should have waited until the end of the day, notified my head master that I wanted to make the changes, scheduled the change and then proceeded to a planned change. Remember to stay humble and grounded. You can prevent these mistakes.

Lesson learned.