Well, I’m dealing with this on my night off. 42 of my Meraki access points are yelling and complaining like a bunch of kids shopping with their mommy during a hot summer day about not finding home.
Yeah, I mean, I’m upset too.
I drove 45 minutes to work (yep, I commute)… Upon arrival, I decided to get my priorities straight, so I started Spotify and played by favorite playlist (lots of hip-hop) of aggressive music.
I then started to TSHOOT by logging into Meraki > Wireless > Monitor > Access Points where I confirmed if any errors were still populating. They were.
I immediately decided that I needed to verify if I added/removed any devices from my network by matching up the dates from when the alerting started and my ticket queue. We decommissioned a few network devices, but we made zero network changes.
Phase II, I RDP’d into my DHCP and DNS server to validate the AP IP addresses. All checked out. I then reviewed DHCP for any “Bad Addresses”. I had 50+ “Bad Addresses”… Yeah, that’s an issue. They were all on the same VLAN (20) that Meraki was claiming DHCP failures on (5/5 transmit failures on VLAN 20).
Okay, so I deleted the “Bad Addresses” since nobody was on campus just to see if we had a stuck entry or caching issue. Most of the IP entries did not come back online. Great. Moving on.
Phase III, I panned over to my DNS server. Wow, okay, I have a lot of clean up that I need to do… PTR entries from 2016!! Okay, I’ll delete most of those entries (since I knew that they were not needed). Checked AP status, we’re almost there, I’m starting to see AP’s come online.
I then decide to go back to DHCP and refresh the lists to see if any entries have been updated. Welp, there she was… ap0016xx.domain.com with a VLAN 20 ip address… I don’t know about you, but I don’t put my access points on access vlans. AP’s belong on the network VLANs.
I take the device name and search Meraki, bing! It pops up immediately with a conflicting IP address! I trace the source port and disable the switchport. The AP goes offline. I refresh my Meraki dashboard and continue to delete the remaining “Bad Addresses” from my DHCP.
Success! All AP’s are online.
I then, physically, traced down the rogue AP in my environment and found that it was coming from our intern VLAN with a DHCP and Print server on it… The dated DNS records was giving our intern server an old Cisco AP name! Several things happened here that could have prevented this issue, however, it was a great reminder that we must stick to our “Maintenance” schedules and keep our network as clean as possible with regular updates and checks of all systems.