Circuit Troubleshooting
Where do you submit City tickets?
The City Communications portal - Creatio
Let your lead know if you do not have access!
What do you do when a circuit is alerting down in PRTG but showing up in Meraki?
Check the device IP address in PRTG to ensure it matches Meraki
What do you do when an HQ goes down?
We don't handle HQ outages! The network team will be paged immediately and they will take care of the problem.
When do you post an outage in On-call reports?
Only WAN/LAN outages
How do you handle the ticket queue while you are on-call?
Distribute non on-call tickets to leads for the appropriate region based on the ticket, assign all On-Call tickets to yourself
What do you do if WAN1/The primary circuit goes down?
Reach out to local team to have circuit modem rebooted and have connections checked. If circuit is still down open a ticket with City.
When do you pause an alert in PRTG?
Never pause an alert! Always use Acknowledge in PRTG.
If you must pause never pause indefinitely.
What do you do if a site is online but a switch is not coming up?
Check switch config against template (if pre-template ask a senior/network team what config is). Ensure proper connections with MX/internet circuit. Rarely you may need to factory reset the switch
When do you post an outage in technology alerts?
Only LAN/full site outages
What is the first thing to have the local team do if the gates or rover is down?
Have them reboot the RaspberryPi
What do you do when a Pepwave goes down?
Power cycle with local team, If still down have you or someone with access check the status in InControl, and then create a ticket with City for the Pepwave
What do you do when a UPS down ticket comes through?
Reboot UPS, check the status through the UPS web portal, have a local contact check the display on the front of the UPS.
What do you do if no one can connect to WiFi?
Check the RADIUS server through Meraki and reboot problem access points.
When should you go over your recent on call shift?
30 minutes before the beginning of every shift with the person going off on-call and 30 minutes before the end of your shift with the next person going on-call
What do you do if a mobile gate is showing down?
Check CarTower to check for the most recent scan, Check Netcloud, reboot PI in the box and if all else fails contact Tammy/enter Velociti ticket
What do you do if a site goes offline completely?
Check for local power outages, then check for power in the cubit/network room, check UPS for power, then reboot circuits if needed.
What do you do when you are paged?
Resolve the PagerDuty alert and Acknowledge the PRTG sensor alert, check for any tickets that have been created and begin troubleshooting the issues.
What to do if you can't find anyone to contact?
After checking Slack, Workday, Orgwiki, reach out to closest site.
When do you join the Morning Tech Huddle?
Every day you are on call to inform of on-going outages (except weekends)
What do you do if the Rover shows down for RFID?
Check Cartower/RFID Dashboard to verify device is online, then contact local team to verify if they have run the rover
What do you do when a Verizon gateway goes down?
Power cycle with local team, If still down have someone with access to the Verizon portal (Ryan has access) check status there before reaching out to Verizon for further troubleshooting.
What should I do if a piece of hardware is flapping? (going online/offline repeatedly and generating multiple tickets)
This is the only time you should pause an alert instead of Acknowledge but this should still never be an indefinite pause!
What should you do when a site goes down off-hours?
Acknowledge alerts, verify what you can, and send a message to local team so they see it in the morning. (Reach out to local RFS immediately if at an IC)
When should you reach out to the networking team?
After reaching out to the site, confirming all hardware is in working order, and City does not see issues on the ISP end, reach out to seniors directly and then the network team may be contacted.
Who should you contact on site when you need local help?
Check Orgwiki/Workday to find the MoM, a lead, or a manager and reach out via phone (if they have a number listed) or Slack. You can call the site as well if it is not completely offline.