T3 Support and Engineering core are observing network and voice traffic, incoming support tickets, and all other metrics similar to where we were prior to Hurricane Ian.
There are still outstanding tickets due to the hurricane's impact.. clients that are impacted by severe structural damage, CenturyLink transport that is still pending repair, or tickets that have lingered due to work load on the support staff.
We are nearly caught up on support tickets, and are closing the Hurricane Incident on the status page as we will no longer be providing updates en masse, instead just to customers that are still impacted.
Posted Nov 08, 2022 - 16:19 EST
Update
T3 Support and Engineering staff continue to work with clients that are still down in the aftermath of Hurricane Ian.
All infrastructure is working optimally at this time, however clients remain down due to premise or field hardware failures, damaged transport, or central office issues. We are working closely with our transport partners to get things fixed as quickly as possible, and rerouting traffic or deploying alternative services as we can.
Posted Oct 25, 2022 - 11:08 EDT
Update
As of this time all of the T3 core network and infrastructure are back online and we are observing normal traffic and communication between systems.
We are focusing heavily on remaining down clients to determine if the issue is our premise equipment or the circuit to the building.
From past experience with Hurricane Irma and other severe storms, we know that aerial connections such as Comcast Cable can take time to repair due to damage to the utility poles in our area and may take longer to restore. T1 and other Copper circuit connections also can take longer to be restored as pedestals, crossboxes and other field repeaters are susceptible to water and storm damage.
The T3 core network being fully restored allows us to divert more resources to the testing of direct circuits, provisioning of replacement hardware, and people to contact transport teams to get these back up and running. We can see who is down in monitoring, and are systematically working through the list to verify that action is being taken.
Posted Oct 18, 2022 - 13:42 EDT
Update
T3 Engineering continues to work on core and infrastructure issues as our client base continue to be impacted by wide area network issues in Lee and Collier Counties. We are working closely with our peering carriers Lumen/CenturyLink, Comcast, Summit Broadband, and Crown Castle to restore services as quickly as possible.
Today there was downtime incurred due to dynamic routing protocols causing problems as exterior legs of the network flapped with the above work going on. This is being addressed so that it does not happen again.
Our Support staff are also working to bring up clients that are still down after outages have cleared, and have been procuring and configuring replacement hardware, opening tickets with transport teams, etc.
We are working as intelligently as we can to have the greatest impact on restoration of services to our clients within each day that passes.
Posted Oct 14, 2022 - 19:06 EDT
Update
T3 Support continues to work directly with clients that are back up and running to revert call flow changes, open tickets on circuits that are still down or damaged, and begin getting back into normal day to day support.
Engineering continues to work to shore up all infrastructure and verify network continuity and health.
At 5pm EST today Engineering is being forced to swing a Fiber jumper at the TANDEM Central Office that will cause a brief drop for CenturyLink Fiber subscribers, and some T1 connections. This is in order to effect permanent resolution on a fiber jumper in that CO, and by order of the CO.
Posted Oct 12, 2022 - 16:30 EDT
Update
Issues impacting WIMAX customers that had gone down post-hurricane have been rectified - and we have seen numerous P2P Radio Subscriber Units coming back online and are reaching out to clients.
Field Operations are scheduling to begin surveying remaining down WIMAX customers for alignment and replacement needs.
Posted Oct 11, 2022 - 14:59 EDT
Update
WIRELESS / WIMAX Customers -
Engineering is currently investigating the wireless sector antennas at the Waterman Broadcasting tower. Once we have confirmation that infrastructure is back to operating standards we will move forward with fixes at individual clients. If your WIMAX connection was working after the storm, and has since failed, Engineering's current investigations are addressing these reports as well.
We have ordered additional stock already for radio replacements - and as early as today will be having Field Operations surveying clients that are impacted to determine alignment or replacement needs.
We apologize for the inconvenience and assure you that we are working tirelessly to restore services as quickly as possible.
DATA CIRCUITS -
Engineering has made strides in full restoration of the T3 core infrastructure, however we continue to work with underlying and upstream carriers at peering points to resolve issues with the greater local data network as a whole. T3 is colocated with CenturyLink, Crown Castle, Comcast, Summit Broadband and other carriers at various Telco Central Offices around Southwest Florida. Many of these COs have have had minor to critical damage from Hurricane Ian and depending what equipment internally there is impacted, the negative effects of this can cascade to other systems.
The Cypress Lakes CO currently has a T3 Engineering team working to restore connectivity to T1 clients that are impacted by flooding and water damage at the Central Office. T3's equipment is not impacted, however the actual facility itself is and we are working to bypass and restore those services.
Outside of this T3 Support has reached the point where we are investigating individual client circuits being down, and opening tickets for dispatch or carrier escalation as appropriate.
We need to confirm power, access hours, and local contact information if someone needed to be onsite to test further. Tickets will be called in with transport teams or Field Operations and need to be further scheduled, at this time we cannot provide ETAs or dispatch time slots. When we have further information we will pass it along.
Posted Oct 11, 2022 - 12:10 EDT
Update
Fax Server issues impacting clients 10/10 have all since been resolved as of 2pm same day.
Today Engineering teams are further evaluating damage at Cypress Lake CO that is impacting some T1 clients in that area, as well as the Waterman Broadcasting WIMAX tower so that we can begin repair of Wireless P2P clients.
Posted Oct 11, 2022 - 09:38 EDT
Update
Engineering is currently investigating reports of fax failures using the T3 Efax service.
Posted Oct 10, 2022 - 13:21 EDT
Update
Today's issues impacting select customers Voice and Fiber circuits have since been resolved.
We are starting to see some WIMAX infrastructure in Cape Coral coming back online, as well as T1s and other services further South towards Naples coming back online.
Monday morning Support staff will be engaged in following up with all clients that have had services restored to fix call flows, as well as identifying clients that are still down to open carriers tickets.
Please understand - our own Field Operations team are working double time to get services both inhouse and at clients back online as quickly as possible, however we will not be dispatching on services or circuits that we know to be impacted by higher level outages. We are also aware that some underlying carriers or carriers we peer with such as CenturyLink/Lumen are NOT dispatching for individual client issues yet. If this is the case and impacts any of our customers, we will communicate that directly and see what alternatives we have to restore service sooner.
Posted Oct 09, 2022 - 22:16 EDT
Update
As repairs continue to local infrastructure we have seen services restored, as well as services that have been restored and continue to bounce due to repairs in the area. We are monitoring the situation closely and communicating with our vendors and upstream and downstream carriers to attempt to track.
We are aware that some other local carriers such as Lumen/CenturyLink are NOT scheduling individual client repair due to the damage to local infrastructure still needing to be fixed.
This morning our Engineering team are investigating issues being reported for call failures, and failures with some client Fiber services that were previously working.
Posted Oct 09, 2022 - 09:40 EDT
Update
As of 9PM EST 10/6 we have been able to restore all services to Central Offices in the area. We've seen a large increase in the amount of restored client data circuits and are now able to more clearly define unique client issues needing to be investigated.
This morning our Support team will be reaching out to notify clients of service restoration and reverting call flow changes as requested.
During an Emergency Maintenance Window today at 6pm EST we will be needing to move a Fiber jumper at the SFTMY CO in order to affect a more permanent fix in that CO. Downtime will be approximately 1minute and impact T1, DSL and some Fiber customers.
Engineering and Field Ops continue to investigate infrastructure related issues and audit physical systems for health and status.
Posted Oct 07, 2022 - 08:01 EDT
Update
We are continuing to work on a fix for this issue.
Posted Oct 07, 2022 - 07:57 EDT
Update
We have been able to restore services to the SFTMY Central Office as of now, and have restored service to over 50 T1, DSL and Fiber connections.
Direct circuits from this CO that are still impacted are now being worked by support with clients to restore service.
Engineering is moving to the next hop in our infrastructure that needs to be repaired.
Cypress Lakes, Bonita Springs, and Naples Central Offices are next in line. We are continuing dispatches this evening to attempt to restore services as quickly as possible.
Posted Oct 06, 2022 - 17:16 EDT
Update
Work continues at SFTMY CO as our techs are still onsite working with other vendors at the site.
Lumen is current outside the CO splicing Fiber to restore services. We are not providing an ETR yet as it would not be a realistic expectation of restoring customer service. We need this telco infrastructure back online so that we can continue to move out from our core network restoring backbone services and then branching out to individual clients.
Support staff at the T3 office continue to field calls and work what we can to assist clients in need.
Posted Oct 05, 2022 - 14:29 EDT
Update
T3 Engineering continues to work on infrastructure outages due to the storm. Some of the repair work is outside of T3's hands as we work with other local carriers and vendors to resolve fiber cuts, and other damage to the wide area network in our area.
As of right now Engineering is focused on the South Fort Myers CO, working with Lumen/CentruyLink on a Dark Fiber cut that is impacting T3 clients that come out of that CO, as well as potentially Lumen's own clients and possibly Fiber+ services.
When that service is restored, we should see a lot of T1, DSL and Fiber connections in Fort Myers coming online - and are making a push to have this done today as soon as possible.
Support team continues to service clients and escalate to our Tier 3 techs to evaluate other down customers or dispatch needs.
Posted Oct 05, 2022 - 09:56 EDT
Update
Work continues at SFTMY CO as we now believe that there is further impact with a backhaul circuit between COs being impacted. This issue is our number one priority at the Engineering level.
Support team continues to service clients calling and emailing into SMC while Engineering is actively working this issue.
No new ETR at this time.
Posted Oct 04, 2022 - 15:13 EDT
Update
Work continues at South Ft Myers CO to resolve infrastructure level issues impacting T1, DSL and Fiber circuits originating from this CO. We are extending the ETR on this to 3pm today as there have been some slow downs for the Engineering team onsite.
Support team in office has been able to bring up numerous clients this morning remotely as power is being restored to our customers, and we are continuing to service client requests as quickly as we can get to them.
Posted Oct 04, 2022 - 11:44 EDT
Update
CenturyLink has rectified issues at the South Ft. Myers CO - allowing T3 to begin working on our equipment there.
We expect to resolve CO level issues impacting T1, DSL and Fiber circuits originating from this CO before 12 noon tomorrow. Once resolved Engineering will continue to move south to the Cypress Lakes, Bonita, and Naples COs while Support staff continue triage and evaluate any circuits out of SFTMY that are still down.
Posted Oct 03, 2022 - 19:50 EDT
Update
We have confirmed that customers utilizing Lumen/CenturyLink Fiber circuits as the underlying transport for their data connectivity are also impacted by the South Fort Myers Central Office outage.
We are pending update from Lumen on an ETR.
When power is restored, clients could come up assuming they have power and there are no further issues between that CO and the client site - again we are pending visibility of regaining access to that equipment to branch further out.
Posted Oct 03, 2022 - 11:21 EDT
Update
We have confirmed that customers receiving service via T1 and DSL circuits are impacted by power failures at central offices in the area.
The South FTMY CO managed by CenturyLink is pending diesel refuel, we have confirmed onsite that CO techs are waiting on master keys for access to the building as the electronic door locks are not operational.
We believe that this will restore service to many clients in the Fort Myers area that utilize those services, but will also need to evaluate issues further once we re-gain visibility to our equipment in that CO.
SMC staff continue to service clients with Call Forwards and are looking into what reported Data outages we can feasibly resolve at this time, while some unfortunately will be waiting for known outages to resolve.
Dispatches are following the previously stated priorities, and requiring us to know that we can resolve an issue - meaning the client being down isnt associated with a higher level problem first.
Posted Oct 03, 2022 - 10:47 EDT
Update
We are continuing to work on a fix for this issue.
Posted Oct 03, 2022 - 10:26 EDT
Update
T3 Engineering is continuing work on bringing up infrastructure to restore service to clients. In our disaster recovery protocols we are still looking at wide-area outages and our staff is working on the largest impacts and then prioritizing customers based on our disaster recovery protocols 911, Medical, Police and Fire being priorities.
Field Operations will be dispatching to Medical clients for emergency equipment replacements today as needed and other clients as available.
Posted Oct 03, 2022 - 08:45 EDT
Update
T3 offices and data enter at 1610 Royal Palm Ave are back up on city power.
Our generator's fuel tank is at ~ 1/2 tank and we are queued for refuel for Monday.
Posted Oct 01, 2022 - 14:04 EDT
Update
As of last evening issues at the Tandem Central Office that were impacting the media gateway have been resolved. This is a step forward in restoring full voice services on the T3 infrastructure (unique client issues withstanding).
As we have advanced our scope of what we can see on the Data network we still have remaining links down to remote CO's across SWFL and are beginning to diagnose and bring these back online as well. We are now at the stage were we could be identifying outside plant infrastructure issues in the area that will require fiber splicing, aerial crews, etc. to repair.
Please bear with us as the area has sustained massive infrastructure damage. As T3 bring up our own links and services, it will illuminate further issues as we work outwards from our core network towards our customer premises. This will require us to work with other carriers and entities in the area who are also working to restore services as quickly as possible. There will be delays here based on man power, safety, and so on that we will have to be understanding of. A good example is that, for instance if you have Comcast Cable for your business which is predominately run as aerial cable backhauls opposed to buried, that Comcast may not even be allowed to begin restoring services in an area until FPL has fully completed repairs to the poles and power.
It is also possible that services can be up and will bounce - frequently remote nodes can be brought up on generator power which will later fail. Nothing will be certain on an given system until we call and all-clear.
Please continue to call 239-333-4357 during the weekend hours as our On Call team will be available to assist with call forwards, etc. However, understand that a vast majority of power and internet services are down in our area, and we will not be able to restore services as quickly as under normal circumstances. Furthermore, we are under our Disaster Recovery doctrines at the moment which prioritize customer issues being worked as:
1. E911 2. Police 3. Fire/Rescue 4. Hospitals 5. Any other “life or death” service situation 6. Military/National Security 7. All other users
Posted Oct 01, 2022 - 13:56 EDT
Update
T3's Fax-to-Email (TFAX/MyCloudVFAX) has been restored as of 10AM EST today.
Continued repairs are being worked simultaneously on the voice and data networks. Voice Engineering has devised a work around to bypass issues for customers utilizing T3 SIP Trunking services and are bringing those clients around for regular call completion, while still working the root problem.
The generator is currently at 3/4 tank of fuel and provisions are already being made to secure additional supply as needed.
Posted Sep 30, 2022 - 11:24 EDT
Identified
With the restoration of power to the data center we are now working to evaluate the rest of the T3 data network to proactively restore services as soon as possible. Hurricane Ian has had an immediately devastating impact on our community, our families, our businesses, and the full impact is yet to even be seen. We are working to audit our systems, monitoring, data network and rest of the T3 infrastructure to get a full scope of where we are today to then determine what reparative actions will need to be taken.
T3's Fax-to-Email (TFAX/MyCloudVFAX) is currently down and being worked on to bring it back online. 3rd party's that are sending to your business numbers will receive a failure that will indicate to them that they need to send again later on, or anything that HAS been accepted by the fax server will be queued for later delivery.
Calls are failing locally for various reasons - T3 Engineering is working to restore services provided by a Media Gateway for voice calls that was brought down by the DC outage. Other calls are failing due to local infrastructure damage or cell service issues. We know that AT&T specifically is extremely degraded.
As clients return to their businesses, power will need to be verified first and foremost, followed by whether or not the internet connection is up and running. Depending on the internet service being delivered, there can be impact of downed telephone poles, flooded pedestals, crossboxes or field repeaters, power issues at central offices or remote sites, or the actual equipment onsite being damaged. We are working to determine what outages are existing OUTSIDE of the widespread power outages as best we can, so that we can begin the repair process as proactively as possible.
Please continue to report your findings to support@t3com.net or 239-333-4357 so that we can get tickets open. If you are reporting voice or data issues please confirm power to network equipment onsite prior to contacting support.
We're in this together and will be doing everything within our power to get everyone back up and running as soon as possible.
Posted Sep 29, 2022 - 21:09 EDT
Update
Engineering has resolved power issues impacting the Fort Myers data center. We are observing T3 infrastructure and customer equipment powered on.
We are continuing to monitor and test to verify all T3 systems are comping up appropriately.
Posted Sep 29, 2022 - 18:17 EDT
Update
T3 Engineering has surveyed Fort Myers colocation and confirmed no water damage after the storm. Facilities are intact and there is no external damage or risk to client equipment.
Transfer Switch managing Generator failover has failed however, and power to colocation clients is down, which is also impacting some T3 services that were local to Fort Myers data center. Electricians are being called out to rectify, and coordinating with local elements of T3 staff.
Due to these issues, further engineering is being dispatched to Orlando DataSite to turn up further resources to compensate.
Posted Sep 29, 2022 - 12:31 EDT
Update
T3 Engineering is on standby to dispatch to additional resources to Ft. Myers Data Center as soon as it is safe to do so. Engineering believes that issues should be resolved relatively quickly at that time.
Posted Sep 28, 2022 - 18:14 EDT
Update
We are currently experiencing an outage impacting Ft. Myers Data Center customers. Engineering continues to investigate and will provide further information and ETR as they are available.
Posted Sep 28, 2022 - 15:58 EDT
Update
Engineering is currently investigating reports of loss of connectivity for some Ft. Myers COLO clients.
Redundancies for Voice and Data Network have been tested and confirmed working. Potential issue appears to be isolated to 1610 Royal Palm Ave. colocation.
Posted Sep 28, 2022 - 14:24 EDT
Monitoring
We have been tracking Hurricane Ian and our Ft. Myers staff have been making preparations for the storm since last week. Our generator at the Ft. Myers datacenter is topped off, emergency contacts are secured in the event we need additional fuel, and the premise has been locked down. DataSite Orlando has also confirmed the same on all accounts. Our work vehicles are fueled and secured, and the local staff is prepped and ready to restore any services that are impacted by the storm, as soon as it is deemed safe to do so.
As T3 is a growing organization, our remote and local staff will be colluding to ensure our customers are as informed and taken care of as possible. As you are aware your favorite T3 techs and account managers are also your neighbors, and we have had to take time and make previsions so that our staff can secure their own homes and families. If/when our team locally is impacted by the storm, our remote members on staff will still be able to answer calls, open tickets and assist as much as possible. In the event that our call center is overwhelmed with calls we will also be adding our California call center to our queues so that they can assist with keeping our clients informed, and helping our local team to triage and prioritize emergencies. If you are having issues reaching support we also ask that you email us at support@t3com.net as this will automatically open a ticket.
We will be monitoring our network for power outages, carrier outages, fiber cuts and so on and providing updated information as we get it so that clients can check here for up to date information.
T3 has been through a few storms over our years, and almost our entire technical staff worked together during Hurricane Irma only a few years ago. We are confident we have made the necessary preparations, and wish everyone the best of luck weathering the storm, and we will be there for you when the time comes to return to work.
A few valuable resources that our team utilize daily may be able to assist you and your businesses: