On the morning of April 6, 2024, an electrical fault in our south wash line transformer took the entire line offline. The fault was in a 480V breaker on the primary side. The breaker had not been inspected in the 11 months since our last full electrical audit, which had cleared it. It tripped during a startup transient and would not reset.
We replaced the breaker. It tripped again. We diagnosed a phase imbalance traced to a developing fault in one of the wash-line motor windings. We replaced the motor. Total downtime: 11 days. We ran the north and central lines on extended shifts and got through the backlog in roughly three weeks.
What we learned from the incident itself
Three things, in order:
- A 480V wash-line motor is single-source from one vendor in our region. Lead time on a replacement motor was 7 of the 11 days of total downtime. We were lucky one was on a shelf.
- Our spare-parts inventory had been quietly drifting. We had a documented spare-parts list dated 2021. The list was wrong in about 30% of line items by 2024.
- Our backup-route plan for orders had assumed we could re-route to the north and central lines. We could, but only at 78% of the south line's capacity, not the 100% the plan had assumed.
What we changed
Spare-parts inventory now audited quarterly, not assumed. The audit is done by the same person who runs the wash bay rather than a separate maintenance person. Their incentive to keep the list accurate is direct.
We pre-bought a spare 480V motor and put it on the shelf. Cost about $4,800. Equivalent to about half a day of downtime at our gross. We considered this insurance.
Our customer communication on the incident was, in retrospect, slower than it should have been. We told customers on day 3. The customers we told on day 3 were uniformly fine. The customers we did not tell were learning by inference from delayed shipments. The new rule is: any unplanned downtime over 24 hours gets a customer email by hour 36.
Capacity-plan numbers are no longer trusted by feel. We did a load test on each wash line under simulated single-line-outage and have actual numbers now.
The general lesson
Most operational incidents are not novel. They are a specific case of an issue that has been quietly worsening for months. The breaker had been due for inspection; the motor winding had been showing minor temperature anomalies on the maintenance dashboard for weeks. The system gave us warning. We did not read it.