The best software review a plant floor ever gives is silence. Nobody calls. Nobody walks over. The bags go where they're meant to go, and the shift ends on time. By that measure, Naveed Kakal got a rave this week in Hong Kong. "Very smooth and calm since we handed it back over," he reports from the floor of Vogue Laundry, Cathay Pacific's laundry. Getting to calm took nine years, one broadcast storm, a server slowed down on purpose, and a 36-hour window that tried hard to become a longer one.
The biggest job yet
When ETECH signed Vogue, it was, in Kakal's words, "our largest project ever at the time." He wasn't there to build it. He left for a stint in Chicago tech while the project was still in design, and he came back in March 2017, just as the plant was being commissioned. So his relationship with the place started backwards. Other people had drawn it, built it and wired it. He walked in through the stage door to a finished building whose problems were already waiting for him.
The storm
Then the plant did what successful plants do. It got busy, and that's when it got sick. "As their production ramped up the issue started," Kakal says, "so it was after we were already off site." ARP storms and broadcast storms rolled across the network. The I/O controllers on the rail system couldn't cope with all the passive traffic hitting their comms ports, and when a controller is choking on chatter that isn't meant for it, it misses inputs.
On a rail system, one missed input is a physical event. "We'd miss a track switch," he says, "and then things wouldn't go to the right spot and cause a huge physical world jam that had to be corrected with people in big lifts." That's the gap between software and steel in one sentence: a dropped packet becomes a crew on a lift, high up in the rail lines, untangling a traffic jam of laundry by hand.
Kakal's first trip to Hong Kong, in December 2018, was a hunt. He took Wireshark captures from different points in the plant and kept collecting until the picture came clear. The culprit was a switch trunk. The eVue server was pushing enough traffic to basically max out the one-gigabit link, and everything else on that trunk, the controllers included, was fighting for what was left. The short-term fix was gloriously counterintuitive: drop the server's ethernet port from a gigabit to 100 megabits, so it physically couldn't flood the link. The biggest laundry project the company had ever built got better when its server got slower.
On credit, Kakal is firm. "This wasn't my fix. I gathered a lot of data and helped colleagues come to that conclusion." He was the one in the switch closets with the captures. The call belonged to the team.
The correct way
Software that runs a plant long enough stops being software. It becomes the way things are done. Between 2017 and 2026 eVue moved through version 6 and on to v26. Over those years, Kakal pulled the Vogue-only special cases out of the shared code so the platform could move on. Vogue stayed put on eVue 5, and its operators did what operators do. They got very, very good at it.
"The operators now consider eVue 5 the 'correct' way things should be," Kakal says, "so every improvement from eVue 6 is a bit of a stand-off till the improvements click." Which leaves him in an odd spot: the person defending the new system against the habits the old one taught. "My job is often to hold the ground on doing it the v26 way, instead of building them ways to keep doing it the eVue 5 way."
Recirculation is a good example of the fight. In the eVue 5 era, sending bags back around through a storage area was rare, and the software treated it that way. It was a manual job: the operator picked the bag, picked where it had to go, and handled the rest. Since then the team has done a lot more recirc, and the software has absorbed it. v26 spots the batches that need recirculating from among the ones trying to leave, so nobody has to move those bags by hand. The scheduling automation can be told recirc is available and pick from mid-line. And months before the trip, he'd already hardened that automation so recirc traffic couldn't gridlock the storage exit. The thing they didn't trust had been tested against exactly what they feared.
During user acceptance testing that was a hard sell. The operators saw their manual controls changed or missing. What wasn't clear to them was that they would no longer have to solve the problem at that level of detail. "They didn't really believe me," Kakal says, "and we went back and forth a lot." So the pitch ended up personal: "Look, I am coming out to Hong Kong explicitly so that nothing like this gets missed. I have done 400 of these laundries and I know all the weird cases. Let me come and make you feel safe and taken care of."
That last line is the job description that never makes it into a job description. The code is necessary, but what actually moved the operators was a person they believed would be standing next to them when it went live.
Thirty-six hours
Don't call it a software upgrade within his hearing. "It's more than a software upgrade. We had layout changes, PLC controller upgrades, network upgrades, new managed switches and fiber optic cable runs. It was a big ordeal, not a patch update." It was months of preparation on both sides of the Pacific, all aimed at one short hole in production.
Then the window opened, and so did the trouble. After months of coordinated IT and network preparation, the plant's IT team ran an upgrade of their own on the morning of the cutover, and it changed the network access rules. It left the team dead in the water on internet access, Kakal says, and stuck doing workaround file transfers for a big chunk of the job. Then a documentation mismatch: twelve panels turned out to have a different controller than the drawings said. That cost a few hours, split between hunting for the hardware they'd expected and waiting for colleagues in the US to wake up. "They ended up waking up to help us," he says, and he adds a heart.
The window was 36 hours. They had most of the system running in under 20.
Too beefy
The new switches and fiber also close the 2018 loop. The trunk that once forced the server down to 100 megabits is gone, buried under new capacity. "The network is much more beefy," Kakal says. "I'd say too beefy, and maybe a bit more complicated than necessary now. But if they understand it, it's on them to maintain, you know."
Eight years after he went through the switch closets with Wireshark, the network belongs to the customer, the server runs at full speed, and recirculation runs itself. The operators haven't brought any big questions or complaints to him on that front, which in this business is as close to applause as it gets. Somewhere in that plant, the new version is quietly on its way to becoming the correct way. Some time in the 2030s, someone will have to fly out and argue with it. ❦