Root Cause Analysis


I met Asiri when he joined our hardware support team, the same time I had been inducted into the documents clearing software team. There were around 10 engineers in hardware support and were quite busy with the numerous installations we had in the emirates. Most of them were either on a call with a customer, or driving to a customer site or was already at the site. It was not that they were busy all the time, there were days when most of them were seen in the department, carrying out the process documentation or inspecting faulty equipment in the lab by wearing anti-static aprons or even vetting newly arrived equipment spares at the store. We, the payments software team gelled well with them. Both teams, the hardware and software support, used to rush to assist each other whenever required at various sites. Those days in the late 90’s, Dubai was devoid of much traffic and we could cover Bur Dubai and Deira within a matter of few minutes.

There were times or days when most of the incidents would happen due to hardware glitches and while some clients thought it was a software problem, others blamed it on the equipment. On such occasions, our support departments wore a deserted look. I thought of writing this post today to highlight the ways and means used to uncover the root cause by our engineers especially Asiri also referred to as the ‘Doctor’. Asiri used to wear his anti-static apron more than anybody else and that is how the term ‘Doctor’ stuck to him. One more reason was due to his style of functioning whenever he reached the site. He used to sit or stand patiently beside the faulty document processor, maybe touching it all the time and getting valuable feedback from the customer as to what caused it to malfunction. He would appear as a doctor posing many a question to the client to understand the root cause before he even thought of opening or checking the machine.

One day, a bank called him while he was on the way to office in the morning, saying that one of their document processors was rejecting all the cheques/checks instead of processing them. He thought it could be a sensor failure issue and reached the client in 20 minutes. In Deira Dubai, those days, you might take only 10 minutes to reach but you may need another 10 minutes to find parking for your car. By the time he reached the site, the document processor which was acting weird was now back to its normal self and processing them without any issue. Asiri cleaned the sensors at the reading bay and watched the machine for another 15 minutes and finding nothing unusual came back.

The next day more or less at the same time, another Engineer received a similar incident call from the same bank around 11 am and rushed to central operations and by the time he reached, the machine as earlier had shook off its symptoms and was processing the cheques at its standard speed of 250 documents per minute. He also sat for around 15 minutes and finding nothing unusual left the client. The client also joked saying that when you doctors come, the machine doesn’t show any symptoms. This incident, if memory serves me right was not communicated by the second engineer to Asiri the same day.

The third day, Asiri is in office and waiting for any potential incident to happen and perhaps working on one of the defective document processor in the lab when he got a call again for the same issue. Since the client had a backup machine and it was not a peak volume day for the bank, he took some time to get out of the lab. By the time he reached his car, the client called back again saying the machine was working fine now and he need not come.

Asiri walked back from the parking space to the office and that is when he met the second engineer who told him about the call he made the earlier day to the bank and how he had to come back since there was no problem with the machine when he arrived. “So that means today is the third day, the machine is behaving this way” said Asiri and checked the time. All three incidents were pointing to the time around 11 AM.

The next day, as was his usual wont, Asiri did not wait for the customer to call but made himself present at around 10:30 AM at the site. Fakhruddin who was managing the clearing operations at central operations welcomed Asiri by saying “today nothing will happen since you are here”. Asiri smiled and asked for a chair and got seated with his eyes riveted on the reading bay sensor where each speeding cheque on the track is sensed and stopped. He just sat there looking at the sensor, who he rightly thought was the root of the cause while the minutes ticked away. At around 11 AM, he saw the rays of the sun striking the sensor through the venetian blinds of the cabin, and the sensor along with the machine went haywire as the cheques on the track started getting rejected and Fakhruddin had to stop the machine by pressing the pause button. A smiling Asiri pointed out to Fakhruddin the root cause which was the venetian blind slanting position and moved across and closed them completely. On the way out, he told Fakhruddin, “You won’t face this issue anytime soon unless someone plays with the blinds again“.

NDP 250
NDP 250 Document Processor

13 thoughts on “Root Cause Analysis

  1. I was totally engrossed in the story…and I was sure there is some silent conspiracy theory that will be unveiled at the end…I did not see the end coming!!! I laughed out aloud when I realised the culprit is the innocent Sun!!! Thank you for this lovely read Sunith!
    P.S: I hope you are doing well. It’s been a while since I found my way to your blog…sorry for being MIA..

      1. By God’s grace, I’m trying staying alive Sunith! That seems to be the only thing that we can do in these difficult times…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s