MAXIS boss explains services disruptionKOTA KINABALU:
Maxis Communications Berhad (Maxis) has launched a full investigation into last week’s state-wide services disruption in Sabah, according to its Chief Operating Officer Jon Eddy Abdullah. Results of the investigations described as “an unprecedented” breakdown of its network and main mobile switching centre (MSC) in Kolombong are expected to be known by the end of July. Speaking to reporters here yesterda
y, Jon said an initial investigation showed that the core fault was with the software but: Maxis’ vendor, Nokia Siemens Network would find out what caused the software to fail. It would however be too: premature to speculate on the root cause of the outage, he said. “The core fault was the software fault and we understood that it was going to take a massive effort to fix it, so at the same time we worked on getting services back to our customers, which was to reroute traffic to Kuching. “Technically we want to find something so we can fix it but since this is such an unprecedented event there’s nothing to fix in the processes of the procedures. Those did not fail us; the people did not fail us. The trouble shooting efforts did not fail us,” he said. Jon explained that there was no failure in the standard processes and procedures of the standard: recovery effort. However, he said Maxis believed it had learned a few things in communications such as “when the mobile phone doesn’t work it is very hard to communicate’. “We are going to investigate how to use more traditional communication methods such as telephones or fax machines if you have a massive outage as other means to contact our dealers and customers,” he said. According to Jon, this was necessary because when the disruption occurred, Maxis had difficulties in informing its customers and that as soon as it had a stable network, it was able to blast the message that there had been a problem. “We are happy that they were the best developers working to try to bring the MSC back up during the eight hours period of time. From 9am to 3pm Monday, we tried everything we could to isolate the problem,” he related. Jon disclosed that basic services were restored to Sabah by Tuesday evening but the network was still congested because there was not enough capacity to move all the traffic. The fault was fixed by 4am on Wednesday which was when the MSC was re-commissioned and Maxis then worked on bringing all traffic back to Kota Kinabalu, a process that was completed by Wednesday evening. By Thursday evening, traffic was back to 100 percent, he added. Nokia Siemens Networks Head of Asia South Joe Dering who was also present at the press conference said within a few minutes of the outage, they had a team in Kuala Lumpur working on the problem. When the team found out that it was an unprecedented event, their experts in Bangkok and Munich were called in for assistance, he said adding that the team in Munich comprised experts as they were the people who actually designed the equipment. The Munich team had in fact logged in two hours after the event on Monday to fmd out what was the root cause of the incident, according to him. The experts from Munich, Dering added, came to Sabah two days after the outage and took two hard disks drive which were hand carried back to Munich where a thorough investigation would be conducted over the next three weeks. “We will put the hard disks in test lab and run the three days scenario three times to find out what had caused the worldwide unique incident,” he said. Jon said the experts would try and recreate the fault as it was very important for Maxis to find the root cause. “My primary concern is that the fault is nowhere in the Maxis network and Joe’s concern is that it is nowhere in any other switch in the world,” he said. Jon took reporters through the sequence of events starting with the fault which occurred at 1.23am on Monday, June 30. According to him, Maxis has a central operation centre that monitors all its elements and when a fault in the network is detected, they will start standard procedures to bring that network element back. “Our team immediately went through the restart processes and we could not -recover through the first few minutes and hours. During that time we contacted our vendor, Nokia Siemens Network and within two hours we had Bangkok and Munich on line,” he said. By 9am, they went into a crisis situation because being unable to restore the network in eight hours means that it was a major problem, he recalled. “We were unable, with all of the help from developers to bring up the MSC, it could not recover. At this point we started a new activity where we tried to isolate parts of the MSC. We knew that we had a software failure so we tried to isolate the part that wasn’t working and at the same time we also tried to replace potential hardware to try and fix things. This went on until 9pm Monday. “We knew that we had a major fault that might not recover and we needed to get services back to our customers. So the engineering team along with the support of MSN came up with an alternative plan at 3pm on Monday, which was to re-route all of the traffic in Sabah to the MSC in Kuching,” Jon said. By 9pm Monday, the first traffic was re-routed to Sarawak and by 4am Tuesday, Maxis successfully re-routed 70 percent of the traffic. The remaining 30percent was completed by 6pm Tuesday. Jon said by 9pm on Tuesday, it was decided after everything had failed to re-commission the MSC. “Maxis decided to re-commission the MSC and what that meant was we reloaded with fresh software. By Wednesday morning the MSC was completely re-commissioned and the core fault had been fixed. We stabilized the switch and began to re-route the traffic back from Kuching to Kota Kinabalu,” he said. “This took Maxis basically all of Wednesday evening until the early hours of Thursday to complete and by that evening, Maxis services in Sabah were restored 100 percent,” he added. Jon expressed gratitude to Maxis customers for being patient, saying they were aware how difficult it was for them during that time. And, as a goodwill gesture of appreciation for their support, registered Maxis postpaid customers in Sabah would enjoy a 50 percent rebate off their monthly commitment fee which would be reflected in the next billing cycle. All Sabah registered Hotlink prepaid customers will receive a seven-day extension of validity for their prepaid accounts, he said, adding that Maxis, in a further gesture of appreciation, has launched a 50 percent off voice calls promotion from July 5 to July 11 where all registered postpaid and prepaid customers in the State would enjoy half price discount on normal domestic voice call tariffs. Sabah, he said, is a priority market for Maxis as their efforts in expansion in the State over the past few months had clearly illustrated. “While the June30 service outage was unfortunate, Maxis is grateful that our Sabah customers have chosen to remain with us,” he said, adding that there has been no significant difference the subscriber base. “We monitor the active subscriber base, we were concerned that we would lose a lot of customers but we are happy to say the active users of the network pre-event and post event are the same,” he disclosed.
y, Jon said an initial investigation showed that the core fault was with the software but: Maxis’ vendor, Nokia Siemens Network would find out what caused the software to fail. It would however be too: premature to speculate on the root cause of the outage, he said. “The core fault was the software fault and we understood that it was going to take a massive effort to fix it, so at the same time we worked on getting services back to our customers, which was to reroute traffic to Kuching. “Technically we want to find something so we can fix it but since this is such an unprecedented event there’s nothing to fix in the processes of the procedures. Those did not fail us; the people did not fail us. The trouble shooting efforts did not fail us,” he said. Jon explained that there was no failure in the standard processes and procedures of the standard: recovery effort. However, he said Maxis believed it had learned a few things in communications such as “when the mobile phone doesn’t work it is very hard to communicate’. “We are going to investigate how to use more traditional communication methods such as telephones or fax machines if you have a massive outage as other means to contact our dealers and customers,” he said. According to Jon, this was necessary because when the disruption occurred, Maxis had difficulties in informing its customers and that as soon as it had a stable network, it was able to blast the message that there had been a problem. “We are happy that they were the best developers working to try to bring the MSC back up during the eight hours period of time. From 9am to 3pm Monday, we tried everything we could to isolate the problem,” he related. Jon disclosed that basic services were restored to Sabah by Tuesday evening but the network was still congested because there was not enough capacity to move all the traffic. The fault was fixed by 4am on Wednesday which was when the MSC was re-commissioned and Maxis then worked on bringing all traffic back to Kota Kinabalu, a process that was completed by Wednesday evening. By Thursday evening, traffic was back to 100 percent, he added. Nokia Siemens Networks Head of Asia South Joe Dering who was also present at the press conference said within a few minutes of the outage, they had a team in Kuala Lumpur working on the problem. When the team found out that it was an unprecedented event, their experts in Bangkok and Munich were called in for assistance, he said adding that the team in Munich comprised experts as they were the people who actually designed the equipment. The Munich team had in fact logged in two hours after the event on Monday to fmd out what was the root cause of the incident, according to him. The experts from Munich, Dering added, came to Sabah two days after the outage and took two hard disks drive which were hand carried back to Munich where a thorough investigation would be conducted over the next three weeks. “We will put the hard disks in test lab and run the three days scenario three times to find out what had caused the worldwide unique incident,” he said. Jon said the experts would try and recreate the fault as it was very important for Maxis to find the root cause. “My primary concern is that the fault is nowhere in the Maxis network and Joe’s concern is that it is nowhere in any other switch in the world,” he said. Jon took reporters through the sequence of events starting with the fault which occurred at 1.23am on Monday, June 30. According to him, Maxis has a central operation centre that monitors all its elements and when a fault in the network is detected, they will start standard procedures to bring that network element back. “Our team immediately went through the restart processes and we could not -recover through the first few minutes and hours. During that time we contacted our vendor, Nokia Siemens Network and within two hours we had Bangkok and Munich on line,” he said. By 9am, they went into a crisis situation because being unable to restore the network in eight hours means that it was a major problem, he recalled. “We were unable, with all of the help from developers to bring up the MSC, it could not recover. At this point we started a new activity where we tried to isolate parts of the MSC. We knew that we had a software failure so we tried to isolate the part that wasn’t working and at the same time we also tried to replace potential hardware to try and fix things. This went on until 9pm Monday. “We knew that we had a major fault that might not recover and we needed to get services back to our customers. So the engineering team along with the support of MSN came up with an alternative plan at 3pm on Monday, which was to re-route all of the traffic in Sabah to the MSC in Kuching,” Jon said. By 9pm Monday, the first traffic was re-routed to Sarawak and by 4am Tuesday, Maxis successfully re-routed 70 percent of the traffic. The remaining 30percent was completed by 6pm Tuesday. Jon said by 9pm on Tuesday, it was decided after everything had failed to re-commission the MSC. “Maxis decided to re-commission the MSC and what that meant was we reloaded with fresh software. By Wednesday morning the MSC was completely re-commissioned and the core fault had been fixed. We stabilized the switch and began to re-route the traffic back from Kuching to Kota Kinabalu,” he said. “This took Maxis basically all of Wednesday evening until the early hours of Thursday to complete and by that evening, Maxis services in Sabah were restored 100 percent,” he added. Jon expressed gratitude to Maxis customers for being patient, saying they were aware how difficult it was for them during that time. And, as a goodwill gesture of appreciation for their support, registered Maxis postpaid customers in Sabah would enjoy a 50 percent rebate off their monthly commitment fee which would be reflected in the next billing cycle. All Sabah registered Hotlink prepaid customers will receive a seven-day extension of validity for their prepaid accounts, he said, adding that Maxis, in a further gesture of appreciation, has launched a 50 percent off voice calls promotion from July 5 to July 11 where all registered postpaid and prepaid customers in the State would enjoy half price discount on normal domestic voice call tariffs. Sabah, he said, is a priority market for Maxis as their efforts in expansion in the State over the past few months had clearly illustrated. “While the June30 service outage was unfortunate, Maxis is grateful that our Sabah customers have chosen to remain with us,” he said, adding that there has been no significant difference the subscriber base. “We monitor the active subscriber base, we were concerned that we would lose a lot of customers but we are happy to say the active users of the network pre-event and post event are the same,” he disclosed.
No comments:
Post a Comment