the account locking mystery

It start last monday 10.40 am when a user notified me that scampoint was down. We did the usual checks, but it was the first time we encountered the symptom - scampoint admin account was locked out, and irregularly, sometimes 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minute, finally stopping at 2 pm. 5 pm it started, locked out a few times and stopped at 6 pm. For the whole day we were checking the various logs, and well... nobody could identify the problem. The application administrator, server administrators, application support vendor. The source IP locking the account was the scampoint server itself.

Tuesday was peaceful. We did a reboot of the servers just to make sure. The server administrator was telling me that the application administrator doesn't even know how to troubleshoot the problem. I told him, administrators are like operators, and if they can do troubleshooting then they are better than others, like himself, he is an server administrator, but he helps us troubleshoot our scampoint issues sometimes. So that was a psychological nudge that I was passing problems to him.

After spending 2 days thinking, I asked these guys what they thought of my speculation, that it's a human triggered action, because it only happened during office hours, has irregular occurrence, and anybody who tries to log in with our user name will lock us out with the wrong password, which is so easy to find out because our admin account name appears all the over place, if you know how to look. What I could think of was just typing a wrong password for a particular user name will lock the user out. Everybody was still clueless, so I escalated to their infra manager, i.e. my infra guy, to ask him for ideas. He told me that he was impossible to prevent an account from getting locked out.

I went back to the administrators to tell them that we will be on our own so I asked the server administrator whether he could write us a script to automatically check our account every minute and unlock it if it is locked, and then send an email notification. His incentive was to feel satisfied that he has contributed to improving the productivity of the team, have lunch in peace, and not having to take turns to monitor whether the account got locked, else we will all have to write an incident report because the intranet is down when everybody is out for lunch. He was helpful enough to agree, and that took some stress out of our plates. My agreement with him was that it's just to help us save time to solve other issues until we can think of something for this issue. Today we had about 10 burning scampoint issues on hand. One of those days where it feels like I am defusing a time bomb, and I cannot make any mistake. The scampoint administrator is already quite stressed out and tickets are piling up.

Luckily for today, the account locking happened 3 times between 10.10 and 11 am, and for the rest of the day it was fine. I was trying to reproduce the problem, but couldn't, this is just self-affirmation I won't pass as a hacker. Thanks to the guy who told me it's impossible to prevent the account from getting locked out, I turned to my friends, and one of them suggested that I lock down the account. I was so glad that he had been thinking about it for 2 days too! So we will try that tomorrow and if it works, we may never know what caused it, and it may be a pity not to know the answer to the mystery.

who wrote it?

Me: [addressing question to vendor project manager] why was this sentence written this way?

Vendor project manager: Hmm, I didn't write this, I am not sure.

Me: Who wrote this paragraph?

Vendor project manager: [vendor technical director] wrote it.

Me: [addressing question to vendor technical director] why was this sentence written this way?

Vendor technical director: I don't know why it was written this way, I didn't write this.

Me: Who wrote this?

Vendor technical director: [vendor project manager] wrote it.

Me: When? [vendor project manager] said he didn't write it.

Vendor technical director: he typed it after the meeting he had with your team.

Users and I all burst out laughing.

strangleton

I am usually patient, calm and good, but when I am sickish, or lack sleep, I become aggressive and any one who annoys me makes them a strangle-ton, adapted the word from singleton.

Me: have we settled the outstanding issues with the user?

Vendor: yes, we have already explained to the user that it cannot be done and customisation effort is required.

Me: and the user agreed?

Vendor: yes, the user agreed to have it in phase 2.

Me: there is no phase 2.

Vendor: yes, there is no phase 2

Me: if we make this application too difficult to use, they wont use it and we don't need to think of any phase 2.

Vendor: yes correct.

Me: the users asked for a breadcrumb trail, why can't we give it to them since it's out of the box?

Vendor: because we didn't give them rights to open the root level folder so the breadcrumb trail doesn't work.

Me: why aren't you giving them rights to the root folder?

Vendor: because giving them rights to open the folder means they will be able to see the files and delete.

Me: can't we just don't give them rights to delete those files?

Vendor: no, because we don't want to break inheritance, and they have delete rights at the root.

Me: what are these files at the root folder?

Vendor: system files, the application pages that the user needs to use the system, that's why we cannot allow them to delete.

Me: why do you store the system files with the user files?

Vendor: because we only used one document library for our applications.

Me: how hard is it to separate the system and user files into two document libraries?

Vendor: it's a simple change of configuration.

Me: then can we do that and give users the breadcrumb?

Vendor: yes

Me: we need to give the users as much productivity features as much as it is within the cost and scope and breadcrumbs is there, and we will be short changing them by your (lousy) application design.

Me: now to the next issue. Why are you telling users to move files one by one instead of using the file explorer bulk moving of files functionality?

Vendor: because we only allow admin to delete/move files in bulk. Users are not admin, so they cant use the function, that's a requirement.

Me: but there is still an option for the user to move files via file explorer.

Vendor: they won't know because we didn't tell them.

Me: they are intelligent people and you think they won't know just because you don't tell them?

Vendor: yes

Me: can we tell them that they can use it?

Vendor: no, they are able to delete files in the file explorer when they don't have rights, but if they use the application they will not be able to delete.

Me: that's a breach of the security model of the product, are you sure you can delete?

His colleague explained to him that the user can click delete but if he refreshes the window, the file will still be there because he only deleted his local cached copy.

Vendor: there is another thing, if they delete from the file explorer there is no audit trail.

Me: audit trail for file explorer is an out of the box feature, why isn't it in the audit trail?

Vendor: because we customised the audit trail so there is no audit trail at the file explorer because we didn't want users to user that function.

Me: since you have already customised the audit trail, there is not much we can do for that, but tell the users that if they use the file explorer to move files, it will not be captured in the audit trail. I don't want the users to use the file explorer and then report the audit trail not capturing the move as a bug.

Vendor: ok.

Me: so think about how you want to tell the user everything we have discussed and we will meet the user next week.

Well the positive side of it is that I know we should be able to sign off the system next week after these issues are resolved. The bad part was to the user I took 2 weeks to get round to this because firstly, the vendors were supposed to be closing all the issues, and secondly, because I was away from work due to yaya's teacher's day school holiday, yaya's HFMD, 1-day course, my own body hibernating, my more ferocious users hunting me down immediately on my return, and of course the 10 chess boards that I rotate playing everyday. Another vendor I met today thought that I only have 1 project with them. Isn't that what every user thinks too? that they have tonnes of work whereas we are idling somewhere escaping from all the work. haha.

The trick is similar to multiplexing, getting the right sampling frequency of the user, and always appearing at a particular frequency to make them feel that you are full time with them, and always there for them, when you are not. And like in networks, you need to have a range of bosses/users/vendors with differet frequencies, so that you don't miss any packet. If they have the same frequency, then we need to double the sampling frequency. Ok digressed too much, the multiplexing bit was just a friday crazy bit, cant really be applied at work, there is no trick. Just pure brain juice being used to play 10 boards of chess concurrently. It helps if your opponent is 10 times slower than you are.

manual work rocks

This happened yesterday, a friday. My colleague asked me to attend her migration meeting at 4 pm but I told her I wasn't free. My phone rang at 5 pm, she asked me to go her meeting because there is an issue. So I went and it was over in 10 min.

Me: what is the issue?

Vendor 1: our migration service request only covers exporting of data to DVD but not importing.

Me: why not import?

Vendor 1: because there is no way to import a data table with attachments.

Me: so how is it supposed to be done?

Vendor 1: you will need to import the spreadsheet first, then manually attach the documents back one by one. It is not in our migration scope to do it. Users are also not going to do it.

Me: and why must I be the one doing it?

Vendor 1: that's the issue now.

Me: do you have any unique identifier to link the data row and attachments?

Vendor 1: no, you need to look at the title.

Me: how is the folder structure going to be like in the DVD?

Vendor 1: just 1, 2, 3, 4, 5, ...

Me: how do you get those numbers?

Vendor 1: we drag out one folder at a time, manually, the numbers are system generated.

Me: if the system has the number, can you export it out to the spreadsheet?

Vendor 1: yes.

Me: if you have the number, then you can create a link to folder containing these files. We just upload all these numbered folders into a folder.

Vendor 2 (vendor 1's boss): oh...

Me: do you understand?

Vendor 1: we can't link to the folder.

Me: we manually pre-fix the URL to the root attachments folder to those unique numbers.

Vendor 2 explained to vendor 1 because he couldn't understand. At this point, the users already understood.

Users: we will use this method.

Vendor 1: we will just give you the spreadsheet and DVD and there will be no links to the attachments.

Me: nevermind, I will show you how do it when the time comes.

He must be still lost, but I think his boss should be able to explain to him. I really don't how they survive as vendors to be recommending manual work. Actually I don't really have any business in this, but that's how my work is.

we don't seem to be hitting the right notes

Recently, when the PM announced the many possibilities that his ministries will be working on to boost the birth rates, it was active action of hitting the keys on the keyboard, but the tune didn't sound nice, just like how we complain about the national day song these days being no match against the national day songs from two decades ago.

There are many facets to the birth rate problems. The more obvious ones are money, time, job security, spouse support, grandparents support, maids, car, preschool, quality of life, standard of living, ...

If we drill down to the blocks of life, what every man wants from life, it's happiness. When we use happiness as the guiding principle, we will be able to see more problems beneath the visible problems.

Firstly, happy people will breed happy people. If the person doesn't feel happy with his life, not necessarily unhappy, he will ask himself why will he want to bring to life a child who will feel the same as him, if not worse.

Secondly, successful people breed successful people. A successful person will feel happier than an unsuccessful person. Those who are unsuccessful and say that they are happier because they are unsuccessful are living in self-denial. Our capitalist society has made us all very competitive. After going through ranking exercises in schools for tests and exams, we go through more ranking exercises at work. It breeds a majority population with low self esteem, and a minority population with high ego. Those who are constantly trying to climb to the top will eventually conclude that having children will add more burden to whatever burden they already have. This burden could be in the form of not being working hard enough to reach the top.

Thirdly, high cost of living is a self-fulfilling prophecy. Everybody around you talks about how expensive it is to raise kids - milk powder, diapers, medical, childcare, tuition, enrichment, insurance, school fees, university fees. Over time, you think that that is the "normal" lifestyle, and when you try to adopt it, realise that it is really a high expenditure pursuit. This reduces one's life satisfaction index, and hence happiness. Innocently, having children implies increasing your already high cost of living.

Fourthly, over sensationalised balloting results in flat applications. People feel that there are not enough flats for everyone, hence being able to get a flat first is seen to be a big achievement, when it really is not. Going through all that trouble really makes people feel unhappier than happy. 

Fifthly, over sensationalised balloting scenarios in primary school admission. Parents even need to do volunteer work to get a chance to ballot for a slot in the school. This gives people an impression that there are insufficient primary schools in singapore, maybe it really is the case, but if it is not, creates that impression. However, I believe that there are insufficient schools that can meet the needs of the aspiring and competitive parents. When others see how grouchy parents become over just getting their child into primary school, they think that it's unlikely their children will be happy as well.

Sixthly, money makes people happy. Those who say that money doesn't make people happy are either filthy rich, or are being supported by donations. If you give someone $100 rebates, he is happier than his PR neighbour who didn't get it. If you give someone $10,000 in baby bonus, he is happier than the foreigner parent who didn't get it. So my point is, the bottomline needs to be raised, the individual needs to have that constant $100, or $10,000 appearing in his bank balance, so that he constantly keeps himself happy, instead of the government making him happy once in a blue moon, if not, only once in a lifetime.

Finally, I grumble once in a while that I have to spend ~S$20k/year on my little girl's childcare, medical, daily expenses. I haven't factored in my manpower cost to look after her, maybe make it cheap, S$10k/year. If I convert this into GDP, I am contributing an additional 30k/year to Singapore's economy. And I am convinced that the government wants me to spend a lot more, so that GDP continues an uptrend, just like their salaries.

who will re-sign something unfavourable?

Yesterday I signed off the user requirements document for a project, then today, the vendor asked me to re-sign it.

Me: Why?

Him: Oh, because we made a change.

Me: What change?

Him: We brought the signed document back and our boss told us to add in a line.

Me: What line?

Him: That you will sign off the functional specs within a week of signing off the user requirements.

Me: No. I am not signing.

Him: But ...

Me: No.

Him: Our boss ...

Me: Just this week, trying to find a date to go through the functional specs with you yielded no dates. I will only agree to such a term if you can promise me that you have the full 5 days blocked out from your other projects, and we meet consecutively 5 days to clear this. We still have a lot of ground to cover.

In the end, he just said ok.

incident prevention and incident response

I had wanted to write about this for quite some time, but had never gotten the chance to type it out. In IT systems support/management, part of the maintenance phase is incident response and prevention. It's always in that sequence. The prevention bit normally comes as a mitigating measure after the incident has happened.

There was a JC class outing I had in 1999 where a group of us went to The Heeren, and wanted to take a group photo at the atrium area in front of HMV (it was HMV then). As it was quite late at night, 9 pm, there wasn't many people around to help us take a photo, so we approached an idle-looking security guard who was standing outside HMV. He said no, I asked him why, he said it's his job to guard the gates - those magnetic gates at HMV that sound of alarms when someone tries to take an unpaid merchandise out. We looked for another person to take the photo for us.

In what happened in a blink of an eye, while we were gathering for a pose, the alarm sounded, the camera flash went off, and a guy was pinned down by the security guard whom we approached earlier, just beside the magnetic gate. Everyone gasped. Another security guard came over, helped to restrain the guy (a young malay boy by the way), and took the guy into a room. Before we could even move out of our formation, that same security guard was back at his original patrol position, and we didn't even notice when he moved!

After that incident, I had a very different perspective towards incident response. Hence I have a different approach to incident response for the systems I support/manage, I have hawk eyes on incident prevention. However, with resource optimisation, and optimising optimised optimisation, incident prevention is almost a non-existent scope of work. You are hired as the security guard to guard the gates, to control what goes in and out of your system, but instead of watching the door, it's likely that you are re-arranging a CD, or processing a customer's payment, or receiving a delivery, or helping a customer look for a soundtrack he wants, anything except standing at the gate, watching. As a result, any thief (incident), almost always escapes your eyes (avoid getting prevented), and becomes an incident (theft case), which you will then need to respond. You then still have to stop re-arranging the CD, or stop processing the payment, or stop receiving the delivery, or stop helping the customer look for his soundtrack, and chase after the thief, in vain.

You then activate the other teams to help you trace that guy, and as you are on it, your boss will ask you to provide a regular status update on the situation, and it's likely that you won't be able to trace the guy (don't know cause of incident), don't know what he took (because it's not RFID, just magnetic strip due to cost cutting measures), and your boss still insists you trace the CD, so you end up having to do an ad hoc stock inventory, tally it against your point-of-sales system for the day (because the IT system doesn't have a report to handle such a scenario), and finally as a mitigating measure, you state that you need to be more vigilent in future, and it's weird if you say that, so your boss helps you say that.

That's why some people look idle, always idle (due to sampling frequency), and are deemed idle. While you are posing for a photo, he would have stopped an incident from happening, and returned to his idle position, without you noticing.