temporal phase shift

"Temporal phase shift" suddenly appeared in my mind while I was on my way home and while I was wondering what it meant, I googled and figured that it most probably was something I learnt in some yucky signals or electrical engineering related module which I didn't comprehend a word.

What was really in my mind at that point in time was "opportunity cost" and "diminishing rate of return".

How I really was feeling was exhaustion due to a very frantic past two months due to massive temporary exodus of keyboard warriors and telephone operators, hence resulting in a lots of interference when calls go hunting down the line.

Why I was feeling as such was most probably due to the fact that I could not complete anything that I had planned to complete in that two months.

Who could have caused that feeling was most probably my upper food chains that have been introducing interrupt requests, race conditions, asynchronous interrupt and acknowledge intervals, buffer overflow, timeout connections, or simply entertaining slow loris'.

Collectively, that could be the enlightened term - "temporal phase shift".

I was looking back at the time I spent on work and divided them into either 1. I didn't learn anything; 2. I learnt something.

Here is my analysis of opportunity costs for the two categories of work:

1. I didn't learn anything. What this probably could be will range from something like copy and paste 1000 times, to explaining the security requirements in the specifications. From an efficiency point of view, I probably was able to complete the unit task in a very short time. By that I don't mean I spend a short time on the task in totality, but when divided to a unit task, it's short. For example, copy and paste 1000 times will take 1 second per copy and paste, as compared with someone who needs to learn how to copy and paste and spend 1 minute per copy and paste. From a productivity point of view, which looks at the time spent on the total task, the longer I spend on a task of this nature, the higher the opportunity cost to the organisation because it means that I am over qualified for the job scope and I had to sacrifice some other work in place for the 1000 copy and paste for example. From an effectiveness point of view of the one who assigns me the task, it's very effective because I can complete the task in a short time and he/she can meet his objective earlier, but from an organisation point of view, localised effectiveness will forsake overall organisational or divisional operational effectiveness. Hence, diminishing rate of return.

2. I learnt something. By now, you should understand what this category means - basically it is something which I had to spend more than average time to get it done. For example, it can range from learning how to fix the table headers for tables that span over many pages, or how to pay an invoice, to learning how to ensure that your boss doesn't screw up and create more work for you. The last part was just an illustration and any resemblance is coincidental. From an efficiency point of view, I will be utterly inefficient, hence rendering me ineffective and unproductive. For example, having to spend 1000 minutes to complete 1000 copy and paste as compared with another keyboard warrior who can complete 1000 copy and paste in 1000 seconds. The organisation pays for my 1000 minutes to learn how to copy and paste 1000 times, and that's the opportunity cost. However, if my main job scope is to specialise in copy and paste, then this would have a lower opportunity cost as I gradually complete my task (over someone whose main job scope isnt copy and paste) and increasing rate of return because the more I practise, the better I will become at copy and paste - potentially 1000 seconds if i master the technique from the other guy who can complete in 1000 seconds.

I wasn't doing 1000 times of copy and paste, but close. The optimal work assignment is to assign the didnt-learn-anything work to people who can complete in short spurts, much like how you queue 4 hours to see a specialist for 2 minutes and pay $200, and allow the specialist to delegate the routine check up work to the i-learnt-something person who will have increasing rate of return. After some time, the i-learnt-something person will feel that he has diminishing rate of returns and that is when his opportunity cost is reduced, the organisation gains the productivity and then move him into something else where he can reset his equilibrium. In this aspect, the specialists need to be given some space to also learn something, else they risk diminishing rate of return and lower organisational effectiveness, even though they charge $200 for 2 minutes of consultation.

the name matters

It has been perhaps 4 months I have been officially switched from the applications team to the infra team and while I was thinking about what sort of transformation will make meaningful use to the organisation, it occurred to me that perhaps a name change will help.

The applications team is called Application Services, and the head is called an Application Service Manager. The infra team is called Technical Services, and the head is called a Technical Service Manager. On paper it seems fine, but let me put up my case.

Firstly, by naming Technical Service for a team implies that the other team is potentially not a Technical Service. I was lucky to be roped in for some interviews in the past few weeks and while I was browsing through resumes (yes, it's my first time browsing resumes), I realised that there are some "Project Managers" who are not technical. What I mean by not technical is that they do coordinating, resource planning, admin work, and the technical work such as writing documentation, designing the system or interfaces are done by "technical or application consultants". Hmph!

Secondly, by naming Technical Service for a team that does infrastructure work, means that the expectation is higher than the capabilities available in the team. When I was in the application team, the only work I assign to my infra team is to buy servers, test connectivitiy, and clear firewalls. I take care of all architectural aspects, interfaces, system design (both hardware and software), security, policy compliance, facing the grumpy users, contracts, vetting documentation, etc. When I joined the infra team, the PM I support pass me stuff like "performance test plan", "security test plan", "functional specs", to vet. When I was a PM myself, I never passed those docs to the infra team because ... well, it's under the scope of the PM, or so I thought. Hmph!

Thirdly, when I asked for expectations of me, I recalled my apps boss told me that my switch to the infra team will help the apps team a lot, and it's because I will be able to do system architecture design and ensure security compliance, which is something he expects the infra team, I mean Technical Service team to be doing. When I asked him whether he feels helpless (because he isn't technical enough) when there are enterprise architecture design issues that are creating down stream system issues, he said no, his job is to find the right resource to do the work. When I spoke to the infra boss, he told me that the apps PM is supposed to be technical, and Technical Service does infra stuff - server and network. Enterprise architecture and application security shouldn't be a Technical Service scope. I tried to ask to do a project to set up a common authorisation interface for systems to adopt in the apps team and I was told that it's the other team's job, and when I went over, and stated my wish, I was told that it's the other's team job. Hmph!

For the longest time, there has been a no man's land between the two teams, and I think I have found the answer. It lies in the name, doesn't it?

If we change the name to Infrastructure Team, then both teams need to be technical.

the nightmare in the IT backroom

The switch from the application team to infrastructure team has been quite seamless, and by seamless, I mean, my work for both sides still function just like any other day, and no impact is felt for everyone except maybe for the infra boss. Almost everyday has been spent revealing secrets and presents hidden all over the network. :P

Treasure hunting could be a competency by itself.

When I first started out in the application team, my first 2 months in every project given to me were treasure island expeditions. I will say that it's no different at the infrastructure team. Which team has it easier? The level of difficulty is the same to me, both are average, not too difficult, not easy as well.

If we look back at the OSI layers, I am just so thankful that layers 1 and 2 are done extremely well. I visited the data centre, checked the cables, switches, patch panels, racks, everything was labelled well, design was systematic, and tracing of the network is really a breeze. Whoever was in charge of it, I am quite sure doesn't exist in our team now. LOL.

I can't say the same for layers 3 to 7. That's where our nightmares are created.

We have strange relationships in the network:
- Configurations for standalone networks are correct at one end, but ended up rout-able at another end.
- Managed-networks with legs in un-managed networks.
- Un-managed servers with legs in managed-networks.
- Bypass-able firewalls because of above 3 points :P

So far, I haven't encountered any blackhole where I feel totally incompetent and unable to understand. Common sense has been the order of the day. Google rocks too.

the strategic thinker

Yesterday, over lunch, I decided to poll my lunch attendees for their opinion towards whether strategic thinking is a skill that could be taught. Without a doubt, everybody nodded that it was a skill that could be learnt, but needs to be nurtured by someone else to develop the skill.

To me, it's not a skill that could be taught, and is just another way of saying how intelligent you are in bigger game of enemies and environment, until this key statement was made that "you may be strategic, but you may not be always right". That really made me think about all the different competencies in the world, and why certain competencies are given the bias of being one where mastery is expected and taken for granted.

For example, if one's competency is "cooking", the societal norm accepts that one may not necessarily be good. However, if one's competency is "strategic thinking/planning", the societal norm expects that one is good at strategic thinking/planning. It may be argued that there are various levels of competencies, for example, amateur, pro, expert, but assuming the highest level, the bias is still present.

When our political leaders assume positions requiring "strategic thinking" competency, I can just imagine what "courses" they will have to attend to build up this competency if they are lacking in judgement accuracy. I am also assuming that some other higher up management will be thinking that the heart, attitude and passion for the job cannot be taught whereas "strategic thinking" can be taught.

With this scenario of lacking in judgement accuracy as a by product of "strategic thinking" going more wrong than right, shouldn't we then also assess the strategic thinker competency with "judgement"? However, judgement isn't a competency, or shall I rephrase, that nobody really teach you how to judge other than a law degree? It's dangerous to also ignore the "strategic thinking" competency and just go after someone with a high "judgement" competency.

(There are many lawyers in our cabinet as well.)

Another conversation I had yesterday was also on the high number of scholars in our defence ministries and its stat boards, sucking away talent from all other industries (like IT for non defence-related industries, or transport, or healthcare). And we can't answer why there are many ex-military in our cabinet? I am sure they are kidding. If all president scholars join SAF, and president scholars are what the country needs, then naturally they will end up in the cabinet, in a matter of time, so it's a no brainer why there are so many ex-military at the top.

Is it desirable? To me, no. It's a clear imbalance of talent distribution that is detrimental to the overall growth of the nation.

And for that matter, those who distribute the cream of the crop talents to the military, shouldn't be expecting the other industries to produce similar talents at same rate as the military, because there will be insufficient talent seeds at the management level to grow any talents.

Therefore, the cabinet will still continue to have a large proportion of ex-military personnel, who are the strategic thinkers, bright, intelligent, no doubt about that, but may not have been in an environment to test their judgement because they spend their whole life doing table top exercises.

a user's confession

We were all gathered for a meeting this morning, with the agenda being sent out to our mobile phones at 11.09 pm yesterday night, saying we need to discuss the various ways to work as one, how to build trust, come with an open mind without prejudice.

One of us said he always remembers our boss' line "please remember to bring your brain when going to work" and she said that it's not a nice thing to say in this context.

I then said, aren't we already working as one? She said no, we are not one. There are times where we are one when everyone contributes without comparing who does what. (read: without comparing how much work she does)

When I said that I still think we are all working as one and she could call it blind faith. Her response was "blind faith does not help as it is blind. Faith builds on honesty works far better". (whatever this means)

Then she said goodnight to everyone, and "we will definitely make it a meaningful meeting for everyone".

The goosebumps kept me away till past midnight, and it didn't help that I was jolted out of bed before my alarm rang.

Our user started her confession, like things are moving too fast at the working level for her to keep up (read: she is management level not working level), she knows her roles and responsibilities but maybe we do not know, she needs a team to work, she cannot be the user PM herself, she has always worked in teams (read: teams have to work for her), she is very involved in the project (like vetting emails, engaging users, etc), yadda yadda yadda...

Seven of us sat there like the seven deadly sins, listening to the human heart.

Towards the end, I finally heard what I wanted to hear, she said that in the past, she had to pressure the IT team to ensure the systems are delivered properly. Now that she is in the IT team, she "cannot be nasty to the IT team anymore".

That unlocked the mystery to her confession - she is being suppressed and cannot be nasty to others.

luck of sorts

It was another lucky day today. I was fiddling about some DB2 drivers to transfer the data from the DB2 to MSSQL. Well, I didn't have to do it, but I just thought that I had been blessed by lots of luck these few days especially, so I need to return some favours.

I started with 1 driver that was using port 446 for the database connection and the error message said connection failed. Network-noob-me went to ask the network admin how to check whether the port is blocked. He took out a spreadsheet, checked a two records and said port is not blocked. I was impressed. Server-noob-me then went to the server admin to ask him how to check why I cannot telnet, he said telnet isn't installed by default from win server 2008 onwards. Network-noob-me went back to ask the network admin why after installing telnet, the connection still could't be established, he said because the other server isn't listening on that port. Ah ha, slapped myself for not knowing network fundamentals. Server-noob-me went back to the server admin to ask him how to check what ports are listener on the source server, and within a few clicks, there you go! A bunch of ports, except 446, so I looked through the list of ports used by the DB2 and picked port 50000.

Googling the port actually revealed that I was using the wrong driver. I was lucky to get the right port 50000. It's quite enlightening actually because DB-noob-me usually has an expert-DBA to help me with all the database connectivity and setup but there was no DB2 DBA this time round. A few more googles, I set up my drivers to connect to the DB2 DB but had problems mapping date fields. Googling didn't help that, so, call-a-friend.

The vendor PM, acclimatised with my incessant random questions and all, found a way out and the DB2 to MSSQL test concluded in just under an hour. Don't I just make things sound so simple? I think it's luck of sorts.

I will need to do my good deeds to return this luck; or maybe I had been doing lots of good deeds to receive this luck; or maybe ...

as an isolationist

The past 2 weeks was possibly one of those weeks where I had dreams every night of all sorts of strange events. I was jolted out of bed once in the morning at 6+ am by a vibration of my phone (silent mode) that was one metre away from me, and I normally wouldn't have noticed. I was woken up at 5+ am almost everyday because yaya was choking over her cough in her sleep. I met up with a network vendor today, the vendor asked why my namecard says applications, and said that nobody will know that I don't have any infra experience. Passed.

So I was just looking back at all the events and thinking about how different I am from others, prompted by a HR guy asking me what kind of education I went through to produce a someone like me (apparently word of mouth has spreaded that I have proven that any applications person without network/infra background can learn and pick up the knowledge easily), because he wants to hire more people like me, which was flattering, but at the same time means I have a higher market value. Hmph.

Background: I was assigned a new role 2 months ago, covering both application and infra (server and network) security, so it's like a third dimension to my work.

I also got hooked on a book, Susan Cain's Quiet, that really helps me understand myself better, and I will extract some content here:

Tom DeMarco, a principal of the Atlantic Systems Guild team of consultants... noticed that some workspaces were a lot more densely packed than others. He wondered what effect all that social interaction had on performance. To find out, De Marco and his colleague Timothy Lister devised a study called the Coding War Games. The purpose of the games was to identify the characteristics of the best and worst computer programmers; more than six hundred developers from ninety-two different companies participated. Each designed, coded, and tested a program, working in his normal office space during business hours. Each participant was also assigned a partner from the same company. The partners worked separately, however, without any communication, a feature of the games that turned out to be critical.

When the results came in, they revealed an enormous performance gap. The best outperformed the worst by a 10:1 ratio. The top performers were also about 2.5 times better than the media. When DeMarco and Lister tried to figure out what accounted for this astonishing range, the factors that you'd think would matter - such as years of experience, salary, even the time spent completing the work - had little correlation to the outcome. Programmers with ten years' experience did no better than those with two years. The half who performed above the median earned less than 10% more than the half below - even though they were almost twice as good. The programers who turned in "zero-defect" work took slightly less, not more, time to complete the exercise than those who made mistakes.

It was a mystery with one intriguing clue: programers from the same companies performed at more or less the same level, even though they hadn't worked together. That's because top performers overwhelmingly worked for companies that gave their workers the most privacy, personal space, control over their physical environments, and freedom from interruption. 62% of the best performers said that their work space was acceptably private, compared to only 19% of the worst performers; 76% of the worst performers but only 38% of the top performers said that people often interrupted them needlessly.

So you know my secret - isolation.

Hint: When your interrupt-ers (irritating-like-hell-mates-or-users) are on leave, you will be able to perform better.

can i have your email address?

Yesterday I received a call from a guy who claimed to be from HP.

Him: Are you the IT manager?

Me: Who are you looking for?

Him: I am looking for the IT manager.

Me: How did you get my number?

Him: Your CTO gave me your number.

Me: What is his name?

Him: *gibberish*

Me: Sorry that's not my CTO.

Him: Sorry I am from India and I am not good with chinese names.

Me: Spell it to me.

Him: *spelt wrongly*

Me: Whose names did he give you?

Him: Lee which is this number and another person *he got the name and number right*

Me: Ok, why didn't you talk with him?

Him: He said he was busy and told me to call you instead.

*running through my head was - my account management scope also covers getting redirected calls when my user is busy*

Me: What do you want?

Him: I want to know whether you have any problems.

Me: What problems do I have?

Him: Do you have any server or network problems.

Me: No.

Him: Are you looking to buy any servers in the next 6 months?

Me: Yes.

Him: How much is your budget?

Me: I don't know.

Him: How much do you intend to spend in the next 6 months?

Me: I don't know.

Him: How big is your company?

Me: Are you really from HP?

Him: Yes I am from HP.

Me: Our account manager should be able to tell you how big we are because they meet us every month.

Him: Are you able to give me a rough estimate?

Me: No.

Him: 100, 1000?

Me: What is our account manager's name?

Him: Ok nevermind. What is your name?

Me: *gave my name*

Him: And your job title?

Me: I thought you said I am the IT manager?

Him: Are you the IT manager?

Me: Well you said the CTO said I am the IT manager.

Him: Ok. Can I have your email address?

Me: How do I check that you are from HP? I would want to confirm your identity before I release my email to you.

Him: Ok I am Frank.

Me: What's your email address?

Him: You don't need my email address, you just need to give me your email address and I will email you our product brochures.

Me: I will email you my email address. What is your email address?

Him: I don't have my email set up yet.

Me: You said you will be emailing me, but you don't have your email set up yet?

Him: We will send you from another email.

Me: Ok, then what is your email address?

Him: I don't have an email address.

Me: Will you be able to give me your colleagues' or boss' email address?

Him: I don't have anyone with me right now.

Me: How am I supposed to believe you?

Him: Ok I will get someone else to call you.

Me: Ok.

He hasn't called back. Maybe his HP doesnt stand for the HP we know of. Did I say I must also thank the CTO for giving my number to an unknown fella.

On this topic, I received a call from supposedly Fuji Xerox few days back. The receptionist called me and said that a lady from Fuji Xerox said that he was looking for a guy called Kelvin who is in charged of printers but she can't find such a guy in our directory. I took the call. As I am the so-called account manager for the receptionists as well, I get called to collect gifts for my boss when his secretary isn't around, and I also get called when they can't find who is in-charge of whatever they need to look for. Anyway, back to the story:

Her: I am conducting a customer satisfaction survey and would like to check whether I can have your email address to send the survey to you.

Me: I may not be the correct party, what product or service is it regarding?

Her: It's for the xerox printers.

Me: I don't think we have a contract with you for xerox printers. Do you have the contact person's name and telephone number?

Her: The person is called Kelvin and the telephone number is the main line.

Me: Ok I don't know any Kelvin who looks after printers.

Her: Can you call me back after you have found the person?

Me: Ok.

Her: *she gave me her number*

I didn't call back. 

troy and elephants

Recently we have been receiving many emails about security vulnerabilities in internet facing websites, almost everyday, which resembles the story about 5 men describing how the elephant is like based on touching the ear, trunk, leg, body, tail.

Email 1: please ensure that all internet applications have proper security.

Email 2: please ensure that all vendors check their applications to ensure that there are nov vulnerabilities. 

Email 3: please conduct your own security tests before monitoring them because any high risk alert detected will be a big issue. 

Email 4: please make sure all vendors are prepared for the xxx scan exercise on auspicious-date. 

Email 5: please make sure applications are secured against sql injection attacks. 

The worst case scenario is if this elephant is role playing the horse in the invasion of troy. Meanwhile we still will be pre occupied with elephants.

recent ramblings on how i talk with common sense

Individual applications need to raise their own SR (service requests) to check their connectivity to the mail server. That was what my colleague told me when I asked whether the connection to the new mail server resolves the email sending problem, he said it didnt resolve his problem, and even though the firewalls have been cleared, individual applications (read: servers) need to raise their own SR to check that the firewall is indeed cleared.

Huh? I was stunned. My colleague gave me a frustrated look and said that he had been arguing with the network guy to no avail, and arrived at this arrangement, so I went to talk with the network guy.

Me: Why do I need to raise 5 people to raise 10 SRs to check the connectivity to 1 stupid mail server?

Network guy: Because each server is different.

Me: And so you mean the firewall clearance is different for each server hence require a different SR to test?

Him: Technically yes.

Me: It's a waste of time. Did you raise 10 requests to clear 10 IPs and ports to 1 stupid mail server? No right? You combined all into 1 request, so why is it now my job to ensure that your firewall request is done, and by that I mean having to test the connectivity for each server to the mail server?

Him: It's ok, I will do it.

Me: Ok good.

I went back to my colleague and told him the network guy said he will do the connectivity test for all the servers. He was flabbergasted because he felt like he wasted his time raising the SR for his own application. That was few days ago.

Today the infra guy (not the network guy) sucked a good few hours of my time trying to convince me that his technical specifications for a project was sufficient. At first I told him just stating the server model and RAM (memory) is not enough, and the specs are too low for me as well, after 2 days, he came back with a revised specs, slightly better, with more information, but still no OS (operating system) information.

Me: So if the vendor gives you unix will you take?

Him: No.

Me: Why? You didn't say you don't want unix you know?

Him: They should know we use windows.

Me: Why should they know when you didn't say?

Him: It's implied that we use microsoft products, so they should know we use windows server.

Me: I can use microsoft products, but my DC (domain controller) is unix, right?

Him: Yes.

Me: So is writing windows server important?

Him: Let me think about it.

Me: What if they tell you they want to install a unix DC because in your specs you didn't say you already have a windows DC? Will you let them install?

Him: No.

Me: So? Are you going to write that you are using a windows DC?

Him: We don't write this in specs normally, let me think about it.

Me: You can think, but I want it in.

No moral of story, just ramblings.

having committee meetings in my head everyday

What I like about psychological tests is that despite all the generalisation, and how unique I feel I am, I still feel that it describes me. My team recently did an assessment on our emergenetics profile and mine says this:

She has two thinking preferences (Analytical and Structural) from the "left brain" and one (Social) from the "right brain". This gives her brain a slight bias for the logical and rational over the intuitive and inspired. She has two thinking preferences (Structural and Social) that are concrete and one (Analytical) that is abstract, giving her concern for details and practical matters a slight edge over theorising and speculating.

The gift of the tri-modal thinker is the ability to empathise with other ways of thinking. She can understand nearly anyone. She can be a catalyst or a facilitator in a group and help promote understanding among the team members.

The stress of having this profile comes from being "Jack of all trades by master of none." She is not always able to sort out her thoughts or feelings about an issue, all sides of the issue make sense. Most likely, making a decision is difficult and time-consuming. As one tri-modal explained, "My brain needs to weight all sides of the question. It's like the committee has to meet, and sometimes the committee fights with itself!"

It really describes how I feel everyday - having committee meetings in my head everyday. It's amazing how it happens, and nobody can really imagine how it feels unless one experience it first hand. It is very tiring to listen to internal debate everyday, and it increases proportionally with my work (read: issues that I have to resolve). I am always the last tower in tower defense terms and I always need to try to influence others to position myself as early as possible in the game and yet still be able to cover the back). Sometimes I see myself as playing 10 boards of chess concurrently everyday. When I was younger, I used to suspect I had a minor split personality disorder. After reading a bunch of literature, I conclude that I wasn't because split personality disorder people are not conscious of the different personalities. I even thought that maybe I was possessed by another being because it became very scary when I had the whole scene visualised in my brain before I actually see the sequence of events happening -  everything happens according to what I plan. As I grew older, I started to accept that it's just the way my brain works, and used this gift to help others understanding themselves better, and lead happier lives.

lessons on interrogation

Today we met up with our vendors to resolve a 1-month old problem, SMTP set up. The last part of the saga was when I went around asking people how they configured their SMTP settings (because I haven't done it before), and also googling and reading how to send SMTP commands, only to have people asking me, "why isn't your vendor doing this for you?" "you are the first person I know who helps their vendor do such things" "why are you helping them?"

Simply because I didn't have a choice. The user calls me every 2 hours the past 2 days to ask me how to resolve the issue, saying that all the web forms must work and the vendors are saying that the web forms can't work without SMTP, and she has promised her boss that the site will be live this friday. After I passed all the information I gathered to the guy, he still said he couldn't get it to work. His boss said that this is an environment factor that is beyond their control, and if we are unable to provide them with settings that work, then they are unable to do anything.

The problem sounds simple. The vendor insisted that we need to give them a username and password to access the SMTP server. The SMTP server works on an unauthenticated mode, meaning no username and password required. I had 3 other systems sending emails through the same SMTP server and all did not need to authenticate with a username and password. Eventually, I sent the question to the helpdesk managing the SMTP server, and we are still waiting to solve the mystery.

That aside, I couldn't understand why the web forms couldn't work if we couldn't send emails. The web forms were storing the data submitted into a database, and the email part was to notify the webmaster that someone had just sent a feedback. They had been explaining and explaining for the past 2 weeks, but I couldn't understand until today, when I asked them, if I don't have any email available, how will the error message look like?

It turned out that the error message misled us into thinking that the form wasn't working! OMG The error message was "we encountered a technical difficulty in submitting the form". When I asked what other error messages he had, he said "we encountered a technical difficulty in saving the form". It was then that it suddenly became clear what the problem was. The forms were saving data into the system, but the notification portion wasn't working, so if we just change the "error message" to "thank you", the user who submitted the feedback will not think that it is an error, the feedback is still captured in the system, and the only thing is the webmaster will need to log in to the system to retrieve the data.

You can just imagine the users' reaction when this was reveal to them. What initially was on a critical path of failure turned out to be something not critical, and all we needed was someone to take the problem apart.

really serious troubleshooting business

I was troubleshooting a 5 month old inconsistent and intermittent problem earlier because I had been relying on the vendors and the issue had not reached the fire-on-the-backside stage.

The problem was easily resolved  in 30 minutes after I figured out the 10 lines of shell script that was fetching the files from the ftp server. I would not have been able to write the shell script from scratch, but I guess I have a good translator in my head.

Before that, 2 external and 2 internal data centre server administrators, 5 application vendors, infra manager, and myself, were all unable to resolve the problem. Just 2 weeks ago, the server admin was asking the infra manager why he is taking so long to solve the problem. The server admin was telling the application team that their code was wrong, the files need to reference the root instead of the sub folder. The application team blamed the script for not copying their files properly. I wasn't contributing constructively by chasing the application vendor for status updates as well.

11 people involved. What a joke, all I did was to tell the shell script to read from the sub folder instead of the root, but to be able to troubleshoot the problem, the person needs to understand shell script and html, which sounds like common sense, but I guess the stars were not aligned.

the account locking mystery

It start last monday 10.40 am when a user notified me that scampoint was down. We did the usual checks, but it was the first time we encountered the symptom - scampoint admin account was locked out, and irregularly, sometimes 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minute, finally stopping at 2 pm. 5 pm it started, locked out a few times and stopped at 6 pm. For the whole day we were checking the various logs, and well... nobody could identify the problem. The application administrator, server administrators, application support vendor. The source IP locking the account was the scampoint server itself.

Tuesday was peaceful. We did a reboot of the servers just to make sure. The server administrator was telling me that the application administrator doesn't even know how to troubleshoot the problem. I told him, administrators are like operators, and if they can do troubleshooting then they are better than others, like himself, he is an server administrator, but he helps us troubleshoot our scampoint issues sometimes. So that was a psychological nudge that I was passing problems to him.

After spending 2 days thinking, I asked these guys what they thought of my speculation, that it's a human triggered action, because it only happened during office hours, has irregular occurrence, and anybody who tries to log in with our user name will lock us out with the wrong password, which is so easy to find out because our admin account name appears all the over place, if you know how to look. What I could think of was just typing a wrong password for a particular user name will lock the user out. Everybody was still clueless, so I escalated to their infra manager, i.e. my infra guy, to ask him for ideas. He told me that he was impossible to prevent an account from getting locked out.

I went back to the administrators to tell them that we will be on our own so I asked the server administrator whether he could write us a script to automatically check our account every minute and unlock it if it is locked, and then send an email notification. His incentive was to feel satisfied that he has contributed to improving the productivity of the team, have lunch in peace, and not having to take turns to monitor whether the account got locked, else we will all have to write an incident report because the intranet is down when everybody is out for lunch. He was helpful enough to agree, and that took some stress out of our plates. My agreement with him was that it's just to help us save time to solve other issues until we can think of something for this issue. Today we had about 10 burning scampoint issues on hand. One of those days where it feels like I am defusing a time bomb, and I cannot make any mistake. The scampoint administrator is already quite stressed out and tickets are piling up.

Luckily for today, the account locking happened 3 times between 10.10 and 11 am, and for the rest of the day it was fine. I was trying to reproduce the problem, but couldn't, this is just self-affirmation I won't pass as a hacker. Thanks to the guy who told me it's impossible to prevent the account from getting locked out, I turned to my friends, and one of them suggested that I lock down the account. I was so glad that he had been thinking about it for 2 days too! So we will try that tomorrow and if it works, we may never know what caused it, and it may be a pity not to know the answer to the mystery.

who wrote it?

Me: [addressing question to vendor project manager] why was this sentence written this way?

Vendor project manager: Hmm, I didn't write this, I am not sure.

Me: Who wrote this paragraph?

Vendor project manager: [vendor technical director] wrote it.

Me: [addressing question to vendor technical director] why was this sentence written this way?

Vendor technical director: I don't know why it was written this way, I didn't write this.

Me: Who wrote this?

Vendor technical director: [vendor project manager] wrote it.

Me: When? [vendor project manager] said he didn't write it.

Vendor technical director: he typed it after the meeting he had with your team.

Users and I all burst out laughing.

strangleton

I am usually patient, calm and good, but when I am sickish, or lack sleep, I become aggressive and any one who annoys me makes them a strangle-ton, adapted the word from singleton.

Me: have we settled the outstanding issues with the user?

Vendor: yes, we have already explained to the user that it cannot be done and customisation effort is required.

Me: and the user agreed?

Vendor: yes, the user agreed to have it in phase 2.

Me: there is no phase 2.

Vendor: yes, there is no phase 2

Me: if we make this application too difficult to use, they wont use it and we don't need to think of any phase 2.

Vendor: yes correct.

Me: the users asked for a breadcrumb trail, why can't we give it to them since it's out of the box?

Vendor: because we didn't give them rights to open the root level folder so the breadcrumb trail doesn't work.

Me: why aren't you giving them rights to the root folder?

Vendor: because giving them rights to open the folder means they will be able to see the files and delete.

Me: can't we just don't give them rights to delete those files?

Vendor: no, because we don't want to break inheritance, and they have delete rights at the root.

Me: what are these files at the root folder?

Vendor: system files, the application pages that the user needs to use the system, that's why we cannot allow them to delete.

Me: why do you store the system files with the user files?

Vendor: because we only used one document library for our applications.

Me: how hard is it to separate the system and user files into two document libraries?

Vendor: it's a simple change of configuration.

Me: then can we do that and give users the breadcrumb?

Vendor: yes

Me: we need to give the users as much productivity features as much as it is within the cost and scope and breadcrumbs is there, and we will be short changing them by your (lousy) application design.

Me: now to the next issue. Why are you telling users to move files one by one instead of using the file explorer bulk moving of files functionality?

Vendor: because we only allow admin to delete/move files in bulk. Users are not admin, so they cant use the function, that's a requirement.

Me: but there is still an option for the user to move files via file explorer.

Vendor: they won't know because we didn't tell them.

Me: they are intelligent people and you think they won't know just because you don't tell them?

Vendor: yes

Me: can we tell them that they can use it?

Vendor: no, they are able to delete files in the file explorer when they don't have rights, but if they use the application they will not be able to delete.

Me: that's a breach of the security model of the product, are you sure you can delete?

His colleague explained to him that the user can click delete but if he refreshes the window, the file will still be there because he only deleted his local cached copy.

Vendor: there is another thing, if they delete from the file explorer there is no audit trail.

Me: audit trail for file explorer is an out of the box feature, why isn't it in the audit trail?

Vendor: because we customised the audit trail so there is no audit trail at the file explorer because we didn't want users to user that function.

Me: since you have already customised the audit trail, there is not much we can do for that, but tell the users that if they use the file explorer to move files, it will not be captured in the audit trail. I don't want the users to use the file explorer and then report the audit trail not capturing the move as a bug.

Vendor: ok.

Me: so think about how you want to tell the user everything we have discussed and we will meet the user next week.

Well the positive side of it is that I know we should be able to sign off the system next week after these issues are resolved. The bad part was to the user I took 2 weeks to get round to this because firstly, the vendors were supposed to be closing all the issues, and secondly, because I was away from work due to yaya's teacher's day school holiday, yaya's HFMD, 1-day course, my own body hibernating, my more ferocious users hunting me down immediately on my return, and of course the 10 chess boards that I rotate playing everyday. Another vendor I met today thought that I only have 1 project with them. Isn't that what every user thinks too? that they have tonnes of work whereas we are idling somewhere escaping from all the work. haha.

The trick is similar to multiplexing, getting the right sampling frequency of the user, and always appearing at a particular frequency to make them feel that you are full time with them, and always there for them, when you are not. And like in networks, you need to have a range of bosses/users/vendors with differet frequencies, so that you don't miss any packet. If they have the same frequency, then we need to double the sampling frequency. Ok digressed too much, the multiplexing bit was just a friday crazy bit, cant really be applied at work, there is no trick. Just pure brain juice being used to play 10 boards of chess concurrently. It helps if your opponent is 10 times slower than you are.

manual work rocks

This happened yesterday, a friday. My colleague asked me to attend her migration meeting at 4 pm but I told her I wasn't free. My phone rang at 5 pm, she asked me to go her meeting because there is an issue. So I went and it was over in 10 min.

Me: what is the issue?

Vendor 1: our migration service request only covers exporting of data to DVD but not importing.

Me: why not import?

Vendor 1: because there is no way to import a data table with attachments.

Me: so how is it supposed to be done?

Vendor 1: you will need to import the spreadsheet first, then manually attach the documents back one by one. It is not in our migration scope to do it. Users are also not going to do it.

Me: and why must I be the one doing it?

Vendor 1: that's the issue now.

Me: do you have any unique identifier to link the data row and attachments?

Vendor 1: no, you need to look at the title.

Me: how is the folder structure going to be like in the DVD?

Vendor 1: just 1, 2, 3, 4, 5, ...

Me: how do you get those numbers?

Vendor 1: we drag out one folder at a time, manually, the numbers are system generated.

Me: if the system has the number, can you export it out to the spreadsheet?

Vendor 1: yes.

Me: if you have the number, then you can create a link to folder containing these files. We just upload all these numbered folders into a folder.

Vendor 2 (vendor 1's boss): oh...

Me: do you understand?

Vendor 1: we can't link to the folder.

Me: we manually pre-fix the URL to the root attachments folder to those unique numbers.

Vendor 2 explained to vendor 1 because he couldn't understand. At this point, the users already understood.

Users: we will use this method.

Vendor 1: we will just give you the spreadsheet and DVD and there will be no links to the attachments.

Me: nevermind, I will show you how do it when the time comes.

He must be still lost, but I think his boss should be able to explain to him. I really don't how they survive as vendors to be recommending manual work. Actually I don't really have any business in this, but that's how my work is.