Balneus

Australian Lefty on Politics, Governance, Science and Info Management

The parliamentary dog ate my homework

Posted by Dave Bath on 2009-01-13


If a note, apparently from a parliamentary worker, to my thoughts about the implications of a dead parliamentary webserver is correct, then going offline without warning to the public, or redirection to a "don’t panic" page, still indicates incompetent IT planning and/or contempt for the public.

When an offered excuse is self-condemning, you have to wonder if it reflects the real reason, or whether the full implications of the excuse are understood… either way, not good!

Richard‘s comment to my report of the dead parliament site and my initial thoughts about the possible causes and consequences raises a number of questions which I’ll explore over the fold, and it certainly doesn’t explain the continuing misbehaviour of a new Senate submissions system, except by pointing to the same root cause of hopeless IT governance, identified as systemic through government in the Gershon Report commissioned by, and promised to be implemented in full by Tanner.

If the comment about scheduled power outage is true, it does rule out some of the possible reasons for the loss of service, but raises others that were so unlikely that I hadn’t initially considered them… but have been partially explored in comments by myself and Robert Merkel (of Benambra and Larvatus Prodeo fame).

The first thing to note is that a scheduled power outage is scheduled, known in advance.  A power outage to the parliament (which is really not so much a building as a highly-fortified small suburb) is a pretty big thing, and should be known days, if not weeks in advance.

Thus, while "Richard" notes that occupants were notified of coming disruptions, the lack of notice given to the public shows contempt and/or negligent management.

Traditionally, sysadmins let all users know that an outage is coming up, even in near emergencies.  As I pointed out elsewhere…

As a sysadmin from the mid 1980’s, for SCHEDULED maintenance (from say 17:00) we’d always gave warnings on login ALL DAY (if not the day before), and even in emergency situations, apart from a kernel panic (see also Screens of Death), we’d wall(1) everyone with typically 5 minutes grace time so (wherever possible) they could save files/records they were editing.

Although you may be correct, a rude shutdown without warnings for “maintenance” is a Bad Thing, should not be tolerated, but is unsurprising in these latter days of poor rigor (but IT cognitive rigor mortis).

After all, when you are in an office, and there is a sudden need to shut servers down, what happens?  You get emails, on screen messages, and IT staff running around like headless chooks telling everyone to save what they are currently doing and not start anything else.

If the outage was planned, and even in the absence of the failover/handover systems to ensure continuous service, the following trivial steps were not taken, and they should have been:

  • A large notice on the parliamentary home page informing the public that the system would be off-line, so they could plan to make submissions at other times;
  • A redirection from all web requests directed at aph.gov.au to even just a static page on another computer, saying "Don’t panic.  Parliament has not been attacked by terrorists.  Scheduled system maintenance is underway and we’ll be back in action by Monday morning at the latest;
  • For those logged in and making submissions (like me),:
    • Extra warning when logging in about the forthcoming outage;
    • Logging of all started sessions (your email address is your login account) and transactions so that followup "Can you please resend what you were trying to upload, or use standard email to [the email of whichever committee you were uploading to]".

No such email confirming my upload (as promised), or apologies and request to resend were sent to me, and it has been a couple of days since the system came back online  This implies that there was no proper planning to trap the information from logs about people making submissions to parliamentary inquiries, meaning that the committees are possible denied all the information they should have when making decisions about recommending legislation or changes to draft bills.

I also bet that they made darn sure that any processing of parliamentarians pay and re-imbursements weren’t corrupted mid-batch!

In essence, if Richard’s comment is true, then it is effectively knowing when the "cleaner will trip over the power cord" and not having a plan to handle it.  Things could only be worse if you don’t know when your cleaner will be disastrously unco-ordinated!

But as Robert Merkel noted elsewhere, and I’m in firm agreement:

In any case, I would have thought that the APH site was important enough that outages of that length are avoided.

Obviously the BCP (Business Continuity Plan) is completely inadequate.  If a planned outage can pull down not only external access, but operations for internal customers (there is no excuse for the latter), then what are the consequences of an unplanned outage caused by meteorites, terrorist crackers, or more mundane terrorists with explosives?  If there is no verified BCP, then what information would be lost?  If losing such information isn’t important enough to have a BCP, then this implies the work of the parliament isn’t that important (you’d lose any paperwork too if there was a meteorite or something similar).

Again, the scheduled outage does not offer an excuse for the misbehaviour (incredibly slow response or session crashes) of the newish submissions system.  Parliamentary IT staff still have to answer for that!  (See description of the problem and comments by various people on causes here.)

So, if the given rationale is from a person-in-the-know, it is such a self-condemnation that it raises into question the other possibility (equally damning of IT security planning)… the system was being cracked and the plug was pulled.  This is just a minor possibility, but if cracking was the cause, the "scheduled shutdown" excuse can be understood, although there should be a fairly detailed investigation by the DSD and lots of resources put into fixing the problem, given that we can only expect more criminal and state-sponsored cracking in the future.

It is worth remembering that parliamentary systems, from cleaning services to IT, are "independent" of the party in power, apart from the resources granted to them.

As noted elsewhere, given that the reworking of various senate systems are reported in the 2006/07 and 2007/08 reports from the Black Rod, and consistent with the attitude of Howard to comments about process by independent parliamentary staff, and given that enterprise architecture and cultural changes (which take into account expected funding) take two years or thereabouts to filter into daily operations and capability, any blame should probably be directed to the previous federal government.

This of course doesn’t mean the current government should sit on its hands.  Parliamentary IT services obviously need greater funding, and much greater oversight by DSD and AGIMO, two agencies that have demonstrated competence time and time again, but are too often sidelined.

I’ll be writing about my concerns to web.senate@aph.gov.au and webmanager@aph.gov.au, but I REALLY WANT YOUR THOUGHTS BEFORE I WRITE, whether you are technologist, service delivery manager, or private citizen.  So please comment on the most appropriate posts.

After all, whether all government systems have the same inadequacies, are exposed to the same untreated risks, or whether it is just the parliamentary IT system, it affects us all.


See Also/Notes:

Advertisements

6 Responses to “The parliamentary dog ate my homework”

  1. Lyn said

    Parliamentary staff may be independent, but the last government didn’t even try to create the impression they were the least bit interested in this intertubes thingy. This government created the impression there’s be technological leaps and bounds, but it’s hard to come up with anything they’ve done right so far.

    I’d been thinking it wouldn’t hurt for them to have some kind of mailing list of blogs and other networked sites to communicate with those most likely to care about such things. If Dave Bath had received notification of this, the whole discussion would have been different.

    So far we have underfunded kiddy laptops, the blog debacle, filtering, and Telstra’s broadband hostage crisis. This outage is small beer in the scheme of things, and barely related, but it does add to the overall bad impression.

    Hopefully, someone somewhere is having a steep learning curve. And hopefully, it’s not us.

  2. Dave Bath said

    Oh dear…. another scaling problem perhaps in the Senate Submissions system. Looks like they aren’t expecting too many visitors!
    At least the speed is a bit better now.
    Server Error in '/Submissions' Application.
    Server was unable to process request. ---> The process cannot access the file 'C:\Inetpub\wwwroot\SCIDWebServices\licence\Aspose.Total.lic' because it is being used by another process.
    Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
    Exception Details: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> The process cannot access the file 'C:\Inetpub\wwwroot\SCIDWebServices\licence\Aspose.Total.lic' because it is being used by another process.
    Source Error:
    An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
    Stack Trace:
    [SoapException: Server was unable to process request. ---> The process cannot access the file 'C:\Inetpub\wwwroot\SCIDWebServices\licence\Aspose.Total.lic' because it is being used by another process.]
    System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) +533462
    System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) +204
    SCIDDocumentUpload.SCIDDocumentUpload.ViewDocument(String Id) +47
    UploadInterface.ViewDocument(String url) +11
    comittees_ViewDocument.Page_Load(Object sender, EventArgs e) +122
    System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +15
    System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) +34
    System.Web.UI.Control.OnLoad(EventArgs e) +99
    System.Web.UI.Control.LoadRecursive() +47
    System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +1061
    Version Information: Microsoft .NET Framework Version:2.0.50727.832; ASP.NET Version:2.0.50727.832

  3. […] to corrupt dataskepticlawyer on Marohasy admits to attempts to corrupt dataDave Bath on The parliamentary dog ate my homeworkMick Sutcliffe on How to save Israeli lives with bombs: target the pubsverdictvolkov on Can you […]

  4. […] Parliament website up and stumblingDraft note to Parliament House webmaster « Balneus on The parliamentary dog ate my homeworkDave Bath on Marohasy admits to attempts to corrupt dataskepticlawyer on Marohasy admits to […]

  5. […] by Dave Bath on 2009-02-08 For those who were following the "Parliamentary dog ate my homework" discussion, (and later posts), I’ve sent in a note based on this […]

  6. […] to Parliament House webmasterPromised note sent to parliamentary webmaster « Balneus on The parliamentary dog ate my homeworkSaving the Goulburn-Murray bill: Overwhelming support « Balneus on Brumby’s nightmare – […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: