What is the Worst Application to Work On?

a trash fire as a metaphor for horrible applications

When I took programming classes, one professor offered this piece of advice: whatever you do, don’t work on a payroll application. Why not? Because if you screw up your company’s payroll, everyone you work with is going to hate you.

I think that advice probably had some solid life experience behind it. If you’ve been a developer for a few years, chances are you’ve been responsible for keeping several different applications running. For those of us who measure our tenure in decades, it’s likely we’ve worked on dozens of applications – and also that at least one of those was really awful.

An incorrect production deployment caused Knight Capital Group to lose close to $500 million in one day. A bug in the initialization code for the Therac-25 radiation machine resulted in multiple deaths from radiation poisoning. Some applications are unappealing because they can leave you, or your organization, exposed.

As one of our solutions architects put it “working on any healthcare applications or ones that can seriously and directly impact someone’s finances would be my worst case scenario.” I would add to this list transportation systems (flight/train/rockets and soon, autonomous vehicles) and possibly, applications for use in the legal profession, or by any other organization likely to sue.

Developers who work in these fields have a good understanding of what could actually cause serious damage, and usually have safeguards in place to prevent that ever happening. Some of those safeguards may have actually been put in place after the fact – for example, when a payment system issued checks for ten times the correct amount due to a decimal point error. Software errors are a fact of life and kudos to the developers who shoulder these kind of responsibilities every day.

Another major factor is an application’s tech stack. Legacy applications, for obvious reasons, rate the worst. Many were designed based on earlier paradigms that allowed combining presentation, business logic and database access into one module. Even if the initial developers designed the architecture with some forethought, it’s unlikely subsequent developers respected their standards. On Quora, Bob Juch wrote:

“One that was written 30 years ago and has been changed by many different people over the years. You generally wind up with spaghetti code that is impossible to understand without spending days deciphering it.”

In cases like this, making changes to code becomes a digital version of Jenga – you never quite know what is going to happen, even with seemingly minor changes.

Another worst case scenario is to be saddled with an application built on technology that is no longer supported. Although internet forums make it easier now, there may no longer be anyone available to help you when you encounter an error. Second to this is migrating an application to a new platform. It will likely be an enormous effort that is undervalued by the organization.

And, of course, even the best application, if it has been developed using an older technology, adds little to your resume, except as it improves your understanding of the business logic.

Which leads to the next factor: the application type.

One of the more interesting things about development is the insight into business processes that you get from working on different applications. Developers tend to be gifted at finding something interesting in the most arcane details of an application.

But some applications are simply more interesting than others, or may just fit better with an individual’s personality. While nobody wants to work on a system that crashes constantly, there are developers who are at their best when firefighting – quick on their feet, fearless and creative. They may prefer having to focus on one big problem or just the excitement that comes from solving a problem under pressure. This type of developer is probably not going to be satisfied supporting an application that features lots of long running jobs, is highly stable and needs only occasional enhancements.

Another factor that can be important is the application’s business cycle. It’s no fun to spend years working on applications that have long development cycles, or those with an end of year crunch time.

End users, too, can make working on an application hellish. Dev’s tend to have antipathy towards their end users, and sometimes it’s for good reasons. Users may not have an understanding of the level of effort involved in maintaining and enhancing their application. They may be defensive, assuming that they’ll be blamed for errors that occur. Or they may be unreliable subject matter experts, or require you to go through management for everything.

Management is, of course, usually the real problem here. On Quora, John McLeod put together an excellent list of problems that (almost all) originate with management:

  • Aggressive deadlines.
  • Feature creep.
  • Long debug cycles.
  • Bad build infrastructure.
  • Failure of the managers to listen to the engineers.
  • Task churn. The inability to ever get anything completed because something more urgent is pushed in to the task list.
  • Toxic work culture of some sort.
  • A large engineering debt.

So what’s the worst application?

If you score your application based on these factors, you may be surprised where you land. For example, I worked on tax systems for several years. No one gets excited at the thought of working on a tax application. However, tax systems have some advantages. There are clear requirements from the IRS, who send out new regulations every year. Deadlines are set in stone. No matter how crazy things get, there’s a guaranteed hard stop. In addition to all of this, taxes have a set expiration date of seven years. That makes it easy to keep on top of versions and backups, and gives your organization a clear path to move off of outdated technology.

There is good reason to believe that the worst application you have ever worked on is the last one. Because most likely you still remember all the things that went wrong: the serious production problems, minor glitches and shortcomings that ultimately led you to move elsewhere. (It’s also likely that you are right now being blamed for all the above, it’s the last developer who touched it rule in action.) Unless you’ve worked on your current application for several years, you may not yet have fallen out of love with it. But, given enough time – and enough bugs – you may get there yet.

14 Comments

  • Much Happier Now says:

    The worst application I’ve ever worked on was for a multi-national health company. The whole system was based around a Cache database with .net frontends and some legacy (IE only) front ends. The whole system looked like it was cobbled together by 8 year olds that had learnt just enough to make a real mess of things and leave it for someone else to fix. Even the newer .net stuff was tarnished, with some single sub procedures around 1500 lines long. Reports could only be run from the legacy system and had to be hand coded in Cache ObjectScript, complete with raw printer commands to position the print head!! I lasted 1 year, then walked. And no, this wasn’t before 2016. Life was better after that horrible ordeal 🙂 If you ever come across Cache ObjectScript, run….run fast…don’t look back…even if they offer you twice the going rate, it’s just not worth the stress.

  • Henrik Thor Christensen says:

    The worst application to work on is, with not doubt what so ever, the one who hasn’t been tested properly. Where you sit and hope that things are going to be OK when it is released. That is the worst feeling you can ever have!!!

  • Anonymous says:

    He said “There are clear requirements from the IRS…” lol

  • ponyboy3 says:

    at a place i worked, there was a master mathematitian that coded a few applications. he must have coded the way his mind worked because no matter who was tasked eith maintaining it understood all of it. there was an email thread with snippets of his code. they were all very good and performant, just unmaintainable. essentially he was writing functional code in c#. everything he did was scrapped or rewritten.

  • In the late 80’s I had to modify a spaghetti direct access disk driver (asynchronous) application under MVS (Pre OS/390, pre Z/OS)… it was horrible so I decided to rewrite it from scratch in BAL. After QA was complete and it was moved into production it kept failing on a regular yet infrequent basis….

    hundreds of hours later I discovered a bug in the original DASD access software which the higher level application (I hadn’t touched it) leveraged in it’s logic.

    Of course the right thing to do would be to fix the application because that’s where the logic failure was… my code worked correctly… but no, it wasn’t possible to touch the application because they couldn’t find the source code… so I had to rewrite a part of my driver code to replicate the error!

    Bankers Trust NY 1980… 250 Vesey St.. the building doesn’t exist any more, the company doesn’t exist any more so I don’t think there’s a tale out of school here!

  • Roseanne Winn says:

    That’s triggering some flashbacks: Morgan Guaranty Trust 1988… West Broadway, and BAL too. Good times, good times!

  • Kevin Kinscherf says:

    Although this article is about software development, the trends noted apply to all forms of product development. The bullet list above is not unique to software and is often discussed in generic business school classes that try to give students a heads up. They are well known in the growing field of Project Management. While coding has its own specific headaches, a good project manager can work on and produce products in just about any industry.

  • Danny says:

    PlanoDan
    Any apps are the worst apps to work on, if they are undocumented apps and have any degree of complexity, which is probably 98% of them. It is relative easy to determine how everything works from reading the source code and help files, if any, given enough time, which of course a developer never has. The reason is that the “whys for procedures” and the original developers “logic or rationale”, are missing, for those who get involved later. Everything becomes legacy code at some point, no matter in what language it is written. Someone will have to maintain and enhance it. I agree with your list of problems caused by management. But the biggest problem I have experienced in corporate environments, is that IT is run as a cost center and they report to a manager who is almost always totally devoid of any in-depth understanding of IT. Managements suffer from the idea that anything can be controlled and made efficient by using good general manage philosophy. That perhaps holds true for the admin aspects of IT, but the managers have no understanding of software technologies, how complexity is impacted by requirements, what languages should be used, the role of good documentation, the difference between good or bad architecture, design methodologies, etc. Therefore, you get all the problems you mentioned, and more. They can make IT a nightmare. I have never seen an engineering department not run by an engineer, an ER unit not run by a doctors, a legal staff not run by a lawyer, etc. There certainly could be exceptions, but not many, in my experience. Therefore, the developers have to cut corner everywhere they can to meet the unrealistic schedules, juggle too many projects for the staff headcounts they are allowed, forced to implement systems before they are ready for production, no time to document anything, they allow users to continuously change their business processes thus constantly changing requirements, and many other obstacles to developing good, functional, and reliable systems. In my experience, companies have bad systems and are dissatisfied with IT because of bad management, not lack of good developers or development practices, period. I have no idea, but I suspect many developers leave the profession or are at least very dissatisfied with their profession, because of constantly having to work under those conditions. I know I was.

  • TBQ says:

    My “best” project was decades ago. Before anyone had uttered the words “pair programming” or “agile” in a software development context, we were a three-man team in a single room with a blackboard (yeah – did I mention it was a while ago?). We worked out our designs together on the blackboard, and had a weekly “deliverable” which was a presentation of the week’s design subject, and later the elements of code. Each week produced a short memo describing designs, choices, etc. We went on to write the documentation (end-user documentation) before the code, and wound up referring back to it regularly – think daily. Design and documentation took some time, but coding was fairly quick and resulted in solid code. The only verified bug was one caused by a firmware bug during debugging with a firmware breakpoint in a specific location, probably due to some page fault. I should add that this was our master’s thesis project, and the weekly presentation was to our professor (later a Turing award recipient… :-)). It probably couldn’t have happened this way in a commercial enterprise.

    One of the “worst” project I’ve worked on was one where the developers were denied access to the tools needed to do the job. This company had a very strict policy on allowable software and versions driven by production version control. While we were allowed to use patched (Service Packs) copies of the compiler and developer environment on our developer workstations, in order to deploy, we had to compile the app in the *original* retail version – no service packs. We had a single shared computer to do that on. The environment was notoriously flaky and it would often require multiple attempts with cold boots in between. Several SPs had been issued, but we couldn’t deploy with any of them. I only stayed there for 6 months, but we were still able to vastly improve the usability of the application I worked on. I learned that later from a colleague who remained there after I left.

    So the moral of the story is that we (as developers) can do good even though the conditions are off. If we don’t like the conditions, we work to change them, look at ourselves, and in the worst case find greener pastures (when available to us… :-))

  • Davyd McColl says:

    Anything involving Polymer. It’s not a framework — though fanbois will want you to accept it as such. It’s nowhere near the level of ease-to-use as Angular (JS or 2+). It’s just (imo), half-baked. Every day is a surprise in how the “framework” can let you down.

  • Tom says:

    Sitecore 7 without a doubt.

  • Anonymous says:

    1st – An application nobody uses
    2nd – Anything containing Lotus Notes
    3rd – Anything containing SharePoint

  • Peter Butler says:

    This was over 50 years ago. But it illustrates management rules preventing a reasonable solution to the goal.

    The rule “We are a COBOL shop. There will be NO assembly language used.” The goal was to write an extremely flexible report generator. The method sounded simple to this very junior programmer. COBOL program reads a template COBOL file, fills in file descriptions and what totals are needed from another file. I could write target programs for the various reports. Given card punch program source (remember card decks anyone?) I could put such a program together in a week. With automation the program would be auto-generated in minutes.

    But there was a problem.

    On that system CDC-3300(?) the COBOL run time could not write a file in a format readable by the COBOL compiler. “Blocker Deblocker” format (card images on disk) was the only format the compiler could read. Alas, that format was not available as a COBOL File Descriptor. (I wonder now if print output could have been directed to a file. It might have been in the proper format. Print files could be spooled.)

    However, 10 lines of assembly code would solve the problem. The Rule™ prevented anybody from thinking about looking for a workaround. A future compiler might be able to compile a file it wrote. But not this one.

    So … I got the job of building the COBOL program that could do almost everything. It was a PIG! Among much else I had to do reverse collating sequence transformations such that an always the same “control card sort” would put the data in the proper order. And I still needed assembly for the variable file description. (Call asm module followed by close file where the assembly code picked up the file descriptor from the close. Return back after close file.) Blah! Got it working with a short data file.

    Over the next two years I was unavailable to consult. Draft notice from Nixon. Vietnam thing in progress. On my return I found out that the program was so big it had to be run standalone at night. Thankfully by my return the program was no longer being used.

  • Dagoberto Vega says:

    Worst case scenario I’ve work is “migrating apps”, old software trying to fit in the new technology.
    No specs, old source code, not very good coding, a mess here, a mess there, if it works don’t touch it and the best of all: customer request “make it work like in production mode, and we want it for next year” (needless to say it was August).
    Eventually application took more than a year thanks to multiple new request from the customer and a lack of knowledge of the application from the QA team of that company (new hires).
    Stick to your plans, your resources and your requirements; customers must understand that we can do job in our way, not their way.

Leave a Reply