"Trust me.” It’s what we want our learners to do, but we have to earn that trust. An e-Learning product may have a beautiful interface and superlative instructional design, but they don’t mean a thing if the product is full of spelling errors, if it is “buggy,” or if it crashes often. Any defect that makes the product appear amateurish, any lack of quality, will have a significant impact on the learner’s gut-level response (trust) to the training.
It all comes down to the quality of development. As Hicks says in Aliens, “It’s a bug hunt.” The answer to the challenge of creating trustworthy e-Learning products is management of any quality issues that arise. Issues can include things like bugs, typos, poor directions, ambiguities, etc.
Any issue management system has two primary goals:
- To ensure that no issue is forgotten, and
- To ensure that each issue is addressed in a timely manner.
Unfortunately, the way many e-Learning development teams handle product quality control does not achieve either of these goals. The issue management system typically consists of a simple lined pad. This system ensures that addressing issues in a timely fashion cannot occur, and that losing issues is likely. Not only that, but it fosters a “blame game” environment between clients, management, and the developers. This is what I refer to as “The Hard Way.”
As a team leader, it has always been fairly obvious to me that I get the best out of my people when I keep them happy. The equally obvious way to deal with a major source of developer unhappiness is bug-reporting software. Unfortunately, on small or one-off projects, or in small e-Learning development companies where capital is hard to come by, no return on investment comes from such an expensive purchase.
Therefore, I developed and deployed a paper-based system, similar to the lined pad used by many teams — but with some innovative additions. If you do not have a bug database available, and cannot afford one, then this manual process will allow you to handle issues more professionally. (Please note: If you can afford bug-reporting software, such as Axosoft’s OnTime, or RMTrack, then I highly recommend making the investment.)
First let’s look at how many e-Learning development companies and departments handle issue management The Hard Way.
The Hard Way
In my experience, many e-Learning development teams use a system to handle issues that arise during e-Learning development that is simplistic at best, as you can see in Figure 1. Developers track bugs on a lined piece of paper.
Figure 1 Issue management tracking The Hard Way uses a lined pad, on which “bugs” are tracked as they are discovered.
Each line has spaces for the number of the bug, a description of the bug, the date found, a signature area for the developer to sign when the issue is resolved, and the date of the resolution.
The first problem is the ability to assign and track the changes. With a wide variety of issues on one sheet, it is difficult to have multiple developers working on the issues contained in that one sheet. You either have one sheet that must be passed around (slowing down the process), or you have photocopies — each with different signatures in different places. It’s easy to see that items can easily be missed or doubled up on with the photocopy solution.
In the single-sheet solution, the sheets themselves could be lost. If a developer hangs onto a sheet because he hasn’t fixed a low priority bug, this stops high priority bugs (listed further down the sheet) from being fixed until much later.
In most cases, you do need to have multiple developers, of different skill levels, working off the one sheet because the order of the bugs is according to the date each was found, not by the order of difficulty (let alone importance). One sheet can contain any number of bugs, from cosmetic display issues to program crash bugs.
The next problem is that it is simply a given that the developer will have fixed the bug on the first attempt. The developer signs the form to say she has fixed the bug, usually seconds after fixing the problem. If the bug is still extant in the next release, or if the fix has caused additional errors, then the blame falls on the developer. There is no record of what the developer attempted to do in an effort to fix the bug. If a different developer is assigned to the issue in the next round, there can be a lot of duplicated effort as the second developer attempts all the failed solutions that the first developer discarded.
There is usually no way to avoid feature creep in this system. There is also the possibility that two testers will have two different ideas of how an interaction works, and after a fix was put in place to satisfy the first tester, the second tester logs a bug. I have seen these issues ping-pong across multiple versions. The developers understandably become upset when asked to change back something they have just “repaired.” The frustration level can only grow when both requests are constantly reiterated.
Under this system, there is no real check in place to prevent unknown bugs from occurring in the test software sent to the client. This is a very serious issue, and one that has caused major friction between client and production house and between developers and the production management teams.
The system is also extremely “blame-centric.” Calling the problems “bugs” identifies each issue as a developer fault, as the term is specifically used to identify “coding errors” — even when the problem may lie in a different area, such as instructional design, feature creep, or logical errors in the functional specification. This can cause a large amount of disharmony in the team. The effect of this is often to cause management to see the development team as inept or lazy, and the development team to see management as overbearing, uncaring, and lacking knowledge about the developmental process.
The Easy Way
As the team leader stuck in the middle of this type of situation, I felt the need to find a better way. The client’s management team turned down requests for industrial-strength bug tracking software because they simply felt the problems would go away if the development team would just do their jobs properly. The only solution left to me was to improve upon the system we already had.
Concerned about the blame-centric nature of the word “bug,” I started an initiative to re-name “Bugs” as “Issues.” While this might sound like a wishy-washy management initiative on par with corporate mission statements, it actually worked. Development team members were more comfortable coming to me saying, “We have an issue with our software,” rather than saying, “We have a bug in our software.” The managers at the client took a little longer to come onboard, but they also came to like the word, as an “issue” was something they felt comfortable dealing with, while a “bug” was something they could only worry about.
The definition of the term “issue” came to include any problems with the product. The primary test of the physical product occurs when comparing it against the Instructional Design Document (IDD) and the Functional Specification, but that left us open to spelling mistakes in the IDD or emergent issues from logical errors in the Functional Specifications. We therefore modified the system to capture errors in the documentation as well. Absolutely any issue to do with the product required the use of an Issue Report. We logged all of the Issue Reports in an Issue Registry so that if one went missing we were at least aware that it was missing and who lost it.
The Issue Registry
We needed a better capture mechanism. I started by assigning the lined pad to the task of tracking Issue Reports. In this way, it became a tool, or Registry, that gave me an overview of where we were in the project.
Each line of the Registry tracks a single Issue Report. We divided the pad into these columns:
- IR Num — The number of the Issue Report
- Issue — The Short Name on the Issue Report
- Reporter — This field contains the name of the person who raised the Issue. The IR must return to this person for final sign off.
- Possession — Who has the Issue Report at the moment (Team Leader, Developer, Auditor, or Reporter)
- Resolved — The date the Issue Report was considered to be resolved
- Resolution/Status — During the life of the Issue, this tracks the status of the Issue. This is simply one word to describe where it is in the process. Once the resolution is completed, this space contains the final resolution, as indicated on the Issue Report.
The Issue Report
The Issue Report is a full page, front and back, that equates to one line on the old pad. As well as the details captured in the old system, the new form includes areas for severity, plus fields for the reporter to describe “what happened,” “what they expected to happen,” “how to replicate the issue,” and any “available workarounds.” The team records details and comments on the back of the page.
This additional information allows developers to home in on the issue much faster than before. It also prevents unnecessary, or ping-pong, rework, as the “expected” field will often highlight reporter misunderstandings (i.e., where the reporter requests a change away from the functional specification). The “available workarounds” allow us to verify the issue severity. See Figure 2.
Figure 2 The Easy Way to track and manage issues still uses paper, but it also expands on the simple lined pad.
A severity checkbox system makes it possible to respond to the most severe issues more quickly. The system includes the following severity options:
- Urgent — Stop work. Resolve problem and release new test version ASAP.
- High — Must be resolved in next release of testing.
- Moderate — Should be resolved in next round of testing but may be delayed.
- Low/Cosmetic — Must be resolved before Beta testing.
In The Hard Way approach to issue management, a major system crash bug could hide in a page of spelling corrections. It is important to note that the severity is only the severity as perceived by the reporter, not the actual severity of the issue. I have seen instructional designers who regard spelling errors as critical, while they are not fazed by the system crashing. The important point for me is that, as team leader, I was able to identify my reporters’ needs and wants and respond to what they considered critical in a timely fashion. Only in two situations have I found it necessary to change a reporter’s selection of urgency:
- . Slab-handed use of the Urgent field. This causes the team to stop everything and rebuild a new test version after the fix is in place. This is not really necessary if the request is only to move an image a couple of pixels.
- Using anything above Low for spelling mistakes. While I understand the fear factor involved, especially from writers, the fact is that you can most effectively batch spelling errors in one spell-check process. As this can take a long time (depending on the size of the product), it’s best to do it only once, at the end, when no new content is likely to appear.
We reserved the bottom of the front side of the page for the issue management process. There we track information such as when the issue was first reported and whether the development staff was able to replicate the issue.
It also includes a number of check boxes for resolutions:
- Resolved — repaired as requested
- Not an Issue — product aligns with specifications, reporter is asking for a change in specs
- Not our Issue — Issue is a result from third-party influence — refer back to third party.
- Changed Documentation — product correct, documentation incorrect, documentation changed in response.
- Deferred — All stakeholders agree to accept that the issue exists, but to do nothing about it in this product release.
This allows a lot of leeway in dealing with issues. The Hard Way only has one method of recording a resolution — the developer’s signature implying that the bug has been “fixed.” If the bug was not a bug at all, a duplicate bug will show up in the next round of testing, and the team leader will also have to deal with an angry tester. Nothing appears to change — but the issue has been closed.
The new sheet also includes signature lines for the developer, the Auditor (more on that below) and the Team Leader if the Team Leader was not the Auditor. This documents that the resolution was reviewed after it was put in place.
The back of the form is one large field. It allows the Developer to describe what he or she did, including logging proposed solutions that did not work — and why they didn’t work, and details of how the Developer resolved the issue. This enables other developers to leapfrog failed approaches if the resolution later did not work or caused additional problems. It also makes it possible for more experienced developers to re-evaluate failed approaches more quickly. If the junior developer gave up on an approach because of a misconception, the reasons why the junior developer swapped to a different resolution will be evident in the notes. A more senior developer may then be able to turn a “failed” approach into a successful one. Also included here are any Auditor comments from testing, if the testing failed.
The Auditor’s role
One of my main aims in developing this system was to create a more harmonious work environment for my team. I strongly believe that a happy team is more productive than an unhappy one. To that effect, I needed a way to ensure that the developers were protected from the constant haranguing that they were receiving from sales people and project managers. (No offense intended to anyone you know. If you’ve ever been a developer, you know how strong the potential for this is.)
I did not subscribe to the view that the developers were lazy and inept. I felt that I had a good team and that they simply needed more support. The most contentious problem between management and development had to do with bugs, especially repeated bugs, or worse, fixes that broke something else. I needed to find a way to ensure that when an issue came back to a reporter for evaluation that it was resolved, and that it had caused no new issues. I created a new role in the team — called the Auditor.
It is the Auditor’s responsibility to sign off on the resolutions. Not only is the Auditor a super-tester but this person also takes on the responsibility for the resolution. If a resolution subsequently fails, it is no longer the developer’s fault; it is the Auditor’s fault. This placed a protective barrier around the developers on my project (who were already working as hard as they could and often putting in ten to twenty hours of un-paid overtime per week). It also ensured that the developers were not being continually distracted or required to justify themselves. Management didn’t take to this idea immediately (and I was required to drag a few sales and project managers away from the developers to the Auditor), but in the end most of them liked the idea.
This Easy Way seems a lot harder at first. There is a lot more paperwork. As team leader, I could no longer just hand out sheets and get back to my own programming. I had to validate the incoming sheets, and register them in the registry. Any sheets that were not legitimate Issues, or not product-related Issues (i.e., not legitimate “bugs”) had to be carefully handled via different processes. Auditing also took up a large amount of time.
It only seems harder. Imagine what would have happened under The Hard Way. We would have had more errors slip through; we would have had at least 20 different examples of feature creep; we would have had ping-pong errors. It took up a lot of time to manage the new system, but we saved so much more, and not only time.
