Kuzuan
Reporting for Duty
One of the major themes that has been carried across the previous entries for Behind the Scenes is the high priority we set for organization and good records keeping. The overall QA process is a much more straightforward affair for the zOMG! team because we have always been a tight-knit group. However, if you imagine that some development teams in other companies may contain hundreds of members (with some members working from different buildings or event countries), then you can see how even small problems can quickly grow in magnitude. Because of this we use a number of different reports as another layer to protect against human error or other mistakes. Each of these reports serves a unique purpose and they all come together to help us bring you a better zOMG! experience.
Each new build that we receive for testing is accompanied by a set of design notes that describe the changes we will test. These notes are understandably brief because the programmers have just spent several weeks bringing the features they are describing to life. 
While QA keeps up with new development as it progresses, this is often our first chance to see the complete list of changes. Because of this, all of us review and discuss the design notes in order to write down any questions we have regarding how new features should function or what the details of a change are. These questions are put together into one report that is then sent to the programmers for additional clarification. Turn around for answers is quick and when we receive the desired clarifications we are better able to fine-tune our testing strategies.
On top of recording all of our bugs and enhancements in our bug tracking database we also use Daily Bug Reports as a way of summarizing our progress for the day, sharing our thoughts about the current build and describing our plans for the next workday. The biggest part of each DBR is our bug activity: the amount of bugs that we opened, closed or re-assigned back to the programmers and the numerical IDs for each bug. These reports are a great way of summarizing the day and organizing your thoughts for tomorrow. 
We put together two types of Feedback Reports: reports on specific game features and a pre-release report. All of you by virtue of providing your opinions about the game contribute to feature reports when they are being put together. Often we'll be asked to read the forums in order to determine user response to a specific feature and provide a summary of the common sentiments. User feedback from the zOMG! main forum provides us an invaluable source of information and a fresh, outside perspective that allows us to make zOMG! an even better game. 
Pre-release reports are created from the feedback on new builds that we ask our users testers to check out. This feedback serves as a sort of mine canary and allows us to judge whether we're on the right track for the wider audience.
Whenever we near the completion of a new build cycle we always put together a "Must Fix" bug list for programmer consideration. This list consists of bugs that QA feels should be fixed before the build is pushed to the live servers. These bugs specifically deal with new features or changes and all of them are ones that have been opened only during the current build cycle.
Often we see many requests to delay fixes for bugs presented on this list. This is understandable since, by this point, everyone wants to push the new build and move onto the next step. We work closely with the programmers to try and work out a compromise on as many issues as possible in order to prevent outstanding fixes from snowballing from build to build and to improve the polish of the new build. Ultimately the reality of the given situation trumps all and we have to prioritize issues and create plans so that we can both meet our deadlines and release a quality product.
Several weeks ago we discussed some of the different test types that we use to find bugs. Among them we covered Acceptance Tests and how this approach helps us increate our coverage of the game.
Acceptance Tests are very importance because if they pass then we have a green-light for starting to test the build further and if they fail a new build must often be assembled before we can continue testing. Because of this a report is put together and sent out to the entire zOMG! team whenever an Acceptance Test is completed. This helps keep everyone in the loop and updates them on whether testing has commenced on the build. In itself the report includes the completed file that lists all the areas we check and notes on that particular acceptance test. These notes can discussion the difficulties that occurred during the test and what problems were found.
I hope you've enjoyed this new addition to our Behind the Scenes series! We have come a long way since our beginning all these weeks ago and have covered all the major aspects of how we test zOMG!. Please post any questions you may have about this or any previous section and we'll be happy to answer them. Thank you!
Previous topics:
zOMG! Behind the Scenes - The Great Cycle of Builds
zOMG! Behind the Scenes - Toolkit of Trade
zOMG! Behind the Scenes - Method in Madness
zOMG! Behind the Scenes - Bug Report's Anatomy, Pt. 1/2
zOMG! Behind the Scenes - Bug Report's Anatomy, Pt. 2/2
One of the major themes that has been carried across the previous entries for Behind the Scenes is the high priority we set for organization and good records keeping. The overall QA process is a much more straightforward affair for the zOMG! team because we have always been a tight-knit group. However, if you imagine that some development teams in other companies may contain hundreds of members (with some members working from different buildings or event countries), then you can see how even small problems can quickly grow in magnitude. Because of this we use a number of different reports as another layer to protect against human error or other mistakes. Each of these reports serves a unique purpose and they all come together to help us bring you a better zOMG! experience.
- The Q List
Each new build that we receive for testing is accompanied by a set of design notes that describe the changes we will test. These notes are understandably brief because the programmers have just spent several weeks bringing the features they are describing to life. While QA keeps up with new development as it progresses, this is often our first chance to see the complete list of changes. Because of this, all of us review and discuss the design notes in order to write down any questions we have regarding how new features should function or what the details of a change are. These questions are put together into one report that is then sent to the programmers for additional clarification. Turn around for answers is quick and when we receive the desired clarifications we are better able to fine-tune our testing strategies.
- Daily Bug Report
On top of recording all of our bugs and enhancements in our bug tracking database we also use Daily Bug Reports as a way of summarizing our progress for the day, sharing our thoughts about the current build and describing our plans for the next workday. The biggest part of each DBR is our bug activity: the amount of bugs that we opened, closed or re-assigned back to the programmers and the numerical IDs for each bug. These reports are a great way of summarizing the day and organizing your thoughts for tomorrow. - Feedback Report
We put together two types of Feedback Reports: reports on specific game features and a pre-release report. All of you by virtue of providing your opinions about the game contribute to feature reports when they are being put together. Often we'll be asked to read the forums in order to determine user response to a specific feature and provide a summary of the common sentiments. User feedback from the zOMG! main forum provides us an invaluable source of information and a fresh, outside perspective that allows us to make zOMG! an even better game. Pre-release reports are created from the feedback on new builds that we ask our users testers to check out. This feedback serves as a sort of mine canary and allows us to judge whether we're on the right track for the wider audience.
- "Must Fix" List
Whenever we near the completion of a new build cycle we always put together a "Must Fix" bug list for programmer consideration. This list consists of bugs that QA feels should be fixed before the build is pushed to the live servers. These bugs specifically deal with new features or changes and all of them are ones that have been opened only during the current build cycle.Often we see many requests to delay fixes for bugs presented on this list. This is understandable since, by this point, everyone wants to push the new build and move onto the next step. We work closely with the programmers to try and work out a compromise on as many issues as possible in order to prevent outstanding fixes from snowballing from build to build and to improve the polish of the new build. Ultimately the reality of the given situation trumps all and we have to prioritize issues and create plans so that we can both meet our deadlines and release a quality product.
- Acceptance Test Report
Several weeks ago we discussed some of the different test types that we use to find bugs. Among them we covered Acceptance Tests and how this approach helps us increate our coverage of the game.Acceptance Tests are very importance because if they pass then we have a green-light for starting to test the build further and if they fail a new build must often be assembled before we can continue testing. Because of this a report is put together and sent out to the entire zOMG! team whenever an Acceptance Test is completed. This helps keep everyone in the loop and updates them on whether testing has commenced on the build. In itself the report includes the completed file that lists all the areas we check and notes on that particular acceptance test. These notes can discussion the difficulties that occurred during the test and what problems were found.
I hope you've enjoyed this new addition to our Behind the Scenes series! We have come a long way since our beginning all these weeks ago and have covered all the major aspects of how we test zOMG!. Please post any questions you may have about this or any previous section and we'll be happy to answer them. Thank you!
Previous topics:
zOMG! Behind the Scenes - The Great Cycle of Builds
zOMG! Behind the Scenes - Toolkit of Trade
zOMG! Behind the Scenes - Method in Madness
zOMG! Behind the Scenes - Bug Report's Anatomy, Pt. 1/2
zOMG! Behind the Scenes - Bug Report's Anatomy, Pt. 2/2
