Intelligent Roaming
Just as I was completing the automation of the regression suite, I was asked to take over testing of IS-136 TDMA intelligent roaming. All I was told was that it was a very large feature and that a co-op was available to assist me.
I spoke with the developer and he indicated there were over 600 numbered requirements as he gave me a copy of the document. The schedule did not permit writing a formal test plan/specification so, I decided to march right through the requirement list writing scripts.
For about the first week, I had the co-op work exclusively with me to get a feeling for the tool, the scripting language and the general test methodology. I then selected some working AMPS scripts and used them as examples when showing him how to read the spec.
I then gave him some AMPS requirements and asked him to study the requirements, study the spec, ask questions and write scripts to cover the requirements. He quickly came up to speed and we split the task with him writing AMPS scripts while I focused on TDMA which was also called DAMPS.
Authentication and message encryption were areas where the tool did well. Both the cellular authentication, voice privacy and encryption (CAVE) algorithm for authentication and the cellular message encryption algorithm (CMEA) for encryption were embedded in the base station meaning, access authentication and message encryption could be verified. The developers were always evasive when I pressed for details about the algorithms but the process seemed to work and as long as a test worked in the lab, it always seemed to work in the field.
Encryption of sensitive user data was also possible but beyond the scope of intelligent roaming. The single major drawback of the test tool was that only messaging could be verified, actual verification of speech on the digital traffic channel and analog voice channel was a future development task that never happened due to carriers migrating away from IS-136 TDMA on the 2G -> 3G path.
Without our test tool, the inhouse developed base station, we wouldn't have been able to do the work nearly as well as we did. For tests such as cell selection, we could program multiple channels having different characteristics such as carrier ID and RF level.
The channels were fed into a combiner which the phone was attached to. In theory, we could program up to 16 different channels for the phone to select from but in reality, we never needed more than three to test any given requirement.
We put in long hours and were able to find a significant number of problems. I believe our failure rate exceeded 10 percent although, since this was the first time a rigorous protocol verification had been attempted on the phone, many of the defects were legacy protocol bugs that we just happened to find while doing intelligent roaming testing.
One aspect of writing scripts so that each script contains all code necessary to execute a test is that the total number of lines of code gets very high. Our scripting language was a hybrid of C and java. One script required about 500 - 750 lines of code. The end result being hundreds of thousands of lines of code. Of course, any single script was 500 - 750 lines but executing all 600+ scripts took about seven hours.
Feedback from the field was positive. Of the 660+ requirements, just over 600 scripts were written with only a single digit number of minor defects reported from the field. The goal of course is always zero defects but in light of the extreme time pressure, lack of test plan, test specification or even written test cases, we exceeded expectations.
