Innovation is important to success.  Sometimes we take on risk because we hope we’ll find a new and useful outcome.  Enter test automation.  There are many pundits who say test automation is the be all end all for a successful, efficient testing process.  While others claim it has limited value and will never replace manual functional testing (these and other experts also raise concerns about its ROI, especially the cost of maintainability as tests evolve along with product maturation). 

The SiteExecutive product team has ventured into the world of automated testing to:

• Accelerate  product development
• Improve build quality for each SiteExecutive release

Although it’s early in the process – we’re nearing the end of phase I of an ongoing program – we’ve had some meaningful results worth sharing.

Testing Framework and Focus

Our objectives for phase I included selecting a testing framework, writing test scripts for our high-priority functions, and evaluating and enforcing Web standards. We quickly settled on Selenium (http://seleniumhq.org/) as our testing platform.  There were two important reasons for this:

• Low (no) Cost – Selenium is open source, with a large community and is widely adopted by major software companies (Google, Mozilla, Oracle to name a few)
• Cross browser support – Selenium works with many different browsers (Internet Explorer, Firefox, WebKit (Chrome & Safari)

At this point, it’s worth noting that even though it has a large and influential user community, working with an open source tool like Selenium is still best for people who are comfortable with a higher level of uncertainty, willing to be self-sufficient and motivated by an opportunity to advance the efforts of the larger community – all of which I fully embrace.

Testing Automation

Once a tool was selected, we mapped out our strategy for tackling the high-priority tests which account for about 20% of our manual functional tests.  Next, we identified reusable tasks that we could generalize into global functions. At this point we realized we could not rely on the out-of-the-box Selenium commands and needed to write code to test code. This meant extending the Selenium API… No problem.

Selenium injects itself into a browser through JavaScript and simulates real human interactions like clicking, typing, etc. – though not always exactly as we would like. One example: by issuing a click command, Selenium didn't check to see if what a prototypical user was clicking really existed.  The workaround we developed has the click command wait for the element being clicked to exist before we issue the next click command.

Web Standards

Web standards presented us with another big challenge.  A Web content management system like SiteExecutive should provide the user with easy-to-use tools for building and maintaining websites and enforcing (ensuring) compliance with Web standards.  Thus, we have a critical need for inspecting HTML for both the expected layout and compliance with the W3C schools DOCTYPE specifications. Before we automated this process, a tester first had to visit http://validator.w3.org to validate the HTML and report if it was not compliant.  The process of complying with the WCAG 2.0 specification for Web accessibility presented a similar challenge. 

As mentioned previously, when people invest in a tool like SiteExecutive they expect it will create websites that comply with Web standards. Likewise, when a new version of Firefox or Internet Explore gets released there is a similar expectation – SiteExecutive will continue to function in the latest release regardless of how much code Mozilla or Microsoft changes. Fortunately, our use of automated testing helps make this happen – and, it also leads to higher quality SiteExecutive releases.

Man vs. Machine

At the end of phase I, we’ve developed automated tests for a number of high-priority functions.  Previously these tests took a human tester four to five days to complete for each platform – they now run in a few hours. Beyond the ROI associated with saving or redirecting manpower, automating SiteExecutive testing has other tangible benefits. For example, as humans, we’re wired to make mistakes.  With automated testing the details of a testing regimen are enforced and the human problem of overlooking problem issues is eliminated. 

Our exploration of automated testing has also given us new skills we can share with clients. Think of it as “Testing as a Service”. This would be especially applicable for clients who have developed or are developing custom modules and applications or otherwise extending SiteExecutive functionality in some unique way. Though they might think about testing their home-grown code during its initial development, they face the same problems Systems Alliance faces whenever third-party vendors release updates to browsers and or other applications in the SiteExecutive stack.

Want to learn more about automated testing – or if you’ve got some custom SiteExecutive code to test, get in touch.