Google

Google Test Automation Conference - 2008


Scheduled Presentations

Advances in Automated Software Testing Technologies and Implementation
Elfriede Dustin

This presentation discusses our various efforts to streamline automated software testing implementations. By “automating the automation” via use of existing open source tools, component “reuse,” and by automating as much as feasible related to automated software testing at IDT.

Using automation to enhance automated software testing is an effective way of removing error-prone manual human interaction as much as possible in order to allow for efficient automated software testing implementation and scalability. Automated testing is not scalable if the implementation is throwaway, and not scalable if components have to be implemented over and over again given different systems-under-test. We achieved automation scalability by automating most all facets of it. This includes:

  • automated test case generation
  • automated test data generation
  • automated Requirements Traceability Matrices (RTMs)
  • use of existing test frameworks, and expanding on those, instead of developing them from scratch
  • use of an Interface Design Language (IDL) to generate automated test code
  • use of compare utilities to automate the expected vs. actual comparison
  • use of available open source log parsers
  • taking advantage of VNCServer technology for automated testing
  • defect tracking automation

Elfriede Dustin leads the efforts in automated software testing research programs at Innovative Defense Technology (www.idtus.com), a company that specializes in the design, development and implementation of Automated Software Testing Solutions. Elfriede has authored multiple books on Software Testing, most notably “Automated Software Testing,” books which have been translated into various languages, has published numerous articles and regularly presents at conferences. Her upcoming book “Implementing Automated Software Testing,” co-authored with IDT’s president Bernie Gauf and co-worker Thom Garrett will be available in bookstores in Feb 2009.

Atom Publishing Protocol, testing your server implementation
David Calavera

The Atom Publishing Protocol was designed as a standard protocol for publishing web content. It was built on the Atom syndication format, but since this format was designed with blog post syndication intentions, specific data extensions appeared in order to allow developers to use the protocol into any web application. This talk will discuss the best practices in order to test AtomPub services. We'll see how to build effective tests that take into account all the capabilities of an extensible standard and how to get started with some useful tools.

David Calavera is a software architect and test lead developer with 11870.com. David is a member of the Spanish Ruby community and co-founder of the Spanish Ruby User Group. He has also contributed to many open source projects like Hudson and Netbeans, and is one of most active commiters to Apache Abdera.

Automated Model-Based Testing Of Web Applications
Atif M. Memon

Today's web user employs sophisticated applications with user-interfaces that offer many degrees of freedom to explore, share, communicate, and retrieve information from the web. The complexity of the web-user interface (WUI), in terms of the degrees of freedom that it offers, makes both development and quality assurance challenging, thus giving rise to defects in the application. Quite often, end-users will perform unanticipated and untested sequences of actions, causing the web application to fail or behave abnormally. The purpose of this presentation is to encourage participants' discussion in the area of web testing, propose solutions to automate it, and share problems encountered. A description of a research project, and associated tool called STATEST, is used as a catalyst to provide a starting point for this discussion. This research project advances the state-of-the-art in web testing by generating automated test cases for web application testing, using a state machine model-based approach.

Oluwaseun Akinmade has a Bachelor of Science degree in Computer Science from the University of Maryland. She has extensive experience with web development and testing. She has served as an intern with Telvent Farradyne Inc. as a software tester for two years. She has developed numerous web applications using Microsoft Visual Studio and ASP.NET Framework. She has focused on Regression Testing and System Testing.

Professor Atif M. Memon is an Associate Professor at the University or Maryland and in the Institute for Advanced Computer Studies. (UMIACS). He has a Ph.D. in Computer Science from the University of Pittsburgh with a dissertation entitled "A Comprehensive Framework for Testing Graphical User Interfaces." Prof. Memon's other numerous awards and honors include a CAREER Award from the US National Science Foundation (2005), a Gold Medal from FAST ICS (1992), two Andrew Mellon pre-doctoral fellowships from the Andrew Mellon Foundation (2000-2001), best paper award at the Australian Software Engineering Conference (ASWEC 2004), and the post-proposal graduate research award from the Department of Computer Science at the University of Pittsburgh (2001). Prof. Memon ranked in the "Top 50 software engineering scholars" in a recent article in the Communications of the ACM. Prof. Memon has served on numerous research and advisory panels and committees, including the US National Academies of Sciences and the US National Science Foundation. He is currently on the Editorial Board of several journals including the Journal of Software Testing, Verification and Reliability (STVR). He is also a National Academy of Sciences panelist in the area of Computer Science and Information Technology, Pakistan-U.S. Science and Technology Cooperative Program, sponsored by United States Agency for International Development (USAID). Prof. Memon is the inventor of the GUI testing software called GUITAR.

Boosting Your Testing Productivity with Groovy
Andres Almiray

Everyone knows that testing takes time, but you shouldn't skip testing because of a hard-to-meet deadline. What can you do to be sure you'll have the following weekend free without worrying that a sudden call will spoil the fun? The answer is to let your testing code be groovier. This session discusses Groovy, a dynamic language for the JVM™ machine whose close integration with the Java™ programming language makes it ideal for testing purposes. With Groovy you can write less code and be more expressive, you can leverage your knowledge of JUnit extensions (such as dbUnit or XMLUnit) to speed up development, and it is also TestNG-friendly.

Andres is a Java developer with more than 8 years of experience in software design and development, currently working for Oracle as a Principal Software Engineer. He has been involved in web and desktop application development since the early days of Java. His current interests include Groovy, Swing, and developer testing. He is a true believer in open source and has participated in popular projects as well as starting his own. Andres maintains a blog at http://jroller.com/aalmiray

Context-driven test automation: How to build the system you really need
Pete Schneider

All tests automation systems have certain common tasks. Different tools tend to focus on some tasks and put less emphasis on other tasks. While examining 11 home-grown test automation tools at F5, we found certain patterns. There were six different tasks that all tools addressed to varying degrees. We found that the tasks emphasized depended on who developed the tool and who the intended users were. Some tools required more work from test-writers and less from the test-framework developers. Others offered more services to the test-writer but required a larger team of framework developers to build and support the tool.

We have grouped the tools examined into four different contexts, depending on who would be writing the tests, who would be looking at the results, and how long the tests would be in use. This talk will describe how the intended audience for the tool should inform your design decisions.

Pete Schneider graduated from Northwestern University’s School of Music and promptly got a job as a test engineer at Microsoft. Over the past 18 years he has worked for numerous companies on a wide range of products including multimedia authoring tools, games, consumer applications, air born email systems, and medical information systems. He currently leads a team of developers building functional automation tools for F5 Networks.

JInjector: a Code Coverage and End-To-End Testing Framework for J2ME and RIM
Julian Harty, Olivier Gaillard, and Michele Sama

A wide range of J2ME applications are GUI-rich with significant code dedicated to handle the user interface (UI). To maximize the chances of detecting bugs, effective testing of such applications require end-to-end testing on devices. Unfortunately there are few software tools suitable for testing J2ME applications. Our tool, JInjector, instruments the application under test in order to run automated tests in emulators and on the actual devices we need to support. At GTAC we will present two of the features provided by JInjector: code coverage, and end-to-end testing.

As a proof-of-concept we will show a demo of an end-to-end test suite applied to Google's Java ME YouTube client running both in the WTK emulator from Sun Microsystems, and on a Nokia N95. The injected code enabled testing of most of the features of the application, including search, login and video upload as well as navigate through the videos. We discovered that the injected code needed be customized to support the custom GUI code, and changes to the GUI code sometimes means the tests have to be updated - an ongoing maintenance task. We will also explain how this end-to-end test can be integrated with a build system.

Julian Harty is a senior test engineer at Google with 2 years of testing mobile wireless applications. His passion is making software work for users far and wide. He's presented at numerous workshops and conferences over the years.

Olivier has been working at Google for 2 years as a software engineer in test. He's been working on creating and improving some of the testing tools used at Google.

Michele is a PhD student in testing and validation for adaptive and context-aware applications. He is also an intern in Google working on testing frameworks for J2ME applications.

Practicing Testability in the Real World
Vishal Chowdhary

There have been some studies both about the concept of Testability and how to implement it, many of which use test hooks. In spite of its obvious benefits, we still find low adoption of testability concepts. Why is there not more emphasis on core testability principles? In this analysis we present our experiences applying testability concepts and lay down guidelines to ensure testability consideration during feature planning and design. We initially lay the groundwork by briefly going over the testability concepts and the SOCK model. We then look into the typical thought process for a Test Developer when trying to plan and automate test cases for a particular feature. Also, here we present some key insights as to why we believe practicing testability is a challenge to many projects. We will present real life examples that we have encountered in our career which exhibit how testability considerations could have made our testing simpler. We also discuss influence of testability on design of a feature. Building from these examples, we present a check-list of for each of the testability principles. Reviewing this testability checklist will ensure that core testability principles are considered while testing a particular feature. The result is simplified testing for that feature and thus reduced costs overall. We finally go over an exercise where we get to apply our principles to an Event Processing component and realize the benefits of applying testability concepts.

Vishal is in his fourth year at Microsoft. He holds a Bachelors degree in Engineering from Bombay University & a Masters in Computer Science from the State University at New York, Stony Brook. He started with the BizTalk Server team, where he owned the new API feature testing, along with parts of BizTalk Server tracking features. Presently, Vishal is involved with designing the test framework and test planning for future versions of BizTalk Server. He is focused on making testing simpler and interesting by employing new testing strategies and processes that eventually prevents defects from entering the product.

Simple tools to fight a big battle: "Continuous integration on a minimalist platform"
Komal Joshi and Patrick Martin

Continuous Integration (CI) is a software development practice where members of a team integrate their work frequently. Usually each person integrates at least daily leading to multiple integrations per day. When integration is verified by an automated build (including test) the gross and obvious integration errors are detected as quickly as possible. This places the focus on the more valuable and less straightforward issues and allows a team to develop cohesive software more rapidly and to higher quality standards. This much is not subject to much debate. However it is perhaps not so well understood that the design of the CI environment and execution of the processes is equally critical to the product's success.

In this presentation we will outline the details of the end to end implementation of a continuous integration environment using 'simple tools' found either in the development tools' toolchain or in the base OS forming part of the development platform. We will also highlight the reasoning and benefits of tools we have used to produce this system.

There are many CI solutions available in the market for the Java and .NET development environment, but implementing continuous integration for Delphi based standalone desktop applications raised some questions. We had to come up with a solution which is minimal, elegant and would tick all the boxes mentioned below:

  • Easy to implement, configure, and maintain
  • Does not introduce a technology not already mandated by the application
  • Could be easily replicated and debugged on any appropriate machine
  • Single click solution for common tasks
  • Has a clear audit trail

We will explain how the overall flow of information within the system was achieved by simple and transparent techniques such as plain text configuration files and scheduled tasks. For example, a single line change in the configuration of a build can fire tests on release mode or highly instrumented builds or indeed both if the machines are available.

Komal Joshi started her career with IBM Rational where she worked on entire Rational Product Suite including Rational Rose, Rational Clear Case, Rational Portfolio Manager and Rational Method Composer (the next generation of Rational Unified Process).She has been very active in generating intellectual property and her two disclosures were rated as publish and are published on IP.com. One of her disclosure related to Automation tools was rated successfully for filing patent by IBM. She has been actively maintaining a website dedicated to software testing www.testinggeek.com . Currently she has started her own venture Atlantis Software Limited offering services of software test consulting, test automation outsourcing, and web development. Atlantis software limited is also a reseller of IBM Rational Robot proxies.

Komal has presented papers in many international software conferences such as Rational User Conference, Software Quality Symposium Asia Pacific. She will also be hosting a workshop on Selenium and Selenium Remote Control in the British Computing Society Special Interest Group in Software Testing Conference in Sep. 2008.

Patrick Martin has worked in a professional capacity in the software development field for 10 years; prior to that he was one of the cave dwellers that inhabited poorly resourced academic research labs gaining a PhD in Solid State Physics. Previous employments have involved 2 start-ups and 3 more established financial services related software firms. During his time in the trenches he has been exposed to technologies ranging from (in no particular order): Native Windows desktop and Web application development, Java development in a number of environments and frameworks, .NET development and architecture and processes including build process configuration, source control configuration, and even a few last-minute scrambles to deploy some systems.

Taming the Beast: How to test an AJAX Application
Markus Clermont and John Thomas

Typically we address the problem of testing an AJAX application through a plethora of big end-to-end tests and (hopefully) high unit-test coverage. In our presentation, we'd like to outline the main problems with this approach. Testing through the GUI is expensive, gives limited insight into the system, and often take only the 'happy paths' into account. While unit tests don't suffer from many of these problems, they alone are not sufficient mainly because they give little insight how the components interact with each other and don't give confidence that the business logic and functionality of the system meets the requirements. In our presentation we would like to demonstrate an effective testing strategy for a sample GWT-based application which goes beyond "testing just through the GUI".

Markus Clermont is currently a Test Engineering Manager at Google in Zurich, being involved in various projects. Markus studied Applied Computer Science at the University of Klagenfurt, Austria, where he also did his PhD in Software Engineering. After working as a Jr. Lecturer at the University Klagenfurt and various consulting jobs in Austria, Markus worked as a Senior Researcher and Director of Postgraduate Studies at the Software Quality Research Lab in Ireland where he researched test data generation from formal specifications. Markus is particularly interested in test automation in Agile Software Development (including tools, process and education), developing useful approaches for test automation and assessing the adequacy of tests.

John Thomas is currently an Engineering Manager at Google in Mountain View, working on the AdWords product focusing on test strategy, infrastructure and engineering productivity. Prior to Google, he was a Senior Software Engineer at Siebel Systems (now part of Oracle) where he led a number of design and development initiatives around tools and test frameworks. Prior to that, he held software engineer positions at SAP Labs and StreamSage (now part of Comcast). John has an MS in Computer Science from the University of Texas at Austin and a BE in Computer Science from University of Mumbai, India. John is interested in agile methodologies and testing approaches related to web consumer applications.

The New Genomics: Software Development at Petabyte Scale
Matt Wood

The Human Genome Project aimed to determine the entire DNA sequence of man. It was completed in 13 years after an international effort and a billion dollar budget. To further our understanding of genes, proteins and their function, the Wellcome Trust Sanger Institute is building a next generation sequencing platform to investigate what makes us who we are.

Biology has become a computing problem. Working with the vast amounts of data produced by new sequencing technologies requires flexible software capable of handling massive throughput. However, maintaining software quality in environments undergoing such rapid change is challenging. This talk will delve into our current process and the tools we use in the software teams to ensure that we meet tight deadlines in a very high throughput, multi-petabyte environment.

Matt Wood heads up software development at the Wellcome Trust Sanger Institute, where he is responsible for the software that drives the Institute’s world class sequencing facility and putting biological data to work to further our knowledge of health and disease. Matt’s interests lie in adding value to such biological data through scientific software. After completing his PhD, he’s built grid services in Nottingham, web-scale search engines in New York City and genes in Hinxton. Matt is an advocate of agile development with Scrum, sensible semantics and, of course, the duck billed platypus.

The Value of Small Tests
Christopher Semturs

Whenever unit-tests are written, chances are good that the code is not isolated. Many tests run against a given class/interface/function. The design didn't consider isolating the dependencies of this function, which makes tests slower, larger, less focused, and likely depending on external dependencies (e.g. database access) which decreases the reliability. In addition, it disables much of the interaction-based testing or verification of behavior at error conditions. In this session I will demonstrate the value of small, isolated tests and explain good engineering practices for how to structure them. I will review examples in several programming languages and domains including C++ and Java, but also on web applications, UMTS Protocol Design and Testing (Device Testing), and other platforms.

Christopher Semturs was working in the Software Testing Industry for the last 9 years. For 8 years, while working as a software engineer at the European software testing company Objentis GmbH, he focused on providing good testing tools for the test process as well as development infrastructures for higher program quality. Within these years he learned a lot about quality in Software Development, both from the process and the code point of view. One year ago he joined Google as a Software Engineer in Test. Since then he focuses on enabling developers to write efficient tests without feeling any overhead in their day-to-day work, to ensure that quality "just happens".

Using Cloud Computing to Automate Full-scale System Tests
Marc-Elian Bégin and Charles Loomis

The ultimate goals of any software-testing effort is to make the software more robust and reliable, thereby reducing support costs and increasing customer satisfaction. Recent trends to use unit testing and other techniques in the software build process help reach those goals. However with the move towards systems of interacting services, full-scale system tests are critical, yet are rarely done as systematically or completely as unit tests because of the large manual effort required to deploy a full-scale system and run the tests.

A couple technical advances have made automation of full-scale system tests possible. First, the advent of commercial cloud computing services, such as Amazon Web Services, provides a convenient, dynamic backend infrastructure avoiding the need to interact directly with system administrators for machine deployment. Second, working prototypes from the ETICS project have shown how to run tests in such an environment and capture the relevant deployment and test information.

Challenges to having an effective testing regime include sociological issues as well. Use of the ETICS system within the EGEE grid-computing project has shown that the “testing team” really consists of a large, distributed community of people ranging from the software designers to the end users. This experience highlights the need for a common, collaborative testing framework that is open to the full testing community, but at the same time, minimally intrusive to existing practices.

Six2 provides a framework, using cloud computing, to automate the deployment of a complete set of services for tests, running those tests, collecting results, and tearing down the test deployment. Using the dynamic pool of resources that clouds provide allows the automatic, low-cost deployment of test configurations.

This presentation briefly presents the need for automated full-scale system tests; it continues with an explanation of the technical aspects of using cloud computing to provide a framework for these types of tests and a summary of early results. The presentation will also discuss sociological issues, exposed in the ETICS/EGEE experience, that impact the implementation of an effective testing regime.

Following Mechanical and Aerospace Engineering studies, Marc-Elian Bégin worked from 1994 to 2004 in industry in the space domain for customers such as the Canadian and European Space Agencies, as well as the Canadian and European Space industries. He specialized in real-time system simulators and test-bed development and contributed to the delivery of no less than a dozen systems. Moving on from the space field in 2004, he joined CERN (the European Laboratory for Particle Physics, in Geneva) in the Grid Computing group on the EGEE project (the largest public grid infrastructure project). In 2006 he joined the CERN team leading the ETICS project, which provides automated build and test services for managing complex software, using on grid technology. In the summer of 2007, Marc-Elian, with co-author Charles Loomis, created Six² a company based in Geneva, developing build, test and integration services for distributed software systems.

©2008 Google - Privacy Policy - Terms and Conditions - About Google