Climbing out of the Box: Bringing Your Quality Career into the Digital Era

Anne Hungate

Deadlines, commitments, policies, and procedures are a few of the many things that are essential for teams and organizations to deliver and grow. But too often, these constructs become constraints on our thinking, our creativity, and our growth as individuals.  Today’s business climate requires fast-paced delivery and real-time interaction with customers; putting our own development second feels necessary for survival.


Join executive and entrepreneur Anne Hungate for a dynamic exploration of how we can drive satisfaction, career growth, and life balance by climbing outside of the box defined by our roles. A dedicated learner and willing risk-taker, Anne will share her story of applying her learning as a Quality professional to help organizations thrive, and provide real examples of others who have stepped outside of their roles to see the value a Quality mindset provides across organizations.


The Digital Era needs daring dedication to accuracy and customer experience – perfect conditions for Quality professionals to thrive… if they’re willing to climb out of the box!

Testing Your Metal – Drawing Parallels Between Testing Concepts And Heavy Music

Paul Grizzaffi

The phrase "test your mettle" means to test what one is made of, what one is prepared to endure. In both testing and automation development, we often have to endure suboptimal situations and difficult challenges such as unrealistic expectations, resistance to change, and an insistence that automation has to come from test cases.


In coping with these challenges, many people find solace in music; listening to and playing music can be a cathartic experience. Music, specifically titles and lyrics, can also be thought-provoking and motivating. Paul Grizzaffi, a self-proclaimed metalhead and rock-a-holic, is one of those people who is greatly affected by music. He not only has experience with testing and automation, he’s quite the aficionado of heavy music. His love of automation, testing, and music inspired the pun in the title.


Join us as Paul takes us on a tour of some impactful lyrics and song titles that frame memorable messages about testing and automation. Among the featured messages: we make our own expectations, automation is software development, and there are important business aspects related to automation and testing that cannot be ignored.



  • Find your friendlies; they can help evangelize your message

  • Automation does not replace testers; it does not have to come from test cases

  • Only automate if there is value, not just because of process.


  • Twitter

The 5 Layers of Security Testing

Alan Koch

“Information Security: Everyone’s job every day” (… and that includes testers!)  Security is too important to ignore, so we will look at all of the ways that we as testers need to collaborate with our Security professionals and our developers to ensure that security is addressed well in our project’s testing and verifications.

We will first lay a foundation of security concepts and terminology.  Then we will walk through the 5 Layers of Security Testing and explore what is involved in doing a good job of each.

  1. Test Security Features.  We will start by looking at some of the security mechanisms our developers design into their code (e.g. encryption) and the ways we can collaborate with them to design good White-Box tests.  Then we will turn our attention to the various types of Security Requirements so we can ensure that our system testing of those requirements is adequate.

  2. Perform Negative Testing.  Not just for quality purposes, negative testing is also an important part of our security testing.  In this context, we will look at both white-box and black-box Fuzz Testing.

  3. Test Misuse and Abuse Cases.  To our consideration of Use Cases, we should add Abuse Cases and Misuse Cases – How might people abuse or misuse the system and how should it respond?

  4. Test for Common Bugs that open Vulnerabilities.  The mistakes the developers make that have the side-effect of opening security vulnerabilities are widely published and well known.  (Think buffer overflow.)  We will look at some to the best sources of this information and discuss how testers can collaborate with their developers to find those mistakes early.

  5. Ensure Readiness for Release.  We will look at the final verifications and testing that can confirm that the system is secure enough to release, including Penetration Testing.

  • Twitter

Re-Shaping The Test Pyramid

Jennifer Bonine

In this session Jennifer Bonine will explore new shifts in testing paradigms. Demonstrate an AI first testing method that integrates with your current manual and automation testing, and understand AI that aids your app teams. Re-think where you want to spend time and money in your testing team in a challenge that plagues most companies of too much to test and too little time.

This will re-position the testing and quality organizations to not be the last part of what happens, but to providing valuable insights and actionable data for your C-Suite to drive business decisions.

Key Takeaways:

  1. Ideas to reshape your test strategies

  2. An understanding of AI solutioning and where to begin implementing

  3. Analysis of available tooling options in the AI space. (Vendor agnostic)

    Behind Your UI Scripts: So Much Room for (Network) Activities

    Andrew Griffin

    Automated UI testing for web apps is commonly implemented across the industry, but concurrently tracking our applications’ network traffic is seen much less. We can squeeze more value out of our already existing automated UI scripts and, with just a little bit of extra effort, add another layer of potential validations that they can perform. The data that we extract can assist us in ensuring the integrity of the data we are sending, checking HTML responses for errors, or even tracking response times for performance. This session will cover some tools we can use to add network traffic monitoring to our tests, and the different types of uses we might have for this extra data.

    • Twitter

    Going Scriptless: Implementing Model-Based Test Automation in a Large Enterprise

    Anita Uppalapati & Joseph Beale

    Over the past year Nationwide Insurance has moved in the direction of using a scriptless model for automating their software testing with a tool called Tosca. Join us for a discussion of this topic with the following highlights:

    Anita Uppalapati of Nationwide will describe the journey, why they made this decision, and how it is being rolled out in waves.

    Joseph Beale of COSQAM will talk about the benefits of scriptless automation versus other methods and go deep into model-based automation using the Tosca tool.

    There will also be a general discussion around how this type of test automation changes the nature of testing teams and contributes to the overall digital transformation process.

    Need for Speed- “How to Performance Test the right way”

    Annie Bhaumik

    Nobody likes a slow application. In today’s world most users abandon a website if it does not load within 3 sec ( based on Google’s research). Performance failures can cost millions of dollars to businesses. There is a tremendous need to performance test the right way. In this talk we will focus on how performance testing is beyond only load testing and how to design realistic performance tests. We will also talk about different types of performance tests and how they can identify specific weaknesses in the system and can prevent performance failures in production.

    The Era of Intelligent Testing: How Machine Intelligence is Shaping Future of QA

    Brian Brady

    Traditional testing isn't keeping pace with the speed of DevOps and Continuous Delivery, but hope is not lost; AI will be used to make testing faster and more accessible to all.


    Existing QA solutions were built for a world where software changed infrequently. Highly adopted tools such as Selenium, Appium, and JUnit require a specialized skillset, and require too much maintenance once you start factoring in the brittle nature of tests, and the infrastructure required to run tests at scale. But there is still hope for QA in machine intelligence. Next-generation ML tools are here to help QA keep up with the agility of modern software delivery practices. You'll learn what ML strategies are being employed to make this possible, and what the future of QA - intelligent testing - looks like.



    • An overview of the QA tooling and vendor landscape

    • Demystifying the buzzwords - what machine intelligence in testing means today and tomorrow

    • How implementing DevTestOps can help increase test coverage and improve product quality

    • Twitter

    Shift-Left Testing: QA in a DevOps World

    David Laulusa

    As we try to shrink cycle times where does QA fit in this new paradigm? Are they the ugly step children that are left in the kitchen to eat only after development has finished the feast? And what of the start up that probably doesn’t even have QA?

    This talk is aimed mainly at the Development and Operations folks who will have to wear the QA hat.

    Deming said ‘Inspection does not improve the quality, nor guarantee quality. Inspection is too late. The quality, good or bad, is already in the product. As Harold F. Dodge said, “You can not inspect quality into a product.”

    And Aristotle said, “Quality is not an act, it is a habit.”

    Larry Smith, in reference to ‘Shift Left Testing’ concept, said, “Bugs are cheaper when caught young.”

    A Glimpse of Test Automation in Hadoop Ecosystem

    Deepika Achary

    Big data is a buzzword that describes a huge volume of both structured and unstructured data that is huge and challenging to process using traditional software engineering techniques. This extensive data is processed through Hadoop ecosystem.

    While big data provides solutions to complex business problems, it poses bigger challenges in testing these scenarios such as effectively managing, maintaining and using the data. This makes testing big data applications different and significantly more complex than testing regular applications. Data is transferred and transformed among many services and needs to be validated at various transition points and processes such as data ingestion, data processing and data storage.


    We will be focusing on how to interact with these components in a test automation framework:

    1. HDFS – Highly fault-tolerant distributed file system for processing large data sets.

    2. HBASE – Highly scalable non-relational database. 

    3. KAFKA - A messaging system that lets you send messages between processes, applications, and servers.

    4. SOLR – Lucene based search platform.

    We will demonstrate the processes and tools we have successfully implemented in our test automation. By using this automation approach, we have achieved faster feedback and low manual intervention during the testing process. This helped us a lot to do effective functional verification and getting accurate results. We will explore different ways and also share tips that can help every tester, new or seasoned, to understand how to use automation as part of testing the above-mentioned Hadoop ecosystem components.

    • Twitter

    Automated unit testing frameworks for ETL projects

    Dennis Colburn

    One important characteristic of a “good” piece of software is that it is testable.  If you can’t test something, then how do you know it works?  How do you know that it still works when making future changes?


    I’ve been working in data warehouse environments for several years on ETL projects.  I’ve had success implementing automated testing in this domain, but one comment that hear frequently from ETL developers is “oh, you can’t test this Informatica workflow…it’s just too complicated…we don’t have data for that…”.  Well, I think the questions in the first paragraph still apply.  How do I know that my ETL process works if I can’t test it?


    In this session we will focus on unit tests that are applicable in the ETL domain.  We will learn how to identify unit tests, how to setup these unit tests in your automated test framework, and how automating these tests might just change how you design your ETL processes.

    Enhancing your Test automation Scenario Coverage Using Selenium

    Eran Kinsbruner

    Success in cross-browser test automation relies on many variables. Today’s reality forces practitioners within DevOps/Agile teams to join effort in assuring quality, removing risks, and releasing fast. To meet these goals, business testers, developers, and test automation engineers need to work together with the proper technology stack that matches their skillset. Join Perfecto’s Chief Evangelist and author, Eran Kinsbruner, in this session as he provides recommendations for high coverage, high reliability, and maintainability of cross-browser test automation. In this session, Eran will walk through the following topics: - Trends in cross-browser test automation. - Introduction to test automation using codeless and BDD Selenium tools. - The material differences between the three approaches: code-based Selenium, BDD, and codeless. Including a live demo of the various approaches.

      I Can't Do That -- Using the Force To Unlock Your True Testing Potential

      Ford Arnett

      One of the most common questions asked in the QA space is "How can I transition from a manual QA to a more automation focused or technical QA?" Using his experience in leading his automation team, as well as teaching dozens of people with all ranges of technical experience, Ford will attempt to answer this question.

      With countless tools to learn and seemingly never ending amount of testing types to master, it can be very intimidating to get started. There are numerous places to start depending on your passions, and what a tester would like to achieve. You don’t have to master five programming languages in order to write your first API test, and you don’t have to be a Selenium expert to create your first record and playback test. Much like you don’t need to be a Jedi to use the force, automation can be a powerful tool in your arsenal even if you are not an automaton master. 

      Join Ford Arnett as we go through a hypothetical automation journey, starting with self doubt and mental barriers to success then moving onto small wins and finally how to become a master of your chosen craft.

      • Twitter
      • Twitter

      A Coach and a Geek:  How to Coach, Test, Measure and Build an Accountable Quality Team!

      Jeff Van Fleet & Mark Adams

      Mark Adams, ESPN's National College Basketball Analyst and VP of Client Success for Lighthouse Technologies and Jeff Van Fleet, President of Lighthouse Technologies combine their passion for coaching, testing, quality and team building in this unique, fun and interactive 60 minute presentation. Learn how a coach and a geek have joined forces to help IT teams effect change, drive better productivity, challenge the status quo and become better quality teams.

      • Twitter

      Why Diversity, Inclusion, and Representation is Necessary in Tech and Quality Assurance

      Jenna Davis

      We all have heard why diversity and inclusion is important to the success of any organization but that also applies to technology and quality assurance. In this session, we will go over the history of diversity in tech, go through examples of how some failures of technology could have been avoided if there was inclusion, and dial down on why specifically diversity & inclusion is important in Quality Assurance. We will discuss implicit biases and ways WE can make a change for the future!

        Automated ADA Testing with Pa11y and Ruby-Cucumber

        Jerren Every

        Session Abstract: ADA compliance is a necessity to that is often a costly and time-consuming part of the software development process. This talk covers the story of how a company can move from manual ADA testing to automated ADA testing utilizing Pa11y and Ruby-Cucumber. 

        Sections include: 

        • Background - Overview of ADA and how it pertains to software development. 

        • Constraints - Limitations of traditional ADA testing ideologies.

        • Pa11y Overview - Discussion of how Pa11y is utilized as a standalone tool.

        • Integration - How to utilize Pa11y within a Ruby/Cucumber framework.

        • Pa11y Customization - How to customize Pa11y for a better end user experience.

        • Limitations - Constraints of current implementation of Pa11y.

        • Future Capabilities - Where we want to go in the future with ADA Automation.

        • Twitter

        Dazzling Dashboards on the Double: 60 minutes to your first Information Radiator

        Jess Lancaster

        Making actionable information visible to your team can drive discussion, improve products, and influence behavior. You want to take steps to improve quality awareness on your team, but don’t know where to begin? Start by using telemetry to engage your team in facing and solving real problems! Walk away from this session with some ideas of what to measure and some tools to build your own information radiator in less than an hour! 


        In this session you will walk away with:

        • The types of telemetry you and your team should consider monitoring

        • The benefits an information radiator can have on your team  

        • How to set up quick and easy information radiators for your team using free tools and data you already have

        • Challenges and successes we encountered launching information radiators at TechSmith

        • Twitter

        Let's focus more on Quality and less on Testing

        Joel Montvelisky

        What value do you provide to your company?  Testing never has value in itself, testing is a service we provide to our teams so that our company can provide value adding products or services.

        Today, due in part to the evolution of development practices and in part due to the adoption of new technologies, engineers who had the exclusive role of being Testing Professionals can now expand their value towards Quality Engineering in the broader sense of the word.

        In this talk we will briefly review the forces pushing forward the changes in the way we write and deliver products today.  We will map the way a traditional tester might fit into this new reality. Then we will explore in depth the additional responsibilities we need to  take as testers in order to provide more value to our organizations as Quality Engineers.

        Some of these points have been developed by Alan Page and Brent Jensen as part of the Modern Testing (MT) principles, you can read more about them here -

        A large number of data, showing the progress of testing comes from our annual State of Testing Survey and Reports, available here -

        Many of the ideas for the future roles and responsibility of testers come from observations and conversations with hundreds of testing organizations around the world, that I have as part of my role.

        • Twitter

        PyTest - The Awesome Parts

        Josh Grant

        Python is a popular language for development, but it's also a great choice for test automation. Here I'll discuss some great plugins and features of PyTest, a Python-based test runner and framework. I'll show how these tools are helpful for Python developers but also why they encourage good test automation approaches and patterns. 

        • Twitter

        Fictional Test Data

        Joshua Russell

        Test data can tell a story at a glance. Most stories are obvious, like users in the QA5 environment with email addresses registered at There is an immediate value in the obvious stories; familiar conventions reveal the intent behind the data and make it easier to recognize and remember. What value can be attributed to stories that are more whimsical and nuanced? Let's talk about Noah Count, the user who never registered, and Maury Wards, who maximizes the loyalty points system. Let's look at the many stories that can be told by test data and turn fake information to fiction. 

        End-to-End Testing with Docker

        Kelsey Shannahan

        Containerization is an efficient way to run applications on your local machine.  By linking containers together we can reproduce our application stack right on our local machines for use with both development and testing.  This talk will cover the basics of Docker and networking, as well as a brief introduction to using Traefik to manage containers.  No more having to fight over limited testing servers!

        Ensuring Accessibility

        Kenny Gilles

        I'm sure you sometimes bump into weird bugs in the wild. Ugly websites and strange UI/UX that just does not make any sense. Your reaction would probably be: "Heh, I would've found that before it reached production.". And then you move on to finish whatever task you were working on.

        This is unfortunately not the case for everyone. Every day millions of people with disabilities are blocked from using a website or widget because of something minor or even unnoticeable to the majority of users. Society has been hard at work trying to make our physical world accessible to all, but we spend just as much time in the digital world and should consider it to the same extent.

        Keeping accessibility in mind throughout the development process, and especially in the testing phase(s), is integral to allowing everyone to use and interact with the web.

        I'll showcase examples of inaccessible widgets, how we can make them accessible and what tools are out there for us to use when testing, analyzing, and reporting on just how accessible our projects really are.

        • Twitter

        Getting to Continuous Testing

        Max Saperstone

        Max will tell the story of how a healthcare company striving to get to continuous releases built up their automation to secure confidence in regular releases. Initially, as no test automation existed, Max was able to take a greenfield test automation opportunity, and in the span of 12 months, develop over 2000 test cases. A testing pipeline was created to verify the integrity of the automated test cases, and to build docker containers for simple execution of the tests. These containers could then be simply re-used by developers and the DevOps team to verify the application. Max will walk through the feedback loop created, which allowed verification of the application go from hours to minutes.

        Max will discuss what processes and paths were taken to achieve continuous testing on this project. While he will cover the tools used and why they were chosen, the main focus will be on the HOW and WHY certain patterns and activities were performed. These choices were critical to achieving continuous testing, rather than just good testing coverage in CI or CD, even allowing a push left for performance and security. Additionally, some time will be spent on the organizational and culture changes that occured, and how he was able to accomplish this push for adoption in an organization that resisted automation, and had major quality problems.

        Quality Engineering Transformation - Open Source

        Mohamed Maktoum Yoosuf

        As a company, how much do you decide to spend on software? And what if I say you can cut your budget unbelievably if you get to eliminate licensed tools and by using open source tools as it is and modifying them accordingly? That is what exactly we did at Sysco, a global leader in selling, marketing, and distributing food products.

        Moving on to the story how a real fortune is saved, Our open source transformation was started back in 2017. By that time, almost all tools associated with quality engineering were commercialized tools. Less than three years, we have transformed a 95% QE tools usage to open source.
        In this transformation, a QE platform has been built in house using the open-source. The beauty of the platform is that it supports the test scripting,  execution, reporting and results analysis.  Also, the platform caters functional and non-functional test and various OSs such as Linux, Windows, Mac and AS400 and tech stacks such as Java, NodeJS, .Net and Cobolt.

        The platform has been used in the more than 50 products development teams and across globally such North America, Europe and Asia.

        Evolving API Testing into API Monitoring

        Patrick Poulin & Sangit Patel

        Today many companies are switching to CI/CD & agile workflows, with the goal of continuously making releases. That aim for continuous integration and deployments is now pushing closer to constant. Learn pointers and tricks from companies with large and successful internal API programs.


        How to write a smarter test.

        How to use dynamic data to improve test coverage.

        Common mistakes made when creating functional tests.

        How to leverage any test into a functional uptime monitor.

        • Twitter

        How to develop and test AI Based Systems

        Raj Subrameyer

        A recent study conducted by Gartner, shows that by 2020, AI will be pervasive in almost all software products and services. The major highlight of the study was how our skills as engineers would have to adapt accordingly. The current roles in companies are going to change significantly and we need to be prepared for it.  The working of AI is a black box-- we do not control or understand how the algorithm forms different relationships and makes decisions -- we just provide different training datasets and monitor the learning/progress. We are trying to make predictions on future values based on learning from past examples, or trying to discover different patterns from datasets.


        Come attend this session where, I discuss the basics of how AI based systems work, what are the key factors to consider when developing and testing these systems and how to come up with an enterprise wide strategy. Finally, I will answer some common questions surrounding AI and also show what are some of the tools available for doing AI based development and testing. This session will help anyone get an understanding of the fundamentals of developing and testing AI based systems.

        Co-Creation Workshops: What Role Do They Play?

        Richard Douglass

        Co-Creation workshops are becoming more prevalent in the product lifecycle. Typically, they are employed early on in the discovery process. Attendees will learn more about Co-Creation workshops: what they are, who is involved, what outputs / deliverables to expect, and how QA professionals can best make use of them.

        • Twitter

        A Lifecycle Of Code Under Test

        Robert Fornal

        When thinking about tests against code, have you "done enough?" How do you know that the code is covered appropriately, that odd bugs aren't going to crop up? Did you cover all the use cases and what if someone changes the code down-the-road?

        This session is about the unit and integration tests that a developer writes to test their code immediately, not about higher level tests that are written by a QA individual or department.

        This session will examine the code and tests holistically against the various lifecycle stages of code and how the testing needs to adjust:

        • Define Inputs and Outputs

        • Write Initial Test Coverage

        • Handling Bug Coverage

        • Refactoring

        • Abstraction

        • Future Work (how are tests affected?)

        Automating the Unknown - A journey of automating external applications

        Ryan Quellhorst

        2 years ago my son started attending daycare, and I started using the daycare’s app that would let me see videos and pictures of my son. Over those two years they’ve amassed over 1,000 photos of my son. To download all of these pictures manually would take me hours on hours of time; not to mention he still has 4 more years of daycare and my daughter has started attending this year. I knew there had to be a better way.


        Follow me on my automation journey as I show you some tips, tricks, and stumbling blocks that I uncovered while trying to preserve my children’s memories. 

        New rules for smarter and faster testing

        Shiva Agolla & Mike Buening

        The World Quality Report is the only global report analyzing software testing and quality engineering trends. It presents an analysis of developments in agile and DevOps, artificial intelligence, automation, test environments, data, security and budgets, showing once again the importance of quality, and of the measures that are put in place to maintain it with sector focus. The 11th Edition of this report captures insight from interviews with 1,725 CIOs and senior technology leaders from 32 countries and 10 industries.


        • Artificial Intelligence can make testing smarter – but broader QA skills are needed

        • Test automation cannot keep pace with the demand for rapid development


        • Elite local panel discussion on the trends that are more pertinent for us here in Central Ohio

        Click here for a link to download the full report The World Quality Report-2019

        Dev(QA)Ops Journey

        Srini Raghavan Ramesh

        DevOps is inconceivable without QA. In this presentation, the role of "QA" in "Devops" world will be discussed and real time project examples will be provided.

        No one likes to witness bugs seeping into production, leading to innumerable issues and wasteful expenditure. At the same time, business teams can ill-afford to wait for months on end to test and release their new features into the hands of ever-demanding customers. This is why many teams have now come to embrace Devops & Agile, as their go to. DevOps represents a significant change and plays a vital role in calling for testing at frequent intervals in a more automated fashion. QA teams have a huge place in a DevOps environment and all it requires is some adaptation and a sound DevOps testing strategy. In traditional approaches, QA has effectively played the role of a gatekeeper for many successful software deliveries, but in a DevOps environment, QA plays a pivotal role in thinning the boundaries among various teams leading to faster deliveries. Traditionally, the role of QA is to detect defects, but in the DevOps age, the prime responsibility of QA team will be more on preventing defects than finding them. This approach, not only helps “speed to market” but also helps in shortening the feedback cycle. 

        • Twitter

        The Automation Firehose: Be Strategic and Tactical

        Thomas Haver

        The widespread adoption of test automation has led to numerous challenges that range from development lifecycle integration to simple scripting strategy. Just because a scenario CAN be automated does not mean it SHOULD be automated. Teams that adopt automation often rush to automate everything they can -- the automation firehose. For those scenarios that should automated, every team must adopt an implementation plan to ensure value is derived from reliable automated test execution. In this session, the audience will learn how to automate both strategically and tactically to maximize the benefits of automation. Entry criteria will be demonstrated for automation in the development lifecycle along with a set of checks to determine automation feasibility & ROI.