Test Automation Strategies in an Agile World

Jul 1, 2024

Test Automation Strategies in an Agile World

Introduction

  • Presenter: Vikas Mittal, Global President of Digital Assurance at CBA
  • Organizer: Rick Bachman, Director of Interactive Communications at STP
  • Occasion: LSP Well Community Webinar
  • Topic: Test Automation Strategies in an Agile World

Important Announcements

  • STPCon Fall 2019: Will be held in Boston, celebrating 10 years in testing

    • Location: Hyatt Regency Boston Harbor
    • Dates: Four days of workshops, keynotes, networking
    • Early Bird Discount: Up to $300 off, expiring on August 2nd
    • Discounts: Can combine early bird, team, and alumni discounts

  • Next Webinar: July 24th, titled "Kino for Experts" with Lean and Agile Consultant Liz Keogh

    • Platform: softwaretestpro.com/training
    • Follow on Twitter: #STPWebinar with mentions to the speaker

Introduction to Speaker: Vikas Mittal

  • Expertise in architecture, design, implementation of test automation frameworks and solutions
  • Background includes roles at Infogain, IBM, CMS, Fiserv
  • Focus on automation solutions using machine learning and predictive analytics
  • Involved in India's Atal Innovation Mission to promote tech-driven innovation and entrepreneurship

Core Points of the Webinar

Current Status of Agile and Automation

  • Surveys (2017 & 2018): World Quality Report shows automation at only 31% in Agile environments
  • Challenges: Release builds every two weeks but low automation percentage; dependency on human factors and budgetary restrictions
  • Essential Questions: Value of automation, metrics for measuring success, alignment with Agile testing needs

Barriers to Effective Automation

  • Low Coverage: Evolving applications, multiple tools, complex test environments, and long regression cycles
  • Complexity of Environments: High setup costs, data privacy/security concerns
  • Automation Analysis Time: Longer than execution time, dependency on human judgment

Steps to Effective Test Automation

Holistic Approach to Automation

  1. Identifying Use Cases: Moving from manual test cases to user journeys and personas. Focus on end-to-end workflows instead of isolated tests.
  2. Technology Layers: Focus on both UI and underlying API/microservices. Aim for a balanced approach (70/30 ratio of API/UI automation).
  3. Programming Language & Tools: Adopt languages understood by development teams, ensuring collective ownership.
  4. Cultural Perspective: Quality as a team responsibility, not limited to testing teams.

Establishing Robust Automation Infrastructure Using CBA's Unified Approach

  • Framework: Integration across all layers using tools like Protractor, Selenium, RestAssured, Appium, etc.
  • Behavior Driven Approach: For contributions by functional engineers
  • DevOps Integration: Performance metrics and continuous integration with feedback loops

Data-driven Test Case Selection and Prioritization

  • Upstream & Downstream Data: Using data from requirements, production telemetry, defect history
  • Sophisticated Algorithms: Combining influence of code churn, execution history, module dependencies, and telemetry to prioritize tests
  • Automated Feedback Loops: Use of heat maps to visualize risk factors and predict failures

Key Metrics for Automation Success

  1. First Pass Percentage: Percentage of test cases that pass on the first run
  2. Parallel Execution Ratio: Percentage of test cases that can run in parallel
  3. Feedback Time: Total time from build to test results analysis
  4. Automated Data Generation: Percentage of data generated automatically for tests
  5. Pipeline Execution: Percentage of tests executed as part of CI/CD pipeline

Case Studies and Examples

  • Regression Cost Reduction: From $1.5 million to $250k in one year through effective automation and predictive analysis
  • Predictive Analytics: Enabled targeted regression, minimized need for human intervention, eliminated hardening sprints

Final Notes

  • Interactive Q&A: Extended Q&A session discussing practical questions on tools, strategies, and specific use cases
  • Contact Information: Provided for further queries and deeper engagement

Key Takeaway

  • Adopting a holistic, data-driven approach to test automation can significantly enhance efficiency, reliability, and ROI in Agile environments.

Full transcript