Rethinking Functional Test Automation

Data is the new Oil. For oil to fetch its right price it needs to be extracted and refined before it can be monetarized. How do we test the extraction and refining of Data? Join us to discuss how to test data collection and data refinement and challenges we faced and the lessons we learnt.

We at FINRA, as the plan processor of the Consolidated Audit Trail (CAT) ingest more than 100 billion orders amounting to more than 1 TB of Data every day to be the world’s largest data repository of information on securities transactions. The CAT System is event driven, spread across multiple components and runs across variety of resources from serverless systems to Hadoop systems. How do we test a system, which is asynchronous? How do you test fault tolerance of the system? How do you test the idempotency of the system? Join us to understand how we tested the data collection and data refinement of the system, different challenges we faced and the lessons we learnt while testing the Data Collection and Data validation.

• Does test pyramid always hold true : though unit tests are important but not always easy to maintain. Should the order be Component tests > Unit tests > end to end tests
• FaaL (Functional tests as Load tests) : using automation framework to write load tests. Does it help in ensuring maintainability of the automation. How different is it from having load testing suite.
• QaOps (QA as part of CI/CD) : Having QA Tests integrated to the code deployment pipeline. Running all the tests vs running smoke tests .
• Test the unexpected: How to test idempotency of the system. How to ensure the system can handle failures. What is functional chaos testing?
• Infrastructure testing : Why do we need to test infrastructure? How to test infrastructure? Automation vs Manual Infrastructure testing
• UX testing of UI Less system : We don’t have an UI? Does User Experience matters. What about the responses by the system.
Lessons learnt
• Automation with asserts vs Parallel Implementation with Asserts
• Testing everything is not possible
• Tests as part of Deployment checks
• Environment independent test automation
• Test with engineered data + Test with real Data

The system built processes more than 100 billion records and uses serverless system along with Big Data Processing through Big Data system hosted in AWS. The QA Team created tests running in jenkins with every deployment and run automated. The tests span across functionality,infrastructure,fault tolerance including component tests and end to end integration tests. As the person involved in designing and automating the tests and the load test system, I can provide you the details on our experience in designing and automating. I also can share various lessons learnt and how we approached testing in contextful way questioning the established principles and applying them where applicable