Why We Invested in Automation and QA Early

Most companies don’t hire someone specifically for quality assurance until well after there’s a working product. After all, isn’t the whole point of a startup to “move fast and break things?”

I was the fifth person to join RapidAPI and a big part of my job, even early on, was to make sure we deliver a high quality product.

While I may be a little biased, I think we made a smart move investing in both QA and automation early on.

Currently, we’re a team of less than 20 people, and we have around 76% coverage in our automated testing. I’m hoping to bring up that number to ~ 90% . Fingers crossed!

Here are the benefits that we’ve seen so far.

1) Faster Iterations

Why Quality Assurance is Essential for RapidAPI

Not every organization needs quality assurance right off the bat. But for us, it’s the nature of the beast! The two factors that make QA so important for RapidAPI are interdependent functionality and our rate of change.

• Interdependent functionality: Our app is mission critical for other apps. If we have any serious errors, it doesn’t just affect RapidAPI’s website, it could affect everyone else’s apps.

• Rate of change: Our web app changes all the time. We add three to five API packages to the marketplace a week. We’re constantly updating, fixing bugs and adding new functionality to the website.

Long story short: there’s a very high cost to any mistake and, since we’re constantly changing, mistakes are likely to pop up. For a more standalone app, or a mobile app that only updated once a month, a full on QA process might not be as necessary early on. For us though, it’s essential.

Automated Testing Leads to Faster Iterations

If you work in a startup or have studied the Lean methodology, you’ve probably heard the phrase “Build, Measure, Learn.” This iterative process of deployment means that you can learn the most about what people value and need from your minimum viable product as quickly as possible.

A bit surprising for some… we realized that as we added a testing step, we were actually able to go through this cycle even faster.

 

Automated testing helps go from “build” to “measure” as consistently and quickly as possible.

Currently, we run code in three environments: dev, staging and production. Before any code can move from development to staging, we test it with sanity scenarios, then move it to staging. In staging, we run automatic tests across devices and browsers–a process that would normally take hours of our engineers time (not to mention, buying devices!). While we’re testing on staging, other developers can get back to what they do best – write code. A process that normally could take a few days to a week now takes less then a day and in some cases less then an hour

2) Reduce Human Error

Running manual QA scenarios can get repetitive. Us mere humans can skip/miss/ forget to run scenarios and details in some of them.

Luckily, we have computers! With automated testing, we can ensure that each test covers every step in each functional scenario.

At RapidAPI, we currently run two types of automated testing:

• Unit tests: Unit tests review individual software components or modules.

• End-to-End tests: End-to-End tests monitor a complete application environment in a situation that mimics real-world use, such as interacting with a database, using network communications, or interacting with other hardware, applications, or systems if appropriate.

Framework

We choose to use the nightwatchjs framework as a base for our general framework in conjunction with browserstack (running multiple browser + os versions).

Nightwatch.js is an easy-to-use Node.js based End-to-End (E2E) testing solution for browser based apps and websites. It uses the powerful Selenium WebDriver API to perform commands and assertions on DOM elements.

 

BrowserStack is a cloud-based cross-browser testing tool that enables developers to test their websites across various browsers on different operating systems and mobile devices, without requiring users to install virtual machines, devices or emulators.

We write scenarios in TestRail, then convert those manual tests to automatic tests using Nightwatch.js. Once we have the automatic script written, we run it in Browserstack.

Here’s an example of a test run – we’re testing a dashboard login flow on a Windows 8 computer with Chrome 52.0.

 

A cool integration we did was to connect our test runs from Browserstack with Slack using Slack’s API. When the automatic test on Browserstack finds an error, an alert will pop up on our Slack channel with a link to the error log and video for quick debugging.

3) The 2AM Bug

Another benefit of automated testing is this scenario. It’s happened to every developer. It’s 2AM and your phone rings. There’s a critical bug and it needs to be fixed RIGHT NOW.

You groggily fix whatever is wrong and head back to bed. Mission accomplished, right?

Not really, since your fix could break the entire application and cause havoc. To ensure this won’t happen, we can run our selected testing suites in minutes and verify things are running smoothly. With automated testing, we reduce our chance that one change will take down the system.

Building QA from the Ground Up

While it’s unusual to have an automated QA process this early in a company’s history, the benefits for RapidAPI make sense. Plus, it’s been really fun to build.

How do you do QA at your company? Do you automate any of your testing? Let me know in the comments below!