GLOSSARY
The terms that appear in our reports and scope documents. Each definition reads on its own; you don’t need one entry to understand another.
Acceptance criteria are the conditions a feature has to meet before it counts as done — written in advance, and each one verifiable on its own.
A good criterion ends the argument: it is either met or it is not. "Should load fast" is not a criterion; "first content visible within 2.5 seconds on a 3G connection" is. We derive test scenarios from these. Where none exist, we test against a reasonable user expectation rather than against the product's current behaviour.
API contract testing checks whether a service actually behaves the way it is documented to: which fields come back, in which types, and with which error codes.
The interface can look correct while the service behind it has quietly changed — a field became optional, an error code moved from 400 to 422. Contract testing catches that before a client does. It covers boundary values, missing fields and unauthorised calls as well as valid requests.
Black-box testing means testing a product purely through its externally visible behaviour, without reading the source code.
The tester sees only what a user sees. The advantage is that assumptions about how the code was written cannot steer the testing — and users hold no such assumptions either. We run functional, compatibility and usability testing as black box. Code access improves performance bottleneck analysis and security review, but is not required.
A compatibility matrix is the table showing which device, operating system and browser versions a product was tested on, and what each one produced.
The matrix states two things at once: what was tested, and what was not. The second is missing from most reports. An emulator alone is not enough — touch targets, keyboard behaviour, memory limits and network switching all behave differently on real hardware.
A coverage matrix shows every scenario run during a test as one cell; the flagged cells are the scenarios that produced findings.
It answers two questions at a glance: how many scenarios were attempted, and where the findings clustered. Findings concentrated in one region usually signal a structural problem in that module rather than a set of unrelated bugs. Every report opens with this matrix.
End-to-end testing exercises a task from where the user starts it to where they finish it, passing through every system in between.
Parts working individually while their combination fails is common: the payment provider responds correctly, the order service records correctly, and a timeout between the two empties the basket. End-to-end testing targets those gaps. Most of our scenarios are of this kind.
Exploratory testing is a time-boxed session in which a tester follows what they notice while using the product, rather than working through a pre-written scenario list.
Scripted testing covers known risks; exploratory testing finds what nobody thought of. The duration and focus are set in advance and what was attempted is written down afterwards — so it is not "clicking around". A significant share of findings comes out of these sessions.
A finding is a single issue identified during testing that can be independently reproduced; an observation that cannot be reproduced never enters the report as one.
We say "finding" rather than "bug" because part of the list is not a coding error: it may be a missing acceptance criterion, an inconsistent behaviour, or a flow that wears the user down. Every finding arrives with reproduction steps, expected versus actual behaviour, environment details and evidence.
A flaky test is one that sometimes passes and sometimes fails on identical code; its result reflects randomness in the test environment rather than the product's behaviour.
The usual causes are fixed sleep durations, data shared between tests, and time-dependent assumptions. Flakiness destroys trust over time: a team gets used to seeing red and stops reacting to a genuine break. When we write automation, flakiness is tracked as an open finding until it is fixed.
Functional testing verifies that a product does what it says it does, by running defined scenarios through it.
The scope is the critical business flows: sign-up, sign-in, search, basket, payment, data entry and authorisation. Not just the happy path — invalid input, half-finished transactions, the back button and concurrent sessions all get exercised too, because all of them happen in real use.
Load testing measures how a system's response time and error rate change under the user volume it is expected to carry.
The goal is not to break the system but to put a number on what a normal and a busy day look like. The output is usually a threshold: "response time stays under 800 ms up to 400 concurrent users, after which the database connection pool saturates."
Localization testing looks for the places where a correct translation is not enough: clipped text, formats that read wrong in the country, and flows that do not fit locally.
A translation can be flawless and the product still wrong. A German label overflows its button, a decimal comma is read as a thousands separator, icons stay unmirrored in a right-to-left language, a payment method nobody there uses sits as the only option. Seeing any of it takes a native speaker in the target country looking at the screen.
The OWASP Top 10 is a regularly updated open reference listing the ten most common and highest-impact categories of security weakness in web applications.
It covers broken access control, cryptographic failures, injection, insecure design and security misconfiguration, among others. It is the baseline framework for our security review. It is not a substitute for a penetration test, and we say so upfront where certification is required.
Payment flow testing confirms with a real card that an order’s money moves exactly once, for the right amount, and leaves the right record behind.
A sandbox makes the happy path look right; the problems appear once money actually moves. A double submit can take two charges, coming back from the 3-D Secure screen can mark an order paid with no charge at all, a partial refund can miss the invoice. The test looks for these with real cards and real bank responses, reconciled against the provider’s dashboard.
Priority says when a finding should be fixed; severity says how much damage it does. They are not the same thing.
A low-severity typo can be high priority if it sits in the middle of the home page. A high-severity crash can wait if it only occurs on an admin screen nobody uses. We supply the severity; the product owner sets the priority.
A re-test round is the re-examination of affected findings after you ship your fixes; closed, still-open and newly-broken items are listed separately.
That third list is absent from most processes and is the expensive one: a fix breaking something else is common. The re-test round is included in all of our packages and is never billed separately.
Regression testing checks whether a new change broke behaviour that previously worked, by running the same suite of scenarios on every release.
The value lies in the repetition and in maintaining the suite: a suite that never changes gradually stops testing the product as it actually is. We review the suite each release and turn closed findings into permanent scenarios. Results are reported so they can be compared version against version.
Severity is the ranking of a finding's impact on the user and the business; ours has four levels — critical, high, medium and low.
Critical means data loss or a workflow stopping outright. High means the user cannot complete the task without finding a workaround. Medium covers cases where the task completes but wrongly or laboriously. Low is inconsistency and polish. The level comes from impact and likelihood together, not from how broken something looks.
A smoke test is a short, shallow set of checks run to decide whether a build is worth testing in depth at all.
Does the application start, can you sign in, does the main flow run end to end — it takes a few minutes. The point is not to find bugs but to avoid spending hours testing a build with something fundamental broken. Every test round of ours opens with one.
Stress testing pushes a system past its capacity to see where and how it fails, and whether it recovers once the load passes.
It differs from load testing in intent: load testing measures a normal day, stress testing looks for the limit. The second half matters most — whether the system returns to normal on its own after the load drops. If it does not, even a campaign ending can leave the service down for hours.
A test environment is a copy that runs separately from the live system but is configured the same way, so testing touches neither real data nor real users.
We can work without one, but then scenarios that create data have to be left out of scope — which narrows coverage considerably. An environment configured differently from production is the most common reason a finding appears live and not in testing.
WCAG 2.2 AA is the commonly required conformance level of the international standard defining how web content is made accessible to people with disabilities.
It covers colour contrast, full keyboard operation, focus visibility, form labels, error identification and screen reader compatibility, among others. Automated scanning catches roughly a third of violations; the rest needs a real screen reader session. Our audit runs both together.
Sign up in the panel, pick what you want tested, pay. The first findings start landing in the same panel within hours.
SendTheCanary provides independent software testing for web, mobile and API products. Ten separate services from functional testing to payments and localization, run by a network of 4,700 testers and delivered as one report ranked by severity.
Services:Functional testingRegression testingCompatibility matrixPerformance & loadSecurity reviewAPI testingPayment testingAccessibilityUsability reviewLocalization testing