SafetyDetectives — How We Test

About SafetyDetectives

<div class="highlight-box green">
<span style="text-decoration: underline;">About SafetyDetectives</span>
</div>

SafetyDetectives is a publishing group of cybersecurity experts, privacy researchers, and technical product reviewers located all over the world. And we all share the same mission — to give our readers accurate and valuable information so they can make informed decisions about staying safe, secure, and protected on the internet.

We test, compare, and review antiviruses, password managers, and virtual private networks (VPNs) using a robust testing methodology. It’s because of these high-stress testing procedures and strict publishing standards that we have become one of the internet’s go-to resources for information about consumer-based cybersecurity products. And that’s not a distinction we take lightly. We aim for the best, and we will never stop bringing our readers the highest quality in our reviews, articles, blog posts, and interviews.

In addition to our product reviews and comparisons, we maintain a distinguished fleet of cybersecurity researchers who employ white-hat hacking methods to find data leaks, security holes, and privacy breaches, oftentimes associated with major online retailers that have millions of users. Our white-hat hacking efforts have been widely featured in tech journals like Wired, TechCrunch, Engadget, and many more.

We’re here to help our readers in any way we can. The SafetyDetectives website is provided as a free resource for everyone to investigate, compare, and inquire about the best products, practices, and cybersecurity information available on the web.

How We Test Cybersecurity Products

Below, you will find our exact testing methodology for the following:


How We Test Antiviruses

How We Test Antiviruses

Many antivirus companies brand their products as “the best product available with 100% malware protection” — but that isn’t always the case.

No one antivirus can offer total protection against all existing viruses and malware. But some antiviruses do a much better job than others at protecting against cyberthreats.

Here’s how we test antivirus products:

1. Purchase & Download — 5%

Our reviewers first pay for an antivirus subscription — usually the most comprehensive package available. We assess the pricing and range of payment options, and we try to find out if there are any hidden costs involved with purchasing each antivirus subscription.

We then download the antivirus installation file to assess the size of each download and whether or not it takes up a lot of storage space on our test devices.

Our reviewers give a score out of 10 based on their experience with purchasing and downloading each product. This score makes up 5% of the final score.

2. Installation & Setup — 10%

We then run the installation process on a variety of systems, including Windows, Mac, Android, and iOS, to see how easy or difficult the antivirus is to set up.

We also assess the amount of time it takes to install the product and whether or not any technical knowledge is required to successfully complete the installation.

Once installed, our reviewers complete any final setup process and ensure that the application has been fully installed.

We score our experiences with installation and setup out of 10. This score makes up 10% of the final score.

3. Malware Protection — 30%

We take an in-depth look at the antivirus technology behind each product to ensure it’s secure, accurate, up-to-date, and effective against the latest malware threats.

We use a variety of methods to test the effectiveness of each antivirus:

3a. Malware Scanner Detection Tests — 15%

Our reviewers run a range of system scans to see how many malware files each product’s anti-malware scanning engine is able to detect.

We use hundreds of malware samples in each malware scanning test. But during this particular test, we’re simply assessing each antivirus program’s ability to detect malware files stored on a system, so we don’t open or execute any malware files.

Each antivirus scanner is tested against a wide range of both old and new malware samples, including viruses, ransomware, spyware, trojans, keyloggers, rootkits, and more.

Here are some examples of malware we include in each test:

  • Ransomware — encrypts/locks up a user’s device and forces the user to pay a ransom to unlock the device.
  • Backdoors/trojans — exploits system vulnerabilities to secretly access a user’s device and steal data.
  • Spyware/Remote access trojan (RAT) — remotely controls a user’s device, accesses sensitive files, and installs harmful malware.
  • Keystroke loggers — tracks a user’s keystrokes, allowing hackers to steal login usernames, passwords, banking details, and more.
  • Webcam & mic hijackers — gives hackers unauthorized access to a user’s webcam or computer microphone to secretly record a user’s activity.

Once the malware scanner detection tests are complete, we see how many malware samples the antivirus scanning engine caught. While we like to see scores of 100%, we consider a 95% malware detection rate to be sufficient enough for us to consider the antivirus as “very secure”.

We convert the overall malware detection percentage to a score out of 10. This score makes up 15% of the final score.

3b. Real-Time Protection Tests — 15%

Our real-time protection tests include a series of simulated attacks to see how well each antivirus real-time protection engine blocks each simulated attack.

Some of the malware attacks we simulate during our tests include:

  • Executed malware files — we run a script that “executes” or “launches” each malware sample one-by-one, and we judge the real-time engine’s capability to block and remove each malware threat as it attempts to infiltrate our test system.
  • Ransomware simulations — we launch various ransomware simulations that mimic legitimate ransomware attacks to see if the real-time protection blocks and disables the ransomware.
  • Zero-day viruses — we simulate attacks using unknown malware files that haven’t been added to a known malware database. We measure how fast each antivirus detects these unknown threats and how each antivirus responds to each attack.

After completing all of the real-time protection tests and simulations, we see how many threats were blocked by each antivirus’s real-time protection engine. While scores of 100% are obviously excellent, we consider a score above 90% to be good enough for most users.

We convert the overall real-time protection percentage to a score out of 10. This score makes up 15% of the final score.

4. Web Protection — 10%

Our reviewers look at what web protections each antivirus offers, and we run a series of tests to ensure these web protections work as intended.

Many antiviruses come with web protection features like network firewalls, anti-phishing protection, privacy leak detection, tracking link blocking, etc.

We test each web protection’s effectiveness against things like:

  • Network attacks — attempts to infiltrate a user’s private network to gain access to network-connected devices.
  • Phishing sites — fake websites that trick users into inputting their personal information, like passwords and bank card details.
  • Malware sites — dangerous websites known to contain malware or malicious scripts.
  • Email phishing scams — scam emails disguised as legitimate emails that contain phishing links.
  • Ad & tracking links — annoying popup ads and tracking links that follow your activity across the web.

We calculate the number of network attacks, phishing links, malware sites, ads, and tracking links blocked by each antivurs’s web protections.

While we give an overall score for web protection in our reviews, we make sure to clearly point out specific web protection features that perform really well and ones that perform poorly.

After we’ve completed our web protection tests, we give an overall score out of 10. This score makes up 10% of the final score.

5. Extra Features — 15%

Most antiviruses come with more than just malware protections. We take a look at all of the extra features and functions offered by each antivirus.

We test each additional feature to assess its functionality, purpose, and usefulness for most users. Many of these extra features increase the overall cost of the product, so we assess  whether or not these features are actually worth this added cost.

Typically, antivirus packages usually offer additional features like a device/performance optimizer, junk file removal, a virtual private network (VPN), a password manager, etc.

In line with our testing standards, we assess each additional feature for the following:

  • Performance — how well the feature does the thing it’s supposed to do.
  • Usefulness — whether users will actually benefit from using the feature or if it’s just an unnecessary add-on.
  • Impact on system performance — how much CPU, RAM, and system energy the feature uses.
  • Value — how much the feature is worth.
  • Standalone comparison — how the feature scores when compared to top-rated standalone apps that have the same purpose (like a VPN/password manager).

We give each feature individual scores based on our findings. We then calculate an average score out of 10. This score makes up 15% of the final score.

6. Impact on System Performance — 10%

Antiviruses need to be effective while not slowing down a user’s system. We measure each antivirus’s impact on CPU, RAM, battery usage, and overall system performance.

To make our research useful and relatable to most users, we look at how each antivirus uses system resources on a variety of old and new systems, including Windows, Mac, and mobile devices to ensure a fair performance average is established.

We assess each antivirus’s performance while idle (not running an active scan), during each scan, and during simulated malware attacks. When testing performance, we look at:

  • CPU usage — how much computer processing power each antivirus uses.
  • RAM usage — how much system memory each antivirus requires to run efficiently.
  • Battery usage — how much battery power each antivirus uses (during full scans and while idle) and whether or not the antivirus maintains an overall energy efficiency.

If an antivirus uses around 20% CPU power, less than 500 MB RAM during a full system scan, and has good energy efficiency (as measured with the task management system), we consider it to be a great choice. Anything above 50% CPU usage, over 2 GB RAM, and/or with poor energy efficiency is generally a bad choice.

After we’ve completed our performance tests, we give an overall score out of 10. This score makes up 10% of the final score.

7. Pricing — 10%

We assess the packages or subscriptions offered by each antivirus company and determine whether or not they’re low-cost, overpriced, or a good value.

Our reviewers determine the value of each product by looking at the quantity and quality of the features offered in each package. We then compare the price of each package to similar products available to see how well each package aligns with the market average.

This method helps us to clearly define which antiviruses offer a good deal and which antiviruses aren’t worth the money.

Based on our assessments, we give an overall score out of 10. This score makes up 10% of each product’s final score.

8. Customer Support & Refunds — 10%

We look at each antivirus company’s support options and test each option by sending numerous questions via email, live chat, phone support, website FAQs, forums, and more.

Our reviewers then qualify how well each company’s support team performed by looking at response times, the quality of each response, and whether or not the response was useful in addressing and solving the issue(s).

We give good scores to support teams that have a wide range of contact options, respond quickly with useful solutions, and ensure users are satisfied with the product.

Finally, we see if each company offers a refund policy or money-back guarantee. We assess the entire refund process by requesting a refund within the correct time frame, judging how easy or difficult it is to get our refund request processed, and how long it takes for the money to arrive back in our account.

We give a score out of 10 based on our overall customer experience. This score makes up 10% of the final score.

Final Scores for Antiviruses

After carrying out extensive testing, our reviewers calculate a final score for each antivirus based on the score of each test category.

Each test category has a varying influence on the final score, so we divide each test score by the following percentages:

  • Purchase & Download: 5%
  • Installation and Setup: 10%
  • Malware Protection: 30%
  • Web Protection: 10%
  • Extra Features & Utilities: 15%
  • Impact on System Performance: 10%
  • Pricing: 10%
  • Customer Support & Refunds: 10%

We then add up the figures to reach a final score for each antivirus, which is shown at the top of each review. Products with an overall score above 8/10 are generally considered to be a great choice for most users. Products scoring around 7/10 are generally pretty good, though there are likely better options elsewhere. Scores around 6/10 and below indicate that our reviewers ran into many issues, and the product probably isn’t the best for most users in its current form.


How We Test Password Managers

How We Test Password Managers

Password managers store and auto-fill passwords so users don’t have to remember or manually enter all of their login credentials every time they’re logging into an online account.

Our reviewers conduct a series of tests in order to find out the pros and cons of each password manager.

Here’s how we test password managers:

1. Purchase & Download — 5%

We purchase the most comprehensive plan offered by each password manager and download it onto our system.

We assess the range of payment options offered, the difficulty of the overall payment process, and whether or not purchasing the product is in any way challenging or off-putting (like hidden costs or upsells).

We then download the password manager installation file onto our system. We look at the size of each download file and see how much storage space is required.

Based on our experience when purchasing and downloading each product, we give a score out of 10. This score makes up 5% of the final score.

2. Installation & Setup — 10%

After downloading, we install the password manager onto a variety of systems using Windows, macOS, Android, and iOS.

Most password managers also come with a browser extension, so we install the password manager on as many web browsers as possible to fully assess its functionality.

Once the password manager has been installed, we then proceed to set up the program. First, we log into the password manager app or browser extension using our previously created login credentials. We always look closely at the master password creation process to assess whether or not each program guarantees that users are making a strong master password.

Based on our experience with the installation and setup of each product, we give a score out of 10. This score makes up 10% of the final score.

3. Security & Encryption — 25%

A proper password manager should provide advanced security options to safely store a user’s passwords, including high-level encryption.

There are a range of things we look for when assessing each password manager’s security, including:

  • Encryptions options — we find what level of encryption each password manager uses (256-bit AES is the standard).
  • Zero-knowledge architecture — we ensure each password manager is designed in such a way that the developers have absolutely no access to a user’s stored data.
  • Two-factor authentication (2FA)/multi-factor authentication (MFA) — we test and assess all of the 2FA/MFA options offered.
  • Account recovery options — we look at the range of account recovery options that enable users to gain access to their data if they lose their master password.

Low scores are given to password managers that don’t include strong encryption standards, aren’t built using zero-knowledge architecture, don’t include a range of 2FA/MFA options, and/or don’t have any account recovery options.

Based on our assessment of each product’s security features and protocols, we give a score out of 10. This score makes up 25% of the final score.

4. Core Features — 25%

Each password manager core feature is assessed closely to ensure it functions as expected.

The core features that we test include:

  • Password storage — how passwords are stored and organized.
  • Auto-fill & auto-save — how well each password manager automatically fills login fields and how well it captures newly entered logins.
  • Password generating — how easy it is to generate new passwords and how strong and versatile the passwords are.
  • Password sharing — how secure and intuitive the password sharing feature is and whether or not there’s a limit on the amount of passwords that can be shared.

Based on our assessment of each product’s core functionalities, we give a score out of 10. This score makes up 25% of the final score.

5. Extra Features — 15%

Many top-rated password managers come with extra features that either offer users additional security or improve productivity.

As the features offered by each brand can differ, we first assess the number of features offered and then decide whether or not each feature works as advertised.

Some of the features that we test include:

  • Dark web monitoring — scans dark web databases to see if a user’s personal details have been leaked on the dark web.
  • Security reports — audits the overall strength of a user’s password database and lets the user know how they can strengthen their passwords.
  • Encrypted file storage — stores large quantities of user files in the company’s encrypted cloud servers.
  • Automatic password changer — allows for changing passwords on certain sites with a single click in the password manager app.
  • Virtual private network (VPN) — helps a user change their IP address and provide an added layer of security to their browsing.

If each additional feature works as advertised, we compare and contrast the feature with other password managers and standalone products that offer the same functionality. We then look at whether or not the extra feature comes at an added cost and determine if the additional feature is worth the added cost.

Based on our experience with each product’s extra features, we give a score out of 10. This score makes up 15% of the final score.

6. Pricing — 10%

Our reviewers assess the cost of each password manager.

We look at the range of packages or subscriptions offered by each company, break down what features come with each package, and determine if each package is worth its price.

Each password manager package is compared to similar packages offered by other password managers to see if the product is low-cost or expensive compared to the competition.

As well as looking at the original price of each package, we look at the available discounts and deals that each company advertises. If available, we’ll let the user know if the discounted price is a good buy or not.

Based on the product’s pricing, we give each password manager a score out of 10. This score makes up 10% of the final score.

7. Customer Support & Refunds — 10%

We look at the support options available to users — the more options, the better.

We give good scores to companies that offer a wide range of support options, like email, live chat, phone, forums, and social media.

Our experts contact each password manager’s support team numerous times with various questions and issues to assess the speed and overall quality of each response.

Finally, we see if each company offers a refund policy or money-back guarantee. We assess the entire refund process by requesting a refund within the correct time frame, judging how easy or difficult it is to get our refund request processed, and how long it takes for the money to arrive back in our account.

Based on how fast each support team responds to our requests and how useful the response is in helping us solve an issue, we give each company a score out of 10. This score makes up 10% of the final score.

Final Scores for Password Managers

Our reviewers calculate a final score for each password manager based on the score of each test category.

Each test category has a varying influence on the final score, so we divide each test score by the following percentages:

  • Purchase & Download: 5%
  • Installation & Setup: 10%
  • Security & Encryption: 25%
  • Core Features: 25%
  • Extra Features: 15%
  • Pricing: 10%
  • Customer Support & Refunds: 10%

We then add up the figures to reach a final score for each password manager, which is shown at the top of each review. Products with an overall score above 8/10 are generally considered to be a great choice for most users. Products scoring around 7/10 are generally pretty good, though there are likely better options elsewhere. Scores around 6/10 and below indicate that our reviewers ran into many issues, and the product probably isn’t the best for most users in its current form.


How We Test Virtual Private Networks (VPNs)

How We Test Virtual Private Networks (VPNs)

Our reviewers conduct a range of tests for each VPN service to determine if each product is actually as secure, easy to use, and fast as advertised by each VPN company.

Here’s how we test VPNs:

1. Purchase & Download — 5%

We purchase the most comprehensive VPN package offered by each brand. During the payment process, we look at how much personal information we need to provide to complete a purchase and whether or not the company offers any anonymous payment methods like cryptocurrency.

A VPN company only needs an email address and payment information to create a user’s account, so if the company asks for more identifying information, we assess whether or not the company has a valid reason to do so.

Once we’ve completed our purchase, we read the system requirements and download the installation file from each company’s website.

Based on our experience when purchasing and downloading each VPN, we give a score out of 10. This score makes up 5% of the final score.

2. Installation & Setup — 10%

We install the VPN on a variety of systems, including Android, iOS, Windows, and Mac. The installation process is closely monitored to determine the difficulty of installing the VPN on each system.

Once installed, we set up the VPN by logging into our account and adjusting the settings as required to ensure the VPN is set up correctly on our systems.

We consider how intuitive the VPN program is and whether or not we think both non-technical and technical users could quickly understand how to use the program’s interface.

Based on our experience with the installation and setup of each product, we give a score out of 10. This score makes up 10% of the final score.

3. Security & Privacy — 30%

We conduct a series of tests to ensure each product indeed protects a user’s information, including browsing activity and IP address, and we strive to find any potential flaws in each VPN’s security features.

Our reviewers look at things like:

  • VPN protocols — we look at the range of VPN protocols offered by each provider, including OpenVPN (UDP), OpenVPN (TCP), IKEv2, WireGuard, and any others that are offered by the VPN service provider.
  • Encryption — we examine the strength and functionality of the encryption method used by the VPN to obscure a user’s browsing data.
  • Obfuscation & government bypassing — we test each VPN’s ability to disguise VPN connections and bypass restrictive government firewalls that block VPN traffic.
  • DNS leak protection — we perform various leak tests to ensure there are no data or IP leaks when using the VPN service.
  • IPv4 & IPv6 visibility — we assess whether or not each VPN is capable of obscuring both IPv4 and IPv6 addresses.
  • Company logging policy — we dissect each provider’s logging policy to ensure the company is as committed to protecting user data as it claims it is.

Once our security assessments are complete, we give an overall score out of 10. This score makes up 30% of the final score.

4. Overall Speeds — 20%

All VPNs slow down a user’s internet connection a little bit. This is due to both the time it takes to reroute user traffic through an intermediary and the time it takes to encrypt that traffic.

Our reviewers test the VPN’s overall speeds with a variety of speed tests. Our methodology includes measuring the percentage difference between the speeds without a VPN and the speeds with the VPN (connected to many servers in many different locations).

To get the most accurate results, we test the overall speed of a VPN by measuring the following:

  • Local server speeds — the download and upload speeds of servers that are located near our reviewers.
  • International server speeds — the download and upload speeds of servers across a wide range of countries all over the world.
  • Ping speed — how long it takes to reach the network host.
  • Streaming speeds — the general experience (buffering, glitches, interruptions, etc.) while streaming in HD.
  • P2P speeds — the download and upload speeds associated with P2P file sharing and torrenting.

Typically, good VPNs only have a slight impact on speed. If a VPN only drops the original download speeds by under 10% for local servers, we consider it as a very good VPN. In general, a good VPN won’t slow down connections by more than 50% on any server (keeping in mind that the further away a server is, the slower the speed will be).

Once our tests are complete, we collate our speed test data and give a score out of 10. When calculating the score, we typically place more emphasis on download speed than upload speed and ping, but certain considerations are made in unique instances (like gaming and torrenting). This score makes up 20% of the final score.

5. Server Locations & Options — 15%

Generally speaking, a good VPN should have a large number of servers all over the world. The more server locations there are, the less likely that any one server will experience a high load at any given time (and impact overall speed and performance).

A wide variety of servers is important to ensure most users can connect to a server that’s near their actual location while also allowing users to stream geo-restricted content from different countries if needed.

We also check to see what kinds of servers the VPN service provides. We look for options like obfuscated servers for bypassing government restrictions and VPN geo-blocks, P2P-optimized servers for secure torrenting, and dedicated streaming servers for fast HD streaming with no lags or buffering.

We give high scores for VPNs that offer a range of server types as well as a large number of server locations. This score makes up 15% of the final score.

6. Pricing — 10%

When judging the price of each VPN, our reviewers look at the range of subscription plans offered by each company, break down what features come with each plan, and determine if each VPN is worth the money.

Each plan is compared to similar packages offered by other VPN providers to see if each product is low-cost or expensive compared to the competition.

As well as looking at the original price of each package, we look at the available discounts and deals that each company advertises. If available, we’ll let the user know if the discounted price is a good buy or not.

We give each VPN provider a score out of 10 for pricing. This score makes up 10% of the final score.

7. Customer Support & Refunds — 10%

Our reviewers contact each VPN provider’s customer support team. VPN providers are given higher scores if they have a wide range of support options, including email, live chat, phone, support forums, social media, and other support channels.

We send questions to each available support channel and judge how fast the support team responds to our request and how useful the response is in helping us solve an issue. The faster and more helpful the response, the higher we score each provider’s customer support.

Subjective experience can play a factor when reporting our customer support experience. However, to mitigate reporting based on one isolated incident, we contact support teams multiple times over the duration of our full VPN testing process to ensure our results are fair and well-rounded.

Finally, we see if each provider offers a refund policy or money-back guarantee. We then request a refund within the correct time frame to see how easy or difficult it is to get our refund processed and how long it takes for the money to arrive back in our account.

This score makes up 10% of the final score.

Final Scores for VPNs

Using the results from each individual test, our reviewers calculate a final score for each VPN based on the score of each test category.

Each test category has a varying influence on the final score, so we divide each test score by the following percentages:

  • Purchase & Download: 5%
  • Installation & Setup: 10%
  • Security & Privacy: 30%
  • Overall Speeds: 20%
  • Server Locations & Options: 15%
  • Pricing: 10%
  • Customer Support & Refunds: 10%

We then add up the figures to reach a final score for each VPN, which is shown at the top of each review. Products with an overall score above 8/10 are generally considered to be a great choice for most users. Products scoring around 7/10 are generally pretty good, though there are likely better options elsewhere. Scores around 6/10 and below indicate that our reviewers ran into many issues, and the product probably isn’t the best for most users in its current form.


Our Commitment to Accuracy

SafetyDetectives is proud to be one of the only unbiased sources of information about the leading cybersecurity products on the internet. If you feel there is any false or misleading information contained within our website content, please don’t hesitate to contact us and let us know. We frequently adjust and revise our reviews based on our readers’ feedback, and we are always on the lookout for the most up-to-date and accurate information.

Honest Expert Reviews
User Evaluations
Thorough Explanations
Real-Time Updates

Our Team

Ben Martens
Ben
Martens
Senior Editor
Sam Boyd
Sam
Boyd
Contributor
Eric Goldstein
Eric
Goldstein
Senior Editor

Contact us

Fill the fields below and we’ll get back to you