SSE Major Vendor Threat Protection Levels Vary, Requiring Caution

Thank you for always reading.

This article is a free translation into Japanese of an article regarding CyberRatings' SSE threat protection test published on SDxCentral on July 17, 2025. Please see the following for the original article.

SSE protection found uneven across major vendors
SSE protection found to be uneven across major vendors

SSE Threat Protection Test by CyberRatings

Researchers reported a significant disparity in the security effectiveness of SSE (Security Service Edge) protection among major vendors.

The non-profit organization CyberRatings.org revealed that in tests of vendor products, security effectiveness ranged from less than 3% to 100%, with only Fortinet, Palo Alto Networks, Versa Networks, and Zscaler receiving a "recommended" rating.

In contrast, the SSE products from Cisco, Cloudflare, and Skyhigh were labeled with "caution," showing "below average" security effectiveness, and end users were advised that they "should consider looking for other solutions." The ratings were based on "failures in critical tests."

Cato Networks and Netskope were excluded from the report, with the former noted for "explicitly showing a stance of refusing engagement" with CyberRatings, and the latter for being "unresponsive."

The tests were conducted by NSS Labs, CyberRatings' official testing partner, which recently revamped its methodology. The test is designed to map the security effectiveness of products, measure relative capabilities, and assess false positive accuracy while accounting for operational overhead.

CyberRatings revealed that the factor with the greatest impact on security effectiveness was the blocking of "evasions," which threat actors use to disguise or modify attacks to bypass defenses. According to the researchers, three of the tested products failed to block evasions.

CyberRatings points out that the cloud-based nature of SSE, which uses continuous integration/continuous deployment (CI/CD) practices, can reduce customer visibility and influence over release quality and change management, making independent testing operations more difficult.

According to research by the Dell'Oro Group, SSE-related revenue increased by 15% in the first quarter of this year, highlighting healthy market growth and the increasing importance of regular testing to ensure consistent network protection and compliance.

SDxCentral requested comments on this report from Cisco, Cloudflare, Skyhigh, Cato Networks, and Netskope.

This concludes the free translation of the SDxCentral article.

Thoughts on this article

This is an independent security evaluation report by the non-profit organization CyberRatings.org. Similar test result reports have been published in the past.

The SSE threat protection tests by CyberRatings.org are as follows:
・Malware detection and protection capabilities...Evaluated using 6,184 actual malware samples in circulation
・Exploit (vulnerability attack) protection...Ability to handle 205 types of known vulnerability attack methods
・Resistance to evasion techniques...Verifying whether security products can be bypassed using 1,154 types of evasion techniques across 37 categories
・Support for encrypted communications (TLS/SSL v1.2/1.3)...Visibility and inspection capabilities for traffic that accounts for 97% of real-world communications
・False positive rate...Presence or absence of false positives caused by over 1,500 legitimate files and applications

The testing methodology reproduces an environment close to a real network using vendor-recommended settings, evaluating from multiple perspectives such as protection rate, false positives, and throughput. To ensure independence and reproducibility, testing tools and actual attack datasets are utilized.

As for key evaluation points and trends,
・Significant differences in defense performance between vendors
The security effectiveness of SSE products varied widely, ranging from 2.95% to 100%. While many products can detect known malware and exploits with high probability, resistance to "evasion techniques" is the biggest differentiator between products, and this significantly impacts actual security effectiveness.
・Recommended products
In the latest comprehensive comparative test (published in July 2025), Fortinet, Palo Alto Networks, Versa Networks, and Zscaler earned "recommended" or "AAA" ratings. Meanwhile, Cisco, Cloudflare, and Skyhigh received "
caution" ratings, with significant test failures observed.
・Undetected threats and evasion techniques
Whether a product can block evasion techniques (stealthy or polymorphic exploits) is directly linked to actual security. Products that cannot detect these face a significantly higher risk of being bypassed by attackers.
・Emphasis on false positives and usabilityThe balance between protection and convenience is also evaluated. Many highly-rated products keep false positives during web browsing and file downloads to almost zero (permitting over 99% of legitimate traffic).

Honestly, CyberRatings evaluation reports are rarely discussed in Japan because they are not picked up as articles by domestic IT media.

However, at the very least, when selecting an SSE, it is certain that while the four vendors Fortinet, Palo Alto Networks, Versa Networks, and Zscaler are "Recommended", the three vendors Cisco, Cloudflare, and Skyhigh are rated as "Caution", and it is correct to say that one "should consider looking for other solutions".

The Magic Quadrant for SSE was released in May 2025, but when selecting an SSE, it is important not only to choose a Gartner Leader but also to refer to market share and objective third-party verification results like those from CyberRatings.

That is all.