68k Phishing Victims are Now Searchable in Have I Been Pwned, Courtesy of CERT Poland

Presently sponsored by: Report URI: Guarding you from rogue JavaScript! Don’t get pwned; get real-time alerts & prevent breaches #SecureYourSite

68k Phishing Victims are Now Searchable in Have I Been Pwned, Courtesy of CERT Poland

Last week I was contacted by CERT Poland. They'd observed a phishing campaign that had collected 68k credentials from unsuspecting victims and asked if HIBP may be used to help alert these individuals to their exposure. The campaign began with a typical email requesting more information:

68k Phishing Victims are Now Searchable in Have I Been Pwned, Courtesy of CERT Poland

In this case, the email contained a fake purchase order attachment which requested login credentials that were then posted back to infrastructure controlled by the attacker:

68k Phishing Victims are Now Searchable in Have I Been Pwned, Courtesy of CERT Poland

All in all, CERT Poland identified 202 other phishing campaigns using the same infrastructure which has subsequently been taken offline. Data accumulated by the malicious activity spanned from October 2022 until just last week.

The advice to impacted individuals is as follows:

  1. Get a digital password manager to help you make all passwords strong and unique
  2. If you've been reusing passwords, change them to strong and unique versions now, starting with the most important services you use
  3. Turn on multi-factor authentication wherever it's available, especially for important accounts such as email, social media and banking
  4. Never open attachments or follow links unless you're confident in the trustworthiness of their origin and if in doubt, delete the email

Data From The Qakbot Malware is Now Searchable in Have I Been Pwned, Courtesy of the FBI

Presently sponsored by: Report URI: Guarding you from rogue JavaScript! Don’t get pwned; get real-time alerts & prevent breaches #SecureYourSite

Data From The Qakbot Malware is Now Searchable in Have I Been Pwned, Courtesy of the FBI

Today, the US Justice Department announced a multinational operation involving actions in the United States, France, Germany, the Netherlands, and the United Kingdom to disrupt the botnet and malware known as Qakbot and take down its infrastructure. Beyond just taking down the backbone of the operation, the FBI began actively intercepting traffic from the botnet and instructing infected machines the uninstall the malware:

To disrupt the botnet, the FBI was able to redirect Qakbot botnet traffic to and through servers controlled by the FBI, which in turn instructed infected computers in the United States and elsewhere to download a file created by law enforcement that would uninstall the Qakbot malware

As part of the operation, the FBI have requested support from Have I Been Pwned (HIBP) to help notify impacted victims of their exposure to the malware. We provided similar support in 2021 with the Emotet botnet, although this time around with a grand total of 6.43M impacted email addresses. These are now all searchable in HIBP albeit with the incident is flagged as "sensitive" so you'll need to verify you control the email address via the notification service first, or you can search any domains you control via the domain search feature. Further, the passwords from the malware will shortly be searchable in the Pwned Passwords service which can either be checked online or via the API. Pwned Passwords is presently requested 5 and a half billion times each month to help organisations prevent people from using known compromised passwords.

Guidance for those impacted by this incident is the same tried and tested advice given after previous malware incidents:

  1. Keep security software such as antivirus up to date with current definitions. I personally use Microsoft Defender which is free, built into Windows and updates automatically via Windows Update.
  2. If you're reusing passwords across services, get a password manager and change them to be strong and unique.
  3. Enable multi-factor authentication where supported, at least for your most important services (email, banking, social, etc.)
  4. For administrators with affected users, CISA has a report which explains the malware in more detail, including links to YARA rules to help identify the presence of the malware within your network.

U.S. Hacks QakBot, Quietly Removes Botnet Infections

The U.S. government today announced a coordinated crackdown against QakBot, a complex malware family used by multiple cybercrime groups to lay the groundwork for ransomware infections. The international law enforcement operation involved seizing control over the botnet’s online infrastructure, and quietly removing the Qakbot malware from tens of thousands of infected Microsoft Windows computers.

Dutch authorities inside a data center with servers tied to the botnet. Image: Dutch National Police.

In an international operation announced today dubbed “Duck Hunt,” the U.S. Department of Justice (DOJ) and Federal Bureau of Investigation (FBI) said they obtained court orders to remove Qakbot from infected devices, and to seize servers used to control the botnet.

“This is the most significant technological and financial operation ever led by the Department of Justice against a botnet,” said Martin Estrada, the U.S. attorney for the Southern District of California, at a press conference this morning in Los Angeles.

Estrada said Qakbot has been implicated in 40 different ransomware attacks over the past 18 months, intrusions that collectively cost victims more than $58 million in losses.

Emerging in 2007 as a banking trojan, QakBot (a.k.a. Qbot and Pinkslipbot) has morphed into an advanced malware strain now used by multiple cybercriminal groups to prepare newly compromised networks for ransomware infestations. QakBot is most commonly delivered via email phishing lures disguised as something legitimate and time-sensitive, such as invoices or work orders.

Don Alway, assistant director in charge of the FBI’s Los Angeles field office, said federal investigators gained access to an online panel that allowed cybercrooks to monitor and control the actions of the botnet. From there, investigators obtained court-ordered approval to instruct all infected systems to uninstall Qakbot and to disconnect themselves from the botnet, Alway said.

The DOJ says their access to the botnet’s control panel revealed that Qakbot had been used to infect more than 700,000 machines in the past year alone, including 200,000 systems in the United States.

Working with law enforcement partners in France, Germany, Latvia, the Netherlands, Romania and the United Kingdom, the DOJ said it was able to seize more than 50 Internet servers tied to the malware network, and nearly $9 million in ill-gotten cryptocurrency from QakBot’s cybercriminal overlords. The DOJ declined to say whether any suspects were questioned or arrested in connection with Qakbot, citing an ongoing investigation.

According to recent figures from the managed security firm Reliaquest, QakBot is by far the most prevalent malware “loader” — malicious software used to secure access to a hacked network and help drop additional malware payloads. Reliaquest says QakBot infections accounted for nearly one-third of all loaders observed in the wild during the first six months of this year.

Qakbot/Qbot was once again the top malware loader observed in the wild in the first six months of 2023. Source: Reliaquest.com.

Researchers at AT&T Alien Labs say the crooks responsible for maintaining the QakBot botnet have rented their creation to various cybercrime groups over the years. More recently, however, QakBot has been closely associated with ransomware attacks from Black Basta, a prolific Russian-language criminal group that was thought to have spun off from the Conti ransomware gang in early 2022.

Today’s operation is not the first time the U.S. government has used court orders to remotely disinfect systems compromised with malware. In May 2023, the DOJ quietly removed malware from computers around the world infected by the “Snake” malware, an even older malware family that has been tied to Russian intelligence agencies.

Documents published by the DOJ in support of today’s takedown state that beginning on Aug. 25, 2023, law enforcement gained access to the Qakbot botnet, redirected botnet traffic to and through servers controlled by law enforcement, and instructed Qakbot-infected computers to download a Qakbot Uninstall file that uninstalled Qakbot malware from the infected computer.

“The Qakbot Uninstall file did not remediate other malware that was already installed on infected computers,” the government explained. “Instead, it was designed to prevent additional Qakbot malware from being installed on the infected computer by untethering the victim computer from the Qakbot botnet.”

The DOJ said it also recovered more than 6.5 million stolen passwords and other credentials, and that it has shared this information with two websites that let users check to see if their credentials were exposed: Have I Been Pwned, and a “Check Your Hack” website erected by the Dutch National Police.

Further reading:

The DOJ’s application for a search warrant application tied to Qakbot uninstall file (PDF)
The search warrant application connected to QakBot server infrastructure in the United States (PDF)
The government’s application for a warrant to seize virtual currency from the QakBot operators (PDF)
A technical breakdown from SecureWorks

Weekly Update 362

Presently sponsored by: Report URI: Guarding you from rogue JavaScript! Don’t get pwned; get real-time alerts & prevent breaches #SecureYourSite

Weekly Update 362

Somehow in this week's video, I forgot to talk about the single blog post I wrote this week! So here's the elevator pitch: Cloudflare's Turnstile is a bot-killing machine I've had enormous success with for the "API" (quoted because it's not meant to be consumed by others), behind the front page of HIBP. It's unintrusive, is super easy to implement and kills bots dead. There you go, how's that for a last minute pitch? 😊

Weekly Update 362
Weekly Update 362
Weekly Update 362
Weekly Update 362

References

  1. Sponsored by: Unpatched devices keeping you up at night? Kolide can get your entire fleet updated in days. It's Device Trust for Okta. Watch the demo!
  2. Fight the bots with Cloudflare's Turnstile (and hey, if you can find a way through it, let me know and I'll pass your feedback on to Cloudflare)
  3. If you enjoy discussing escorts on public forums, you may be in the ECCIE breach (along with your email and IP address 😳)
  4. But you probably won't be in the Atmeltomo breach (unless you're Japanese and looking for a friend)
  5. The Duolingo scrape from earlier this year is now doing the rounds (that's a 100% hit rate with other breaches)
  6. And SevenRooms had their near half a TB breach from December start circulating (that's one of the largest we've seen in a long time)

Where to Find Cybersecurity Help in Zimbabwe: What to Do When You Get Hacked

  1. Cybersecurity Help in Zimbabwe:

a. The Ministry of ICT: The Ministry of ICT in Zimbabwe is a valuable resource for individuals and businesses seeking cybersecurity assistance. Their primary role revolves around providing guidance, awareness, and support to protect against cyber threats. The ministry offers various programs, workshops, and initiatives aimed at empowering citizens with knowledge on cybersecurity best practices.

b. ICT companies and service providers: Several ICT companies and service providers in Zimbabwe offer cybersecurity services and solutions to help individuals and businesses address their security concerns. These companies specialize in advising and implementing robust security measures, conducting risk assessments, and providing incident response services.

c. Computer Emergency Response Teams (CERTs): CERTs are organizations dedicated to managing cybersecurity incidents and emergencies. Zimbabwe has its own national Computer Emergency Response Team, known as ZIMCERT. Their role includes identifying threats, responding to incidents, and providing advice and guidance to affected parties.

d. Cybersecurity firms and consultants: Several independent cybersecurity firms and consultants operate in Zimbabwe and offer a wide range of services, including penetration testing, vulnerability assessments, and cybersecurity audits. These professionals can assist in evaluating and strengthening your organization's security posture.

  1. What to Do When You Get Hacked:

a. Disconnect from the network: If you suspect that your device has been compromised, disconnect it from the network immediately. This will help prevent further damage or unauthorized access.

b. Report the incident: Contact the relevant authorities, such as ZIMCERT or your local law enforcement agency, to report the hacking incident. Provide them with all the necessary details and cooperate fully during the investigation process.

c. Change passwords: Change all your passwords, including those for email accounts, social media platforms, and any compromised websites or online services. Ensure that your new passwords are strong and unique for each account.

d. Scan for malware: Run a reputable antivirus or anti-malware software scan on your device to identify and remove any malicious programs or files.

e. Strengthen security measures: Review and enhance your security measures by implementing additional layers of protection. This may include enabling two-factor authentication, updating software and operating systems, and regularly backing up important data.

  1. Zimbabwe Agricultural Show and Cybersecurity:

The Zimbabwe Agricultural Show, scheduled to take place this year, presents an excellent opportunity for individuals and businesses to explore cybersecurity resources and platforms. The Ministry of ICT, in collaboration with various stakeholders, will showcase initiatives, and offer educational materials on cybersecurity best practices. Attending this event can provide valuable insights and tools to enhance your cybersecurity knowledge and mitigate potential risks.

Conclusion:

As cyber threats continue to evolve, it is crucial for individuals and businesses in Zimbabwe to be prepared and proactive in safeguarding their digital assets. Whether through seeking assistance from the Ministry of ICT, engaging cybersecurity companies, or attending events like the Zimbabwe Agricultural Show, accessing the right resources and platforms will help enhance security measures. Remember, staying vigilant, adopting best practices, and seeking appropriate assistance are key to protecting yourself against cyber attacks in Zimbabwe.

Kroll Employee SIM-Swapped for Crypto Investor Data

Security consulting giant Kroll disclosed today that a SIM-swapping attack against one of its employees led to the theft of user information for multiple cryptocurrency platforms that are relying on Kroll services in their ongoing bankruptcy proceedings. And there are indications that fraudsters may already be exploiting the stolen data in phishing attacks.

Cryptocurrency lender BlockFi and the now-collapsed crypto trading platform FTX each disclosed data breaches this week thanks to a recent SIM-swapping attack targeting an employee of Kroll — the company handling both firms’ bankruptcy restructuring.

In a statement released today, New York City-based Kroll said it was informed that on Aug. 19, 2023, someone targeted a T-Mobile phone number belonging to a Kroll employee “in a highly sophisticated ‘SIM swapping’ attack.”

“Specifically, T-Mobile, without any authority from or contact with Kroll or its employees, transferred that employee’s phone number to the threat actor’s phone at their request,” the statement continues. “As a result, it appears the threat actor gained access to certain files containing personal information of bankruptcy claimants in the matters of BlockFi, FTX and Genesis.”

T-Mobile has not yet responded to requests for comment.

Countless websites and online services use SMS text messages for both password resets and multi-factor authentication. This means that stealing someone’s phone number often can let cybercriminals hijack the target’s entire digital life in short order — including access to any financial, email and social media accounts tied to that phone number.

SIM-swapping groups will often call employees on their mobile devices, pretend to be someone from the company’s IT department, and then try to get the employee to visit a phishing website that mimics the company’s login page.

Multiple SIM-swapping gangs have had great success using this method to target T-Mobile employees for the purposes of reselling a cybercrime service that can be hired to divert any T-Mobile user’s text messages and phone calls to another device.

In February 2023, KrebsOnSecurity chronicled SIM-swapping attacks claimed by these groups against T-Mobile employees in more than 100 separate incidents in the second half of 2022. The average cost to SIM swap any T-Mobile phone number was approximately $1,500.

The unfortunate result of the SIM-swap against the Kroll employee is that people who had financial ties to BlockFi, FTX, or Genesis now face increased risk of becoming targets of SIM-swapping and phishing attacks themselves.

And there is some indication this is already happening. Multiple readers who said they got breach notices from Kroll today also shared phishing emails they received this morning that spoofed FTX and claimed, “You have been identified as an eligible client to begin withdrawing digital assets from your FTX account.”

A phishing message targeting FTX users that went out en masse today.

A major portion of Kroll’s business comes from helping organizations manage cyber risk. Kroll is often called in to investigate data breaches, and it also sells identity protection services to companies that recently experienced a breach and are grasping at ways to demonstrate that they doing something to protect their customers from further harm.

Kroll did not respond to questions. But it’s a good bet that BlockFi, FTX and Genesis customers will soon enjoy yet another offering of free credit monitoring as a result of the T-Mobile SIM swap.

Kroll’s website says it employs “elite cyber risk leaders uniquely positioned to deliver end-to-end cyber security services worldwide.” Apparently, these elite cyber risk leaders did not consider the increased attack surface presented by their employees using T-Mobile for wireless service.

The SIM-swapping attack against Kroll is a timely reminder that you should do whatever you can to minimize your reliance on mobile phone companies for your security. For example, many online services require you to provide a phone number upon registering an account, but that number can often be removed from your profile afterwards.

Why do I suggest this? Many online services allow users to reset their passwords just by clicking a link sent via SMS, and this unfortunately widespread practice has turned mobile phone numbers into de facto identity documents. Which means losing control over your phone number thanks to an unauthorized SIM swap or mobile number port-out, divorce, job termination or financial crisis can be devastating.

If you haven’t done so lately, take a moment to inventory your most important online accounts, and see how many of them can still have their password reset by receiving an SMS at the phone number on file. This may require stepping through the website’s account recovery or lost password flow.

If the account that stores your mobile phone number does not allow you to delete your number, check to see whether there is an option to disallow SMS or phone calls for authentication and account recovery. If more secure options are available, such as a security key or a one-time code from a mobile authentication app, please take advantage of those instead. The website 2fa.directory is a good starting point for this analysis.

Now, you might think that the mobile providers would share some culpability when a customer suffers a financial loss because a mobile store employee got tricked into transferring that customer’s phone number to criminals. But earlier this year, a California judge dismissed a lawsuit against AT&T that stemmed from a 2017 SIM-swapping attack which netted the thieves more than $24 million in cryptocurrency.

Top ten Cyber Security Tips for Companies

Cyber Security in Companies 1. Develop a Strong Cybersecurity Culture: Foster a culture of cybersecurity awareness within your organization. Train employees regularly on threats, best practices, and the importance of data protection.

  1. Implement Multi-Factor Authentication (MFA): Require employees and clients to use MFA, which adds an extra layer of security by asking for multiple forms of identification before granting access.

  2. Regularly Update and Patch Software: Ensure that all software, including operating systems and applications, are up to date with the latest security patches and updates.

  3. Secure Wi-Fi Networks: Protect your Wi-Fi networks by using strong passwords, changing default router settings, and regularly updating firmware.

  4. Establish Strong Password Policies: Enforce complex passwords that include a mix of upper and lower-case letters, numbers, and special characters. Encourage employees to use password managers to securely store their passwords.

  5. Regularly Backup Data: Implement a regular data backup strategy to ensure that critical information is not lost in the event of a cyber attack or system failure.

  6. Use Secure Cloud Storage: If you utilize cloud storage, opt for reputable providers that offer strong encryption and robust security measures to protect your clients' data.

  7. Conduct Regular Security Audits: Regularly assess your systems and networks for vulnerabilities, performing penetration tests and security audits to identify potential weaknesses.

  8. Educate Employees on Phishing and Social Engineering: Provide training on spotting and avoiding phishing attacks and social engineering tactics, as these are common methods used by cybercriminals to gain unauthorized access to sensitive data.

  9. Develop an Incident Response Plan: Establish a clear plan for responding to cybersecurity incidents. This includes steps to contain, investigate, and recover from an attack, as well as steps to communicate with affected clients and stakeholders.

By implementing these top 10 cybersecurity practices, businesses in Zimbabwe can significantly enhance the protection of their clients' data. Remember, cybersecurity is an ongoing effort, and staying informed about emerging threats and regularly updating security practices is essential. Safeguarding client data not only helps protect your business but also maintains the trust and confidence of your clients in an increasingly digital world.

Tourists Give Themselves Away by Looking Up. So Do Most Network Intruders

In large metropolitan areas, tourists are often easy to spot because they’re far more inclined than locals to gaze upward at the surrounding skyscrapers. Security experts say this same tourist dynamic is a dead giveaway in virtually all computer intrusions that lead to devastating attacks like data theft and ransomware, and that more organizations should set simple virtual tripwires that sound the alarm when authorized users and devices are spotted exhibiting this behavior.

In a blog post published last month, Cisco Talos said it was seeing a worrisome “increase in the rate of high-sophistication attacks on network infrastructure.” Cisco’s warning comes amid a flurry of successful data ransom and state-sponsored cyber espionage attacks targeting some of the most well-defended networks on the planet.

But despite their increasing complexity, a great many initial intrusions that lead to data theft could be nipped in the bud if more organizations started looking for the telltale signs of newly-arrived cybercriminals behaving like network tourists, Cisco says.

“One of the most important things to talk about here is that in each of the cases we’ve seen, the threat actors are taking the type of ‘first steps’ that someone who wants to understand (and control) your environment would take,” Cisco’s Hazel Burton wrote. “Examples we have observed include threat actors performing a ‘show config,’ ‘show interface,’ ‘show route,’ ‘show arp table’ and a ‘show CDP neighbor.’ All these actions give the attackers a picture of a router’s perspective of the network, and an understanding of what foothold they have.”

Cisco’s alert concerned espionage attacks from China and Russia that abused vulnerabilities in aging, end-of-life network routers. But at a very important level, it doesn’t matter how or why the attackers got that initial foothold on your network.

It might be zero-day vulnerabilities in your network firewall or file-transfer appliance. Your more immediate and primary concern has to be: How quickly can you detect and detach that initial foothold?

The same tourist behavior that Cisco described attackers exhibiting vis-a-vis older routers is also incredibly common early on in ransomware and data ransom attacks — which often unfurl in secret over days or weeks as attackers methodically identify and compromise a victim’s key network assets.

These virtual hostage situations usually begin with the intruders purchasing access to the target’s network from dark web brokers who resell access to stolen credentials and compromised computers. As a result, when those stolen resources first get used by would-be data thieves, almost invariably the attackers will run a series of basic commands asking the local system to confirm exactly who and where they are on the victim’s network.

This fundamental reality about modern cyberattacks — that cybercriminals almost always orient themselves by “looking up” who and where they are upon entering a foreign network for the first time — forms the business model of an innovative security company called Thinkst, which gives away easy-to-use tripwires or “canaries” that can fire off an alert whenever all sorts of suspicious activity is witnessed.

“Many people have pointed out that there are a handful of commands that are overwhelmingly run by attackers on compromised hosts (and seldom ever by regular users/usage),” the Thinkst website explains. “Reliably alerting when a user on your code-sign server runs whoami.exe can mean the difference between catching a compromise in week-1 (before the attackers dig in) and learning about the attack on CNN.”

These canaries — or “canary tokens” — are meant to be embedded inside regular files, acting much like a web beacon or web bug that tracks when someone opens an email.

The Canary Tokens website from Thinkst Canary lists nearly two-dozen free customizable canaries.

“Imagine doing that, but for file reads, database queries, process executions or patterns in log files,” the Canary Tokens documentation explains. “Canarytokens does all this and more, letting you implant traps in your production systems rather than setting up separate honeypots.”

Thinkst operates alongside a burgeoning industry offering so-called “deception” or “honeypot” services — those designed to confuse, disrupt and entangle network intruders. But in an interview with KrebsOnSecurity, Thinkst founder and CEO Haroon Meer said most deception techniques involve some degree of hubris.

“Meaning, you’ll have deception teams in your network playing spy versus spy with people trying to break in, and it becomes this whole counterintelligence thing,” Meer said. “Nobody really has time for that. Instead, we are saying literally the opposite: That you’ve probably got all these [security improvement] projects that are going to take forever. But while you’re doing all that, just drop these 10 canaries, because everything else is going to take a long time to do.”

The idea here is to lay traps in sensitive areas of your network or web applications where few authorized users should ever trod. Importantly, the canary tokens themselves are useless to an attacker. For example, that AWS canary token sure looks like the digital keys to your cloud, but the token itself offers no access. It’s just a lure for the bad guys, and you get an alert when and if it is ever touched.

One nice thing about canary tokens is that Thinkst gives them away for free. Head over to canarytokens.org, and choose from a drop-down menu of available tokens, including:

-a web bug / URL token, designed to alert when a particular URL is visited;
-a DNS token, which alerts when a hostname is requested;
-an AWS token, which alerts when a specific Amazon Web Services key is used;
-a “custom exe” token, to alert when a specific Windows executable file or DLL is run;
-a “sensitive command” token, to alert when a suspicious Windows command is run.
-a Microsoft Excel/Word token, which alerts when a specific Excel or Word file is accessed.

Much like a “wet paint” sign often encourages people to touch a freshly painted surface anyway, attackers often can’t help themselves when they enter a foreign network and stumble upon what appear to be key digital assets, Meer says.

“If an attacker lands on your server and finds a key to your cloud environment, it’s really hard for them not to try it once,” Meer said. “Also, when these sorts of actors do land in a network, they have to orient themselves, and while doing that they are going to trip canaries.”

Meer says canary tokens are as likely to trip up attackers as they are “red teams,” security experts hired or employed by companies seeking to continuously probe their own computer systems and networks for security weaknesses.

“The concept and use of canary tokens has made me very hesitant to use credentials gained during an engagement, versus finding alternative means to an end goal,” wrote Shubham Shah, a penetration tester and co-founder of the security firm Assetnote. “If the aim is to increase the time taken for attackers, canary tokens work well.”

Thinkst makes money by selling Canary Tools, which are honeypots that emulate full blown systems like Windows servers or IBM mainframes. They deploy in minutes and include a personalized, private Canarytoken server.

“If you’ve got a sophisticated defense team, you can start putting these things in really interesting places,” Meer said. “Everyone says their stuff is simple, but we obsess over it. It’s really got to be so simple that people can’t mess it up. And if it works, it’s the best bang for your security buck you’re going to get.”

Further reading:

Dark Reading: Credential Canaries Create Minefield for Attackers
NCC Group: Extending a Thinkst Canary to Become an Interactive Honeypot
Cruise Automation’s experience deploying canary tokens

Fighting API Bots with Cloudflare's Invisible Turnstile

Presently sponsored by: Report URI: Guarding you from rogue JavaScript! Don’t get pwned; get real-time alerts & prevent breaches #SecureYourSite

Fighting API Bots with Cloudflare's Invisible Turnstile

There's a "hidden" API on HIBP. Well, it's not "hidden" insofar as it's easily discoverable if you watch the network traffic from the client, but it's not meant to be called directly, rather only via the web app. It's called "unified search" and it looks just like this:

Fighting API Bots with Cloudflare's Invisible Turnstile

It's been there in one form or another since day 1 (so almost a decade now), and it serves a sole purpose: to perform searches from the home page. That is all - only from the home page. It's called asynchronously from the client without needing to post back the entire page and by design, it's super fast and super easy to use. Which is bad. Sometimes.

To understand why it's bad we need to go back in time all the way to when I first launched the API that was intended to be consumed programmatically by other people's services. That was easy, because it was basically just documenting the API that sat behind the home page of the website already, the predecessor to the one you see above. And then, unsurprisingly in retrospect, it started to be abused so I had to put a rate limit on it. Problem is, that was a very rudimentary IP-based rate limit and it could be circumvented by someone with enough IPs, so fast forward a bit further and I put auth on the API which required a nominal payment to access it. At the same time, that unified search endpoint was created and home page searches updated to use that rather than the publicly documented API. So, 2 APIs with 2 different purposes.

The primary objective for putting a price on the public API was to tackle abuse. And it did - it stopped it dead. By attaching a rate limit to a key that required a credit card to purchase it, abusive practices (namely enumerating large numbers of email addresses) disappeared. This wasn't just about putting a financial cost to queries, it was about putting an identity cost to them; people are reluctant to start doing nasty things with a key traceable back to their own payment card! Which is why they turned their attention to the non-authenticated, non-documented unified search API.

Let's look at a 3 day period of requests to that API earlier this year, keeping in mind this should only ever be requested organically by humans performing searches from the home page:

Fighting API Bots with Cloudflare's Invisible Turnstile

This is far from organic usage with requests peaking at 121.3k in just 5 minutes. Which poses an interesting question: how do you create an API that should only be consumed asynchronously from a web page and never programmatically via a script? You could chuck a CAPTCHA on the front page and require that be solved first but let's face it, that's not a pleasant user experience. Rate limit requests by IP? See the earlier problem with that. Block UA strings? Pointless, because they're easily randomised. Rate limit an ASN? It gets you part way there, but what happens when you get a genuine flood of traffic because the site has hit the mainstream news? It happens.

Over the years, I've played with all sorts of combinations of firewall rules based on parameters such as geolocations with incommensurate numbers of requests to their populations, JA3 fingerprints and, of course, the parameters mentioned above. Based on the chart above these obviously didn't catch all the abusive traffic, but they did catch a significant portion of it:

Fighting API Bots with Cloudflare's Invisible Turnstile

If you combine it with the previous graph, that's about a third of all the bad traffic in that period or in other words, two thirds of the bad traffic was still getting through. There had to be a better way, which brings us to Cloudflare's Turnstile:

With Turnstile, we adapt the actual challenge outcome to the individual visitor or browser. First, we run a series of small non-interactive JavaScript challenges gathering more signals about the visitor/browser environment. Those challenges include, proof-of-work, proof-of-space, probing for web APIs, and various other challenges for detecting browser-quirks and human behavior. As a result, we can fine-tune the difficulty of the challenge to the specific request and avoid ever showing a visual puzzle to a user.

"Avoid ever showing a visual puzzle to a user" is a polite way of saying they avoid the sucky UX of CAPTCHA. Instead, Turnstile offers the ability to issue a "non-interactive challenge" which implements the sorts of clever techniques mentioned above and as it relates to this blog post, that can be an invisible non-interactive challenge. This is one of 3 different widget types with the others being a visible non-interactive challenge and a non-intrusive interactive challenge. For my purposes on HIBP, I wanted a zero-friction implementation nobody saw, hence the invisible approach. Here's how it works:

Fighting API Bots with Cloudflare's Invisible Turnstile

Get it? Ok, let's break it down further as it relates to HIBP, starting with when the front page first loads and it embeds the Turnstile widget from Cloudflare:

<script src="https://challenges.cloudflare.com/turnstile/v0/api.js" async defer></script>

The widget takes responsibility for running the non-interactive challenge and returning a token. This needs to be persisted somewhere on the client side which brings us to embedding the widget:

<div ID="turnstileWidget" class="cf-turnstile" data-sitekey="0x4AAAAAAADY3UwkmqCvH8VR" data-callback="turnstileCompleted"></div>

Per the docs in that link, the main thing here is to have an element with the "cf-turnstile" class set on it. If you happen to go take a look at the HIBP HTML source right now, you'll see that element precisely as it appears in the code block above. However, check it out in your browser's dev tools so you can see how it renders in the DOM and it will look more like this:

Fighting API Bots with Cloudflare's Invisible Turnstile

Expand that DIV tag and you'll find a whole bunch more content set as a result of loading the widget, but that's not relevant right now. What's important is the data-token attribute because that's what's going to prove you're not a bot when you run the search. How you implement this from here is up to you, but what HIBP does is picks up the token and sets it in the "cf-turnstile-response" header then sends it along with the request when that unified search endpoint is called:

Fighting API Bots with Cloudflare's Invisible Turnstile

So, at this point we've issued a challenge, the browser has solved the challenge and received a token back, now that token has been sent along with the request for the actual resource the user wanted, in this case the unified search endpoint. The final step is to validate the token and for this I'm using a Cloudflare worker. I've written a lot about workers in the past so here's the short pitch: it's code that runs in each one of Cloudflare's 300+ edge nodes around the world and can inspect and modify requests and responses on the fly. I already had a worker to do some other processing on unified search requests, so I just added the following:

const token = request.headers.get('cf-turnstile-response');

if (token === null) {
    return new Response('Missing Turnstile token', { status: 401 });
}

const ip = request.headers.get('CF-Connecting-IP');

let formData = new FormData();
formData.append('secret', '[secret key goes here]');
formData.append('response', token);
formData.append('remoteip', ip);

const turnstileUrl = 'https://challenges.cloudflare.com/turnstile/v0/siteverify';
const result = await fetch(turnstileUrl, {
    body: formData,
    method: 'POST',
});
const outcome = await result.json();

if (!outcome.success) {
    return new Response('Invalid Turnstile token', { status: 401 });
}

That should be pretty self-explanatory and you can find the docs for this on Cloudflare's server-side validation page which goes into more detail, but in essence, it does the following:

  1. Gets the token from the request header and rejects the request if it doesn't exist
  2. Sends the token, your secret key and the user's IP along to Turnstile's "siteverify" endpoint
  3. If the token is not successfully verified then return 401 "Unauthorised", otherwise continue with the request

And because this is all done in a Cloudflare worker, any of those 401 responses never even touch the origin. Not only do I not need to process the request in Azure, the person attempting to abuse my API gets a nice speedy response directly from an edge node near them 🙂

So, what does this mean for bots? If there's no token then they get booted out right away. If there's a token but it's not valid then they get booted out at the end. But can't they just take a previously generated token and use that? Well, yes, but only once:

If the same response is presented twice, the second and each subsequent request will generate an error stating that the response has already been consumed.

And remember, a real browser had to generate that token in the first place so it's not like you can just automate the process of token generation then throw it at the API above. (Sidenote: that server-side validation link includes how to handle idempotency, for example when retrying failed requests.) But what if a real human fails the verification? That's entirely up to you but in HIBP's case, that 401 response causes a fallback to a full page post back which then implements other controls, for example an interactive challenge.

Time for graphs and stats, starting with the one in the hero image of this page where we can see the number of times Turnstile was issued and how many times it was solved over the week prior to publishing this post:

Fighting API Bots with Cloudflare's Invisible Turnstile

That's a 91% hit rate of solved challenges which is great. That remaining 9% is either humans with a false positive or... bots getting rejected 😎

More graphs, this time how many requests to the unified search page were rejected by Turnstile:

Fighting API Bots with Cloudflare's Invisible Turnstile

That 990k number doesn't marry up with the 476k unsolved ones from before because they're 2 different things: the unsolved challenges are when the Turnstile widget is loaded but not solved (hopefully due to it being a bot rather than a false positive), whereas the 401 responses to the API is when a successful (and previously unused) Turnstile token isn't in the header. This could be because the token wasn't present, wasn't solved or had already been used. You get more of a sense of how many of these rejected requests were legit humans when you drill down into attributes like the JA3 fingerprints:

Fighting API Bots with Cloudflare's Invisible Turnstile

In other words, of those 990k failed requests, almost 40% of them were from the same 5 clients. Seems legit 🤔

And about a third were from clients with an identical UA string:

Fighting API Bots with Cloudflare's Invisible Turnstile

And so on and so forth. The point being that the number of actual legitimate requests from end users that were inconvenienced by Turnstile would be exceptionally small, almost certainly a very low single-digit percentage. I'll never know exactly because bots obviously attempt to emulate legit clients and sometimes legit clients look like bots and if we could easily solve this problem then we wouldn't need Turnstile in the first place! Anecdotally, that very small false positive number stacks up as people tend to complain pretty quickly when something isn't optimal, and I implemented this all the way back in March. Yep, 5 months ago, and I've waited this long to write about it just to be confident it's actually working. Over 100M Turnstile challenges later, I'm confident it is - I've not seen a single instance of abnormal traffic spikes to the unified search endpoint since rolling this out. What I did see initially though is a lot of this sort of thing:

Fighting API Bots with Cloudflare's Invisible Turnstile

By now it should be pretty obvious what's going on here, and it should be equally obvious that it didn't work out real well for them 😊

The bot problem is a hard one for those of us building services because we're continually torn in different directions. We want to build a slick UX for humans but an obtrusive one for bots. We want services to be easily consumable, but only in the way we intend them to... which might be by the good bots playing by the rules!

I don't know exactly what Cloudflare is doing in that challenge and I'll be honest, I don't even know what a "proof-of-space" is. But the point of using a service like this is that I don't need to know! What I do know is that Cloudflare sees about 20% of the internet's traffic and because of that, they're in an unrivalled position to look at a request and make a determination on its legitimacy.

If you're in my shoes, go and give Turnstile a go. And if you want to consume data from HIBP, go and check out the official API docs, the uh, unified search doesn't work real well for you any more 😎

Weekly Update 361

Presently sponsored by: Report URI: Guarding you from rogue JavaScript! Don’t get pwned; get real-time alerts & prevent breaches #SecureYourSite

Weekly Update 361

This week hasd been manic! Non-stop tickets related to the new HIBP domain subscription service, scrambling to support invoicing and resellers, struggling our way through some odd Stripe things and so on and so forth. It's all good stuff and there have been very few issues of note (and all of those have merely been people getting to grips with the new model), so all in all, it's happy days 😊

Weekly Update 361
Weekly Update 361
Weekly Update 361
Weekly Update 361

References

  1. Sponsored by: Unpatched devices keeping you up at night? Kolide can get your entire fleet updated in days. It's Device Trust for Okta. Watch the demo!
  2. Brett Adams built a really cool Splunk app using the new domain search API (and he talked me into adding a couple of other ones too)
  3. iMenu360 had 3.4M customer records appear in a breach (and ignored every single attempt made to disclose it 🤷‍♂️)
  4. We now have a model for education facilities, non-profits and charities (for now, it boils down to "log a ticket and we'll help you out")