住宅代理正在抓取您的网站?耗尽它们的带宽。
Residential proxies scraping your site? Drain their bandwidth

原始链接: https://layer3intel.com/blog/residential-proxy-bandwidth-drain

Layer3 Intel 为其 Tripwire 代理检测工具推出了“带宽消耗”(bandwidth drain)功能,旨在使网页抓取在经济上变得不可行。 大多数反机器人工具仅会阻止可疑流量,而抓取者可以通过轮换 IP 地址来绕过这些限制。相比之下,Tripwire 的“带宽消耗”功能会识别出高置信度的住宅代理连接,并强制其下载最多 10MB 的数据。由于住宅代理提供商按千兆字节收费,这会显著增加每次请求的成本,从而将常规抓取转化为一项昂贵的操作。 该功能效果显著:在测试中,它迫使抓取者支付了超过 11GB 的不必要流量费用,导致其在 48 小时内停止了抓取活动。为防止误报,该功能采用选择性加入机制,仅在严格匹配 Layer3 的住宅代理数据库,且确认流量为转发代理连接时才会激活。 通过将经济负担转嫁给操作者,Tripwire 超越了简单的拦截,从根本上改变了大规模抓取的经济成本。现有用户只需更新其 Tripwire 代码片段参数即可启用此功能。

Hacker News 最新 | 过往 | 评论 | 提问 | 展示 | 招聘 | 提交 登录 住宅代理正在抓取您的网站?耗尽它们的带宽 (layer3intel.com) 6 分,Rasbora 发布于 2 小时前 | 隐藏 | 过往 | 收藏 | 讨论 | 帮助 指南 | 常见问题 | 列表 | API | 安全 | 法律 | 申请 YC | 联系 搜索:
相关文章

原文

Today we're launching bandwidth drain in Tripwire, our proxy detection tool.

Most anti-bot tools block suspicious traffic or show it a challenge. A scraper can respond by changing IPs and trying again. The block works, but it does not make the next attempt more expensive.

Tripwire's bandwidth drain adds a cost. When Tripwire confirms a residential proxy connection, it sends up to 10MB through that connection. Residential proxy providers charge their customers for those bytes.

Residential proxies charge by the gigabyte

Proxy customers can rotate through a large supply of residential IPs. Layer3 Intel sees 70–80 million live IPs at any point in time, and even the seizure of a 33-million-IP network had little effect on the wider pool.

Bandwidth is harder to avoid because major providers bill by the gigabyte. Bright Data lists $4.00/GB pay-as-you-go, dropping to $2.50/GB at a 798GB commitment. Oxylabs lists $6/GB at entry and $2.50/GB at a terabyte a month. Changing IPs does not change those rates.

The cost of a drain

Assume each page transfers 100KB. Adding a full 10MB drain makes each request use about 100 times more bandwidth. Here is the cost of one million requests at current list prices:

Residential ratePlan1M pages1M pages, drained
$2.50 / GB1 TB monthly commit$250$25,250
$4.00 / GBPay as you go$400$40,400
$6.00 / GBEntry tier$600$60,600

Rotating to another IP does not help: Tripwire checks the new connection too. Ending the connection early reduces the drain, but the proxy provider still bills for bytes already transferred.

What happened on our site

layer3intel.com was receiving steady scraper traffic through residential proxies. On August 20th, we enabled bandwidthDrain=true in our Tripwire snippet. This is the unedited dashboard result:

Tripwire dashboard card titled Bandwidth drained: no activity from August 17 to 19, large bars on August 20 and 21, a sliver on August 22 and nothing on August 23. Totals read 11.6 GB drained across 1,510 sessions, 21.8 Mbps average peak rate and 12 seconds average time on the wire
  • Aug 17–19: The drain was off.
  • Aug 20: We enabled it and sent about 6 GB through confirmed residential proxies.
  • Aug 21: Tripwire sent another 5.46 GB across 704 sessions.
  • Aug 22: The scraper traffic fell sharply.
  • Aug 23: The scraper traffic stopped.

The result

The scraping stopped within two days.

Tripwire drained 11.6 GB across 1,510 confirmed residential proxy connections. The average was 7.9MB per connection, or about 80 times the assumed 100KB page size. The average fell below the 10MB cap because crawlers often left before the transfer finished.

At the rates above, those two days added an estimated $31 to $75 to the operator's bill. The important change is the cost per request, which scales with a larger crawl.

Each drain also occupied the connection for an average of 12 seconds. During that time, the residential device had less capacity for other proxy traffic.

How Tripwire limits false positives

The feature is off by default. It runs only when all three conditions are met:

  • You enable it. Each deployment must opt in.
  • Tripwire detects a forwarded connection. Its WebSocket test measures a delay caused by the additional hop created by a forwarded connection or TCP-terminating middlebox.
  • The IP is a known residential proxy. It must match Layer3 Intel's residential proxy database, which receives more than 500 million unique signals a day.

Neither signal is enough on its own to authorize a bandwidth drain. Requiring both limits draining to Tripwire's highest-confidence residential proxy classification and reduces the risk of draining a legitimate visitor.

Layer3 Intel serves the drain payload. It does not use your origin or CDN bandwidth.

Why it applies only to residential proxies

Tripwire can identify ISP proxies and datacenter proxies, but it does not drain them. Those services often use flat-rate or unmetered bandwidth, so sending extra data would cost us without adding much cost for the proxy operator. The feature works because residential proxy bandwidth is metered.

Turning it on

Add one parameter to the Tripwire snippet:

<script
  src="https://cdn.layer3intel.com/tripwire.min.js?key=YOUR_PUBLIC_KEY&bandwidthDrain=true"
></script>

Bandwidth drain requires the visitor to execute Tripwire's JavaScript, so its effectiveness depends on how your site serves protected content. For stricter enforcement, you can place a Tripwire interstitial at the edge using a Cloudflare Worker or other middleware and require a successful assessment before serving the requested content.

A drained assessment returns its encrypted token after the transfer finishes. If a form waits for that token, run the drain on page load and validate the token already stored for the session. See the bandwidth drain documentation for configuration options and per-call overrides.

Pricing

Tripwire's free plan includes 100,000 assessments a month. Each eligible assessment can send up to 10MB, for a maximum of one terabyte of drain traffic. Layer3 Intel pays to serve that traffic; it does not count against your bandwidth.

What this changes

Bandwidth drain will not stop every scraper but it changes the economics. Defenders have long been able to consume resources on hostile proxy connections with custom engineering. Bandwidth drain turns that tactic into a one-parameter Tripwire feature: in minutes, a site can begin imposing metered bandwidth costs on confirmed residential proxy connections.

联系我们 contact @ memedata.com