Home 零对冲(ZeroHedge)每日HackerNews

Rust 版 SearXNG
SearXNG in Rust

原始链接: https://github.com/MikeLuu99/searxng-rust

本项目是一个使用 Rust 构建的高性能 SearXNG 风格元搜索引擎。其工作原理是将用户查询并发分发至多个搜索引擎（DuckDuckGo、Brave、Startpage 和 Yahoo）。 主要特性包括： * **HTML 抓取**：使用 `reqwest` 和 `scraper`（基于 `html5ever`）来获取并解析搜索结果。 * **智能去重**：通过剔除追踪参数、移除地区前缀以及排序查询字符串来规范化 URL，从而准确合并重复结果。 * **RRF 排序**：采用倒数排名融合（Reciprocal Rank Fusion）算法汇总来自多个引擎的评分，确保在多个来源中出现的页面获得优先展示。 该引擎既可作为 Rust 自定义应用的开发库，也提供开箱即用的 Web 服务器。它专为可扩展性设计，开发者只需定义结构体并实现 `SearchEngine` 特征（trait），即可轻松接入新的搜索提供商。项目包含完善的单元测试和实时测试套件，以确保在目标搜索引擎结构发生变更时依然稳健可靠。

```Hacker News最新 | 过往 | 评论 | 提问 | 展示 | 招聘 | 提交登录SearXNG（使用 Rust 重写）(github.com/mikeluu99)4 点，由 dluuuu 发布于 51 分钟前 | 隐藏 | 过往 | 收藏 | 1 条评论 帮助 satvikpendem 4 分钟前 [–] 我正好在找类似的东西，因为我不想把原版的 SearXNG 作为 Python 包嵌入。回复 考虑申请 YC 2026 年秋季批次！申请截止日期为 7 月 27 日。 准则 | 常见问题 | 列表 | API | 安全 | 法律 | 申请 YC | 联系 搜索：```
相关文章
原文

A SearXNG-style metadata search engine written in Rust. Fans out queries to multiple search engines concurrently, scrapes their HTML results, deduplicates by normalized URL, and ranks using Reciprocal Rank Fusion (RRF).

  1. A search request arrives at GET /search?q=<query>
  2. The query is sent concurrently to DuckDuckGo, Brave, Startpage, and Yahoo via reqwest
  3. Each engine parses the HTML response with scraper (CSS selectors over Mozilla's html5ever)
  4. Results are deduplicated by normalized URL (tracking params stripped, locale prefixes removed, query params sorted)
  5. Duplicate URLs are merged and scored with RRF (score = Σ 1/(60 + rank) across engines) — pages returned by multiple engines rank higher
  6. The top results are returned as JSON
cargo add metadata-search-engine-rs

Or add manually to Cargo.toml:

[dependencies]
metadata-search-engine-rs = "0.1"

As a server (from source)

git clone https://github.com/MikeLuu99/searxng-rust
cd metadata-search-engine-rs
cargo build --release
PORT=8080 MAX_RESULTS=20 cargo run --release

Enable debug logging:

Add to your Cargo.toml:

[dependencies]
metadata-search-engine-rs = "0.1"
tokio = { version = "1", features = ["full"] }
use std::sync::Arc;
use metadata_search_engine_rs::engines::{DuckDuckGoEngine, SearchEngine, build_http_client};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Arc::new(build_http_client()?);
    let engine = DuckDuckGoEngine { client };

    let results = engine.search("rust programming", 5).await?;
    for r in results {
        println!("{}\n  {}", r.title, r.url);
    }
    Ok(())
}

Fan out to all engines and get RRF-ranked results

use std::sync::Arc;
use metadata_search_engine_rs::{
    aggregator::{aggregate, query_all_engines},
    engines::{BraveEngine, DuckDuckGoEngine, SearchEngine, StartpageEngine, YahooEngine, build_http_client},
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Arc::new(build_http_client()?);
    let engines: Vec<Arc<dyn SearchEngine>> = vec![
        Arc::new(DuckDuckGoEngine { client: Arc::clone(&client) }),
        Arc::new(BraveEngine     { client: Arc::clone(&client) }),
        Arc::new(StartpageEngine { client: Arc::clone(&client) }),
        Arc::new(YahooEngine     { client: Arc::clone(&client) }),
    ];

    let (successes, failures) = query_all_engines(&engines, "rust programming", 10).await;
    for (name, err) in &failures {
        eprintln!("engine {name} failed: {err}");
    }

    let results = aggregate(successes, 10);
    for r in &results {
        println!("[{:.3}] ({}) {}", r.score, r.engines.join(", "), r.title);
        println!("        {}", r.url);
    }
    Ok(())
}

Use only specific engines

use std::sync::Arc;
use metadata_search_engine_rs::{
    aggregator::{aggregate, query_all_engines},
    engines::{BraveEngine, DuckDuckGoEngine, SearchEngine, build_http_client},
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let client = Arc::new(build_http_client()?);
    let engines: Vec<Arc<dyn SearchEngine>> = vec![
        Arc::new(DuckDuckGoEngine { client: Arc::clone(&client) }),
        Arc::new(BraveEngine     { client: Arc::clone(&client) }),
    ];

    let (successes, _) = query_all_engines(&engines, "tokio async rust", 5).await;
    for r in aggregate(successes, 5) {
        println!("{} — {}", r.title, r.url);
        if let Some(snippet) = r.snippet {
            println!("  {snippet}");
        }
    }
    Ok(())
}
curl http://localhost:3000/health
curl "http://localhost:3000/search?q=rust"
{
  "query": "rust",
  "results": [
    {
      "title": "Rust Programming Language",
      "url": "https://rust-lang.org/",
      "snippet": "A language empowering everyone to build reliable and efficient software.",
      "engines": ["duckduckgo", "brave", "startpage", "yahoo"],
      "score": 0.049
    }
  ],
  "engines_queried": ["duckduckgo", "brave", "startpage", "yahoo"],
  "engines_failed": []
}

Error responses:

Case Status Body
Missing q 400 {"error": "query parameter 'q' is required"}
Empty q 400 {"error": "query parameter 'q' cannot be empty"}
All engines fail 503 {"error": "all engines failed to respond"} 
# All unit tests
cargo test

# Specific module
cargo test normalizer
cargo test aggregator
cargo test engines::duckduckgo
cargo test engines::brave
cargo test engines::startpage
cargo test engines::yahoo
cargo test server::handlers

# Live tests (hit real search engines — requires internet)
cargo test -- --ignored test_live

Live tests are marked #[ignore] so they don't run in CI by default. Run them manually to verify HTML selectors still work against the real sites.

A ratatui-based TUI is available as a crate to install here. Access the code via github

stx TUI

Adding a new search engine

  1. Create src/engines/<name>.rs
  2. Define a struct holding Arc<reqwest::Client>
  3. Implement the SearchEngine trait:
impl SearchEngine for MyEngine {
    fn name(&self) -> &'static str { "myengine" }

    fn search<'a>(
        &'a self,
        query: &'a str,
        max_results: usize,
    ) -> BoxFuture<'a, Result<Vec<SearchResult>, EngineError>> {
        Box::pin(async move {
            // fetch HTML, parse with scraper, return Vec<SearchResult>
        })
    }
}

Add it to engines/mod.rs and wire it in main.rs

联系我们 contact @ memedata.com