Show HN: Filament – 用 Go 编写的高速数据传输引擎

Show HN: Filament – 用 Go 编写的高速数据传输引擎
Show HN: Filament – Fast data movement engine in Go

原始链接: https://github.com/galaxy-io/filament

Filament 是一个模块化、事件驱动的数据复制引擎，专为数据源与目标端之间可靠的数据传输而设计。它支持全量、增量以及变更数据捕获（CDC）复制，并具备强大的检查点机制、批量验证和无缝断点续传能力。Filament 使用 Go 语言构建，并配有 React/TypeScript 界面，提供了一种灵活的架构，其状态存储和事件传输组件均可替换。用户既可以将 Filament 作为独立服务部署，也可以将其集成到现有的 Go 应用程序中，或通过 Helm 在 Kubernetes 上运行。通过简单的工具链可支持本地开发，而针对 Railway、Render 和 DigitalOcean 等平台提供的一键部署功能则简化了生产环境的部署流程。该系统通过将记录提取为批次进行处理，验证双方的写入情况，并利用 PostgreSQL 和 NATS JetStream 确保进度持久化。作为一个开源且以社区为中心的系统，Filament 欢迎开发者为其引擎、连接器和文档做出贡献，为管理复杂的数据流水线提供了一种可扩展的解决方案。

Galaxy 开源了 **Filament**，这是一个采用 Go 语言编写、基于 Apache 2.0 协议的高性能数据传输引擎。该工具目前支持 PostgreSQL、MySQL、ClickHouse、Iceberg 和 S3 的连接器。该项目旨在为现有的开源数据传输方案提供更快的替代方案，团队提供了公开的基准测试工具来支持其性能声明。联合创始人 Leon 指出，这是初始版本，团队正积极寻求社区反馈以规划产品路线图。在 Hacker News 的讨论中，社区成员指出了该名称与现有的其他项目（FilamentPHP 和 Google 的 Filament 渲染引擎）之间可能存在的命名冲突。用户还询问了 Filament 与 Vector.dev 等现有工具的区别。感兴趣的开发者可以通过文档或团队的介绍博客进一步了解该项目。

原文

Filament is pluggable data replication with checkpointing, batching, and integrity events.

It moves data from sources to sinks using full, incremental, or change data capture replication. Filament keeps progress durable as data moves, verifies batches on both sides of a write, and safely pauses or resumes runs from their checkpoints.

Sources, sinks, state storage, and event transport are replaceable interfaces. Run Filament as a service with its API and web UI, deploy it to your own infrastructure, or embed the engine in a Go application.

curl -fsSL https://getgalaxy.io/filament/install | sh

On macOS via Homebrew

brew install galaxy-io/tap/filament

Prebuilt binaries for Linux and macOS are on the releases page.

To run Filament locally, install Go, Docker, just, Node.js, and pnpm. On macOS, the repository's Brewfile installs the toolchain:

For Podman users, set CONTAINER_ENGINE=podman to use Podman with the commands below. See engine configuration.

Start PostgreSQL and NATS, then run Filament:

Open http://localhost:5173 to use the web UI.

When you are finished, stop the local infrastructure with:

Kubernetes with Helm is the recommended way to run Filament in production. See the Kubernetes deployment guide for installation instructions and the Helm chart README for all supported values.

For a simpler hosted deployment without Kubernetes, deploy with 1 click:

Read the Railway, Render, and DigitalOcean guides for the deployed resources, runtime behavior, and production checklist.

Warning

The DigitalOcean deployment uses filament/standalone, an all-in-one image that includes the control-plane, server, and SQLite datastore.

A Filament pipeline connects a source to a sink and defines how selected resources should be replicated. During a run, Filament extracts records into bounded batches, writes and verifies each batch, and advances durable checkpoints only after successful work.

The runtime is event-driven: the server accepts pipeline and run requests, the control plane schedules and dispatches work, and workers execute individual runs. PostgreSQL stores durable state and NATS JetStream carries lifecycle events between components.

Filament is written in Go, with a React and TypeScript web UI. Contributions to the engine, connectors, UI, deployment tooling, tests, and documentation are welcome.

See CONTRIBUTING.md for repository setup, code generation, tests, and pull request guidance.

Show HN: Filament – 用 Go 编写的高速数据传输引擎 Show HN: Filament – Fast data movement engine in Go

Show HN: Filament – 用 Go 编写的高速数据传输引擎
Show HN: Filament – Fast data movement engine in Go