資深後端工程師 (爬蟲工程師) Sr. Backend Engineer (Scraper Engineer)

薪資範圍:1,400,000 ~ 1,750,000 TWD / year

[本職缺僅接受CreatorDB網站投遞]

[This position only accepts applications through the CreatorDB website]

請至CreatorDB招募網站投遞個人履歷表，此職缺履歷登錄網址:
Please submit your resume via the CreatorDB recruitment website. Application link for this position:

https://CreatorDB.teamdoor.io/s/3F87eKWd

關於我們

我們是一家數據驅動型公司，通過大規模網絡爬蟲為企業提供可操作的洞察。我們的爬蟲團隊是我們數據獲取策略的骨幹，構建和維護複雜的系統，從網絡各處收集、處理和構建數據。

職位概述

作為我們爬蟲團隊的高級後端工程師，您將設計、開發和維護用於大規模數據獲取的複雜系統。您將負責構建強大的爬蟲解決方案，管理複雜的數據管道，並指導初級工程師。根據您的團隊分配，您將專注於維護現有系統或開發新的爬蟲功能。

主要職責

- 維護重點

管理爬蟲目標的格式變更，確保數據流的連續性
設計並實施結構化信息捕獲的數據定義
開發創作者聚合系統，整合來自各種來源的數據
構建和維護下游團隊的數據服務接口
與 AI 團隊密切合作，將爬取的數據整合到機器學習管道中

- 開發重點

構建和維護特定平台的爬蟲，用於社交媒體和網絡內容
開發和增強我們的網絡爬行蜘蛛系統
創建和維護數據清理和轉換工具
隨著平台的發展，設計和實施新的爬蟲方法
逆向工程網站和 API，創建高效的數據提取方法

- 共同職責

領導我們爬蟲基礎設施主要組件的設計和實施
指導後端工程師並審核他們的代碼
與 DevOps 團隊合作確保系統可靠性
應對需要技術專長的緊急情況
與產品和 API 團隊合作，使數據收集與業務需求保持一致
開發技術規格和架構文檔
參與系統架構討論和設計決策
為我們工程實踐的持續改進做出貢獻

必要資格

計算機科學、工程或相關領域的學士學位，或同等經驗
5 年以上後端開發經驗，其中至少 2 年專注於網絡爬蟲
3 年以上 TypeScript 經驗
大型系統架構、設計、實施經驗
熟悉分布式系統和高容量數據處理
性能優化和可擴展性開發
精通用於爬蟲的最佳實踐、錯誤處理
深入理解 HTTP/HTTPS 協議、請求標頭和網絡認證機制
具備瀏覽器自動化工具經驗（Puppeteer、Playwright、Selenium）
了解數據處理管道和 ETL 概念
逆向工程網站和 API 的能力
雲基礎設施經驗（AWS、GCP 或 Azure）
容器化技術經驗（Docker、Kubernetes）
具備 NoSQL 數據庫經驗（MongoDB、Firestore、Cassandra、ScyllaDB）

優先資格

SDK 開發經驗
監控系統經驗 (Prometheus, ELK Stack, Grafana)
了解無頭瀏覽和 JavaScript 渲染
理解反機器人技術和緩解策略
具備代理管理和 IP 輪換技術經驗
了解數據倉庫概念
具備 AI/ML 數據管道經驗
Monorepo (Nx) 開發環境經驗
理解數據隱私法規和道德爬蟲實踐

我們提供

具有競爭力的薪資和福利待遇
來自經驗豐富的資深後端工程師的指導
使用尖端數據獲取技術的實踐經驗
協作的團隊環境
專業發展機會
靈活的工作安排
明確的晉升至資深後端工程師的職業發展路徑
有機會參與具有實際應用的重要項目

重要提示

我們致力於道德爬蟲實踐和尊重網站服務條款。我們的爬蟲活動遵循法律指南和行業最佳實踐。

About Us

We are a data-driven company that leverages web scraping at scale to provide actionable insights for businesses. Our Scraper Team is the backbone of our data acquisition strategy, building and maintaining sophisticated systems that collect, process, and structure data from across the web.

Job Overview

As a Senior Backend Engineer on our Scraper team, you will design, develop, and maintain sophisticated systems for data acquisition at scale. You will be responsible for building robust scraping solutions, managing complex data pipelines, and mentoring junior engineers. Depending on your team assignment, you will focus on either maintenance of existing systems or development of new scraping capabilities.

Key Responsibilities

- Maintenance Focus

Manage format changes in our scraping targets to ensure continuous data flow
Design and implement data definitions for structured information capture
Develop creator aggregation systems to consolidate data from various sources
Build and maintain Data Service interfaces for downstream teams
Work closely with AI teams to integrate scraped data into machine learning pipelines

- Development Focus

Build and maintain platform-specific scrapers for social media and web content
Develop and enhance our spider systems for web crawling
Create and maintain utilities for data cleaning and transformation
Design and implement new scraping methodologies as platforms evolve
Reverse engineer websites and APIs to create efficient data extraction methods

- Common Responsibilities

Lead the design and implementation of major components of our scraping infrastructure
Mentor Backend Engineers and review their code
Collaborate with the DevOps team to ensure system reliability
Respond to emergency situations requiring technical expertise
Work with Product and API teams to align data collection with business needs
Develop technical specifications and architecture documentation
Participate in system architecture discussions and design decisions
Contribute to the continuous improvement of our engineering practices

Required Qualifications

Bachelor's degree in Computer Science, Engineering, or related field, or equivalent experience
5+ years of experience in backend development with at least 2 years focused on web scraping
3+ years of experience in TypeScript
Big system architect, design, implementation experience
Familiarity with distributed systems and high-volume data processing
Background in performance optimization and scalability
Strong proficiency in scraping with best practices, error handling...etc
Deep understanding of HTTP/HTTPS protocols, request headers, and web authentication mechanisms
Experience with browser automation tools (Puppeteer, Playwright, Selenium)
Knowledge of data processing pipelines and ETL concepts
Proven ability to reverse engineer websites and APIs
Experience with cloud infrastructure (AWS, GCP, or Azure)
Experience with containerization technologies (Docker, Kubernetes)
Experience with noSQL databases (MongoDB, Firestore, Cassandra, ScyllaDB)
Experience mentoring junior engineers

Preferred Qualifications

Experience with SDK development
Experience with Observability (Prometheus, ELK Stack, Grafana)
Knowledge of headless browsing and JavaScript rendering
Understanding of anti-bot technologies and mitigation strategies
Experience with proxy management and IP rotation techniques
Knowledge of data warehousing concepts
Experience with AI/ML data pipelines
Experience with monorepo development environments (Nx)

What We Offer

Competitive salary and benefits package
Mentorship from experienced Senior Backend Engineers
Hands-on experience with cutting-edge data acquisition technologies
Collaborative team environment
Professional development opportunities
Flexible work arrangements
Clear career path to Senior Backend Engineer
Opportunity to work on impactful projects with real-world applications

Important Note

We are committed to ethical scraping practices and respect for website terms of service. Our scraping activities follow legal guidelines and industry best practices.

關於我們

CreatorDB 是一個專注於網紅行銷的軟體平台與服務團隊。

我們正在擴展網紅合作業務，並希望與更多直播社群媒體創作者（如 Twitch.tv）、TikTok 網紅以及 Instagram 創作者合作。

我們的目標是透過詳細的數據分析、分類與篩選，幫助品牌與行銷機構更好地了解社群媒體上的影響者。

近幾個月來，我們的業務大幅成長，並希望持續保持這一趨勢。

我們的辦公室位於台北，擁有來自世界各地的團隊成員，打造了一個友善且國際化的工作環境。

About Us

CreatorDB is an influencer marketing software platform and services team.

We are expanding our influencer operations and are looking to work with more live streaming social media creators (eg. Twitch.tv), TikTok talent, and Instagrammers.

We aim to help brands and marketing agencies better understand social media influencers through detailed data, categorization, and filtering.

We have grown significantly in recent months and are looking to continue that trend.

Our office, located in Taipei, is comprised of team members from many countries in a very friendly and international environment.

公司地址:

台灣台北市大安區忠孝東路四段209號12F

其他:

None-2025-04-01

應徵