Senior Data Engineer

Company: Lambda Inc.
Location: San Francisco
Posted on: January 23, 2025

Job Description:

In 2012, Lambda started with a crew of AI engineers publishing research at top machine-learning conferences. We began as an AI company built by AI engineers. That hasn't changed. Today, we're on a mission to be the world's top AI computing platform. We equip engineers with the tools to deploy AI that is fast, secure, affordable, and built to scale. Whether they need powerhouse GPU hardware on-site or the flexibility of cloud-based solutions, we've got the horsepower to make it happen. Lambda's AI Cloud has been adopted by the world's leading companies and research institutions including Anyscale, Rakuten, The AI Institute, and multiple enterprises with over a trillion dollars of market capitalization. Our goal is to make computation as effortless and ubiquitous as electricity.About the JobJoin our team as a Senior Data Engineer, where you will collaborate with a small high-performing team of Analytics Engineers and Data Engineers to design data pipelines, a data warehouse, an automated data catalog and internal tools. Your output will directly power product analytics, marketing, supply chain, investor relations, and many more. You will have the opportunity to design and implement best practices while working directly with stakeholders. This position reports to the Data Platform Manager.What You'll Do

• Data Pipeline Development: Design, build, and maintain scalable, robust, and efficient data pipelines to process large volumes of structured and unstructured data from multiple sources.
• Data Architecture: Work with product engineers and system admins to design, document, and build data flows and infrastructure. You will also be a technical subject matter expert and recommend patterns to fix data gaps and ensure data quality.
• ETL Processes: Implement and optimize Extract, Transform, and Load (ETL) processes to integrate data across various systems, using industry-standard cloud-based tools.
• Data Warehousing: Design and manage data warehouses and data lakes, ensuring optimal storage, governance, cataloging, data lifecycle and retrieval performance.
• Data Quality & Reliability: Ensure the accuracy and consistency of data across multiple systems, performing regular data quality assessments. Implement testing, CICD and IT General Controls to ensure security and reliability of data flows.
• Collaboration: Work closely with business leaders, analysts, and other stakeholders to understand data requirements and deliver appropriate solutions. This will include participating early in product-design discussions, and staying involved with teams across the company.
• Automation & Optimization: Automate manual data processes and continuously optimize data workflows for performance, cost efficiency, and scalability.
• Technology Evaluation: Stay up-to-date with the latest industry trends, tools, and technologies to improve data engineering practices and infrastructure.
• Mentorship: Provide technical leadership and mentorship to junior data engineers, analysts, and business users, sharing best practices and fostering a collaborative team environment.You
  • Have a technical bachelor's degree (computer science, information technology, analytics, or similar) that equips you to intersect business intuition with technical skills.
  • Have 5+ years of experience in Data Engineer, Software Engineer, Analytics Engineer or Sr Data Analyst roles.
  • Are interested in being part of a small super-versatile data team that does data engineering, data analysis, and data science at executive, system and project levels.
  • Are willing to roll up your sleeves and do architecture or data janitorial work that can have significant business impact.
  • Are highly proficient in SQL and experience working with large-scale relational databases (e.g., PostgreSQL, MySQL, etc.).
  • Have extensive experience with cloud data platforms (AWS, GCP, Azure) and related services such as S3, IAM, Redshift, BigQuery, etc.
  • Have intermediate to advanced experience in Python or similar programming languages.
  • Have experience with data orchestration tools like Apache Airflow, Prefect, or similar.
  • Are knowledgeable in data modeling, database design, and performance optimization techniques.
  • Have a working understanding of data security, privacy, and compliance requirements.
  • Show a proven ability to work in a fast-paced, agile environment and manage multiple projects simultaneously.Nice to Have
    • Master's degree in a technical or analytical discipline or business discipline that supplements a technical undergraduate degree.
    • Experience with Master Data Management (MDM) practices.
    • Experience with data modeling in DBT (Data Build Tool).
    • Knowledge of data visualization tools and BI platforms (e.g., Tableau, Looker, Power BI).
    • Experience with CICD build tools (gitlab CICD, github actions, jenkins or similar).
    • Experience with IaC tools (Terraform, or similar).
    • Experience with event streaming technologies (kafka, SNS, rabbitMQ, SQS).
    • General cloud computing proficiency, such as knowledge of compute resources, databases, disk, networking, logging, alerting etc as it pertains to clouds, such as AWS and GCP.
    • Experience in iPaaS tools (e.g. Workato, Zapier, Celigo or similar).
    • Experience with creating unified master customer data records across systems.
    • Experience with containerization and orchestration tools (Docker, Kubernetes).
    • Familiarity with machine learning pipelines or data science workflows.
    • Familiarity with bash. Ability to navigate, and perform basic functions within a Unix shell such as terminal.Salary Range InformationBased on market data and other factors, the annual salary range for this position is $151,000-$218,000. However, a salary higher or lower than this range may be appropriate for a candidate whose qualifications differ meaningfully from those listed in the job description.About Lambda
      • We offer generous cash & equity compensation.
      • Investors include US Innovative Technology, Gradient Ventures, Mercato Partners, SVB.
      • We are experiencing extremely high demand for our systems, with quarter over quarter, year over year profitability.
      • Our have been accepted into top machine learning and graphics conferences, including NeurIPS, ICCV, SIGGRAPH, and TOG.
      • We have a wildly talented team of 350+, and growing fast.
      • Health, dental, and vision coverage for you and your dependents.
      • Commuter/Work from home stipends for select roles.
      • 401k Plan with 2% company match.
      • Flexible Paid Time Off Plan that we all actually use.A Final Note:You do not need to match all of the listed expectations to apply for this position. We are committed to building a team with a variety of backgrounds, experiences, and skills.Equal Opportunity EmployerLambda is an Equal Opportunity employer. Applicants are considered without regard to race, color, religion, creed, national origin, age, sex, gender, marital status, sexual orientation and identity, genetic information, veteran status, citizenship, or any other factors prohibited by local, state, or federal law.
        #J-18808-Ljbffr

Keywords: Lambda Inc., Rancho Cordova , Senior Data Engineer, Engineering , San Francisco, California