Example Company
Software engineer
- Location
- San Francisco
- Type
- Full time
- Compensation
- $150k-$200k
- Posted
- 2 days ago
Join a team building useful software. Work closely with engineering and design to turn new ideas into products.
Loading job details
Senior AI Engineer LLM and Agentic Workflows
- Posted
- 23D AGO
About the role
ELLKAY started out providing connectivity solutions to laboratories and within a few years, grew to also provide data management solutions to ambulatory organizations. ELLKAY is now a trusted data management partner in five healthcare segments. ELLKAY’s solutions continue to serve laboratories and ambulatory practices and have expanded to empower hospitals and health systems, healthcare IT vendors, ambulatory practices, health plans, and other healthcare organizations with cutting-edge technologies and solutions that drive their growth and interoperability strategies.
Today, ELLKAY remains true to our core values, building strong partner relationships and offering unparalleled service and support while providing innovative, scalable solutions to the challenges our customers face in today’s data-rich world.
ELLKAY's experience, customer-focused approach, and reputation for innovation, speed, and accuracy differentiate ELLKAY as a premier partner for your interoperability needs and data management strategy.
Job Description:
Build production LLM applications and agent workflows for implementation assistance, connector capability discovery, migration operations, and archive access. The engineer will implement retrieval, tool integration, workflow state, citations, approval checkpoints, and automated evaluation. The role connects assistants to existing ELLKAY services and the Central DME Portal while preserving tenant, purpose, and data-access boundaries.
Essential Duties and Responsibilities:
- Build RAG and tool-calling workflows for implementation assistance, connector capability discovery, error explanation, and guided migration operations.
- Create context assembly from approved documentation, run status, mapping metadata, and evidence. Return citations and structured outputs.
- Implement MCP clients/tool integrations and thin adapters over platform contracts. Handle async request IDs, retries, timeouts, and duplicate action prevention with the orchestration team.
- Implement permission-aware retrieval, approval checkpoints, prompt-injection defenses, bounded tool use, and explicit failure or abstention behavior.
- Capture agent traces and user feedback, and add regression tests for prompts, tool selection, retrieval quality, and task completion.
- Improve retrieval and context assembly using measured failures. Evaluate rerankers or model adaptation when simpler prompting and retrieval changes do not meet quality needs.
- Partner with Data Platform and product engineers, clinical/data SMEs, DME/QA, and security teams. Build reusable services for ELLKAY products and preserve source evidence, approved data-use rights, and tenant isolation. Model or agent proposals follow established human approval and platform release workflows.
- 5+ years in software/AI engineering, including production LLM applications and backend services. The total experience requirement does not imply five years in generative AI alone.
- Strong Python, REST APIs, asynchronous programming, data handling, software testing, and production debugging.
- Hands-on RAG, hybrid/vector search, embeddings, reranking, structured outputs, context management, and tool calling.
- Experience implementing agent state, timeouts, bounded retries, approval checkpoints, fallback, and duplicate-action prevention.
- Ability to evaluate retrieval relevance, groundedness, tool selection, task completion, latency, and unit cost using representative tests.
- Experience deploying services to a cloud environment and implementing authorization-aware retrieval and PHI-safe telemetry. A relevant degree or equivalent practical experience is acceptable.
Preferred qualifications
MCP, LangGraph or equivalent agent orchestration, FastAPI, AWS, Docker/Kubernetes, HL7/FHIR familiarity, enterprise authorization patterns, and integrations with project or engineering systems such as Asana/JTS.
Expected outcomes
- First 90 days: deliver a synthetic-data agent prototype with tool contracts, traces, and regression tests covering correct execution and failure cases.
- By six months: pilot no-PHI capability, status, and failure-explanation tools with a design partner, recording task completion, groundedness, response time, and cost per task.
- By 12 months: target at least 80% successful completion of agreed supported tasks on a held-out benchmark within approved cost/latency limits. Require passing authorization tests, traceability for all tool actions, and confirmation for exposed action/bulk tools.
- Quality, performance, and business-impact targets will be calibrated to the selected use cases and their initial baselines. Delivery depends on the corresponding platform capabilities and approved customer pilot scope.
Engineering approach
Use rules, classical ML, retrieval, or LLMs according to measured task performance. Document model/data limitations and route uncertain outputs for review. Retain versioned evaluations and release evidence, and improve the service through production feedback. Training on customer data requires approved rights. Clinical delivery paths remain independent of AI processing.
Additional Information:
ELLKAY is committed to fostering a collaborative and high-performance work environment that supports innovation, teamwork, and professional growth. Most roles are designed to operate from our office locations to encourage effective collaboration and engagement across teams.
Any alternative work arrangements may be considered at the company’s discretion based on role requirements and business needs.
For more information about our company, please visit www.ELLKAY.com.
ELLKAY is a Smoke-Free Workplace.
ELLKAY, LLC provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.
Similar roles
Founding Engineer (AI + Backend) - Remote, IndiaOpenbenchmarks · Remote
Operational Excellence EngineerDover
Senior Software Engineer - JavaAvalara · Remote
Parts Engineer IINASA Jet Propulsion Laboratory
Principal Engineer, Service DeliveryDell Technologies · Gurugram
Senior Controls & Automation Engineer, Kansas City PlantProcter & Gamble