NirGeron98Claude Code
curl -fsSL https://raw.githubusercontent.com/vibe-md/vibe.md/7cb0620fbd80f9829155d8866328b58cd18d5aa7/scripts/install.sh \
| bash -s -- --group llm \
--agent claude-code --ref 7cb0620fbd80f9829155d8866328b58cd18d5aa7LLM Engineering
Package · 3 skillsAvailablePractical skills for building and checking features that use language models.
Three skills that help your agent build, measure and maintain features that call language models. llm-eval-builder builds a repeatable evaluation set, llm-prompt-review reviews prompts and proposes specific edits, and llm-model-migration-check plans a safe switch from one model to another.
AI / AgentsTesting
- Claude Code
- Codex CLI
- GitHub Copilot
- Cursor
- Gemini CLI
Maintained by
The problem
Changing a prompt or a model can change every answer an AI feature gives, and "I think the new prompt is better" is a guess, not a measurement. A prompt can also ask for an output format that the code reading it cannot parse.
Skills in LLM Engineering
LLM Engineering has 3 skills. Open one to see when to use it and how to install it on its own.
LLM eval builder
llm-eval-builderBuilds a small, repeatable evaluation set and runner for an LLM feature, with deterministic checks, optional rubric grading, and baseline comparison that catches regressions after a prompt or model change.Builds a small, repeatable evaluation set and runner for an LLM feature, with deterministic checks, optional rubric grading, and baseline comparison that catches regressions after a prompt or model change. Use when adding evals to an AI feature, checking whether a prompt or model change made outputs better or worse, or when the user asks how to measure LLM output quality. Not for security review, general unit tests, or one-off prompt wording.
When to use
- The user is building or changing an LLM feature and wants to know whether outputs are getting better or worse.
- A prompt, model, or retrieval setting has changed and the user needs to check for regressions.
- The user asks how to test or measure LLM output quality.
Do not use this skill:
- to review a feature for security problems. Use
security-agent-guardfor AI-specific risks. - to write general unit tests for ordinary code. Use the project's normal test practice.
- to tweak wording in one prompt without a measurable goal. Measure first.
- to run evals against production systems, real customer data, or live accounts.
Examples
User: "Our invoice extractor sometimes returns the wrong total. How do I know if my prompt change fixed it?"
Agent: Asks for three correct extractions and the most costly error. Finds the function that builds the prompt and parses the response. Collects 30 synthetic invoices that include a missing total, two currencies, and a multi-page layout. Writes deterministic checks: the total equals the expected value, the currency is one of the allowed codes, and the output parses. Creates
evals/llm/invoice-extractor.jsonland a runner, runs five cases after confirming the cost, and reports that the total check passed four of five with the failing invoice's id. Proposes a baseline and a regression rule for the user to confirm.User: "Make a quality check for our support reply summaries. They should be short, accurate, and polite."
Agent: Explains that "accurate" needs the source ticket to check against, and asks for 20 tickets with the summaries the team accepted. Uses deterministic checks for length and for required fields, and a three-criteria rubric for politeness and accuracy with pass and fail examples. Asks the user to label 10 cases by hand before the rubric is trusted, and reports the agreement rate after calibration.
User: "Check that my agent doesn't leak secrets in its answers."
Agent: Does not build an eval for this. Explains that the question is about security, and suggests
security-agent-guardfor the review andsecurity-access-regressionfor access tests.View SKILL.mdInstall only this skill
Command for Claude Code. To change the agent, use the picker in the installation section. To get every skill in LLM Engineering, install the whole package instead.
Terminalcurl -fsSL https://raw.githubusercontent.com/vibe-md/vibe.md/7cb0620fbd80f9829155d8866328b58cd18d5aa7/scripts/install.sh \ | bash -s -- --skill llm-eval-builder \ --agent claude-code --ref 7cb0620fbd80f9829155d8866328b58cd18d5aa7LLM model migration check
llm-model-migration-checkPlans a safe switch from one language model to another.Plans a safe switch from one language model to another. Finds every model ID and model-specific setting in the project, checks the provider's current documentation for deprecations and API differences, compares eval results on the current and target models, and drafts a rollout and rollback plan. Use when a provider retires a model, when the team wants to upgrade or change models, or when model IDs are scattered through the code. Does not change production settings or deploy.
When to use
- A provider has announced that a model the project uses will be deprecated or retired.
- The team wants to move to a newer, cheaper, or faster model.
- The user is not sure which models the project uses, or where they are set.
Do not use this skill:
- to build the evaluation set itself. Use
llm-eval-builderfirst if the project has no evals. - to improve a prompt for the current model. Use
llm-prompt-review. - to make the switch in production. This skill plans and checks; the user decides and deploys.
Examples
User: "Our provider says the model we use is being retired next quarter. What do we need to do?"
Agent: Reads the provider's deprecation page and records the date. Finds three call sites, one of which uses an even older model in a backfill script nobody remembered. Lists that the target model renames a token-limit parameter and handles JSON mode differently. Proposes running the existing evals on both models, after showing the cost, and drafts a flag-based rollout with a rollback condition.
User: "Switch everything to the newest model."
Agent: Explains that it will plan and check the switch, not deploy it. Produces the inventory and the list of differences, recommends measuring with the project's evals first, and offers to make the code edits one call site at a time if the user approves.
View SKILL.mdInstall only this skill
Command for Claude Code. To change the agent, use the picker in the installation section. To get every skill in LLM Engineering, install the whole package instead.
Terminalcurl -fsSL https://raw.githubusercontent.com/vibe-md/vibe.md/7cb0620fbd80f9829155d8866328b58cd18d5aa7/scripts/install.sh \ | bash -s -- --skill llm-model-migration-check \ --agent claude-code --ref 7cb0620fbd80f9829155d8866328b58cd18d5aa7LLM prompt review
llm-prompt-reviewReviews the prompts and prompt templates behind an LLM feature for a clear task, an output format that matches the code that parses it, handling of edge cases, and testability, then proposes specific edits.Reviews the prompts and prompt templates behind an LLM feature for a clear task, an output format that matches the code that parses it, handling of edge cases, and testability, then proposes specific edits. Use when the user asks to review or improve a prompt, before merging a prompt change, or when an LLM feature gives inconsistent or unparseable output. Read-only unless the user asks for edits. Not for prompt-injection or other security review.
When to use
- The user asks to review, improve, or tighten a prompt or prompt template.
- A prompt change is about to be merged.
- An LLM feature gives inconsistent answers, ignores instructions, or returns output the code fails to parse.
Do not use this skill:
- to look for prompt injection or other security risks. Use
security-agent-guard. - to build an evaluation set. Use
llm-eval-builder, and come back here to improve the prompt it measures. - to choose or switch models. Use
llm-model-migration-check.
Examples
User: "Our summary feature sometimes returns a bulleted list and sometimes a paragraph. Can you fix the prompt?"
Agent: Finds the template and the code that displays the summary. Reports that the prompt asks for "a short summary" with no format, and that one of the two examples uses bullets while the other does not. Proposes a stated format, a length limit, and consistent examples, as a diff. Recommends measuring with
llm-eval-builderbefore merging.User: "Review this prompt before I merge it."
Agent: Reviews the assembled prompt and the parser. Finds that the prompt requests a field named
category, while the code readstype, and that nothing says what to return when the input is empty. Reports those two findings with edits, and notes that the rest of the prompt is clear.User: "Can someone trick this prompt into leaking data?"
Agent: Explains that this is a security question and suggests
security-agent-guard.View SKILL.mdInstall only this skill
Command for Claude Code. To change the agent, use the picker in the installation section. To get every skill in LLM Engineering, install the whole package instead.
Terminalcurl -fsSL https://raw.githubusercontent.com/vibe-md/vibe.md/7cb0620fbd80f9829155d8866328b58cd18d5aa7/scripts/install.sh \ | bash -s -- --skill llm-prompt-review \ --agent claude-code --ref 7cb0620fbd80f9829155d8866328b58cd18d5aa7
Install the whole package
Installs every skill in LLM Engineering. To install a single skill, open it above and copy its own command.
One command
Install directly into your project.
Open Terminal in your project folder, then run this command. No ZIP, moving files, or prompt needed.
- 1Open your project folder in Terminal
- 2Run the command above
- 3Reload your agent and ask to use LLM Engineering
Works on macOS, Linux, and Windows through WSL or Git Bash. The installer checks destinations and stops if existing files conflict.
What it does
Builds an eval set
llm-eval-builder agrees with you on what good output looks like, collects 20 to 50 realistic cases, writes the cheapest reliable checks first, and creates an eval file and a runner.
Catches regressions
It saves a baseline and a regression rule you confirm, so the next prompt or model change can be compared against it.
Reviews prompts
llm-prompt-review checks the prompt as the model receives it for a clear task, an output format that matches the parser, edge cases and conflicting rules, then proposes small edits as a diff.
Plans model switches
llm-model-migration-check reads the provider's current documentation, finds every model ID and model-specific setting, compares evals on both models, and drafts a rollout and rollback plan.
Asks before paid calls
Before any model call that costs money, your agent shows a cost estimate and asks you, and starts with a small sample.
Usage examples
Type these to your agent in plain language. Here's what to expect.
you › Our invoice extractor sometimes returns the wrong total. How do I know if my prompt change fixed it?
Expected workflow
- 01Asks for three correct extractions and the most costly error.
- 02Finds the function that builds the prompt and parses the response.
- 03Collects 30 synthetic invoices and writes checks, such as the total equals the expected value and the output parses.
- 04Runs five cases after confirming the cost and reports the failing case.
- 05Proposes a baseline and a regression rule for you to confirm.
you › Review this prompt before I merge it.
Expected workflow
- 01Reviews the assembled prompt and the code that parses its output.
- 02Finds that the prompt asks for a field named category while the code reads type, and that nothing says what to return for empty input.
- 03Reports both findings with proposed edits.
you › Our provider says the model we use is being retired next quarter. What do we need to do?
Expected workflow
- 01Reads the provider's deprecation page and records the date.
- 02Finds every call site, including an older model in a backfill script nobody remembered.
- 03Lists what differs on the target model.
- 04Proposes running the evals on both models after showing the cost, and drafts a rollout with a rollback condition.
How it works
Each skill starts from your code: the prompt, the function that calls the model, and the code that uses the result. llm-eval-builder measures that whole path, not just the raw model call, and llm-prompt-review reviews the prompt as the model receives it.
The skills work together: build evals with llm-eval-builder, then use them to measure a prompt edit or a model switch. None of them claims an improvement without a measurement.
Safety and limitations
Required permissions
- Read your code and configuration.
- Create an eval file and a runner, after you confirm where they go.
- Read the model provider's current documentation online, to check model status.
- Run model calls that cost money, only after you approve the cost.
Limitations
- They do not change your production prompt, model settings or application code unless you ask, and they do not commit, push or deploy.
- An eval set of 20 to 50 cases shows direction, not statistical proof.
- API keys come from your own environment. They are never written into the repository or printed.
- They do not do security review. For prompt injection and other AI risks, use security-agent-guard from the Security package.