CRISP: Complex Reasoning with Interpretable Step-based Plans

Complex Reasoning with Interpretable Step-based Plans — a training dataset of short high-level plans for math and code problems

Matan Vetzler, Koren Lazar, Guy Uziel, Eran Hirsch, Ateret Anaby-Tavor, Leshem Choshen · arXiv · 2025

In one sentence

CRISP is a 122,000-plan multi-domain dataset of validated high-level plans for math and code problems, showing that a LoRA fine-tuned 8B planner writes better plans than a few-shot 70B model and cuts downstream error over chain-of-thought by up to 28%.

Abstract

Recent advancements in large language models (LLMs) underscore the need for stronger reasoning capabilities to solve complex problems effectively. While Chain-of-Thought (CoT) reasoning has been a step forward, it remains insufficient for many domains. A promising alternative is explicit high-level plan generation, but existing approaches largely assume that LLMs can produce effective plans through few-shot prompting alone, without additional training. In this work, we challenge this assumption and introduce CRISP (Complex Reasoning with Interpretable Step-based Plans), a multi-domain dataset of high-level plans for mathematical reasoning and code generation. The plans in CRISP are automatically generated and rigorously validated--both intrinsically, using an LLM as a judge, and extrinsically, by evaluating their impact on downstream task performance. We demonstrate that fine-tuning a small model on CRISP enables it to generate higher-quality plans than much larger models using few-shot prompting, while significantly outperforming Chain-of-Thought reasoning. Furthermore, our out-of-domain evaluation reveals that fine-tuning on one domain improves plan generation in the other, highlighting the generalizability of learned planning capabilities.

Questions this paper answers

can a small model that has been trained to outline solutions write better outlines than a much bigger model given a few examples?
does LoRA fine-tuning an 8B instruct model for high-level plan generation outperform a 70B few-shot planner on math and code benchmarks?
how do I get good solution plans without paying for a 70B planner at inference time?
should I fine-tune my own 8B planner or just prompt a 70B model with a few plan examples?
A LoRA fine-tuned Granite-3.1-8B-Instruct planner beats a few-shot Llama-3.1-70B-Instruct planner on all four benchmarks. With the small solver it reaches 64.8 vs 62.0 Pass@1 on MBPP, 76.4 vs 73.1 on HumanEval, 87.1 vs 85.7 on GSM8K and 60.6 vs 53.2 on MATH.
Holds for: Granite-3.1-8B-Instruct as solver in zero-shot; 5 epochs of LoRA (R=32, alpha=16, lr 1e-5) on CRISP; planner trained on both domains.
The CRISP effect reproduces on a second small model. A LoRA fine-tuned Llama-3.1-8B-Instruct planner scores 64.9 on MBPP, 76.1 on HumanEval, 86.3 on GSM8K and 54.5 on MATH, above both its vanilla self and a vanilla Llama-70B planner.
Holds for: Llama-3.1-8B-Instruct serving as both planner and solver; error reductions versus CoT range from 6.2% on MATH to 13.4% on HumanEval, smaller than those obtained with Granite-3.1-8B-Instruct.
how much more accurate is asking a model to sketch a plan first compared with letting it think step by step?
what error reduction over chain-of-thought prompting does a fine-tuned high-level planner give on GSM8K, MATH, MBPP and HumanEval?
how do I cut errors on math word problems and code generation beyond what chain-of-thought gives me?
is switching my pipeline from chain-of-thought to plan-then-solve worth it for math and coding tasks?
Plans from the CRISP fine-tuned Granite-3.1-8B-Instruct planner cut error relative to Chain-of-Thought prompting by up to 28.1% on GSM8K and 23.4% on HumanEval when Llama-3.1-70B-Instruct is the solver.
Holds for: Relative error reduction against the CoT (no plan) baseline with the same solver; four benchmarks (MBPP, HumanEval, GSM8K, MATH); solvers used zero-shot without fine-tuning.
Plans from the CRISP fine-tuned small planner give an average error reduction over Chain-of-Thought of 17.1% with the small solver and 19.65% with the large solver, so both solver sizes gain similarly.
Holds for: Averaged over MBPP, HumanEval, GSM8K and MATH; 'small' is Granite-3.1-8B-Instruct and 'large' is Llama-3.1-70B-Instruct.
can handing a model a plan for the problem actually make its answer worse than no plan at all?
do low-quality few-shot generated plans degrade solver accuracy below a chain-of-thought baseline?
how do I tell whether the plans I feed my solver are helping or hurting accuracy?
if I prompt a small model for plans instead of fine-tuning one, am I risking worse results than plain chain-of-thought?
Plans from a vanilla few-shot Granite-3.1-8B-Instruct planner do not help a Llama-3.1-70B-Instruct solver: error rises by up to 4.3% versus Chain-of-Thought, with MATH accuracy falling from 67.2 to 65.8.
Holds for: Untrained small planner paired with the large solver; with the small solver the same vanilla plans give small gains, and vanilla large-model plans improve over CoT by up to 9.1%.
if a model learns to outline solutions to math problems, does that skill carry over to writing code?
does planner fine-tuning on math-only plan data transfer out of domain to code generation benchmarks, and is the transfer symmetric?
how do I train a planner that helps on coding when the plan data I have is all math?
I only have plan training data for one domain, can I still expect gains on the other?
Planning ability transfers across domains: a CRISP math-only fine-tuned planner reaches 84.6 Pass@1 on HumanEval, only 0.4 points below the code-only fine-tuned planner's 85.0, and both beat the vanilla Llama-3.1-70B-Instruct planner's 82.2.
Holds for: Llama-3.1-70B-Instruct as solver; single-domain LoRA fine-tuning of Granite-3.1-8B-Instruct; transfer measured only between math and coding-and-debugging.
Math-to-code transfer is stronger than code-to-math: the average gap in error reduction between the out-of-domain and in-domain planner is 2.8% on the code benchmarks but 8% on the math benchmarks.
Holds for: Granite-3.1-8B-Instruct planners fine-tuned on one CRISP domain, Llama-3.1-70B-Instruct solver, MBPP/HumanEval versus GSM8K/MATH; the paper calls the two directions not completely comparable.
what should I read about teaching a language model to sketch out a solution before it answers?
is there a public plan-generation corpus and prior work treating high-level planning as a fine-tunable skill rather than an emergent prompting behaviour?
where do I start if I want to train a planner model for math and code reasoning?
CRISP treats high-level plan generation as a trainable skill rather than an emergent one, in contrast to plan-and-solve prompting work that assumes few-shot prompting alone yields good plans.
Holds for: Positioning as of the 2025 arXiv preprint; argued for myopic problems solvable by a fixed sequence of steps, not for interactive feedback-dependent planning.
CRISP is a dataset of 65,800 math and 56,200 coding-and-debugging high-level plans, distilled from the detailed solutions of Magpie-Reasoning-V1-150K and filtered by an LLM judge plus a downstream-task check.
Holds for: Two domains only (math, coding and debugging); plans generated by Mixtral-8x22B-Instruct-v0.1, judged by Llama-3.1-70B-Instruct; 3.8 steps per math plan, 4.4 per code plan.
how many solution outlines for math and coding problems are in the CRISP collection, and what were they built from?
what is the size and provenance of the CRISP high-level plan corpus for math and coding-and-debugging?
where can I get tens of thousands of high-level solution plans to fine-tune a planner on?
is there enough plan data in CRISP for me to fine-tune an 8B planner without generating my own?
CRISP is a dataset of 65,800 math and 56,200 coding-and-debugging high-level plans, distilled from the detailed solutions of Magpie-Reasoning-V1-150K and filtered by an LLM judge plus a downstream-task check.
Holds for: Two domains only (math, coding and debugging); plans generated by Mixtral-8x22B-Instruct-v0.1, judged by Llama-3.1-70B-Instruct; 3.8 steps per math plan, 4.4 per code plan.
are plans with more steps in them better for getting the right answer?
does plan step count correlate with downstream solver accuracy for fine-tuned versus few-shot planners?
should I prompt for more detailed multi-step plans or shorter ones when I want better solver accuracy?
my planner emits long verbose plans, is that a problem?
Better plans are shorter, not longer: the vanilla Llama-3.1-70B-Instruct planner produces 1.3 more steps than the CRISP fine-tuned small planner on coding benchmarks and 0.8 more on math benchmarks, yet performs worse.
Holds for: Average step counts across MBPP, HumanEval, GSM8K and MATH; plans constrained by the generation prompt to 3-5 steps.
is it worth throwing away machine-generated training plans that fail a quality check, given you end up with less data?
does intrinsic LLM-judge filtering plus extrinsic downstream-task filtering of synthetic plan data improve benchmark accuracy net of the data loss?
how do I decide whether to filter my synthetic plan training set or keep all of it?
I have a synthetic plan dataset and a limited budget, should I spend it on judge-based filtering?
Both CRISP filtering stages improve downstream accuracy despite shrinking the data: intrinsic LLM-judge filtering adds 0.72 points on GSM8K and MATH and 0.44 on MBPP and HumanEval. Extrinsic filtering adds a further 0.28 points for math and 0.32 for code.
Holds for: Intrinsic filtering removed 7,412 math and 5,592 code plans (about 9% of the original dataset); extrinsic filtering removed a further 1,089 math and 4,612 code plans, keeping only plans that raised the number of correct answers over 10 sampled solutions.
when a large model compares plans written by a smaller trained model against its own, which does it say are clearer and more complete?
in an LLM-as-a-judge pairwise evaluation on clarity, coherence, conciseness and completeness, how often are fine-tuned 8B plans preferred over the judge's own few-shot plans?
how do I show that my fine-tuned planner writes better plans without relying only on downstream accuracy?
can I trust a 70B judge to grade plans fairly when it is comparing against plans it wrote itself?
Llama-3.1-70B-Instruct, acting as judge against its own few-shot plans, preferred the CRISP fine-tuned Granite-3.1-8B-Instruct plans in 73.3% of cases on average across the four datasets for clarity, coherence, conciseness and completeness.
Holds for: Pairwise LLM-as-judge with the competitor model also serving as judge, a setup biased in the competitor's favour; per-dataset preference was highest on MATH and HumanEval.
in a two-model setup where one writes the plan and another solves it, which model matters more for the final answer?
does solver capacity dominate planner capacity for end-task accuracy in a decoupled plan-and-solve pipeline?
I have compute for one strong model in a plan-and-solve pipeline, where do I put it?
should I spend my budget upgrading the model that writes the plan or the one that produces the solution?
Solver choice outweighs planner choice: a vanilla small planner with the Llama-3.1-70B-Instruct solver scores higher on all four benchmarks than a CRISP fine-tuned small planner with the Granite-3.1-8B-Instruct solver.
Holds for: Comparison of two configurations in the same table; planner and solver drawn from Granite-3.1-8B-Instruct and Llama-3.1-70B-Instruct only.

Claims and scope

Common misreadings

Terminology in this paper

high-level plan
A 3-5 step abstract statement of the logical strategy for solving a problem, with equations, function names and numerical computations deliberately omitted.
myopic problem
A task solvable by a predefined sequence of steps, without adaptive decision-making or real-time environment feedback.
planner and solver
In a plan-and-solve pipeline, the planner is the model that emits the high-level plan and the solver is the separate model that receives the plan plus the task description and produces the answer.
extrinsic validation
Keeping a generated plan only if including it in the prompt yields more correct answers on the original task than solving without it, measured over 10 sampled solutions.
Err down
Relative reduction in error rate compared with a Chain-of-Thought no-plan baseline using the same solver model.

How to cite

@article{vetzler2025crisp,
  title={CRISP: Complex Reasoning with Interpretable Step-based Plans},
  author={Vetzler, Matan and Lazar, Koren and Uziel, Guy and Hirsch, Eran and Anaby-Tavor, Ateret and Choshen, Leshem},
  journal={arXiv preprint arXiv:2507.08037},
  year={2025}
}

References

See the full reference list in the paper.