Scrum or Kanban? Pick One and Stop Arguing

Who should read it?

Project leads, data science managers, and stakeholders who wonder why the data team is "still not done"

Intro

There is a moment in every project when someone says "we should be more agile" and everyone nods, and nothing changes. The word has become a wish, not a method. Meanwhile the actual problems grind on: requirements nobody wrote down, time estimates invented in a hallway, and teams who cannot finish a thought between meetings.

Why requirements fail before the work starts

Nearly every project begins with people who need to understand what is actually being asked, and why. This is requirements management, and it predates data science by decades, because software teams learned the hard way that unstated expectations do not stay unstated. They reappear later as "feature creep," the slow accumulation of extra requests that pushes a project past its time and budget.

The classic process is unglamorous: gathering requirements, analyzing, documenting, and loop back with stakeholders and developers until the mental picture matches. In the traditional version, stakeholders write a requirements specification document (what should be built, what is scope), the team answers how it will built and deliver it.

Be aware, even with perfect preparation, requirements will change. The skill is not preventing change, it is making the consequences of each change visible before agreeing to it.

The agile version: a frame instead of a fence

Before the project starts, stakeholders usually have only a vague idea of what the result will look like, because the work has a research character: you learn what the data supports along the way, and you cannot fully specify that in advance.

Agile management responds with iteration instead of specification. The core principles, drawn from the Agile Manifesto and lean thinking:

• Everything you build should create value for a customer; work that does not gets deprioritized or cut.
• Requirements are captured as user stories ("As [role], I want [goal], so that [benefit]"), grouped into larger epics, each with acceptance criteria that define what "done" means.
• Work happens in small increments, so a functioning product appears early and improves from there. Incremental delivery hands customers something fast. Each increment gets the customer and you closer to a great product.
• Teams reflect regularly in retrospectives and improve continuously.
• Teams organize themselves wherever central rules add nothing.

What agile does not fix

Not every task fits into a clean user story with a defined endpoint; research tasks resist getting packaged and a truely organizational transformation takes patience.

Scrum vs. Kanban: the actual decision

Both methods use a board with columns (backlog, in progress, review or test, done). The difference is the rhythm.

Scrum chops time into sprints, usually two weeks. Each sprint has a fixed set of user stories, no changes mid-sprint, a daily standup, a review with stakeholders at the end, and a retrospective for the team. It gives you fast, tangible progress and clear increments. It costs you: fixed roles (one product owner, a scrum master, and development team) and the discipline of saying no during a sprint.

Kanban originated come from Toyota's production system, and it optimizes flow instead of rhythm. There are no sprints and no fixed roles. The task list flows continuously, and the key mechanism is the WIP limit: a cap on how many tasks may sit in a column at once, so the team finishes things instead of starting everything. Change is allowed at any time. Improvement comes from measuring how long tasks take and finding the bottlenecks.

Decision table:

Scrum fits when you are building a product or application that will run long-term, where you can package work into stories and stakeholders want regular demos. Kanban fits smaller teams doing unpredictable work, such as an ongoing analytics function with a stream of requests, where the priority order of the backlog is the main lever. Pick one per project. The hybrid, Scrumban, where sprints carry a flow column on the side, is the compromise you should avoid. A team running both ends up serving neither.

Do not work in a method for the method's sake; the method exists to deliver what stakeholders actually want.

The estimation problem, and why adding people makes it worse

Here is the uncomfortable math of project planning. In a TU Munich study of IT project budgets, fewer than half of projects (<50%) stayed within 10% of plan. That was 2005.

Data science inherits the IT estimation problem and adds research uncertainty on top. The people cost dominates the budget, so time errors are money errors.

What measurably helps:

• Planning poker
  • The team estimates each story in parallel with cards, then discusses the disagreements. Slower than one person guessing, but consistently more accurate, and tracking estimated versus actual hours per story calibrates the team over time.
• Context switching awareness
  • Splitting 1 person across 5 projects does not give you 5 workers; context switching eats so much time that total effective output collapses. The same effect appears at the daily scale when attention jumps between meetings, chats, and calls. Paul Graham's distinction between a maker's schedule (needs long uninterrupted blocks) and a manager's schedule (hourly slots) explains why a dev teams can be busy all day and produce nothing.
• Brooks's law
  • Adding people to a late project makes it later. New people need onboarding, communication overhead grows with every addition, and tasks are interdependent.

Protect long focus blocks for the people doing the work, and staff them on as few projects at once as possible.

The team bubble

Stakeholders do not usually know how long work takes, so the gap between their imagined timeline and the real one is often larger than the gap between plan and reality. Impatient stakeholders then contact the people directly, each interruption costs a lost focus block, and the project slows exactly because someone is pushing it.

You need someone who shields the team, channels communication, pre-filters requests, and gives stakeholders a single competent contact. Requirements flow in through that person, results flow out through and the team gets uninterrupted time.

Shielding is not isolation: the team should still get context about why a task matters, because abstract tasks without purpose kill motivation fast, and understanding the value of the work raises it.

Tools that take an afternoon, not a transformation

Team health checks. A short anonymous survey (feelings of the team, mutual support, clarity of goals, handling of technical hurdles, sense of creating value, room to learn), rated weekly on a 1-10 or traffic-light scale, with an optional free-text field.

Only two rules matter: anonymity is non-negotiable, because otherwise results are distorted and HR will object; and the numbers must lead to action, ideally a retrospective that looks at what is going well, what is not, and what to change.

The AI Project Canvas. Jan Zawadzki's one-pager, adapted from Osterwalder's Business Model Canvas. Its power is not the document, it is the gaps it exposes before you have spent anything.

The team fills in:

• "Who are the customers?"
• "what value does the project offer to cusotmers?"
• "Who are the stakeholders?"
• "What data & skills are needed?"
• "What is outcome?"
• "What does it cost?"
• "What does it earn?"

The requirements checklist. It is a list of questions for a customer conversation, grouped into three blocks. Pick the questions that fit the customer; do not interrogate.

List of questions:

1. Name the problem fields
   1. "What frustrates you about the current situation?"
   2. "Why is this not already solved?"
   3. "What have you tried?"
   4. "What keeps you up at night?"
2. Determine the challenge
   1. "What must absolutely not happen?"
   2. "What exists already?"
   3. "Which strengths must not be endangered?"
   4. "Who must be involved and when?"
3. Describe the value
   1. "Who benefits?"
   2. "What savings in time or money would count as success?"
   3. "By which verifiable facts will you judge the project a success?"

One more thing the checklist quietly does: it professionalizes the word "no". The checklist gives you the evidence to propose the simpler, cheaper alternative and be believed.

Key Takeaways

If your data function is a continuous stream of mixed requests, set up a Kanban board with a WIP limit of two and a weekly health check; that is a one-week change. If you are building a product with a stable team and stakeholders who want demos, run Scrum, but keep the focus-block protection and the estimation feedback loop; those are the parts that actually move delivery dates. And whichever you choose, write the requirements conversation down. Memory is not a requirements document.