How to Evaluate a Software Development Agency: Questions, Red Flags, and a Scoring Framework

Choosing a software development agency is one of the highest-stakes vendor decisions a company makes. You are picking a team that will build something your business depends on. Get it wrong and you lose months and money. Get it right and you gain a partner that accelerates your business for years.

Most advice on this topic is vague. "Check their portfolio." "Ask about their process." That is not enough to make a $100,000+ decision.

Here is a concrete framework for evaluating agencies, including specific questions to ask, what the answers should sound like, and a scoring rubric you can actually use.

Step 1: Define Your Problem Before You Evaluate Anyone

Before you look at a single agency website, write down the problem you are solving. One paragraph. If you cannot explain it clearly, you are not ready to hire.

The best agency relationships start with a shared understanding of the problem — not a 40-page requirements document. A clear statement of what is broken, who it affects, and what success looks like.

This paragraph becomes your filter. Every agency you talk to will respond to it differently, and those responses will tell you more than any portfolio page.

Step 2: Look Past the Website

Agency websites are marketing materials, not portfolios. The real question is whether their work holds up under scrutiny, not whether their homepage has smooth animations.

What to Ask For

Case studies with specifics. Not "we helped a client increase efficiency." That means nothing. You want:

• What was the problem?
• What was the timeline and budget?
• What technology did they use and why?
• What were the measurable results?

If they cannot provide this level of detail for at least 2-3 projects, they either do not track outcomes or do not have outcomes worth sharing.

A reference call with a past client. Any agency that refuses is a red flag. When you get on that call, ask two questions:

1. Did the project finish within 20% of the original timeline and budget?
2. Would you hire them again?

The second question tells you everything. If there is hesitation, dig deeper.

Code samples or a technical walkthrough. Ask to see how they structure a project, how they handle testing, and how they manage deployments. You do not need to be technical to listen — pay attention to whether the explanation is clear and organized, or hand-wavy and vague.

Step 3: Evaluate the Team, Not Just the Company

The 10 Questions to Ask

Ask these during your initial calls. The answers — and the way they answer — will tell you what working with them will actually be like.

#	Question	Good Answer	Warning Sign
1	Who will work on my project day to day?	Specific names and roles	"We'll assign the right team later"
2	Will these people stay on the project or rotate?	Committed team for the duration	"We rotate based on availability"
3	What technology would you recommend and why?	Recommendation fits your problem	Same stack regardless of the problem
4	How do you run projects?	Specific process: sprint length, demo schedule, communication cadence	Vague: "We're agile" with no details
5	What happens when something goes wrong?	Defined escalation path, examples of past issues resolved	"That doesn't really happen"
6	What is included in the price?	Itemized: design, dev, QA, PM, DevOps	Unclear — things assumed included are actually add-ons
7	How do you handle scope changes?	Change request process with impact assessment	"We'll figure it out as we go"
8	Who owns the code?	You own it, clearly stated in the contract	Unclear IP terms
9	What happens after launch?	Defined support period, maintenance options, SLA	Silence — you are on your own
10	What percentage of revenue comes from repeat clients?	50%+ (means clients come back)	Low repeat rate or unwillingness to share

Step 4: Understand the Pricing Models

Agencies price in three ways. Each has tradeoffs.

Fixed Price

How it works: Defined scope, defined price. You pay $X for Y deliverables.

Best for: Well-defined projects with clear requirements. Marketing websites, simple mobile apps, specific integrations.

Risk: Scope always grows. Fixed-price contracts create friction when it does. Either the agency says no to changes, or they say yes and build resentment because they are doing unpaid work.

Time and Materials (T&M)

How it works: You pay for hours worked at agreed rates. Scope can flex.

Best for: Complex projects where requirements will evolve. Most custom software falls here.

Risk: Without discipline, costs drift upward. Requires trust and regular check-ins.

Monthly Retainer

How it works: Set number of hours per month at a fixed rate.

Best for: Ongoing relationships — maintenance, iteration, long-term product development.

Risk: Hours may not align with actual needs month-to-month. Unused hours are typically lost.

What to Watch For in Pricing

• Ask what is included. Design? QA? Project management? DevOps? Infrastructure setup?
• Ask about overtime and rush work pricing
• Ask what happens if the project takes longer than estimated
• Get it in writing before signing

Step 5: Spot the Red Flags

These patterns predict problems. If you see two or more, keep looking.

They say yes to everything. If every answer to "can you do this?" is enthusiastic agreement, nobody is thinking critically about your project. The agencies that deliver the best results are the ones that say "you could, but here is a simpler way" or "that feature is not worth building yet." Agreement is easy. Honest assessment is valuable.

No discovery phase. Any agency that jumps straight to coding without understanding the problem will build the wrong thing. Discovery takes 1-4 weeks and saves months of rework.

Unclear IP terms. If the contract does not clearly state that you own the code, do not sign. Some agencies retain IP rights and essentially hold your software hostage.

They cannot explain their process. If an agency cannot clearly describe how they run projects in concrete terms (sprint length, demo frequency, communication tools), they do not have a real process.

No post-launch plan. Software needs maintenance from day one. If they have no answer for "what happens after launch," expect to be on your own when bugs appear.

The sales team vanishes after signing. Ask: will the person I am talking to now be involved after the contract is signed? If the answer is no, ask to meet the project manager and technical lead before committing.

The Agency Evaluation Scorecard

Score each agency you are considering on a 1-5 scale. The weights reflect what we have seen matter most in practice — portfolio and team quality get 3x because they are the strongest predictors of project success. Pricing and post-launch support get 1x because they are easier to negotiate after you have found a technically strong team.

Criteria	Weight	Why This Weight	Agency A	Agency B	Agency C
Relevant portfolio (similar complexity/industry)	3x	Past work is the best predictor of future results	_	_	_
Team quality (experience of assigned team)	3x	You are hiring people, not a brand name	_	_	_
Communication (responsiveness, clarity)	2x	Poor communication causes more project failures than poor code	_	_	_
Process maturity (sprint cadence, demos, docs)	2x	Process keeps complex projects from drifting	_	_	_
Technical honesty (pushback, recommendations)	2x	Honest advice saves more money than a lower hourly rate	_	_	_
Pricing transparency	1x	Important but negotiable — and you can always get a second quote	_	_	_
Client references (satisfaction, repeat rate)	2x	Repeat clients mean the relationship worked beyond the first project	_	_	_
Post-launch support plan	1x	Necessary but can be defined in the contract regardless of initial score	_	_	_
Weighted Total	/80		_	_	_

65-80: Strong candidate. 50-64: Decent but has gaps — investigate the weakest scores before committing. Below 50: Keep looking.

Print this out or copy it into a spreadsheet. Fill it in after each agency conversation. It forces structured evaluation and makes it much easier to explain your choice to stakeholders.

The Small Things That Reveal a Lot

Response time during the sales process. If they take a week to reply now, expect two weeks after you sign. Sales-phase responsiveness is the best predictor of project-phase responsiveness.

How they handle "I don't know." Good agencies say "I don't know, but here is how I would find out." Bad agencies make something up or redirect.

Whether they ask you questions. The best agencies ask more questions than they answer in early conversations. They are trying to understand your problem. Agencies that jump straight to solutions are pitching, not listening.

Documentation quality. Ask to see a sample project document — a scope doc, a technical spec, anything. If it is clear and organized, their work probably is too.

Key Takeaways

1. Define your problem in one paragraph before talking to any agency
2. Ask for specific case studies with timelines, budgets, and measurable results
3. Use the 10-question framework during evaluation calls
4. Understand the pricing model and get inclusions/exclusions in writing
5. Watch for red flags: yes to everything, no discovery, unclear IP, no post-launch plan
6. Use the weighted scorecard to compare agencies objectively

If you want to talk about your next project, get in touch.

Related reading: Build vs buy: when custom makes sense | Web app development: timelines and costs