How do you design software for a new project?

What and why are the most important questions an engineer can ask. How often is the one an engineer starts with. Be sure to understand what you are building and why it is important before you start with how you will build it. Avoid the trap of building "a perfectly good implementation of the wrong solution" trap.

The first step in software design aligns closely with my role as a solution architect: understanding the core problem we're aiming to solve. This involves not just figuring out what the issue is, but also ensuring everyone on the team is focused on it. Part of my responsibility is identifying the key people who should be involved, such as important stakeholders, an experienced business analyst, and a project manager. By having these crucial team members together in one room, we can more effectively gather and analyse the project's requirements, goals, constraints, and user needs. This approach helps us define the software's scope, features, and success criteria without getting sidetracked or getting in to 'rabbit holes'.

Analyse & discuss with business, extract important use cases which will help build the core of the solution. Prototyping: At the same time start prototyping a brute fore version of the application using readily available tools and technologies. Anything that helps business understand our approach to the functional/non functional requirements. Design: Start with high level diagrams of various components/modules of the application using flow charts, swimlanes, network diagrams etc. that help us understand how the modules interact with each other. Followed by Low level detail design of the individual components and corresponding tech stack.

While trying to understand, define and document the problem statement, it's of utmost importance to keep the target user at the epicenter of all discussions. The solution must solve a user's basic need(s). It is equally important to organize and prioritize the different needs, wants, preferences, pain points, features, use cases etc. and address the ones that offer the highest rates of return on spending. The smallest such feature set that can be delivered to users is often referred to as Minimum Viable Product (MVP). In modern day software development, starting with MVP is very common. All other features are planned to be developed and delivered incrementally.

I am confident that I understand the problem when I can build an analysis model of the business objects in the problem domain, and especially the relationships between these objects. The process of building this model will often expose vague or ambiguous requirements.

It's a potential fallacy to choose "the architecture" of your software. The components, the relationships and the communication patterns collectively define the architecture of a system. While a single architectural style may be appropriate to start with, solution architects must consider the expandability and portability of the system along with the other attributes referenced. Rollout of systems to end users often unveil scenarios and use cases that were not considered at the onset. Keeping room to expand the software using a combination of architectural styles in the long run can turn into longer shelf life of any software.

Before choosing the architecture, a team should focus on defining and validating the solution to the suspected problem using a minimum viable prototype. The goal is to get a version of the solution in the hands of your potential users as quickly as possible. Even a mockup or interactive prototype can help validate the hypothesis and proposed solutions. Until then, any architectural decisions are moot and will only add unnecessary cost to the project. You can choose the greatest architecture in the world and spend years building it only to discover that your definition of the problem or solution were completely wrong.

In my experience, defining a software’s architecture is a continuous process. A high level architecture should suffice to put an idea to implementation quickly using the minimal viable prototype. While efficiency and cost should be the main drivers for any software architecture, security and maintainability should always be the pillars in the modern world. Kind of architecture would not matter much if we get the above 4 parameters covered.No one architectural pattern is best for all cases.

Which quality attributes to consider at the structural level will lay the foundation of the resulting system. For secondary attributes, the options will be limited as a consequence. All of these choices will be based on hypotheses about the future system. Those must be validated in PoCs, starting with the riskiest. Wargaming the architecture, maybe based on the "residuality theory" of Barry M O’Reilly or just with classical scenarios allows to check the architecture early on. Documenting it should exend the existing domain map with a context map, sketching the solution.

Beware the trap of "microservices for everything" Often times when I'm starting an architecture from scratch, I prefer a monolith since the data model is unknown. If you're working on an application that already has a well understood data model, then consider microservices but only if the bounded contexts are truly loosely coupled and highly cohesive.

There appears to be some confusion about what an API exposes here, sorry @zach but you are wrong. The API should mimic the domain model, not the data model. You should always start with the real world, the domain model and the conceptual data model then move to API to express the domain model and the database to solve the needs of the conceptual data model. They should meet in the middle. You should never expose your data model to the world, nor should you be limited by it, but instead express the real world domains in your API.

Do not, I repeat, DO NOT, start out with the API. It's like starting out with form rather than function. The API comes from the data model, not the other way around. Spend some time whiteboarding the entities in the data model. Use the UL (ubiquitous language) with the same terminology of the end users. Once you have it up on a whiteboard, iterated on it two or three times, then you have a somewhat workable model. Don't concern yourself with traditional design patterns before you have a solid understanding of how the data will flow. Then use GraphQL and thank me later :)

After settling on an architecture, the next step is designing the software components, which are the nuts and bolts executing specific functions. Upholding principles like modularity ensures they're distinct and reusable. Embracing industry standard design patterns such as MVC, Layered,Cloud, Microservice can resolve recurring issues and elevate code quality. Utilizing tools like IDEs streamlines design and development, making your code consistent and efficient. In essence, this phase breaks the architecture into manageable parts that function seamlessly together.

Having the right foundation at this stage could be key where you modularize your software and modularize it such it fits best to the overall value and scalability goal. Not getting this right early and having to change later after some adoption from customers could be costly in terms of incompatible change where then you are managing technical migrations rather than focusing on delivering customer value. Exhaustive validation by leading scenarios is key here. This is not to say that you are able to scale and change later but the foundation should be such that it is easily extendable. It is good to distribute responsibility in terms of function and independent use and scale of that responsibility.

Test the software as you build it. Capture the requirements as tests that can be automatically executed. Let the tests drive the design.

Testing is often one of the most problematic areas in companies that don't have a TDD culture built in. Ensuring that you're focusing on implementing all portions of the testing pyramid (Unit Tests as the foundation, Integration Tests as the center, E2E/UI testing as the top), and their corresponding weights will help you achieve optimal results. Over focusing on different sections of the pyramid can make your testing out of sync. You get better integration tests by having good unit tests. You don't need as many E2E tests, if you can assert various functionality from Integration tests. Each step up the pyramid has a higher cost, so invest in the foundations.

Tests should be the first things that are written, in the spirit of "TDD as if you meant it". This may be test-driven development but can also stand for type-driven development which lays a lot of the burden of executing the tests on the compilers shoulders. What to test should come out of the hypotheses and their associated risks. The goal must be a set of highly relevant tests rather than full coverage.

Having the right amount of automation here would define how quickly you can manage iterations of delivery and be really agile in delivering value. In a cloud world, you should have CI and CD such that you have exhaustive automated tests and you can deploy at the frequency of every day so that you can ship features as they are ready enabling quick feedback cycles as well.

One thing I have found useful, which is warranty of the software, there should be team on stand by for the software stability and addressing issues, which might arise after deployment.

Software is a battle of continuous learning and frustration and rewards. Design ceremonies into your SDLC that allow you to learn directly from end users, unfiltered through the biases, both positive and inimical, that come from the Rashomon Effect of VPs and managers telling you what end users do without every letting you actually see what they do. I prefer having a ride-along or shadowing session. We've saved companies tens of thousands of dollars with discoveries made in these sessions, that there is no way to reproduce or simulate otherwise.

Gather Requirements: I guess one of the important aspects of designing software is to gather requirements. Once I have a good understanding of the problem, I would gather requirements from the users and stakeholders. This would involve understanding their specific needs and expectations, as well as any non-functional requirements, such as performance, security, and scalability. It also helps to prioritize developing the most important feature first. By taking the time to gather and document requirements, you can ensure that the software will meet the needs of its users.

When I first read the title, I thought all about design. Not sure why to talk about software testing in the design phase. I understand if the section that is talking about software testing is to build the test plan document as part of the design phase. Same thing for software delivery. It’s good to change the article title to SDLC

Try to collect information about the real dev/devops who will implement the project. Usually the the latest part, but the most relevant for success. I mean, previous experience, old team or new team, possible learning. Taking into account and understand the human team available (not the one you would like to participate) will be key also when designing the architecture, and defining the technologies/languages to use. Selecting the correct team may be 60% of the success (may be the data is very opinionated), and I think the technical team skills (including soft) may have 80% of the weight in the performance result (Pareto) , because usually the lower technical details, technical debt, etc, in a project can delay dates and rise the costs...

Adapt to change. Change is the only constant as the saying goes. Making your solutions flexible allows to for adjusting to emerging requirements and contextual shifts.