You're a Systems Design professional. Which cloud-based tools should you be using?

When choosing tools for systems design, consider whether you need modeling or just diagramming. Do you require a comprehensive approach like C4 (Context, Containers, Components, Code)? You might also need a tool that supports modeling of architecture, system design, databases, and cloud infrastructure. In my experience, Lucidchart is a top choice, offering versatility for various modeling needs, from high-level architecture to detailed designs. Another notable option is Visual Paradigm, known for its extensive modeling capabilities.

Azure offers a comprehensive set of cloud services for building, deploying, and managing applications. It includes services like Azure Virtual Machines, Azure Blob Storage, Azure SQL Database, Azure Functions, Azure Cosmos DB, and Azure Kubernetes Service (AKS).

- Use tools like Lucidchart or Microsoft Visio for creating system architecture diagrams and flowcharts. - For database modeling, consider Amazon Web Services (AWS) Schema Conversion Tool or Oracle SQL Developer Data Modeler. - For UML and BPMN models, tools like Creately or Cacoo can be helpful.

For diagrams - Miroboard is a good option if you just need a whiteboard to draw your thoughts. It is user friendly and provides a good interface. It also helps you to collaborate with your teammate. If you want a more comprehensive solution, you can go for Lucidchart, GenMyModel.

Unified Modeling Language (UML): Widely used for specifying, visualizing, constructing, and documenting the artifacts of software systems. Entity-Relationship Diagrams (ERDs): Useful for modeling databases, showing the relationships between entities. Data Flow Diagrams (DFDs): Illustrate the flow of data through a system and are useful for understanding system processes. Structured Analysis and Design Technique (SADT): A diagramming method used to model the functions and data involved in a system. Business Process Model and Notation (BPMN): Used to visually represent business processes and workflows. ArchiMate: A modeling language for enterprise architecture that provides concepts to describe, analyze, and visualize architectures. ,SysML

1. AWS Device Farm or Azure DevTest Labs for simulating different environments and testing applications. 2. For network simulation, Cisco Modeling Labs (CML) or GNS3 offer cloud-based options. 3. MATLAB and Simulink in the cloud for complex engineering simulations.

CloudSim: Specifically designed for modeling and simulating cloud computing infrastructures and services. It enables the modeling of data centers, virtualized resources, and application workflows in cloud environments. DISCUS: A discrete event simulation tool used for modeling and analyzing complex systems, including cloud-based systems. It allows users to simulate various scenarios and evaluate system performance and scalability. SimGrid: A toolkit for the simulation of distributed applications and infrastructures, including cloud environments. NetSim: A network simulation tool that can be used to simulate cloud computing environments by modeling network infrastructure, communication protocols, and application behavior.

Most of the time the tool for this would be JMeter. It is quite simple to use and provides insightful data regarding your app performance, and reliability in different scenarios. Nowadays, we can also do similar things with just Postman, which is great because we are already familiar with this tool.

1. SonarQube or Coverity for code quality and security analysis. 2. BrowserStack or Sauce Labs for cross-browser and cross-platform testing. 3. Postman or SoapUI for API testing and verification.

Google integrated JUnit, Selenium, and SonarQube into its CI/CD pipelines for robust system design verification. JUnit facilitated unit testing, Selenium automated web testing across browsers, and SonarQube ensured continuous code quality checks. This combination, alongside cloud-based CI/CD tools like Google Cloud Build, automated the verification process, providing immediate feedback on code quality and functionality. The outcome was a significant reduction in manual testing time, enhanced code quality, and improved developer productivity. This strategy allowed Google to maintain high-quality standards across its services while accelerating development cycles.

Selenium is a time tested solution to automate your test cases. Postman is very useful if you have an API based service and it provides tons of features including monitoring and load testing. You can also integrate your code repo with a code scanner like Sonarqube to check code quality and coverage. Post that you can integrate your CI/CD pipeline with Security scans like SAST(Static Application Software Testing), DAST(Dynamic Application Software Testing) and SCA (Software Composition Analysis) to make sure you dont push any KEVs (Known Expoitable Vulnerabilities) to the next stage.

- Em nosso processo de desenvolvimento, adotamos o SonarQube. Ele nos ajuda a testar e resolver bugs, além de melhorar com boas práticas o código.

To begin with, every tech stack would probably have different tools for verification of the code quality. Regarding that point, we can list these categories: - Linters and formatters help you quickly validate your code with standards, typos, and best practices for your current tech stack. - static analysis tools like SonarQube, if we integrate it with our repository as a verification check step then we are most likely good to go. - test automation following the test pyramid. Regardless of the platform, and tool you use(CircleCI, GitHub Actions, Jenkins) test automation should be there, that way we can create artifacts ready to be used if passed all the levels with have on our project.

1. AWS CodeDeploy or Azure DevOps for automated deployment pipelines. 2. Jenkins or CircleCI for continuous integration and delivery (CI/CD) setups. 3. Terraform or CloudFormation for infrastructure as code (IaC) to manage cloud resources.

For Deployment you have plenty of options. If you are part of a big company, chances are you have an inhouse system to take care of this. Other options are Openshift, Bare Kubernetes if you are comfortable with that. For containerisation you can go for docker or podman. Ansible is a great solution if you have servers in the scale of 1000s or more(either ansible pull or push depending on your architecture). If your application is hosted in cloud, there are excellent inhouse options in AWS, GCP, Azure etc.

- Usamos o Docker para lidar com o gerenciamento dos nossos ambientes de desenvolvimento. Tem nos atendido bem, principalmente porque temos muitos ambientes e diferentes tecnologias para desenvolvimento.

Most of the PaaS available would take care of the deployment process, which makes our life easier. Now if we have an IaaS or On-premise infrastructure we handle ourselves, this probably has two scenarios. - Multi-cloud providers: probably the best match is Terraform. - single cloud: we still could use Terraform here for sure, but in my experience, it becomes easier to use the cloud platform IaC tool. Aws => CloudFormation, Azure => ARM Templates. It is by far easier to work with Kubernetes for deployment with the help of tools like helm charts, argocd, flux, etc. That way we solve scaling, fault tolerance, and many other issues.

1. Datadog or New Relic for performance monitoring and alerting. 2. Splunk or Loggly for log management and analysis. 3. Grafana or AWS CloudWatch for data visualization and dashboarding.

As a systems design professional, I wouldn't rely solely on cloud-based monitoring systems. AWS CloudWatch is good, but problems begin when customization is needed. I would combine cloud solutions with custom monitoring tools like Prometheus, Grafana, Victoria Metrics, etc. If there's a need to collect specific metrics, build complex dashboards, and attach an alerting system, that's one task; basic monitoring of cloud status is another.

1. Prometheus Pros: Open-source and highly customizable. Strong support for time series data. Efficient storage and powerful query language (PromQL). Cons: Visualization capabilities are limited, often requiring integration with Grafana. Managing high-availability setups can be complex. 2. Grafana Pros: Open-source with rich visualization options. Supports a wide range of data sources including Prometheus, CloudWatch, and more. Offers alerting and notifications. Cons: Primarily a visualization tool; relies on other tools for data collection. Can become complex as dashboards and configurations grow. Grafana’s adaptability and integration capabilities make it an invaluable tool for system monitoring across various scenarios.

1. Appdynamics provides good visibility into your application. It is infact very powerful and has very little overhead. Although you need to install the Appd agent in your system and that can use a bit of CPU. 2. Splunk for log handling and analytics. It has good dashboarding feature. 3. Thousand eyes gives good visibility into your network if you are in that domain. It can also perform synthetic monitoring of your APIs with agents in geographically dispersed locations if that helps. 4. Grafana and Prometheus if you want an opensource solution

1. Integration: Ensure the tools you choose integrate well with each other and with your existing workflows. 2. Cost and Scalability: Consider the cost of these tools and their scalability as your system grows. 3. Security and Compliance: Verify that the tools meet your security requirements and comply with relevant regulations. 4. User Experience: Choose tools that are user-friendly and have good support communities or documentation. 5. Vendor Lock-in: Be aware of potential vendor lock-in and evaluate the portability of your systems with these tools.

For system design in big tech companies, use: Microsoft Azure: For comprehensive cloud services and DevOps. AWS: For scalable computing and storage solutions. Google Cloud Platform (GCP): For data analytics and machine learning. GitHub: For version control and collaboration. Lucidchart/Draw.io: For cloud-based diagramming. Terraform: For infrastructure as code across clouds. Best Practices: Automate with Infrastructure as Code (IaC). Implement CI/CD for efficiency. Use Docker and Kubernetes for container management. Prioritize security with cloud-native tools. Optimize costs and review performance regularly. Collaborate using cloud-based tools for documentation and design.

Never rely on cloud services only for your critical subsystems events logging. Even if your hosting provider guarantees 99.99% monthly uptime, this means more than 4 minutes of continuous acceptable downtime (in reality it can be much more without any consequences for ISPs, see SLA) when the cloud will be inaccessible (while cloud system outages can also happen). Having properly rotated local logs will help you investigate the problems, and using the right tools such as fluentbit will make it easy to switch between logging clouds.

There is plethora of tools as others called out. However, everything may not be available/applicable for you. You need to consider your organization's policies, tool's features, your comfort level, your experience with tools, outcome from tool to choose right one for your situation and use case.