How do you handle large or complex data sets in GIS?

I always recommend to go for a geodatabase or geopackage to store the data for a GIS project, as it reduces the amount of files/folders needed when shapefiles are used instead. It is also good idea to connect a GIS database such as PostGIS, and retrieve only the data needed to perform the analysis.

I think the choices should be based on the needs of the situation and how the data is going to be accessed. Personally I find geojson easier to handle than kml/gml and it is relatively independent of traditional gis applications so offers more flexibility.

Regarding data compression, it is always good idea to backup in zip format, in an external hard-drive or the cloud if possible. This is relevant when you need to revisit the project or update it in a regular basis (yearly or so). If you are more interested in keeping the maps or other figures as reference, JPEG or PDF will do great.

Spatial indexing is essential to improve GIS efficiency. Software like ArcGIS Pro, identify datasets that are not indexed and offers to do it as part of the loading process.

Data aggregation is a powerful way to reduce the size of your GIS projects. This will require planning, you would need to have an objective, identify data sources, variables of interest. Once you analyse the data you have and your objectives you could aggregate and also discard the data you do not need. Recently, I did a project where I aggregated six variables into 1, they were % Electricity, %Water, %Sewage, %Gas, %Garbage_recolection and %Internet in a group of Colombian towns. I made sense to aggregate them to understand their impact on the community, as they can be complementary and work as a whole.

The best way to visualize the big or complex data is via a server. One of the popular map servers is Geoserver. I have used the Geoserver to host the 10GB of single raster and get the average performance on visualization. I have tested it with a 7K dataset, the average size of each dataset was ~100Mb and it easily handled visualization work. On the visualization side, I always prefer Web Map Service (WMS) or the Web Map Tile Service (WMTS). You should not use the Web Feature Service (WFS) or Web Coverage Service (WCS) in case of a large dataset for visualization, otherwise, you get poor performance on the client side.

GIS are know for they capability to communicate visual information using maps, however it is important to know the scale and the symbols you would like to use. There is a technique called generalization that helps to summarize data in one symbol, depending on the scale of a map. GIS software also comes with graphs to show the data as a histogram, or point clouds, or pie charts, which are also very helpful to summarize data and convey information. On the other hand, software such as Power BI or Tableau allow you to connect to Esri maps or geodatabases.

Got a feeling this question has already been asked. Get the impression the AI training needs to add some subtle different phrase in order to relate similar responses.