What numerical issues are common in simulation algorithms?

Random numbers are not truly random, and sometimes can generate spurious results. PRN Generators have periodic components and correlation which can show up in scatter plots.

Unfortunately it's impossible to represent any irrational or transcendental number using *any* number system, let alone binary. Any calculations that recursively use such approximations are going to add some numerical noise. In other numerical feedback systems such as IIR filters there's also a risk that numerical noise can trigger instability.

I would argue that with today's 64-bit CPUs, the inherent challenge of floating point errors are less of an issue. But, depending on your domain, you might still care about it. If you are simulating a physical process and you want to compare the results with a physical experiment, then you might care. Even then, you are likely looking for some agreement, trend, or equivalency, not an exact match. If you are working with very small or very large numbers, then apply a scaling factor. For example, use 1 AU to scale distance if you are simulating planetary orbits. If it still really matters to you, use a native language like C++ or Fortran and set your FP rounding heuristic at compile time. Or choose a different number system, like log scale.

I also feel this isn't in general as big a problem it was years ago. I often work with probabilities of extreme values where, rather than multiplying a lot of numbers very close to unity, it's much safer to add their logarithms.

Regarding the comment below on irrational or transcendental numbers - yes you can. For example, suppose you want to represent numbers that include sqrt(2), then what you do is use a pair (a,b) that is supposed to represent a+b*sqrt(2). Addition is componentwise, while multiplication is (a,b)*(c,d)=(ac+2bd, ad+bc). This is easiest to do with roots of polynomial equation (P(z)), as then you take whatever field F you can either represent and use F[z]/P(z). This is essentially the path to computer algebra systems, but you do not have to reimplement the whole Mathematica/Maple - often just a small piece will do the job and run much faster.

This is an area I obsess in. Your method may be theoretically very accurate, but if the algorithm is unstable that theoretical accuracy means nothing. My advice is to learn some of the classical methods for studying stability (like the difference between L-2 and L-infinity stability for ODEs, or learning how to numerically stabilize orthogonalization methods). Studying classical problems is a good way to understand how algorithms can go unstable and how you can avoid the pitfalls of just blindly implementing a method that looks good on paper.

The usual workflow is you got your magic algorithm to actually work but it takes too long to give you a result. Regardless of your choice of language or target CPU, the first step in code optimization is to profile your code to understand how the memory is growing and which functions are consuming the most time on the CPU. Choice of data structures can have an impact on performance, but so do things like pass-by-value, division, and transcendental functions. After some tuning, you have to decide how much further down the rabbit hole you want to go. Look for opportunities to unroll loops or embarrassingly parallel sections that can exploit multiple cores. And if you can't avoid square roots, use the magic of Quake! Google it.

There multiple factors to consider for an algorithm efficiency. Some efficiencies are rooted in how you code an algorithm. Are you using proper data structure and methods? For example, Are you using an implementation that requires O(nlogn) or O(n^2). Same numerical algorithm implemented wrongly can reduce your efficiency. The other is the numerical algorithm it self. Some iterative algorithms converge faster than the others. For example, when solving a linear system of equations, jacobi iteration takes very long to converge for most practical and real world cases. But conjugate gradient method is by far requires much less iterations. Even better, algebraic multigrid methods converge in less iteration

Input error is also an issue in numerical simulation. It maybe the errors of material properties, forces, displacements, and so on to input into the numerical models to calculate the output with not negligible error. It may be a real world data coming into the model with a time delay which can accumulate after a certain amount of time to affect the output.