Python: Key things to prevent application crashes

Poor memory management is the main reason that caused application crashes. Here are the important techniques to optimize memory usage.

1. Lazy evaluation

Lazy evaluation techniques to process data on demand instead of loading it all into memory at one.

a. Generators with the yield keyword

def generate_squares(limit):
    for x in range(limit):
        yield x ** 2
squares = generate_squares(1_000_000)

# Inefficient: Consumes memory for the entire list
squares = [x ** 2 for x in range(1_000_000)]        

b. Iterators with itertools.islice

from itertools import islice
subset = islice(large_list, 1000)
for item in subset:
    print(item)  # Process items one at a time        

2. Batch Processing

Process data in smaller chunks instead of loading everything into memory at once.

with open("large_file.txt", "r") as file:
    for line in file:  # Processes one line at a time
        process(line)

# Inefficient: Loads the entire file into memory
with open("large_file.txt", "r") as file:
    data = file.readlines()        

It is important to use "with" statements to ensure files and connections are properly closed.

3. Use NumPy for numerical computations

NumPy arrays are more memory- and time-efficient than Python lists for numerical operations.

import numpy as np
data = np.arange(1000000) ** 2

# Inefficient: Using Python lists
data = [x ** 2 for x in range(1000000)]        

4. Work with Views Instead of Copies

Considering "views" of data instead of copies when possible to reduce memory usage

import pandas as pd
df = pd.DataFrame({"A": range(10)})
subset = df[df["A"] > 5]

# Inefficient: Creating a new DataFrame copy
subset = df[df["A"] > 5].copy()        

5. Avoid Duplicating Data

If you need to access the same data in multiple places, use references instead of copying it.

def process_reference(data):
    # Perform operations on the original data
    
# Inefficient: Creates a duplicate list
def process_copy(data):
    data_copy = data[:]
    # Perform operations on the copy