Linux physical memory concept:NODE

Nodes

As I have mentioned, each node in memory is described by a pg_data_t, which is a typedef for a struct pglist_data. When allocating a page, Linux uses a node-local allocation policy to allocate memory from the node closest to the running CPU. Because processes tend to run on the same CPU, it is likely the memory from the current node will be used. The struct is declared as follows in <linux/mmzone.h>:

 typedef struct pglist_data {
   zone_t node_zones[MAX_NR_ZONES];
     zonelist_t node_zonelists[GFP_ZONEMASK+1];
     int nr_zones;
     struct page *node_mem_map;
     unsigned long *valid_addr_bitmap;
     struct bootmem_data *bdata;
     unsigned long node_start_paddr;
     unsigned long node_start_mapnr;
     unsigned long node_size;
     int node_id;
     struct pglist_data *node_next;
 } pg_data_t;

I now briefly describe each of these fields:

node_zones The zones for this node are ZONE_HIGHMEM, ZONE_NORMAL, ZONE_DMA.

node_zonelists This is the order of zones that allocations are preferred from. build_zonelists() in mm/page_alloc.c sets up the order when called by free_area_init_core(). A failed allocation in ZONE_HIGHMEM may fall back to ZONE_NORMAL or back to ZONE_DMA.

nr_zones This is the number of zones in this node between one and three. Not all nodes will have three. A CPU bank may not have ZONE_DMA, for example.

node_mem_map This is the first page of the struct page array that represents each physical frame in the node. It will be placed somewhere within the global mem_map array.

valid_addr_bitmap This is a bitmap that describes "holes" in the memory node that no memory exists for. In reality, this is only used by the Sparc and Sparc64 architectures and is ignored by all others.

bdata This is only of interest to the boot memory allocator .

node_start_paddr This is the starting physical address of the node. An unsigned long does not work optimally because it breaks for ia32 with Physical Address Extension (PAE) and for some PowerPC variants such as the PPC440GP. PAE is discussed further. A more suitable solution would be to record this as a Page Frame Number (PFN). A PFN is simply an index within physical memory that is counted in page-sized units. PFN for a physical address could be trivially defined as (page_phys_addr >> PAGE_SHIFT).

node_start_mapnr This gives the page offset within the global mem_map. It is calculated in free_area_init_core() by calculating the number of pages between mem_map and the local mem_map for this node called lmem_map.

node_size This is the total number of pages in this zone.

node_id This is the Node ID (NID) of the node and starts at 0.

node_next Pointer to next node in a NULL terminated list.

All nodes in the system are maintained on a list called pgdat_list. The nodes are placed on this list as they are initialized by the init_bootmem_core() function, which is described later . Up until late 2.4 kernels (> 2.4.18), blocks of code that traversed the list looked something like the following:

pg_data_t * pgdat;
pgdat = pgdat_list;
do {
      /* do something with pgdata_t */
      ...
} while ((pgdat = pgdat->node_next));

In more recent kernels, a macro for_each_pgdat(), which is trivially defined as a for loop, is provided to improve code readability.