内存管理器（十）kernel内存管理----数据结构

布满荆棘的人生 2022-08-10 00:36 304阅读 0赞

# 内存管理器（十） kernel内存管理----概况与数据结构 #

## 前言 ##

正式开始学习内核的内存管理了，先学习下接口函数，每一个例字都必须写内核模块实验，然后深入到函数的内部研究源码，最后写写练习的小程序。我发现最近我的废话越来越多了。

## 页 ##

说起内存管理，自然就是两个方面，物理内存管理，虚拟内存管理，我们先看看物理内存管理。  
内核将物理页作为内存管理的基本单位。从虚拟内存的角度来看，页就是最小单位。32位机一般支持的是4KB的页，64位的机器一般就支持8KB的页，内核使用struct page 结构标示系统中的每一个物理页：  
<linux/mm\_types.h>

\[c\]  
struct page \{  
//First double word block  
unsigned long flags; //Atomic flags, some possibly  
//页的状态 updated asynchronously 这里至少有32种状态 <linux/page-flags.h>  
union \{  
struct address\_space *mapping; /* If low bit clear, points to  
\* inode address\_space, or NULL.  
\* If page mapped as anonymous  
\* memory, low bit is set, and  
\* it points to anon\_vma object:  
\* see PAGE\_MAPPING\_ANON below.  
*/  
void \*s\_mem; /* slab first object */  
\};  
/* Second double word */  
struct \{  
union \{  
pgoff\_t index; /* Our offset within mapping. */  
void \*freelist; /* sl\[aou\]b first free object */  
bool pfmemalloc; /* If set by the page allocator,  
\* ALLOC\_NO\_WATERMARKS was set  
\* and the low watermark was not  
\* met implying that the system  
\* is under some pressure. The  
\* caller should try ensure  
\* this page is only used to  
\* free other pages.  
\*/  
\};

union \{  
\#if defined(CONFIG\_HAVE\_CMPXCHG\_DOUBLE) && \\  
defined(CONFIG\_HAVE\_ALIGNED\_STRUCT\_PAGE)  
/\* Used for cmpxchg\_double in slub */  
unsigned long counters;  
\#else  
/*  
\* Keep \_count separate from slub cmpxchg\_double data.  
\* As the rest of the double word is protected by  
\* slab\_lock but \_count is not.  
\*/  
unsigned counters;  
\#endif  
struct \{  
union \{  
/\*  
\* Count of ptes mapped in  
\* mms, to show when page is  
\* mapped & limit reverse map  
\* searches.  
\*  
\* Used also for tail pages  
\* refcounting instead of  
\* \_count. Tail pages cannot  
\* be mapped and keeping the  
\* tail page \_count zero at  
\* all times guarantees  
\* get\_page\_unless\_zero() will  
\* never succeed on tail  
\* pages.  
\*/  
atomic\_t \_mapcount;  
struct \{ /\* SLUB \*/  
unsigned inuse:16;  
unsigned objects:15;  
unsigned frozen:1;  
\};  
int units; /\* SLOB \*/  
\};  
atomic\_t \_count; /\* Usage count, see below. \*/  
\};  
unsigned int active; /\* SLAB \*/  
\};  
\};  
atomic\_t \_count; //页的引用计数，也就是这个页被引用了多少次  
//当计数值标变成-1 的时候就表示内核没有引用这一页，page\_count()函数可以得到这个页的引用情况。  
/\* Third double word block \*/  
union \{  
struct list\_head lru; /\* Pageout list, eg. active\_list  
\* protected by zone->lru\_lock !  
\* Can be used as a generic list  
\* by the page owner.  
\*/  
//包含页的最近最少使用双向链表的指针  
struct \{ /\* slub per cpu partial pages \*/  
struct page \*next; /\* Next partial slab \*/  
\#ifdef CONFIG\_64BIT  
int pages; /\* Nr of partial slabs left \*/  
int pobjects; /\* Approximate \# of objects \*/  
\#else  
short int pages;  
short int pobjects;  
\#endif  
\};  
struct slab \*slab\_page; /\* slab fields \*/  
struct rcu\_head rcu\_head; /\* Used by SLAB  
\* when destroying via RCU  
\*/  
/\* First tail page of compound page \*/  
struct \{  
compound\_page\_dtor \*compound\_dtor;  
unsigned long compound\_order;  
\};  
\#if defined(CONFIG\_TRANSPARENT\_HUGEPAGE) && USE\_SPLIT\_PMD\_PTLOCKS  
pgtable\_t pmd\_huge\_pte; /\* protected by page->ptl \*/  
\#endif  
\};  
void \*virtual; //页的虚拟地址  
\[/c\]

这里只节选了这个结构体的一部分代码,不得不说换了博客以后真的有点不适应，这篇排版有点难看...

## 区 ##

先说说几种体系架构吧（以下架构说明来自互联网）  
SMP：  
SMP(Symmetric Multi-Processor)

所谓对称多处理器结构，是指服务器中多个CPU对称工作，无主次或从属关系。各CPU共享相同的物理内存，每个 CPU访问内存中的任何地址所需时间是相同的，因此SMP也被称为一致存储器访问结构(UMA：Uniform Memory Access)。对SMP服务器进行扩展的方式包括增加内存、使用更快的CPU、增加CPU、扩充I/O(槽口数与总线数)以及添加更多的外部设备(通常是磁盘存储)。

SMP服务器的主要特征是共享，系统中所有资源(CPU、内存、I/O等)都是共享的。也正是由于这种特征，导致了SMP服务器的主要问题，那就是它的扩展能力非常有限。对于SMP服务器而言，每一个共享的环节都可能造成SMP服务器扩展时的瓶颈，而最受限制的则是内存。由于每个CPU必须通过相同的内存总线访问相同的内存资源，因此随着CPU数量的增加，内存访问冲突将迅速增加，最终会造成CPU资源的浪费，使 CPU性能的有效性大大降低。实验证明，SMP服务器CPU利用率最好的情况是2至4个CPU

内存毕竟是线性的模型，个人觉得寻址页是线性，确实是线性的，不过内核很聪明，它把内存根据地址情况画分成了不同的区。这里更多的是因为硬件的关系，导致内核并不能对所有的页一视同仁。每个内存节点的物理内存被划分为了不同的区。

NUMA：  
NUMA服务器的基本特征是具有多个CPU模块，每个CPU模块由多个CPU(如4个)组成，并且具有独立的本地内存、I/O槽口等。由于其节点之间可以通过互联模块(如称为Crossbar Switch)进行连接和信息交互，因此每个CPU可以访问整个系统的内存(这是NUMA系统与MPP系统的重要差别)。显然，访问本地内存的速度将远远高于访问远地内存(系统内其它节点的内存)的速度，这也是非一致存储访问NUMA的由来。由于这个特点，为了更好地发挥系统性能，开发应用程序时需要尽量减少不同CPU模块之间的信息交互。利用NUMA技术，可以较好地解决原来SMP系统的扩展问题，在一个物理服务器内可以支持上百个CPU。比较典型的NUMA服务器的例子包括HP的Superdome、SUN15K、IBMp690等。

但NUMA技术同样有一定缺陷，由于访问远地内存的延时远远超过本地内存，因此当CPU数量增加时，系统性能无法线性增加。如HP公司发布Superdome服务器时，曾公布了它与HP其它UNIX服务器的相对性能值，结果发现，64路CPU的Superdome (NUMA结构)的相对性能值是20，而8路N4000(共享的SMP结构)的相对性能值是6.3。从这个结果可以看到，8倍数量的CPU换来的只是3倍性能的提升。

MPP：  
和NUMA不同，MPP提供了另外一种进行系统扩展的方式，它由多个SMP服务器通过一定的节点互联网络进行连接，协同工作，完成相同的任务，从用户的角度来看是一个服务器系统。其基本特征是由多个SMP服务器(每个SMP服务器称节点)通过节点互联网络连接而成，每个节点只访问自己的本地资源(内存、存储等)，是一种完全无共享(Share Nothing)结构，因而扩展能力最好，理论上其扩展无限制，目前的技术可实现512个节点互联，数千个CPU。目前业界对节点互联网络暂无标准，如 NCR的Bynet，IBM的SPSwitch，它们都采用了不同的内部实现机制。但节点互联网仅供MPP服务器内部使用，对用户而言是透明的。

在MPP系统中，每个SMP节点也可以运行自己的操作系统、数据库等。但和NUMA不同的是，它不存在异地内存访问的问题。换言之，每个节点内的CPU不能访问另一个节点的内存。节点之间的信息交互是通过节点互联网络实现的，这个过程一般称为数据重分配(Data Redistribution)。

但是MPP服务器需要一种复杂的机制来调度和平衡各个节点的负载和并行处理过程。目前一些基于MPP技术的服务器往往通过系统级软件(如数据库)来屏蔽这种复杂性。举例来说，NCR的Teradata就是基于MPP技术的一个关系数据库软件，基于此数据库来开发应用时，不管后台服务器由多少个节点组成，开发人员所面对的都是同一个数据库系统，而不需要考虑如何调度其中某几个节点的负载。

主要原因有下边的几个：  
1.一些硬件只能用某些特定的内存地址来执行DMA（直接内存访问），所以虽然现在CPU有46位寻址，但并不代表计算机上所有的硬件都能寻址46位。  
2.一些体系结构的内存寻址能力比虚拟地址范围大的多。  
因为存在这些，Linux主要采用了几种分区：  
\[c\]  
enum zone\_type \{

# ifdef CONFIG\_ZONE\_DMA #

/*
     * ZONE_DMA is used when there are devices that are not able
     * to do DMA to all of addressable memory (ZONE_NORMAL). Then we
     * carve out the portion of memory that is needed for these devices.
     * The range is arch specific.
     *
     * Some examples
     *
     * Architecture     Limit
     * ---------------------------
     * parisc, ia64, sparc  <4G
     * s390         <2G
     * arm          Various
     * alpha        Unlimited or 0-16MB.
     *
     * i386, x86_64 and multiple other arches
     *          <16M.
     */
    ZONE_DMA,    //这个区包含的页只能执行DMA操作，注释中已经举出了不同体系结构下的限制

# endif #

# ifdef CONFIG\_ZONE\_DMA32 #

/*
     * x86_64 needs two ZONE_DMAs because it supports devices that are
     * only able to do DMA to the lower 16M but also 32 bit devices that
     * can only do DMA areas below 4G.
     */
    ZONE_DMA32,//32位机上是0，64位机上是4GB

# endif #

/*
     * Normal addressable memory is in ZONE_NORMAL. DMA operations can be
     * performed on pages in ZONE_NORMAL if the DMA devices support
     * transfers to all addressable memory.
     */
    ZONE_NORMAL, //可以正常寻址的区

# ifdef CONFIG\_HIGHMEM #

/*
     * A memory area that is only addressable by the kernel through
     * mapping portions into its own address space. This is for example
     * used by i386 to allow the kernel to address the memory beyond
     * 900MB. The kernel will set up special mappings (page
     * table entries on i386) for each page that the kernel needs to
     * access.
     */
    ZONE_HIGHMEM, //标记了超出内核段的物理内存，动态映射的页

# endif #

ZONE_MOVABLE,//伪内存域，防止产生内存碎片的时候需要使用
    __MAX_NR_ZONES //充当结束标记

\};

\[/c\]

机智的linux 内核把系统的页划分为区，形成不同的内存池，这样就可以根据用途或者阶段来分配了，不过DMA区页时可以用来进行正常分配的，不过一般情况下会首先保证DMA够用。

现在我们再来看看 区的数据结构

\[c\]  
struct zone \{  
/\* Read-mostly fields \*/

/* zone watermarks, access with *_wmark_pages(zone) macros */
    unsigned long watermark[NR_WMARK]; //界线标识
    
    /*
     * We don't know if the memory that we're going to allocate will be freeable
     * or/and it will be released eventually, so to avoid totally wasting several
     * GB of ram we must reserve some of the lower zone memory (otherwise we risk
     * to run OOM on the lower zones despite there's tons of freeable ram
     * on the higher zones). This array is recalculated at runtime if the
     * sysctl_lowmem_reserve_ratio sysctl changes.
     */
    long lowmem_reserve[MAX_NR_ZONES]; //分配不可失败的页

# ifdef CONFIG\_NUMA //支持非一致性访问 #

int node;

# endif #

/*
     * The target ratio of ACTIVE_ANON to INACTIVE_ANON pages on
     * this zone's LRU.  Maintained by the pageout code.
     */
    unsigned int inactive_ratio;
    
    struct pglist_data  *zone_pgdat;
    struct per_cpu_pageset __percpu *pageset;
    
    /*
     * This is a per-zone reserve of pages that should not be
     * considered dirtyable memory.
     */
    unsigned long       dirty_balance_reserve;/

# ifndef CONFIG\_SPARSEMEM #

/*
     * Flags for a pageblock_nr_pages block. See pageblock-flags.h.
     * In SPARSEMEM, this map is stored in struct mem_section
     */
    unsigned long       *pageblock_flags;

# endif /\* CONFIG\_SPARSEMEM \*/ #

# ifdef CONFIG\_NUMA #

/*
     * zone reclaim becomes active if more unmapped pages exist.
     */
    unsigned long       min_unmapped_pages;
    unsigned long       min_slab_pages;

# endif /\* CONFIG\_NUMA \*/ #

/* zone_start_pfn == zone_start_paddr >> PAGE_SHIFT */
    unsigned long       zone_start_pfn;
    
    /*
     * spanned_pages is the total pages spanned by the zone, including
     * holes, which is calculated as:
     *  spanned_pages = zone_end_pfn - zone_start_pfn;
     *
     * present_pages is physical pages existing within the zone, which
     * is calculated as:
     *  present_pages = spanned_pages - absent_pages(pages in holes);
     *
     * managed_pages is present pages managed by the buddy system, which
     * is calculated as (reserved_pages includes pages allocated by the
     * bootmem allocator):
     *  managed_pages = present_pages - reserved_pages;
     *
     * So present_pages may be used by memory hotplug or memory power
     * management logic to figure out unmanaged pages by checking
     * (present_pages - managed_pages). And managed_pages should be used
     * by page allocator and vm scanner to calculate all kinds of watermarks
     * and thresholds.
     *
     * Locking rules:
     *
     * zone_start_pfn and spanned_pages are protected by span_seqlock.
     * It is a seqlock because it has to be read outside of zone->lock,
     * and it is done in the main allocator path.  But, it is written
     * quite infrequently.
     *
     * The span_seq lock is declared along with zone->lock because it is
     * frequently read in proximity to zone->lock.  It's good to
     * give them a chance of being in the same cacheline.
     *
     * Write access to present_pages at runtime should be protected by
     * mem_hotplug_begin/end(). Any reader who can't tolerant drift of
     * present_pages should get_online_mems() to get a stable value.
     *
     * Read access to managed_pages should be safe because it's unsigned
     * long. Write access to zone->managed_pages and totalram_pages are
     * protected by managed_page_count_lock at runtime. Idealy only
     * adjust_managed_page_count() should be used instead of directly
     * touching zone->managed_pages and totalram_pages.
     */
    unsigned long       managed_pages;
    unsigned long       spanned_pages;
    unsigned long       present_pages;
    
    const char      *name;
    
    /*
     * Number of MIGRATE_RESERVE page block. To maintain for just
     * optimization. Protected by zone->lock.
     */
    int         nr_migrate_reserve_block;

# ifdef CONFIG\_MEMORY\_ISOLATION #

/*
     * Number of isolated pageblock. It is used to solve incorrect
     * freepage counting problem due to racy retrieving migratetype
     * of pageblock. Protected by zone->lock.
     */
    unsigned long       nr_isolate_pageblock;

# endif #

# ifdef CONFIG\_MEMORY\_HOTPLUG #

/* see spanned/present_pages for more description */
    seqlock_t       span_seqlock;

# endif #

/*
     * wait_table       -- the array holding the hash table
     * wait_table_hash_nr_entries   -- the size of the hash table array
     * wait_table_bits  -- wait_table_size == (1 << wait_table_bits)
     *
     * The purpose of all these is to keep track of the people
     * waiting for a page to become available and make them
     * runnable again when possible. The trouble is that this
     * consumes a lot of space, especially when so few things
     * wait on pages at a given time. So instead of using
     * per-page waitqueues, we use a waitqueue hash table.
     *
     * The bucket discipline is to sleep on the same queue when
     * colliding and wake all in that wait queue when removing.
     * When something wakes, it must check to be sure its page is
     * truly available, a la thundering herd. The cost of a
     * collision is great, but given the expected load of the
     * table, they should be so rare as to be outweighed by the
     * benefits from the saved space.
     *
     * __wait_on_page_locked() and unlock_page() in mm/filemap.c, are the
     * primary users of these fields, and in mm/page_alloc.c
     * free_area_init_core() performs the initialization of them.
     */
    wait_queue_head_t   *wait_table;
    unsigned long       wait_table_hash_nr_entries;
    unsigned long       wait_table_bits;
    
    ZONE_PADDING(_pad1_)
    /* free areas of different sizes */
        //通常由页面回收扫描程序访问的段
    struct free_area    free_area[MAX_ORDER];
        //伙伴系统，内存空间实例，每一数据元素都表示一些连续的某些固定长度的内存区。
    /* zone flags, see below */
    unsigned long       flags;
    
    /* Write-intensive fields used from the page allocator */
    spinlock_t      lock;
    
    ZONE_PADDING(_pad2_)
    
    /* Write-intensive fields used by page reclaim */
    
    /* Fields commonly accessed by the page reclaim scanner */
    spinlock_t      lru_lock;
    struct lruvec       lruvec;
    
    /* Evictions & activations on the inactive file list */
    atomic_long_t       inactive_age;
    
    /*
     * When free pages are below this point, additional steps are taken
     * when reading the number of free pages to avoid per-cpu counter
     * drift allowing watermarks to be breached
     */
    unsigned long percpu_drift_mark;

# if defined CONFIG\_COMPACTION || defined CONFIG\_CMA #

/* pfn where compaction free scanner should start */
    unsigned long       compact_cached_free_pfn;
    /* pfn where async and sync compaction migration scanner should start */
    unsigned long       compact_cached_migrate_pfn[2];

# endif #

# ifdef CONFIG\_COMPACTION #

/*
     * On compaction failure, 1<<compact_defer_shift compactions
     * are skipped before trying again. The number attempted since
     * last failure is tracked with compact_considered.
     */
    unsigned int        compact_considered;
    unsigned int        compact_defer_shift;
    int         compact_order_failed;

# endif #

# if defined CONFIG\_COMPACTION || defined CONFIG\_CMA #

/* Set to true when the PG_migrate_skip bits should be cleared */
    bool            compact_blockskip_flush;

# endif #

ZONE_PADDING(_pad3_)
    /* Zone statistics */
    atomic_long_t       vm_stat[NR_VM_ZONE_STAT_ITEMS];

\} \_\_\_\_cacheline\_internodealigned\_in\_smp;

\[/c\]

以上就是内核内存分配区的全部结构体，可以使用uname -a 来查看自己的电脑是什么体系结构的，我的电脑是SMP,根据上面所说的SMP 对称结构应当就是UMA 体系结构，所以我的系统一共就这样的三个区。关于NUMA的数据结构暂时不关心。

### mlock ###

这里再说这样一个系统调用吧，mlock() 这个系统调用可以将，指定的内存页锁定在物理内存上，不让它进入交换区。对于对时间有严格要求的程序，这个操作很用。

查看原文：[http://zmr.lezifang.cn/2015/11/05/%e5%86%85%e5%ad%98%e7%ae%a1%e7%90%86%e5%99%a8%ef%bc%88%e5%8d%81%ef%bc%89kernel%e5%86%85%e5%ad%98%e7%ae%a1%e7%90%86-%e6%95%b0%e6%8d%ae%e7%bb%93%e6%9e%84/][http_zmr.lezifang.cn_2015_11_05_e5_86_85_e5_ad_98_e7_ae_a1_e7_90_86_e5_99_a8_ef_bc_88_e5_8d_81_ef_bc_89kernel_e5_86_85_e5_ad_98_e7_ae_a1_e7_90_86-_e6_95_b0_e6_8d_ae_e7_bb_93_e6_9e_84]

[http_zmr.lezifang.cn_2015_11_05_e5_86_85_e5_ad_98_e7_ae_a1_e7_90_86_e5_99_a8_ef_bc_88_e5_8d_81_ef_bc_89kernel_e5_86_85_e5_ad_98_e7_ae_a1_e7_90_86-_e6_95_b0_e6_8d_ae_e7_bb_93_e6_9e_84]: http://zmr.lezifang.cn/2015/11/05/%e5%86%85%e5%ad%98%e7%ae%a1%e7%90%86%e5%99%a8%ef%bc%88%e5%8d%81%ef%bc%89kernel%e5%86%85%e5%ad%98%e7%ae%a1%e7%90%86-%e6%95%b0%e6%8d%ae%e7%bb%93%e6%9e%84/