numa 局部性调用API

拼搏现实的明天。 2022-01-13 23:35 245阅读 0赞

转载自 [http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-1/index.html][http_docs.oracle.com_cd_E19082-01_820-1691_lgroups-1_index.html]

Memory and Thread Placement Optimization Developer's Guide

*  [**Previous**: Chapter 2 MPO Observability Tools][Previous_ Chapter_2 MPO Observability Tools]

# Chapter 3 Locality Group APIs #

his chapter describes the APIs that applications use to interact with locality groups.

This chapter discusses the following topics:

[erifying the Interface Version][] describes the functions that give information about the interface.

*  [Initializing the Locality Group Interface][] describes function calls that initialize and shut down the portion of the interface that is used to traverse the locality group hierarchy and to discover the contents of a locality group.
 *  [Locality Group Hierarchy][] describes function calls that navigate the locality group hierarchy and functions that get characteristics of the locality group hierarchy.
 *  [Locality Group Contents][] describes function calls that retrieve information about a locality group's contents.
 *  [Locality Group Characteristics][] describes function calls that retrieve information about a locality group's characteristics.
 *  [Locality Groups and Thread and Memory Placement][] describes how to affect the locality group placement of a thread and its memory.
 *  [Examples of API Usage][] contains code that performs example tasks by using the APIs that are described in this chapter.

# Verifying the Interface Version #

The [lgrp\_version(3LGRP)][lgrp_version_3LGRP] function must be used to verify the presence of a supported lgroup interface before using the lgroup API. The**lgrp\_version()** function has the following syntax:

#include <sys/lgrp_user.h>
    int lgrp_version(const int version);

The **lgrp\_version()** function takes a version number for the lgroup interface as an argument and returns the lgroup interface version that the system supports. When the current implementation of the lgroup API supports the version number in the version argument, the**lgrp\_version()** function returns that version number. Otherwise, the **lgrp\_version()** function returns LGRP\_VER\_NONE.

--------------------

##### Example 3–1 Example of lgrp\_version() Use #####

#include <sys/lgrp_user.h>
    if (lgrp_version(LGRP_VER_CURRENT) != LGRP_VER_CURRENT) {
        fprintf(stderr, "Built with unsupported lgroup interface %d\n",
            LGRP_VER_CURRENT);
        exit (1);
        }

--------------------

# Initializing the Locality Group Interface #

Applications must call [lgrp\_init(3LGRP)][lgrp_init_3LGRP] in order to use the APIs for traversing the lgroup hierarchy and to discover the contents of the lgroup hierarchy. The call to **lgrp\_init()** gives the application a consistent snapshot of the lgroup hierarchy. The application developer can specify whether the snapshot contains only the resources that are available to the calling thread specifically or the resources that are available to the operating system in general. The **lgrp\_init()** function returns a cookie that is used for the following tasks:

*  Navigating the lgroup hierarchy
 *  Determining the contents of an lgroup
 *  Determining whether the snapshot is current

## Using lgrp\_init() ##

The **lgrp\_init()** function initializes the lgroup interface and takes a snapshot of the lgroup hierarchy.

#include <sys/lgrp_user.h>
    lgrp_cookie_t lgrp_init(lgrp_view_t view);

When the **lgrp\_init()** function is called with LGRP\_VIEW\_CALLER as the view, the function returns a snapshot that contains only the resources that are available to the calling thread. When the **lgrp\_init()** function is called with LGRP\_VIEW\_OS as the view, the function returns a snapshot that contains the resources that are available to the operating system. When a thread successfully calls the **lgrp\_init()**function, the function returns a cookie that is used by any function that interacts with the lgroup hierarchy. When a thread no longer needs the cookie, call the **lgrp\_fini()** function with the cookie as the argument.

The lgroup hierarchy consists of a root lgroup that contains all of the machine's CPU and memory resources. The root lgroup might contain other locality groups bounded by smaller latencies.

The **lgrp\_init()** function can return two errors. When a view is invalid, the function returns EINVAL. When there is insufficient memory to allocate the snapshot of the lgroup hierarchy, the function returns ENOMEM.

## Using lgrp\_fini() ##

The [lgrp\_fini(3LGRP)][lgrp_fini_3LGRP] function ends the usage of a given cookie and frees the corresponding lgroup hierarchy snapshot.

#include <sys/lgrp_user.h>
    int lgrp_fini(lgrp_cookie_t cookie);

The **lgrp\_fini()** function takes a cookie that represents an lgroup hierarchy snapshot created by a previous call to **lgrp\_init()**. The**lgrp\_fini()** function frees the memory that is allocated to that snapshot. After the call to **lgrp\_fini()**, the cookie is invalid. Do not use that cookie again.

When the cookie passed to the **lgrp\_fini()** function is invalid, **lgrp\_fini()** returns EINVAL.

# Locality Group Hierarchy #

The APIs that are described in this section enable the calling thread to navigate the lgroup hierarchy. The lgroup hierarchy is a directed acyclic graph that is similar to a tree, except that a node might have more than one parent. The root lgroup represents the whole machine and contains all of that machine's resources. The root lgroup is the lgroup with the highest latency value in the system. Each of the child lgroups contains a subset of the hardware that is in the root lgroup. Each child lgroup is bounded by a lower latency value. Locality groups that are closer to the root have more resources and a higher latency. Locality groups that are closer to the leaves have fewer resources and a lower latency. An lgroup can contain resources directly within its latency boundary. An lgroup can also contain leaf lgroups that contain their own sets of resources. The resources of leaf lgroups are available to the lgroup that encapsulates those leaf lgroups.

## Using lgrp\_cookie\_stale() ##

The [lgrp\_cookie\_stale(3LGRP)][lgrp_cookie_stale_3LGRP] function determines whether the snapshot of the lgroup hierarchy represented by the given cookie is current.

#include <sys/lgrp_user.h>
    int lgrp_cookie_stale(lgrp_cookie_t cookie);

The cookie returned by the **lgrp\_init()** function can become stale due to several reasons that depend on the view that the snapshot represents. A cookie that is returned by calling the **lgrp\_init()** function with the view set to LGRP\_VIEW\_OS can become stale due to changes in the lgroup hierarchy such as dynamic reconfiguration or a change in a CPU's online status. A cookie that is returned by calling the**lgrp\_init()** function with the view set to LGRP\_VIEW\_CALLER can become stale due to changes in the calling thread's processor set or changes in the lgroup hierarchy. A stale cookie is refreshed by calling the **lgrp\_fini()** function with the old cookie, followed by calling**lgrp\_init()** to generate a new cookie.

The **lgrp\_cookie\_stale()** function returns EINVAL when the given cookie is invalid.

## Using lgrp\_view() ##

The [lgrp\_view(3LGRP)][lgrp_view_3LGRP] function determines the view with which a given lgroup hierarchy snapshot was taken.

#include <sys/lgrp_user.h>
    lgrp_view_t lgrp_view(lgrp_cookie_t cookie);

The **lgrp\_view()** function takes a cookie that represents a snapshot of the lgroup hierarchy and returns the snapshot's view of the lgroup hierarchy. Snapshots that are taken with the view LGRP\_VIEW\_CALLER contain only the resources that are available to the calling thread. Snapshots that are taken with the view LGRP\_VIEW\_OS contain all the resources that are available to the operating system.

The **lgrp\_view()** function returns EINVAL when the given cookie is invalid.

## Using lgrp\_nlgrps() ##

The [lgrp\_nlgrps(3LGRP)][lgrp_nlgrps_3LGRP] function returns the number of locality groups in the system. If a system has only one locality group, memory placement optimizations have no effect.

#include <sys/lgrp_user.h>
    int lgrp_nlgrps(lgrp_cookie_t cookie);

The **lgrp\_nlgrps()** function takes a cookie that represents a snapshot of the lgroup hierarchy and returns the number of lgroups available in the hierarchy.

The **lgrp\_nlgrps()** function returns EINVAL when the cookie is invalid.

## Using lgrp\_root() ##

The [lgrp\_root(3LGRP)][lgrp_root_3LGRP] function returns the root lgroup ID.

#include <sys/lgrp_user.h>
    lgrp_id_t lgrp_root(lgrp_cookie_t cookie);

The **lgrp\_root()** function takes a cookie that represents a snapshot of the lgroup hierarchy and returns the root lgroup ID.

## Using lgrp\_parents() ##

The [lgrp\_parents(3LGRP)][lgrp_parents_3LGRP] function takes a cookie that represents a snapshot of the lgroup hierarchy and returns the number of parent lgroups for the specified lgroup.

#include <sys/lgrp_user.h>
    int lgrp_parents(lgrp_cookie_t cookie, lgrp_id_t child,
                     lgrp_id_t *lgrp_array, uint_t lgrp_array_size);

If lgrp\_array is not NULL and the value of lgrp\_array\_size is not zero, the **lgrp\_parents()** function fills the array with parent lgroup IDs until the array is full or all parent lgroup IDs are in the array. The root lgroup has zero parents. When the **lgrp\_parents()** function is called for the root lgroup, lgrp\_array is not filled in.

The **lgrp\_parents()** function returns EINVAL when the cookie is invalid. The **lgrp\_parents()** function returns ESRCH when the specified lgroup ID is not found.

## Using lgrp\_children() ##

The [lgrp\_children(3LGRP)][lgrp_children_3LGRP] function takes a cookie that represents the calling thread's snapshot of the lgroup hierarchy and returns the number of child lgroups for the specified lgroup.

#include <sys/lgrp_user.h>
    int lgrp_children(lgrp_cookie_t cookie, lgrp_id_t parent,
                      lgrp_id_t *lgrp_array, uint_t lgrp_array_size);

If lgrp\_array is not NULL and the value of lgrp\_array\_size is not zero, the **lgrp\_children()** function fills the array with child lgroup IDs until the array is full or all child lgroup IDs are in the array.

The **lgrp\_children()** function returns EINVAL when the cookie is invalid. The **lgrp\_children()** function returns ESRCH when the specified lgroup ID is not found.

# Locality Group Contents #

The following APIs retrieve information about the contents of a given lgroup.

The lgroup hierarchy organizes the domain's resources to simplify the process of locating the nearest resource. Leaf lgroups are defined with resources that have the least latency. Each of the successive ancestor lgroups of a given leaf lgroup contains the next nearest resources to its child lgroup. The root lgroup contains all of the resources that are in the domain.

The resources of a given lgroup are contained directly within that lgroup or indirectly within the leaf lgroups that the given lgroup encapsulates. Leaf lgroups directly contain their resources and do not encapsulate any other lgroups.

## Using lgrp\_resources() ##

The **lgrp\_resources()** function returns the number of resources contained in a specified lgroup.

#include <sys/lgrp_user.h>
    int lgrp_resources(lgrp_cookie_t cookie, lgrp_id_t lgrp, lgrp_id_t *lgrpids,
                       uint_t count, lgrp_rsrc_t type);

The **lgrp\_resources()** function takes a cookie that represents a snapshot of the lgroup hierarchy. That cookie is obtained from the**lgrp\_init()** function. The **lgrp\_resources()** function returns the number of resources that are in the lgroup with the ID that is specified by the value of the lgrp argument. The **lgrp\_resources()** function represents the resources with a set of lgroups that directly contain CPU or memory resources. The lgrp\_rsrc\_t argument can have the following two values:

LGRP\_RSRC\_CPU

The **lgrp\_resources()** function returns the number of CPU resources.

LGRP\_RSRC\_MEM

The **lgrp\_resources()** function returns the number of memory resources.

When the value passed in the lgrpids\[\] argument is not null and the count argument is not zero, the **lgrp\_resources()** function stores lgroup IDs in the lgrpids\[\] array. The number of lgroup IDs stored in the array can be up to the value of the count argument.

The **lgrp\_resources()** function returns EINVAL when the specified cookie, lgroup ID, or type are not valid. The **lgrp\_resources()**function returns ESRCH when the function does not find the specified lgroup ID.

## Using lgrp\_cpus() ##

The [lgrp\_cpus(3LGRP)][lgrp_cpus_3LGRP] function takes a cookie that represents a snapshot of the lgroup hierarchy and returns the number of CPUs in a given lgroup.

#include <sys/lgrp_user.h>
    int lgrp_cpus(lgrp_cookie_t cookie, lgrp_id_t lgrp, processorid_t *cpuids,
                  uint_t count, int content);

If the cpuid\[\] argument is not NULL and the CPU count is not zero, the **lgrp\_cpus()** function fills the array with CPU IDs until the array is full or all the CPU IDs are in the array.

The content argument can have the following two values:

LGRP\_CONTENT\_ALL

The **lgrp\_cpus()** function returns IDs for the CPUs in this lgroup and this lgroup's descendants.

LGRP\_CONTENT\_DIRECT

The **lgrp\_cpus()** function returns IDs for the CPUs in this lgroup only.

The **lgrp\_cpus()** function returns EINVAL when the cookie, lgroup ID, or one of the flags is not valid. The **lgrp\_cpus()** function returnsESRCH when the specified lgroup ID is not found.

## Using lgrp\_mem\_size() ##

The [lgrp\_mem\_size(3LGRP)][lgrp_mem_size_3LGRP] function takes a cookie that represents a snapshot of the lgroup hierarchy and returns the size of installed or free memory in the given lgroup. The **lgrp\_mem\_size()** function reports memory sizes in bytes.

#include <sys/lgrp_user.h>
    lgrp_mem_size_t lgrp_mem_size(lgrp_cookie_t cookie, lgrp_id_t lgrp,
                                  int type, int content)

The type argument can have the following two values:

LGRP\_MEM\_SZ\_FREE

The **lgrp\_mem\_size()** function returns the amount of free memory in bytes.

LGRP\_MEM\_SZ\_INSTALLED

The **lgrp\_mem\_size()** function returns the amount of installed memory in bytes.

The content argument can have the following two values:

LGRP\_CONTENT\_ALL

The **lgrp\_mem\_size()** function returns the amount of memory in this lgroup and this lgroup's descendants.

LGRP\_CONTENT\_DIRECT

The **lgrp\_mem\_size()** function returns the amount of memory in this lgroup only.

The **lgrp\_mem\_size()** function returns EINVAL when the cookie, lgroup ID, or one of the flags is not valid. The **lgrp\_mem\_size()** function returns ESRCH when the specified lgroup ID is not found.

# Locality Group Characteristics #

The following API retrieves information about the characteristics of a given lgroup.

## Using lgrp\_latency\_cookie() ##

The [lgrp\_latency(3LGRP)][lgrp_latency_3LGRP] function returns the latency between a CPU in one lgroup to the memory in another lgroup.

#include <sys/lgrp_user.h>
    int lgrp_latency_cookie(lgrp_cookie_t cookie, lgrp_id_t from, lgrp_id_t to.
                            lat_between_t between);

The **lgrp\_latency\_cookie()** function takes a cookie that represents a snapshot of the lgroup hierarchy. The **lgrp\_init()** function creates this cookie. The **lgrp\_latency\_cookie()** function returns a value that represents the latency between a hardware resource in the lgroup given by the value of the from argument and a hardware resource in the lgroup given by the value of the to argument. If both arguments point to the same lgroup, the **lgrp\_latency\_cookie()** function returns the latency value within that lgroup.

--------------------

**Note –**

The latency value returned by the **lgrp\_latency\_cookie()** function is defined by the operating system and is platform-specific. This value does not necessarily represent the actual latency between hardware devices. Use this value only for comparison within one domain.

--------------------

When the value of the between argument is LGRP\_LAT\_CPU\_TO\_MEM, the **lgrp\_latency\_cookie()** function measures the latency from a CPU resource to a memory resource.

The **lgrp\_latency\_cookie()** function returns EINVAL when the lgroup ID is not valid. When the **lgrp\_latency\_cookie()** function does not find the specified lgroup ID, the “from” lgroup does not contain any CPUs, or the “to” lgroup does not have any memory, the**lgrp\_latency\_cookie()** function returns ESRCH.

# Locality Groups and Thread and Memory Placement #

This section discusses the APIs used to discover and affect thread and memory placement with respect to lgroups.

*  The [lgrp\_home(3LGRP)][lgrp_home_3LGRP] function is used to discover thread placement.
 *  The [meminfo(2)][meminfo_2] system call is used to discover memory placement.
 *  The MADV\_ACCESS flags to the [madvise(3C)][madvise_3C] function are used to affect memory allocation among lgroups.
 *  The [lgrp\_affinity\_set(3LGRP)][lgrp_affinity_set_3LGRP] function can affect thread and memory placement by setting a thread's affinity for a given lgroup.
 *  The affinities of an lgroup may specify an order of preference for lgroups from which to allocate resources.
 *  The kernel needs information about the likely pattern of an application's memory use in order to allocate memory resources efficiently.
 *  The **madvise()** function and its shared object analogue madv.so.1 provide this information to the kernel.
 *  A running process can gather memory usage information about itself by using the **meminfo()** system call.

## Using lgrp\_home() ##

The **lgrp\_home()** function returns the home lgroup for the specified process or thread.

#include <sys/lgrp_user.h>
    lgrp_id_t lgrp_home(idtype_t idtype, id_t id);

The **lgrp\_home()** function returns EINVAL when the ID type is not valid. The **lgrp\_home()** function returns EPERM when the effective user of the calling process is not the superuser and the real or effective user ID of the calling process does not match the real or effective user ID of one of the threads. The **lgrp\_home()** function returns ESRCH when the specified process or thread is not found.

## Using madvise() ##

The **madvise()** function advises the kernel that a region of user virtual memory in the range starting at the address specified in addr and with length equal to the value of the len parameter is expected to follow a particular pattern of use. The kernel uses this information to optimize the procedure for manipulating and maintaining the resources associated with the specified range. Use of the **madvise()** function can increase system performance when used by programs that have specific knowledge of their access patterns over memory.

#include <sys/types.h>
    #include <sys/mman.h>
    int madvise(caddr_t addr, size_t len, int advice);

The **madvise()** function provides the following flags to affect how a thread's memory is allocated among lgroups:

MADV\_ACCESS\_DEFAULT

This flag resets the kernel's expected access pattern for the specified range to the default.

MADV\_ACCESS\_LWP

This flag advises the kernel that the next LWP to touch the specified address range is the LWP that will access that range the most. The kernel allocates the memory and other resources for this range and the LWP accordingly.

MADV\_ACCESS\_MANY

This flag advises the kernel that many processes or LWPs will access the specified address range randomly across the system. The kernel allocates the memory and other resources for this range accordingly.

The **madvise()** function can return the following values:

EAGAIN

Some or all of the mappings in the specified address range, from addr to addr\+len, are locked for I/O.

EINVAL

The value of the addr parameter is not a multiple of the page size as returned by [sysconf(3C)][sysconf_3C], the length of the specified address range is less than or equal to zero, or the advice is invalid.

EIO

An I/O error occurs while reading from or writing to the file system.

ENOMEM

Addresses in the specified address range are outside the valid range for the address space of a process or the addresses in the specified address range specify one or more pages that are not mapped.

ESTALE

The NFS file handle is stale.

## Using meminfo() ##

The **meminfo()** function gives the calling process information about the virtual memory and physical memory that the system has allocated to that process.

#include <sys/types.h>
    #include <sys/mman.h>
    int meminfo(const uint64_t inaddr[], int addr_count,
        const uint_t info_req[], int info_count, uint64_t outdata[],
        uint_t validity[]);

The **meminfo()** function can return the following types of information:

MEMINFO\_VPHYSICAL

The physical memory address corresponding to the given virtual address

MEMINFO\_VLGRP

The lgroup to which the physical page corresponding to the given virtual address belongs

MEMINFO\_VPAGESIZE

The size of the physical page corresponding to the given virtual address

MEMINFO\_VREPLCNT

The number of replicated physical pages that correspond to the given virtual address

MEMINFO\_VREPL|n

The **n**th physical replica of the given virtual address

MEMINFO\_VREPL\_LGRP|n

The lgroup to which the **n**th physical replica of the given virtual address belongs

MEMINFO\_PLGRP

The lgroup to which the given physical address belongs

The **meminfo()** function takes the following parameters:

inaddr

An array of input addresses.

addr\_count

The number of addresses that are passed to **meminfo()**.

info\_req

An array that lists the types of information that are being requested.

info\_count

The number of pieces of information that are requested for each address in the inaddr array.

outdata

An array where the **meminfo()** function places the results. The array's size is equal to the product of the values of the info\_req andaddr\_count parameters.

validity

An array of size equal to the value of the addr\_count parameter. The validity array contains bitwise result codes. The **0**th bit of the result code evaluates the validity of the corresponding input address. Each successive bit in the result code evaluates the validity of the response to the members of the info\_req array in turn.

The **meminfo()** function returns EFAULT when the area of memory to which the outdata or validity arrays point cannot be written to. The**meminfo()** function returns EFAULT when the area of memory to which the info\_req or inaddr arrays point cannot be read from. The**meminfo()** function returns EINVAL when the value of info\_count exceeds 31 or is less than 1. The **meminfo()** function returns EINVALwhen the value of addr\_count is less than zero.

--------------------

##### Example 3–2 Use of meminfo() to Print Out Physical Pages and Page Sizes Corresponding to a Set of Virtual Addresses #####

void
    print_info(void **addrvec, int how_many)
    {
            static const int info[] = {
                    MEMINFO_VPHYSICAL,
                    MEMINFO_VPAGESIZE};
            uint64_t * inaddr = alloca(sizeof(uint64_t) * how_many);
            uint64_t * outdata = alloca(sizeof(uint64_t) * how_many * 2;
            uint_t * validity = alloca(sizeof(uint_t) * how_many);
    
            int i;
    
            for (i = 0; i < how_many; i++)
                    inaddr[i] = (uint64_t *)addr[i];
    
            if (meminfo(inaddr, how_many,  info,
                        sizeof (info)/ sizeof(info[0]),
                        outdata, validity) < 0)
                    ...
    
            for (i = 0; i < how_many; i++) {
                    if (validity[i] & 1 == 0)
                            printf("address 0x%llx not part of address
                                            space\n",
                                    inaddr[i]);
    
                    else if (validity[i] & 2 == 0)
                            printf("address 0x%llx has no physical page
                                            associated with it\n",
                                    inaddr[i]);
    
                    else {
                            char buff[80];
                            if (validity[i] & 4 == 0)
                                    strcpy(buff, "<Unknown>");
                            else
                                    sprintf(buff, "%lld", outdata[i * 2 +
                                                    1]);
                            printf("address 0x%llx is backed by physical
                                            page 0x%llx of size %s\n",
                                            inaddr[i], outdata[i * 2], buff);
                    }
            }
    }

--------------------

## Locality Group Affinity ##

The kernel assigns a thread to a locality group when the lightweight process (LWP) for that thread is created. That lgroup is called the thread's**home lgroup**. The kernel runs the thread on the CPUs in the thread's home lgroup and allocates memory from that lgroup whenever possible. If resources from the home lgroup are unavailable, the kernel allocates resources from other lgroups. When a thread has affinity for more than one lgroup, the operating system allocates resources from lgroups chosen in order of affinity strength. Lgroups can have one of three distinct affinity levels:

1.  LGRP\_AFF\_STRONG indicates strong affinity. If this lgroup is the thread's home lgroup, the operating system avoids rehoming the thread to another lgroup if possible. Events such as dynamic reconfiguration, processor, offlining, processor binding, and processor set binding and manipulation might still result in thread rehoming.
2.  LGRP\_AFF\_WEAK indicates weak affinity. If this lgroup is the thread's home lgroup, the operating system rehomes the thread if necessary for load balancing purposes.
3.  LGRP\_AFF\_NONE indicates no affinity. If a thread has no affinity to any lgroup, the operating system assigns a home lgroup to the thread .

The operating system uses lgroup affinities as advice when allocating resources for a given thread. The advice is factored in with the other system constraints. Processor binding and processor sets do not change lgroup affinities, but might restrict the lgroups on which a thread can run.

### Using lgrp\_affinity\_get() ###

The [lgrp\_affinity\_get(3LGRP)][lgrp_affinity_get_3LGRP] function returns the affinity that a LWP has for a given lgroup.

#include <sys/lgrp_user.h>
    lgrp_affinity_t lgrp_affinity_get(idtype_t idtype, id_t id, lgrp_id_t lgrp);

The idtype and id arguments specify the LWP that the **lgrp\_affinity\_get()** function examines. If the value of idtype is P\_PID, the**lgrp\_affinity\_get()** function gets the lgroup affinity for one of the LWPs in the process whose process ID matches the value of the idargument. If the value of idtype is P\_LWPID, the **lgrp\_affinity\_get()** function gets the lgroup affinity for the LWP of the current process whose LWP ID matches the value of the id argument. If the value of idtype is P\_MYID, the **lgrp\_affinity\_get()** function gets the lgroup affinity for the current LWP.

The **lgrp\_affinity\_get()** function returns EINVAL when the given lgroup or ID type is not valid. The **lgrp\_affinity\_get()** function returns EPERM when the effective user of the calling process is not the superuser and the ID of the calling process does not match the real or effective user ID of one of the LWPs. The **lgrp\_affinity\_get()** function returns ESRCH when a given lgroup or LWP is not found.

### Using lgrp\_affinity\_set() ###

The [lgrp\_affinity\_set(3LGRP)][lgrp_affinity_set_3LGRP] function sets the affinity that a LWP or set of LWPs have for a given lgroup.

#include <sys/lgrp_user.h>
    int lgrp_affinity_set(idtype_t idtype, id_t id, lgrp_id_t lgrp,
                          lgrp_affinity_t affinity);

The idtype and id arguments specify the LWP or set of LWPs the **lgrp\_affinity\_set()** function examines. If the value of idtype is P\_PID, the **lgrp\_affinity\_set()** function sets the lgroup affinity for all of the LWPs in the process whose process ID matches the value of the idargument to the affinity level specified in the affinity argument. If the value of idtype is P\_LWPID, the **lgrp\_affinity\_set()** function sets the lgroup affinity for the LWP of the current process whose LWP ID matches the value of the id argument to the affinity level specified in the affinityargument. If the value of idtype is P\_MYID, the **lgrp\_affinity\_set()** function sets the lgroup affinity for the current LWP or process to the affinity level specified in the affinity argument.

The **lgrp\_affinity\_set()** function returns EINVAL when the given lgroup, affinity, or ID type is not valid. The **lgrp\_affinity\_set()**function returns EPERM when the effective user of the calling process is not the superuser and the ID of the calling process does not match the real or effective user ID of one of the LWPs. The **lgrp\_affinity\_set()** function returns ESRCH when a given lgroup or LWP is not found.

# Examples of API Usage #

This section contains code for example tasks that use the APIs that are described in this chapter.

--------------------

##### Example 3–3 Move Memory to a Thread #####

The following code sample moves the memory in the address range between addr and addr\+len near the next thread to touch that range.

#include <stdio.h>
    #include <sys/mman.h>
    #include <sys/types.h>
    
    
    
    /*
     * Move memory to thread
     */
    void
    mem_to_thread(caddr_t addr, size_t len)
    {
    	if (madvise(addr, len, MADV_ACCESS_LWP) < 0)
    		perror("madvise");
    }

--------------------

##### Example 3–4 Move a Thread to Memory #####

This sample code uses the **meminfo()** function to determine the lgroup of the physical memory backing the virtual page at the given address. The sample code then sets a strong affinity for that lgroup in an attempt to move the current thread near that memory.

#include <stdio.h>
    #include <sys/lgrp_user.h>
    #include <sys/mman.h>
    #include <sys/types.h>
    
    
    /*
     * Move a thread to memory
     */
    int
    thread_to_memory(caddr_t va)
    {
    	uint64_t    addr;
    	ulong_t     count;
    	lgrp_id_t   home;
    	uint64_t    lgrp;
    	uint_t      request;
    	uint_t      valid;
    
    	addr = (uint64_t)va;
    	count = 1;
    	request = MEMINFO_VLGRP;
    	if (meminfo(&addr, 1, &request, 1, &lgrp, &valid) != 0) {
    		perror("meminfo");
    		return (1);
    	}
    
    	if (lgrp_affinity_set(P_LWPID, P_MYID, lgrp, LGRP_AFF_STRONG) != 0) {
    		perror("lgrp_affinity_set");
    		return (2);
    	}
    
    	home = lgrp_home(P_LWPID, P_MYID);
    	if (home == -1) {
    		perror ("lgrp_home");
    		return (3);
    	}
    
    	if (home != lgrp)
    		return (-1);
    
    	return (0);
    }

--------------------

##### Example 3–5 Walk the lgroup Hierarchy #####

The following sample code walks through and prints out the lgroup hierarchy.

#include <stdio.h>
    #include <stdlib.h>
    #include <sys/lgrp_user.h>
    #include <sys/types.h>
    
    
    /*
     * Walk and print lgroup hierarchy from given lgroup
     * through all its descendants
     */
    int
    lgrp_walk(lgrp_cookie_t cookie, lgrp_id_t lgrp, lgrp_content_t content)
    {
    	lgrp_affinity_t    aff;
    	lgrp_id_t          *children;
    	processorid_t      *cpuids;
    	int                i;
    	int                ncpus;
    	int                nchildren;
    	int                nparents;
    	lgrp_id_t          *parents;
    	lgrp_mem_size_t    size;
    
    	/*
    	 * Print given lgroup, caller's affinity for lgroup,
    	 * and desired content specified
    	 */
    	printf("LGROUP #%d:\n", lgrp);
    
    	aff = lgrp_affinity_get(P_LWPID, P_MYID, lgrp);
    	if (aff == -1)
    		perror ("lgrp_affinity_get");
    	printf("\tAFFINITY: %d\n", aff);
    
    	printf("CONTENT %d:\n", content);
    
    	/*
    	 * Get CPUs
    	 */
    	ncpus = lgrp_cpus(cookie, lgrp, NULL, 0, content);
    	printf("\t%d CPUS: ", ncpus);
    	if (ncpus == -1) {
    		perror("lgrp_cpus");
    		return (-1);
    	} else if (ncpus > 0) {
    		cpuids = malloc(ncpus * sizeof (processorid_t));
    		ncpus = lgrp_cpus(cookie, lgrp, cpuids, ncpus, content);
                    if (ncpus == -1) {
    			free(cpuids);
                   	        perror("lgrp_cpus");
    			return (-1);
    		}
    		for (i = 0; i < ncpus; i++)
    			printf("%d ", cpuids[i]);
    		free(cpuids);
    	}
    	printf("\n");
    
    	/*
    	 * Get memory size
    	 */
    	printf("\tMEMORY: ");
    	size = lgrp_mem_size(cookie, lgrp, LGRP_MEM_SZ_INSTALLED, content);
    	if (size == -1) {
    		perror("lgrp_mem_size");
    		return (-1);
    	}
    	printf("installed bytes 0x%llx, ", size);
    	size = lgrp_mem_size(cookie, lgrp, LGRP_MEM_SZ_FREE, content);
            if (size == -1) {
    		perror("lgrp_mem_size");
    		return (-1);
    	}
    	printf("free bytes 0x%llx\n", size);
    
    	/*
    	 * Get parents
    	 */
    	nparents = lgrp_parents(cookie, lgrp, NULL, 0);
    	printf("\t%d PARENTS: ", nparents);
    	if (nparents == -1) {
    		perror("lgrp_parents");
    		return (-1);
    	} else if (nparents > 0) {
    		parents = malloc(nparents * sizeof (lgrp_id_t));
    		nparents = lgrp_parents(cookie, lgrp, parents, nparents);
                   	if (nparents == -1) {
    			free(parents);
                            perror("lgrp_parents");
    			return (-1);
                   	}
    		for (i = 0; i < nparents; i++)
    			printf("%d ", parents[i]);
    		free(parents);
    	}
    	printf("\n");
    
    	/*
    	 * Get children
    	 */
    	nchildren = lgrp_children(cookie, lgrp, NULL, 0);
    	printf("\t%d CHILDREN: ", nchildren);
    	if (nchildren == -1) {
    		perror("lgrp_children");
    		return (-1);
    	} else if (nchildren > 0) {
    		children = malloc(nchildren * sizeof (lgrp_id_t));
    		nchildren = lgrp_children(cookie, lgrp, children, nchildren);
                   	if (nchildren == -1) {
    			free(children);
                            perror("lgrp_children");
    			return (-1);
                   	}
    		printf("Children: ");
    		for (i = 0; i < nchildren; i++)
    			printf("%d ", children[i]);
    		printf("\n");
    
    		for (i = 0; i < nchildren; i++)
    			lgrp_walk(cookie, children[i], content);
    
    		free(children);
    	}
    	printf("\n");
    
    	return (0);
    }

--------------------

##### Example 3–6 Find the Closest lgroup With Available Memory Outside a Given lgroup #####

#include <stdio.h>
    #include <stdlib.h>
    #include <sys/lgrp_user.h>
    #include <sys/types.h>
    
    
    #define	INT_MAX	2147483647
    
    
    /*
     * Find next closest lgroup outside given one with available memory
     */
    lgrp_id_t
    lgrp_next_nearest(lgrp_cookie_t cookie, lgrp_id_t from)
    {
    	lgrp_id_t          closest;
    	int                i;
    	int                latency;
    	int                lowest;
    	int                nparents;
    	lgrp_id_t          *parents;
    	lgrp_mem_size_t    size;
    
    
    	/*
    	 * Get number of parents
    	 */
    	nparents = lgrp_parents(cookie, from, NULL, 0);
    	if (nparents == -1) {
    		perror("lgrp_parents");
    		return (LGRP_NONE);
    	}
    
    	/*
    	 * No parents, so current lgroup is next nearest
    	 */
    	if (nparents == 0) {
    		return (from);
    	}
    
    	/*
    	 * Get parents
    	 */
    	parents = malloc(nparents * sizeof (lgrp_id_t));
    	nparents = lgrp_parents(cookie, from, parents, nparents);
    	if (nparents == -1) {
    		perror("lgrp_parents");
    		free(parents);
    		return (LGRP_NONE);
            }
    
    	/*
    	 * Find closest parent (ie. the one with lowest latency)
    	 */
    	closest = LGRP_NONE;
    	lowest = INT_MAX;
    	for (i = 0; i < nparents; i++) {
    		lgrp_id_t	lgrp;
    
    		/*
    		 * See whether parent has any free memory
    		 */
    		size = lgrp_mem_size(cookie, parents[i], LGRP_MEM_SZ_FREE,
    		    LGRP_CONTENT_ALL);
    		if (size > 0)
    			lgrp = parents[i];
    		else {
    			if (size == -1)
    				perror("lgrp_mem_size");
    
    			/*
    			 * Find nearest ancestor if parent doesn't
    			 * have any memory
    			 */
    			lgrp = lgrp_next_nearest(cookie, parents[i]);
    			if (lgrp == LGRP_NONE)
    				continue;
    		}
    
    		/*
    		 * Get latency within parent lgroup
    		 */
    		latency = lgrp_latency_cookie(lgrp, lgrp);
    		if (latency == -1) {
    			perror("lgrp_latency_cookie");
    			continue;
    		}
    
    		/*
    		 * Remember lgroup with lowest latency
    		 */
    		if (latency < lowest) {
    			closest = lgrp;
    			lowest = latency;
    		}
    	}
    
    	free(parents);
    	return (closest);
    }
    
    
    /*
     * Find lgroup with memory nearest home lgroup of current thread
     */
    lgrp_id_t
    lgrp_nearest(lgrp_cookie_t cookie)
    {
    	lgrp_id_t	home;
    	longlong_t	size;
    
    	/*
    	 * Get home lgroup
    	 */
    	home = lgrp_home(P_LWPID, P_MYID);
    
    	/*
    	 * See whether home lgroup has any memory available in its hierarchy
    	 */
    	size = lgrp_mem_size(cookie, home, LGRP_MEM_SZ_FREE,
    	    LGRP_CONTENT_ALL);
    	if (size == -1)
    		perror("lgrp_mem_size");
    
    	/*
    	 * It does, so return the home lgroup.
    	 */
    	if (size > 0)
    		return (home);
    
    	/*
    	 * Otherwise, find next nearest lgroup outside of the home.
    	 */
    	return (lgrp_next_nearest(cookie, home));
    }

--------------------

##### Example 3–7 Find Nearest lgroup With Free Memory #####

This example code finds the nearest lgroup with free memory to a given thread's home lgroup.

lgrp_id_t
    lgrp_nearest(lgrp_cookie_t cookie)
    {
            lgrp_id_t         home;
            longlong_t        size;
    
            /*
             * Get home lgroup
             */
    
            home = lgrp_home();
    
            /*
             * See whether home lgroup has any memory available in its hierarchy
             */
    
            if (lgrp_mem_size(cookie, lgrp, LGRP_MEM_SZ_FREE,
                LGRP_CONTENT_ALL, &size) == -1)
                    perror("lgrp_mem_size");
    
            /*
             * It does, so return the home lgroup.
             */
    
            if (size > 0)
                    return (home);
    
            /*
             * Otherwise, find next nearest lgroup outside of the home.
             */
    
            return (lgrp_next_nearest(cookie, home));
    }

[http_docs.oracle.com_cd_E19082-01_820-1691_lgroups-1_index.html]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-1/index.html
[Previous_ Chapter_2 MPO Observability Tools]: http://docs.oracle.com/cd/E19082-01/820-1691/gevrx/index.html
[erifying the Interface Version]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-27/index.html
[Initializing the Locality Group Interface]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-4/index.html
[Locality Group Hierarchy]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-29/index.html
[Locality Group Contents]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-30/index.html
[Locality Group Characteristics]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-31/index.html
[Locality Groups and Thread and Memory Placement]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-32/index.html
[Examples of API Usage]: http://docs.oracle.com/cd/E19082-01/820-1691/lgroups-33/index.html
[lgrp_version_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-version-3lgrp/index.html
[lgrp_init_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-init-3lgrp/index.html
[lgrp_fini_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-fini-3lgrp/index.html
[lgrp_cookie_stale_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-cookie-stale-3lgrp/index.html
[lgrp_view_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-view-3lgrp/index.html
[lgrp_nlgrps_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-nlgrps-3lgrp/index.html
[lgrp_root_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-root-3lgrp/index.html
[lgrp_parents_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-parents-3lgrp/index.html
[lgrp_children_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-children-3lgrp/index.html
[lgrp_cpus_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-cpus-3lgrp/index.html
[lgrp_mem_size_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-mem-size-3lgrp/index.html
[lgrp_latency_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-latency-3lgrp/index.html
[lgrp_home_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-home-3lgrp/index.html
[meminfo_2]: http://docs.oracle.com/docs/cd/E19963-01/821-1463/meminfo-2/index.html
[madvise_3C]: http://docs.oracle.com/docs/cd/E19963-01/821-1465/madvise-3c/index.html
[lgrp_affinity_set_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-affinity-set-3lgrp/index.html
[sysconf_3C]: http://docs.oracle.com/docs/cd/E19963-01/821-1465/sysconf-3c/index.html
[lgrp_affinity_get_3LGRP]: http://docs.oracle.com/docs/cd/E19963-01/821-1468/lgrp-affinity-get-3lgrp/index.html