1 |
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% |
2 |
|
% $Id$ |
3 |
|
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% |
4 |
\chapter{Slab Allocator} |
\chapter{Slab Allocator} |
5 |
|
|
6 |
The majority of memory allocation requests in the kernel are for small, |
The majority of memory allocation requests in the kernel are for small, |
13 |
outlined in Bonwick's~\cite{slab} paper. Some terminology: |
outlined in Bonwick's~\cite{slab} paper. Some terminology: |
14 |
|
|
15 |
\begin{description} |
\begin{description} |
16 |
\idn{cache} It is a store of recently used objects of the same type. In the slab allocator, it is the highest logical unit of storage. It has a human parseable name like dentry\_cache etc. |
\idn{cache} It is a store of recently used objects of the same type. In the slab allocator, it is the highest logical unit of storage. It has a human parse-able name like dentry\_cache etc. |
17 |
\idn{slab} A slab is a container for objects and is made up of one or more page frames. A cache consists of a number of slabs. |
\idn{slab} A slab is a container for objects and is made up of one or more page frames. A cache consists of a number of slabs. |
18 |
\idn{object} This is the smallest unit. It resides on the slab and would be something like a single dentry. |
\idn{object} This is the smallest unit. It resides on the slab and would be something like a single dentry. |
19 |
\end{description} |
\end{description} |
20 |
|
|
21 |
The objective is that a single page can now be used to contain a number of |
The objective is that a single page can now be used to contain a number of |
22 |
objects thus saving memory and avoiding internal fragmentation. The slabs |
objects thus saving memory and avoiding internal fragmentation. The slabs |
23 |
are organised into three types, full slabs, partial slabs and empty ones. |
are organized into three types, full slabs, partial slabs and empty ones. |
24 |
Partial slabs are used if available to avoid fragmentation. To see all |
Partial slabs are used if available to avoid fragmentation. To see all |
25 |
information on caches and slabs available in a system, type {\bf cat |
information on caches and slabs available in a system, type {\bf cat |
26 |
/proc/slabinfo} to see a list. The fields correspond to: |
/proc/slabinfo} to see a list. The fields correspond to: |
57 |
this helps in bringing down the number of hardware cache-coherency issues: |
this helps in bringing down the number of hardware cache-coherency issues: |
58 |
if more than one CPU references some particular piece of data (and therefore |
if more than one CPU references some particular piece of data (and therefore |
59 |
has it in its on-chip CPU cache) the SMP hardware has to worry about keeping |
has it in its on-chip CPU cache) the SMP hardware has to worry about keeping |
60 |
those CPU caches synchronised between them. |
those CPU caches synchronized between them. |
61 |
|
|
62 |
Hence, each cache has a short per-cpu array called {\bf cpudata} of type |
Hence, each cache has a short per-cpu array called {\bf cpudata} of type |
63 |
{\bf cpucache\_t}. This is a very simple struct with only two members: |
{\bf cpucache\_t}. This is a very simple struct with only two members: |
72 |
To simplify access to this array, a macro called {\bf cc\_data} is provided. |
To simplify access to this array, a macro called {\bf cc\_data} is provided. |
73 |
Most allocs and frees will be taken out of this per-CPU cache until it |
Most allocs and frees will be taken out of this per-CPU cache until it |
74 |
overflows. Once it overflows, half of the entries are placed in a global |
overflows. Once it overflows, half of the entries are placed in a global |
75 |
cache minimising the amount of spinlock operations required. |
cache minimizing the amount of spinlock operations required. |
76 |
|
|
77 |
\newpage |
\newpage |
78 |
\section{Cache Structure} |
\section{Cache Structure} |
115 |
SLAB\_DEBUG\_FREE & Perform (expensive) checks on free \\ |
SLAB\_DEBUG\_FREE & Perform (expensive) checks on free \\ |
116 |
SLAB\_DEBUG\_INITIAL & Call constructor even if slab is a bogus creation \\ |
SLAB\_DEBUG\_INITIAL & Call constructor even if slab is a bogus creation \\ |
117 |
SLAB\_RED\_ZONE & Red zone objs in a cache to check for overflows \\ |
SLAB\_RED\_ZONE & Red zone objs in a cache to check for overflows \\ |
118 |
SLAB\_POISON & Poison objects with known pattern for trapping uninitialised data access \\ |
SLAB\_POISON & Poison objects with known pattern for trapping uninitialized data access \\ |
119 |
\end{tabularx} |
\end{tabularx} |
120 |
|
|
121 |
\vspace{10pt} |
\vspace{10pt} |
224 |
$list\rightarrow{prev}$ points to slab\_t (the slab it is part of). So given an object, |
$list\rightarrow{prev}$ points to slab\_t (the slab it is part of). So given an object, |
225 |
we can easily find the associated cache and slab through these pointers. |
we can easily find the associated cache and slab through these pointers. |
226 |
|
|
227 |
\section{Initialisation} |
\section{Initialization} |
228 |
|
|
229 |
The first function called from \emph{start\_kernel} is {\bf |
The first function called from \emph{start\_kernel} is {\bf |
230 |
kmem\_cache\_init()}. This takes the following very simple steps |
kmem\_cache\_init()}. This takes the following very simple steps |
231 |
|
|
232 |
\begin{itemize} |
\begin{itemize} |
233 |
\item Initialise a mutex for access to the cache chain |
\item Initialize a mutex for access to the cache chain |
234 |
\item Initialise the linked list for the cache chain |
\item Initialize the linked list for the cache chain |
235 |
\item Initialise the cache\_cache |
\item Initialize the cache\_cache |
236 |
\item Sets the cache\_cache colour |
\item Sets the cache\_cache colour |
237 |
\end{itemize} |
\end{itemize} |
238 |
|
|
239 |
The term \emph{cache chain} is simply a fancy name for a circular linked list |
The term \emph{cache chain} is simply a fancy name for a circular linked list |
240 |
of caches the slab allocator knows about. It then goes on to initialise |
of caches the slab allocator knows about. It then goes on to initialize |
241 |
a cache of caches called {\bf kmem\_cache}. This is a cache of objects of |
a cache of caches called {\bf kmem\_cache}. This is a cache of objects of |
242 |
type {\bf kmem\_cache\_t} which describes information about the cache itself. |
type {\bf kmem\_cache\_t} which describes information about the cache itself. |
243 |
|
|
244 |
\subsection{Initialising cache\_cache} |
\subsection{Initializing cache\_cache} |
245 |
|
|
246 |
This cache is initialised as follows |
This cache is initialized as follows |
247 |
|
|
248 |
\begin{verbatim} |
\begin{verbatim} |
249 |
static kmem_cache_t cache_cache = { |
static kmem_cache_t cache_cache = { |
264 |
slabs\_free & Standard list init \\ |
slabs\_free & Standard list init \\ |
265 |
objsize & Size of the struct. See the kmem\_cache\_s struct \\ |
objsize & Size of the struct. See the kmem\_cache\_s struct \\ |
266 |
flags & Make sure this cache can't be reaped \\ |
flags & Make sure this cache can't be reaped \\ |
267 |
spinlock & Initialise unlocked \\ |
spinlock & Initialize unlocked \\ |
268 |
colour\_off & Align the objects to the L1 Cache \\ |
colour\_off & Align the objects to the L1 Cache \\ |
269 |
name & Name of the cache \\ |
name & Name of the cache \\ |
270 |
\end{tabularx} |
\end{tabularx} |
271 |
|
|
272 |
\subsection{Initialising cache\_sizes} |
\subsection{Initializing cache\_sizes} |
273 |
|
|
274 |
\emph{kmem\_cache\_sizes\_init()} is called to create a set of caches of |
\emph{kmem\_cache\_sizes\_init()} is called to create a set of caches of |
275 |
different sizes. On a system with a page size of 4096, the smallest chunk |
different sizes. On a system with a page size of 4096, the smallest chunk |
280 |
size of 131072 bytes are created. These will be used by \emph{kmalloc} later. |
size of 131072 bytes are created. These will be used by \emph{kmalloc} later. |
281 |
Refer to section~\ref{fun:kcsi} for the implementation details. |
Refer to section~\ref{fun:kcsi} for the implementation details. |
282 |
|
|
283 |
\section{Initialising Objects} |
\section{Initializing Objects} |
284 |
\subsection{Function kmem\_cache\_init\_objs()} |
\subsection{Function kmem\_cache\_init\_objs()} |
285 |
\textit{File: }\url{mm/slab.c}\\ |
\textit{File: }\url{mm/slab.c}\\ |
286 |
\textit{Prototype: } |
\textit{Prototype: } |
289 |
slab_t * slabp, |
slab_t * slabp, |
290 |
unsigned long ctor_flags) |
unsigned long ctor_flags) |
291 |
\end{verbatim} |
\end{verbatim} |
292 |
This function is called to initialise all the objects on a slab once |
This function is called to initialize all the objects on a slab once |
293 |
by \texttt{kmem\_cache\_grow} when creating a new slab. |
by \texttt{kmem\_cache\_grow} when creating a new slab. |
294 |
\begin{verbatim} |
\begin{verbatim} |
295 |
int i; |
int i; |
409 |
\begin{supertabular}{lp{10cm}} |
\begin{supertabular}{lp{10cm}} |
410 |
|
|
411 |
\id{SLAB\_NOFS}& This flag tells the page free logic to not make any |
\id{SLAB\_NOFS}& This flag tells the page free logic to not make any |
412 |
calls to the filesystem layer. This is important for the |
calls to the file-system layer. This is important for the |
413 |
allocation of buffer heads for instance where it is important |
allocation of buffer heads for instance where it is important |
414 |
the filesystem does not end up recursively calling itself. \\ |
the file-system does not end up recursively calling itself. \\ |
415 |
|
|
416 |
\id{SLAB\_NOIO} & Do not start any IO. For example, in |
\id{SLAB\_NOIO} & Do not start any IO. For example, in |
417 |
\texttt{try\_to\_free\_buffers()}, no attempt to write out |
\texttt{try\_to\_free\_buffers()}, no attempt to write out |
432 |
\id{SLAB\_KERNEL}& Used when the caller just wants the object to be allocated |
\id{SLAB\_KERNEL}& Used when the caller just wants the object to be allocated |
433 |
and are not particular about what needs to be done to get |
and are not particular about what needs to be done to get |
434 |
it. The caller will perform IO, sleep and can make calls to |
it. The caller will perform IO, sleep and can make calls to |
435 |
teh filesystem. \\ |
the file-system. \\ |
436 |
|
|
437 |
\id{SLAB\_NFS} & Supplied to provide a mapping to GFP\_NFS. In reality, it is |
\id{SLAB\_NFS} & Supplied to provide a mapping to GFP\_NFS. In reality, it is |
438 |
never used. The only caller that needs it uses GFP\_NFS |
never used. The only caller that needs it uses GFP\_NFS |
445 |
\end{supertabular} |
\end{supertabular} |
446 |
|
|
447 |
\vspace{10pt} |
\vspace{10pt} |
448 |
For completness, there are two other SLAB flags which exist. They are: |
For completeness, there are two other SLAB flags which exist. They are: |
449 |
|
|
450 |
\vspace{10pt} |
\vspace{10pt} |
451 |
\begin{supertabular}{lp{8cm}} |
\begin{supertabular}{lp{8cm}} |
493 |
The macro \texttt{kmem\_cache\_alloc\_one} which will be described in |
The macro \texttt{kmem\_cache\_alloc\_one} which will be described in |
494 |
section~\ref{mac:kcao} allocates an object if there is a partially allocated |
section~\ref{mac:kcao} allocates an object if there is a partially allocated |
495 |
or completely free slab available. \texttt{local\_irq\_save} disables |
or completely free slab available. \texttt{local\_irq\_save} disables |
496 |
interrupts and saves the flags. This will guarantee synchronisation which |
interrupts and saves the flags. This will guarantee synchronization which |
497 |
is needed for \texttt{kmem\_cache\_alloc\_one}. A spinlock can not be used |
is needed for \texttt{kmem\_cache\_alloc\_one}. A spinlock can not be used |
498 |
because an interrupt handler can not take out a spinlock and an interrupt |
because an interrupt handler can not take out a spinlock and an interrupt |
499 |
handler can call this function. |
handler can call this function. |
584 |
} |
} |
585 |
\end{verbatim} |
\end{verbatim} |
586 |
|
|
587 |
If a cpucache is not availiable, just allocate one object in the same way |
If a cpucache is not available, just allocate one object in the same way |
588 |
a UP does it except that a spinlock is held. |
a UP does it except that a spinlock is held. |
589 |
|
|
590 |
\begin{verbatim} |
\begin{verbatim} |
657 |
\end{verbatim} |
\end{verbatim} |
658 |
|
|
659 |
Once a slab is found that can be used, \emph{kmem\_cache\_alloc\_one\_tail()} |
Once a slab is found that can be used, \emph{kmem\_cache\_alloc\_one\_tail()} |
660 |
is called. The main complexity in this funcion is in the debugging so lets |
is called. The main complexity in this function is in the debugging so lets |
661 |
examine it in pieces: |
examine it in pieces: |
662 |
|
|
663 |
\begin{verbatim} |
\begin{verbatim} |
678 |
\end{verbatim} |
\end{verbatim} |
679 |
|
|
680 |
\textit{s\_mem} is the pointer to the beginning of the objects within the slab and |
\textit{s\_mem} is the pointer to the beginning of the objects within the slab and |
681 |
\textit{free} is the index of the first object on the slab's freelist. Multiplying |
\textit{free} is the index of the first object on the slab's free-list. Multiplying |
682 |
it by the size of each object will make objp the address of a free object. |
it by the size of each object will make objp the address of a free object. |
683 |
slab\_bufctl is a macro which casts kmem\_bufctl\_t to slab\_t and adds 1 to it |
slab\_bufctl is a macro which casts kmem\_bufctl\_t to slab\_t and adds 1 to it |
684 |
effectively giving the address of the next free object. |
effectively giving the address of the next free object. |
823 |
of flags against the CREATE\_MASK as the caller may request flags that are |
of flags against the CREATE\_MASK as the caller may request flags that are |
824 |
simply not available. |
simply not available. |
825 |
|
|
826 |
The arguements to kmem\_cache\_create are as follows |
The arguments to kmem\_cache\_create are as follows |
827 |
|
|
828 |
\vspace{10pt} \noindent \begin{tabularx}{15cm}{lX} |
\vspace{10pt} \noindent \begin{tabularx}{15cm}{lX} |
829 |
const char *name & Human readable name of the cache \\ |
const char *name & Human readable name of the cache \\ |
861 |
|
|
862 |
Request a kmem\_cache\_t from the cache\_cache. Remember this is a cache |
Request a kmem\_cache\_t from the cache\_cache. Remember this is a cache |
863 |
of cache descriptors. It's not a catch 22 problem as the cache\_cache is |
of cache descriptors. It's not a catch 22 problem as the cache\_cache is |
864 |
statically initialised. |
statically initialized. |
865 |
|
|
866 |
\begin{verbatim} |
\begin{verbatim} |
867 |
/* Check that size is in terms of words. |
/* Check that size is in terms of words. |
911 |
\end{verbatim} |
\end{verbatim} |
912 |
|
|
913 |
If the cache is SLAB\_HWCACHE\_ALIGN, it's aligning on the size of |
If the cache is SLAB\_HWCACHE\_ALIGN, it's aligning on the size of |
914 |
L1\_CACHE\_BYES which is quiet large, 32 bytes on an intel. So, align is |
L1\_CACHE\_BYES which is quiet large, 32 bytes on an Intel. So, align is |
915 |
adjusted to that two objects could fit in a cache line. If 2 would fit, |
adjusted to that two objects could fit in a cache line. If 2 would fit, |
916 |
then try 4, until as many objects are packed in. Then size is adjusted to |
then try 4, until as many objects are packed in. Then size is adjusted to |
917 |
the new alignment |
the new alignment |
974 |
break; /* Acceptable internal fragmentation. */ |
break; /* Acceptable internal fragmentation. */ |
975 |
\end{verbatim} |
\end{verbatim} |
976 |
|
|
977 |
8 appears to be an arbitary figure. |
8 appears to be an arbitrary figure. |
978 |
|
|
979 |
\begin{verbatim} |
\begin{verbatim} |
980 |
next: |
next: |
1058 |
|
|
1059 |
\end{verbatim} |
\end{verbatim} |
1060 |
|
|
1061 |
This just copies the information into the kmem\_cache\_t and initialises |
This just copies the information into the kmem\_cache\_t and initializes |
1062 |
it's fields. The kmem\_find\_general\_cachep despite it's funny name just |
it's fields. The kmem\_find\_general\_cachep despite it's funny name just |
1063 |
goes through the sized caches used by kmalloc until it finds one big enough |
goes through the sized caches used by kmalloc until it finds one big enough |
1064 |
to store the slab\_t . |
to store the slab\_t . |
1249 |
kmem_cache_init_objs(cachep, slabp, ctor_flags); |
kmem_cache_init_objs(cachep, slabp, ctor_flags); |
1250 |
\end{verbatim} |
\end{verbatim} |
1251 |
|
|
1252 |
This function, described earlier will initialise each object that can fit |
This function, described earlier will initialize each object that can fit |
1253 |
on the slab. |
on the slab. |
1254 |
|
|
1255 |
\begin{verbatim} |
\begin{verbatim} |
1258 |
\end{verbatim} |
\end{verbatim} |
1259 |
|
|
1260 |
Lock the cache so the slab can be inserted on the list and say that we are not |
Lock the cache so the slab can be inserted on the list and say that we are not |
1261 |
growing any more so that the cache will be considered for reapin again later. |
growing any more so that the cache will be considered for reaping again later. |
1262 |
|
|
1263 |
\begin{verbatim} |
\begin{verbatim} |
1264 |
/* Make slab active. */ |
/* Make slab active. */ |