The illusion of control is the tendency for people to overestimate their ability to control events. It was named by U.S. psychologist Ellen Langer and is thought to influence gambling behavior and belief in the paranormal. Along with illusory superiority and optimism bias, the illusion of control is one of the positive illusions.
Definition
The
illusion of control is the tendency for people to overestimate their
ability to control events, for example, when someone feels a sense of
control over outcomes that they demonstrably do not influence.
The illusion might arise because a person lacks direct introspective insight into whether they are in control of events. This has been called the introspection illusion.
Instead, they may judge their degree of control by a process which is
often unreliable. As a result, they see themselves as responsible for
events to which there is little or no causal link.
For example, in one study, college students were in a virtual reality
setting to treat a fear of heights using an elevator. Those who were
told that they had control, yet had none, felt as though they had as
much control as those who actually did have control over the elevator.
Those who were led to believe they did not have control said they felt
as though they had little control.
History
Psychological
theorists have consistently emphasized the importance of perceptions of
control over life events. One of the earliest instances was when Alfred Adler argued that people strive for proficiency in their lives. Heider
later proposed that humans have a strong motive to control their
environment and Wyatt Mann hypothesized a basic competence motive that
people satisfy by exerting control. Wiener, an attribution theorist, modified his original theory of achievement motivation to include a controllability dimension. Kelley
then argued that people's failure to detect noncontingencies may result
in their attributing uncontrollable outcomes to personal causes. Nearer
to the present, Taylor and Brown argued that positive illusions, including the illusion of control, foster mental health.
The effect was named by U.S. psychologist Ellen Langer and has been replicated in many different contexts.
Occurrence
The illusion is more common in familiar situations, and in situations where the person knows the desired outcome.
Feedback that emphasizes success rather than failure can increase the
effect, while feedback that emphasizes failure can decrease or reverse
the effect. The illusion is weaker for depressed individuals and is stronger when individuals have an emotional need to control the outcome. The illusion is strengthened by stressful and competitive situations, including financial trading.
Although people are likely to overestimate their control when the
situations are heavily chance-determined, they also tend to
underestimate their control when they actually have it, which runs
contrary to some theories of the illusion and its adaptiveness.
People also showed a higher illusion of control when they were allowed
to become familiar with a task through practice trials, make their
choice before the event happens like with throwing dice, and when they
can make their choice rather than have it made for them with the same
odds. People are more likely to show control when they have more
answers right at the beginning than at the end, even when the people had
the same number of correct answers.
Being in a position of power enhances the illusion of control, which may lead to overreach in risk taking.
By proxy
At
times, people attempt to gain control by transferring responsibility to
more capable or “luckier” others to act for them. By forfeiting direct
control, it is perceived to be a valid way of maximizing outcomes. This
illusion of control by proxy is a significant theoretical extension of
the traditional illusion of control model. People will of course give up
control if another person is thought to have more knowledge or skill in
areas such as medicine where actual skill and knowledge are involved.
In cases like these it is entirely rational to give up responsibility to
people such as doctors. However, when it comes to events of pure
chance, allowing another to make decisions (or gamble) on one's behalf,
because they are seen as luckier is not rational and would go against
people's well-documented desire for control in uncontrollable
situations. However, it does seem plausible since people generally
believe that they can possess luck and employ it to advantage in games
of chance, and it is not a far leap that others may also be seen as
lucky and able to control uncontrollable events.
In a study conducted in Singapore, the perception of control, luck, and
skill when gambling led to an increase in gambling behavior.
In one instance, a lottery pool at a company decides who picks
the numbers and buys the tickets based on the wins and losses of each
member. The member with the best record becomes the representative until
they accumulate a certain number of losses and then a new
representative is picked based on wins and losses. Even though no member
is truly better than the other and it is all by chance, they still
would rather have someone with seemingly more luck to have control over
them.
In another real-world example, in the 2002 Olympics men's and
women's hockey finals, Team Canada beat Team USA. Prior to the match, a Canadian coin
was secretly placed under the ice before the game, an action which the
players and officials believed would bring them luck. The members of
Team Canada were the only people who knew the coin had been placed
there. The coin was later put in the Hockey Hall of Fame where there was
an opening so people could touch it. People believed they could
transfer luck from the coin to themselves by touching it, and thereby
change their own luck..
Demonstration
The
illusion of control is demonstrated by three converging lines of
evidence: 1) laboratory experiments, 2) observed behavior in familiar
games of chance such as lotteries, and 3) self-reports of real-world
behavior.
Laboratory experiments
One
kind of laboratory demonstration involves two lights marked "Score" and
"No Score". Subjects have to try to control which one lights up. In one
version of this experiment, subjects could press either of two buttons. Another version had one button, which subjects decided on each trial to press or not.
Subjects had a variable degree of control over the lights, or none at
all, depending on how the buttons were connected. The experimenters made
clear that there might be no relation between the subjects' actions and
the lights.
Subjects estimated how much control they had over the lights. These
estimates bore no relation to how much control they actually had, but
was related to how often the "Score" light lit up. Even when their
choices made no difference at all, subjects confidently reported
exerting some control over the lights.
Observed behavior in games
Ellen
Langer's research demonstrated that people were more likely to behave
as if they could exercise control in a chance situation where "skill
cues" were present.
By skill cues, Langer meant properties of the situation more normally
associated with the exercise of skill, in particular the exercise of
choice, competition, familiarity with the stimulus and involvement in
decisions. One simple form of this effect is found in casinos: when rolling dice in a craps game people tend to throw harder when they need high numbers and softer for low numbers.
In another experiment, subjects had to predict the outcome of
thirty coin tosses. The feedback was rigged so that each subject was
right exactly half the time, but the groups differed in where their
"hits" occurred. Some were told that their early guesses were accurate.
Others were told that their successes were distributed evenly through
the thirty trials. Afterwards, they were surveyed about their
performance. Subjects with early "hits" overestimated their total
successes and had higher expectations of how they would perform on
future guessing games. This result resembles the irrational primacy effect in which people give greater weight to information that occurs earlier in a series.
Forty percent of the subjects believed their performance on this chance
task would improve with practice, and twenty-five percent said that
distraction would impair their performance.
Another of Langer's experiments replicated by other researchers
involves a lottery. Subjects are either given tickets at random or
allowed to choose their own. They can then trade their tickets for
others with a higher chance of paying out. Subjects who had chosen their
own ticket were more reluctant to part with it. Tickets bearing
familiar symbols were less likely to be exchanged than others with
unfamiliar symbols. Although these lotteries were random, subjects
behaved as though their choice of ticket affected the outcome. Participants who chose their own numbers were less likely to trade their ticket even for one in a game with better odds.
Self reported behavior
Yet
another way to investigate perceptions of control is to ask people
about hypothetical situations, for example their likelihood of being
involved in a motor vehicle accident. On average, drivers regard
accidents as much less likely in "high-control" situations, such as when
they are driving, than in "low-control" situations, such as when they
are in the passenger seat. They also rate a high-control accident, such
as driving into the car in front, as much less likely than a low-control
accident such as being hit from behind by another driver.
Explanations
Ellen
Langer, who first demonstrated the illusion of control, explained her
findings in terms of a confusion between skill and chance situations.
She proposed that people base their judgments of control on "skill
cues". These are features of a situation that are usually associated
with games of skill, such as competitiveness, familiarity and individual
choice. When more of these skill cues are present, the illusion is
stronger.
In 1998, Suzanne Thompson and colleagues argued that Langer's
explanation was inadequate to explain all the variations in the effect.
As an alternative, they proposed that judgments about control are based
on a procedure that they called the "control heuristic".
This theory proposes that judgments of control depend on two
conditions; an intention to create the outcome, and a relationship
between the action and outcome. In games of chance, these two conditions
frequently go together. As well as an intention to win, there is an
action, such as throwing a die or pulling a lever on a slot machine,
which is immediately followed by an outcome. Even though the outcome is
selected randomly, the control heuristic would result in the player
feeling a degree of control over the outcome.
Self-regulation theory
offers another explanation. To the extent that people are driven by
internal goals concerned with the exercise of control over their
environment, they will seek to reassert control in conditions of chaos,
uncertainty or stress. One way of coping with a lack of real control is
to falsely attribute oneself control of the situation.
The core self-evaluations (CSE) trait is a stable personality trait composed of locus of control, neuroticism, self-efficacy, and self-esteem.
While those with high core self-evaluations are likely to believe that
they control their own environment (i.e., internal locus of control), very high levels of CSE may lead to the illusion of control.
In 1988 Taylor and Brown have argued that positive illusions,
including the illusion of control, are adaptive as they motivate people
to persist at tasks when they might otherwise give up. This position is supported by Albert Bandura's
claim in 1989 that "optimistic self-appraisals of capability, that are
not unduly disparate from what is possible, can be advantageous, whereas
veridical judgements can be self-limiting".
His argument is essentially concerned with the adaptive effect of
optimistic beliefs about control and performance in circumstances where
control is possible, rather than perceived control in circumstances
where outcomes do not depend on an individual's behavior.
In 1997 Bandura also suggested that:
"In activities where the margins of error are narrow and
missteps can produce costly or injurious consequences, personal
well-being is best served by highly accurate efficacy appraisal."
Taylor and Brown argue that positive illusions are adaptive,
since there is evidence that they are more common in normally mentally
healthy individuals than in depressed individuals. However, in 1998
Pacini, Muir and Epstein showed that this may be because depressed
people overcompensate for a tendency toward maladaptive intuitive
processing by exercising excessive rational control in trivial
situations, and note that the difference with non-depressed people
disappears in more consequential circumstances.
There is also empirical evidence
that high self-efficacy can be maladaptive in some circumstances. In a
scenario-based study, Whyte et al. showed in 1997 that participants in
whom they had induced high self-efficacy were significantly more likely
to escalate commitment to a failing course of action.
In 1998 Knee and Zuckerman challenged the definition of mental health
used by Taylor and Brown and argue that lack of illusions is associated
with a non-defensive personality oriented towards growth and learning
and with low ego involvement in outcomes. They present evidence that self-determined individuals are less prone to these illusions.
In the late 1970s, Abramson and Alloy
demonstrated that depressed individuals held a more accurate view than
their non-depressed counterparts in a test which measured illusion of
control.
This finding held true even when the depression was manipulated
experimentally. However, when replicating the findings Msetfi et al.
(2005, 2007) found that the overestimation of control in nondepressed
people only showed up when the interval was long enough, implying that
this is because they take more aspects of a situation into account than
their depressed counterparts.
Also, Dykman et al. (1989) showed that depressed people believe they
have no control in situations where they actually do, so their
perception is not more accurate overall.
Allan et al. (2007) has proposed that the pessimistic bias of
depressives resulted in "depressive realism" when asked about estimation
of control, because depressed individuals are more likely to say no
even if they have control.
A number of studies have found a link between a sense of control and health, especially in older people.
This link for older people having improved health because of a sense of
control was discussed in a study conducted in a nursing home. As the
residents at the nursing home were encouraged to make more choices for
themselves, there was more sense of control over their daily lives. This
increase in control increased their overall happiness and health
compared to those not making as many decisions for themselves. It was
even speculated that with results so promising could slow down or
reverse cognitive decline that may occur with aging.
Fenton-O'Creevy et al. argue, as do Gollwittzer and Kinney in 1998,
that while illusory beliefs about control may promote goal striving,
they are not conducive to sound decision-making. Illusions of control
may cause insensitivity to feedback, impede learning and predispose
toward greater objective risk taking (since subjective risk will be
reduced by illusion of control).
Applications
Psychologist Daniel Wegner argues that an illusion of control over external events underlies belief in psychokinesis, a supposed paranormal ability to move objects directly using the mind. As evidence, Wegner cites a series of experiments on magical thinking in which subjects were induced to think they had influenced external events. In one experiment, subjects watched a basketball player taking a series of free throws. When they were instructed to visualise him making his shots, they felt that they had contributed to his success.
A study published in 2003 examined traders working in the City of London's investment banks.
They each watched a graph being plotted on a computer screen, similar
to a real-time graph of a stock price or index. Using three computer
keys, they had to raise the value as high as possible. They were warned
that the value showed random variations, but that the keys might have
some effect. In fact, the fluctuations were not affected by the keys.
The traders' ratings of their success measured their susceptibility to
the illusion of control. This score was then compared with each trader's
performance. Those who were more prone to the illusion scored
significantly lower on analysis, risk management and contribution to profits. They also earned significantly less.
An oversimplification of how a kernel connects application software to the hardware of a computer
The kernel is a computer program at the core of a computer's operating system and generally has complete control over everything in the system. It is the portion of the operating system code that is always resident in memory
and facilitates interactions between hardware and software components. A
full kernel controls all hardware resources (e.g. I/O, memory,
cryptography) via device drivers,
arbitrates conflicts between processes concerning such resources, and
optimizes the utilization of common resources e.g. CPU & cache
usage, file systems, and network sockets. On most systems, the kernel is
one of the first programs loaded on startup (after the bootloader). It handles the rest of startup as well as memory, peripherals, and input/output (I/O) requests from software, translating them into data-processing instructions for the central processing unit.
The critical code of the kernel is usually loaded into a separate area of memory, which is protected from access by application software
or other less critical parts of the operating system. The kernel
performs its tasks, such as running processes, managing hardware devices
such as the hard disk, and handling interrupts, in this protected kernel space.
In contrast, application programs such as browsers, word processors, or
audio or video players use a separate area of memory, user space. This separation prevents user data and kernel data from interfering with each other and causing instability and slowness,
as well as preventing malfunctioning applications from affecting other
applications or crashing the entire operating system. Even in systems
where the kernel is included in application address spaces, memory protection is used to prevent unauthorized applications from modifying the kernel.
There are different kernel architecture designs. Monolithic kernels run entirely in a single address space with the CPU executing in supervisor mode, mainly for speed. Microkernels run most but not all of their services in user space, like user processes do, mainly for resilience and modularity. MINIX 3 is a notable example of microkernel design. Instead, the Linux kernel is monolithic, although it is also modular, for it can insert and remove loadable kernel modules at runtime.
This central component of a computer system is responsible for
executing programs. The kernel takes responsibility for deciding at any
time which of the many running programs should be allocated to the
processor or processors.
Random-access memory
Random-access memory (RAM) is used to store both program instructions and data.
Typically, both need to be present in memory in order for a program to
execute. Often multiple programs will want access to memory, frequently
demanding more memory than the computer has available. The kernel is
responsible for deciding which memory each process can use, and
determining what to do when not enough memory is available.
Input/output devices
I/O devices include peripherals such as keyboards, mice, disk drives, printers, USB devices, network adapters, and display devices.
The kernel provides convenient methods for applications to use these
devices which are typically abstracted by the kernel so that
applications do not need to know their implementation details.
Resource management
Key aspects necessary in resource management are defining the execution domain (address space) and the protection mechanism used to mediate access to the resources within a domain. Kernels also provide methods for synchronization and inter-process communication
(IPC). These implementations may be located within the kernel itself or
the kernel can also rely on other processes it is running. Although the
kernel must provide IPC in order to provide access to the facilities
provided by each other, kernels must also provide running programs with a
method to make requests to access these facilities. The kernel is also
responsible for context switching between processes or threads.
The kernel has full access to the system's memory and must allow
processes to safely access this memory as they require it. Often the
first step in doing this is virtual addressing, usually achieved by paging and/or segmentation.
Virtual addressing allows the kernel to make a given physical address
appear to be another address, the virtual address. Virtual address
spaces may be different for different processes; the memory that one
process accesses at a particular (virtual) address may be different
memory from what another process accesses at the same address. This
allows every program to behave as if it is the only one (apart from the
kernel) running and thus prevents applications from crashing each other.
On many systems, a program's virtual address may refer to data
which is not currently in memory. The layer of indirection provided by
virtual addressing allows the operating system to use other data stores,
like a hard drive, to store what would otherwise have to remain in main memory (RAM).
As a result, operating systems can allow programs to use more memory
than the system has physically available. When a program needs data
which is not currently in RAM, the CPU signals to the kernel that this
has happened, and the kernel responds by writing the contents of an
inactive memory block to disk (if necessary) and replacing it with the
data requested by the program. The program can then be resumed from the
point where it was stopped. This scheme is generally known as demand paging.
Virtual addressing also allows creation of virtual partitions of
memory in two disjointed areas, one being reserved for the kernel (kernel space) and the other for the applications (user space).
The applications are not permitted by the processor to address kernel
memory, thus preventing an application from damaging the running kernel.
This fundamental partition of memory space has contributed much to the
current designs of actual general-purpose kernels and is almost
universal in such systems, although some research kernels (e.g., Singularity) take other approaches.
Device management
To perform useful functions, processes need access to the peripherals connected to the computer, which are controlled by the kernel through device drivers. A device driver is a computer program encapsulating, monitoring and controlling a hardware device (via its Hardware/Software Interface (HSI))
on behalf of the OS. It provides the operating system with an API,
procedures and information about how to control and communicate with a
certain piece of hardware. Device drivers are an important and vital
dependency for all OS and their applications. The design goal of a
driver is abstraction; the function of the driver is to translate the
OS-mandated abstract function calls (programming calls) into
device-specific calls. In theory, a device should work correctly with a
suitable driver. Device drivers are used for e.g. video cards, sound
cards, printers, scanners, modems, and Network cards.
At the hardware level, common abstractions of device drivers include:
Using a lower-level device driver (file drivers using disk drivers)
Simulating work with hardware, while doing something entirely different
And at the software level, device driver abstractions include:
Allowing the operating system direct access to hardware resources
Only implementing primitives
Implementing an interface for non-driver software such as TWAIN
Implementing a language (often a high-level language such as PostScript)
For example, to show the user something on the screen, an application
would make a request to the kernel, which would forward the request to
its display driver, which is then responsible for actually plotting the
character/pixel.
A kernel must maintain a list of available devices. This list may be known in advance (e.g., on an embedded system
where the kernel will be rewritten if the available hardware changes),
configured by the user (typical on older PCs and on systems that are not
designed for personal use) or detected by the operating system at run
time (normally called plug and play). In plug-and-play systems, a device manager first performs a scan on different peripheral buses, such as Peripheral Component Interconnect (PCI) or Universal Serial Bus (USB), to detect installed devices, then searches for the appropriate drivers.
As device management is a very OS-specific
topic, these drivers are handled differently by each kind of kernel
design, but in every case, the kernel has to provide the I/O to allow drivers to physically access their devices through some port
or memory location. Important decisions have to be made when designing
the device management system, as in some designs accesses may involve context switches, making the operation very CPU-intensive and easily causing a significant performance overhead.
In computing, a system call is how a process requests a service from
an operating system's kernel that it does not normally have permission
to run. System calls provide the interface between a process and the
operating system. Most operations interacting with the system require
permissions not available to a user-level process, e.g., I/O performed
with a device present on the system, or any form of communication with
other processes requires the use of system calls.
A system call is a mechanism that is used by the application program to request a service from the operating system. They use a machine-code
instruction that causes the processor to change mode. An example would
be from supervisor mode to protected mode. This is where the operating
system performs actions like accessing hardware devices or the memory management unit.
Generally the operating system provides a library that sits between the
operating system and normal user programs. Usually it is a C library such as Glibc
or Windows API. The library handles the low-level details of passing
information to the kernel and switching to supervisor mode. System calls
include close, open, read, wait and write.
To actually perform useful work, a process must be able to access
the services provided by the kernel. This is implemented differently by
each kernel, but most provide a C library or an API, which in turn invokes the related kernel functions.
The method of invoking the kernel function varies from kernel to
kernel. If memory isolation is in use, it is impossible for a user
process to call the kernel directly, because that would be a violation
of the processor's access control rules. A few possibilities are:
Using a software-simulated interrupt. This method is available on most hardware, and is therefore very common.
Using a call gate.
A call gate is a special address stored by the kernel in a list in
kernel memory at a location known to the processor. When the processor
detects a call to that address, it instead redirects to the target
location without causing an access violation. This requires hardware
support, but the hardware for it is quite common.
Using a special system call instruction. This technique requires special hardware support, which common architectures (notably, x86)
may lack. System call instructions have been added to recent models of
x86 processors, however, and some operating systems for PCs make use of
them when available.
Using a memory-based queue. An application that makes large numbers
of requests but does not need to wait for the result of each may add
details of requests to an area of memory that the kernel periodically
scans to find requests.
The mechanisms or policies provided by the kernel can be
classified according to several criteria, including: static (enforced at
compile time) or dynamic (enforced at run time); pre-emptive or post-detection; according to the protection principles they satisfy (e.g., Denning);
whether they are hardware supported or language based; whether they
are more an open mechanism or a binding policy; and many more.
Support for hierarchical protection domains is typically implemented using CPU modes.
Many kernels provide implementation of "capabilities", i.e.,
objects that are provided to user code which allow limited access to an
underlying object managed by the kernel. A common example is file
handling: a file is a representation of information stored on a
permanent storage device. The kernel may be able to perform many
different operations, including read, write, delete or execute, but a
user-level application may only be permitted to perform some of these
operations (e.g., it may only be allowed to read the file). A common
implementation of this is for the kernel to provide an object to the
application (typically so called a "file handle") which the application
may then invoke operations on, the validity of which the kernel checks
at the time the operation is requested. Such a system may be extended to
cover all objects that the kernel manages, and indeed to objects
provided by other user applications.
An efficient and simple way to provide hardware support of capabilities is to delegate to the memory management unit (MMU) the responsibility of checking access-rights for every memory access, a mechanism called capability-based addressing. Most commercial computer architectures lack such MMU support for capabilities.
An alternative approach is to simulate capabilities using
commonly supported hierarchical domains. In this approach, each
protected object must reside in an address space that the application
does not have access to; the kernel also maintains a list of
capabilities in such memory. When an application needs to access an
object protected by a capability, it performs a system call and the
kernel then checks whether the application's capability grants it
permission to perform the requested action, and if it is permitted
performs the access for it (either directly, or by delegating the
request to another user-level process). The performance cost of address
space switching limits the practicality of this approach in systems with
complex interactions between objects, but it is used in current
operating systems for objects that are not accessed frequently or which
are not expected to perform quickly.
If the firmware does not support protection mechanisms, it is
possible to simulate protection at a higher level, for example by
simulating capabilities by manipulating page tables, but there are performance implications. Lack of hardware support may not be an issue, however, for systems that choose to use language-based protection.
An important kernel design decision is the choice of the
abstraction levels where the security mechanisms and policies should be
implemented. Kernel security mechanisms play a critical role in
supporting security at higher levels.
One approach is to use firmware and kernel support for fault
tolerance (see above), and build the security policy for malicious
behavior on top of that (adding features such as cryptography mechanisms where necessary), delegating some responsibility to the compiler. Approaches that delegate enforcement of security policy to the compiler and/or the application level are often called language-based security.
The lack of many critical security mechanisms in current
mainstream operating systems impedes the implementation of adequate
security policies at the application abstraction level.
In fact, a common misconception in computer security is that any
security policy can be implemented in an application regardless of
kernel support.
According to Mars Research Group developers, a lack of isolation is one of the main factors undermining kernel security. They propose their driver isolation framework for protection, primarily in the Linux kernel.
Hardware- or language-based protection
Typical
computer systems today use hardware-enforced rules about what programs
are allowed to access what data. The processor monitors the execution
and stops a program that violates a rule, such as a user process that
tries to write to kernel memory. In systems that lack support for
capabilities, processes are isolated from each other by using separate
address spaces.
Calls from user processes into the kernel are regulated by requiring
them to use one of the above-described system call methods.
An alternative approach is to use language-based protection. In a language-based protection system, the kernel will only allow code to execute that has been produced by a trusted language compiler.
The language may then be designed such that it is impossible for the
programmer to instruct it to do something that will violate a security
requirement.
Advantages of this approach include:
No need for separate address spaces. Switching between address
spaces is a slow operation that causes a great deal of overhead, and a
lot of optimization work is currently performed in order to prevent
unnecessary switches in current operating systems. Switching is
completely unnecessary in a language-based protection system, as all
code can safely operate in the same address space.
Flexibility. Any protection scheme that can be designed to be
expressed via a programming language can be implemented using this
method. Changes to the protection scheme (e.g. from a hierarchical
system to a capability-based one) do not require new hardware.
Disadvantages include:
Longer application startup time. Applications must be verified
when they are started to ensure they have been compiled by the correct
compiler, or may need recompiling either from source code or from bytecode.
Inflexible type systems. On traditional systems, applications frequently perform operations that are not type safe.
Such operations cannot be permitted in a language-based protection
system, which means that applications may need to be rewritten and may,
in some cases, lose performance.
Examples of systems with language-based protection include JX and Microsoft's Singularity.
Process cooperation
Edsger Dijkstra proved that from a logical point of view, atomiclock and unlock operations operating on binary semaphores are sufficient primitives to express any functionality of process cooperation. However this approach is generally held to be lacking in terms of safety and efficiency, whereas a message passing approach is more flexible.
A number of other approaches (either lower- or higher-level) are
available as well, with many modern kernels providing support for
systems such as shared memory and remote procedure calls.
I/O device management
The
idea of a kernel where I/O devices are handled uniformly with other
processes, as parallel co-operating processes, was first proposed and
implemented by Brinch Hansen (although similar ideas were suggested in 1967). In Hansen's description of this, the "common" processes are called internal processes, while the I/O devices are called external processes.
Similar to physical memory, allowing applications direct access
to controller ports and registers can cause the controller to
malfunction, or system to crash. With this, depending on the complexity
of the device, some devices can get surprisingly complex to program, and
use several different controllers. Because of this, providing a more
abstract interface to manage the device is important. This interface is
normally done by a device driver
or hardware abstraction layer. Frequently, applications will require
access to these devices. The kernel must maintain the list of these
devices by querying the system for them in some way. This can be done
through the BIOS, or through one of the various system buses (such as
PCI/PCIE, or USB). Using an example of a video driver, when an
application requests an operation on a device, such as displaying a
character, the kernel needs to send this request to the current active
video driver. The video driver, in turn, needs to carry out this
request. This is an example of inter-process communication (IPC).
Kernel-wide design approaches
The above listed tasks and features can be provided in many ways that differ from each other in design and implementation.
The principle of separation of mechanism and policy is the substantial difference between the philosophy of micro and monolithic kernels. Here a mechanism
is the support that allows the implementation of many different
policies, while a policy is a particular "mode of operation". Example:
Mechanism: User login attempts are routed to an authorization server
Policy: Authorization server requires a password which is verified against stored passwords in a database
Because the mechanism and policy are separated, the policy can be easily changed to e.g. require the use of a security token.
In minimal microkernel just some very basic policies are included,
and its mechanisms allows what is running on top of the kernel (the
remaining part of the operating system and the other applications) to
decide which policies to adopt (as memory management, high level process
scheduling, file system management, etc.). A monolithic kernel instead tends to include many policies, therefore restricting the rest of the system to rely on them.
Per Brinch Hansen presented arguments in favour of separation of mechanism and policy.
The failure to properly fulfill this separation is one of the major
causes of the lack of substantial innovation in existing operating
systems, a problem common in computer architecture. The monolithic design is induced by the "kernel mode"/"user mode" architectural approach to protection (technically called hierarchical protection domains), which is common in conventional commercial systems; in fact, every module needing protection is therefore preferably included into the kernel. This link between monolithic design and "privileged mode" can be reconducted to the key issue of mechanism-policy separation;
in fact the "privileged mode" architectural approach melds together the
protection mechanism with the security policies, while the major
alternative architectural approach, capability-based addressing, clearly distinguishes between the two, leading naturally to a microkernel design (see Separation of protection and security).
While monolithic kernels execute all of their code in the same address space (kernel space), microkernels try to run most of their services in user space, aiming to improve maintainability and modularity of the codebase.
Most kernels do not fit exactly into one of these categories, but are
rather found in between these two designs. These are called hybrid kernels. More exotic designs such as nanokernels and exokernels are available, but are seldom used for production systems. The Xen hypervisor, for example, is an exokernel.
In a monolithic kernel, all OS services run along with the main
kernel thread, thus also residing in the same memory area. This approach
provides rich and powerful hardware access. UNIX developer Ken Thompson stated that "it is in [his] opinion easier to implement a
monolithic kernel".
The main disadvantages of monolithic kernels are the dependencies
between system components – a bug in a device driver might crash the
entire system – and the fact that large kernels can become very
difficult to maintain; Thompson also stated that "It is also easier for
[a monolithic kernel] to turn into a mess in a hurry as it is modified."
Monolithic kernels, which have traditionally been used by
Unix-like operating systems, contain all the operating system core
functions and the device drivers. A monolithic kernel is one single
program that contains all of the code necessary to perform every
kernel-related task. Every part which is to be accessed by most programs
which cannot be put in a library is in the kernel space: Device
drivers, scheduler, memory handling, file systems, and network stacks.
Many system calls are provided to applications, to allow them to access
all those services. A monolithic kernel, while initially loaded with
subsystems that may not be needed, can be tuned to a point where it is
as fast as or faster than the one that was specifically designed for the
hardware, although more relevant in a general sense.
Modern monolithic kernels, such as the Linux kernel, the FreeBSD kernel, the AIX kernel, the HP-UX kernel, and the Solaris kernel, all of which fall into the category of Unix-like operating systems, support loadable kernel modules,
allowing modules to be loaded into the kernel at runtime, permitting
easy extension of the kernel's capabilities as required, while helping
to minimize the amount of code running in kernel space.
Most work in the monolithic kernel is done via system calls.
These are interfaces, usually kept in a tabular structure, that access
some subsystem within the kernel such as disk operations. Essentially
calls are made within programs and a checked copy of the request is
passed through the system call. Hence, not far to travel at all. The
monolithic Linux kernel can be made extremely small not only because of
its ability to dynamically load modules but also because of its ease of
customization. In fact, there are some versions that are small enough to
fit together with a large number of utilities and other programs on a
single floppy disk and still provide a fully functional operating system
(one of the most popular of which is muLinux). This ability to miniaturize its kernel has also led to a rapid growth in the use of Linux in embedded systems.
These types of kernels consist of the core functions of the
operating system and the device drivers with the ability to load modules
at runtime. They provide rich and powerful abstractions of the
underlying hardware. They provide a small set of simple hardware
abstractions and use applications called servers to provide more
functionality. This particular approach defines a high-level virtual
interface over the hardware, with a set of system calls to implement
operating system services such as process management, concurrency and
memory management in several modules that run in supervisor mode.
This design has several flaws and limitations:
Coding in kernel can be challenging, in part because one cannot use common libraries (like a full-featured libc), and because one needs to use a source-level debugger like gdb.
Rebooting the computer is often required. This is not just a problem
of convenience to the developers. When debugging is harder, and as
difficulties become stronger, it becomes more likely that code will be
"buggier".
Bugs in one part of the kernel have strong side effects; since every
function in the kernel has all the privileges, a bug in one function
can corrupt data structure of another, totally unrelated part of the
kernel, or of any running program.
Kernels often become very large and difficult to maintain.
Even if the modules servicing these operations are separate from the
whole, the code integration is tight and difficult to do correctly.
Since the modules run in the same address space, a bug can bring down the entire system.
In the microkernel approach, the kernel itself only provides basic functionality that allows the execution of servers, separate programs that assume former kernel functions, such as device drivers, GUI servers, etc.
Microkernel (also abbreviated μK or uK) is the term describing an
approach to operating system design by which the functionality of the
system is moved out of the traditional "kernel", into a set of "servers"
that communicate through a "minimal" kernel, leaving as little as
possible in "system space" and as much as possible in "user space". A
microkernel that is designed for a specific platform or device is only
ever going to have what it needs to operate. The microkernel approach
consists of defining a simple abstraction over the hardware, with a set
of primitives or system calls to implement minimal OS services such as memory management, multitasking, and inter-process communication. Other services, including those normally provided by the kernel, such as networking, are implemented in user-space programs, referred to as servers. Microkernels are easier to maintain than monolithic kernels, but the large number of system calls and context switches might slow down the system because they typically generate more overhead than plain function calls.
Only parts which really require being in a privileged mode are in
kernel space: IPC (Inter-Process Communication), basic scheduler, or
scheduling primitives, basic memory handling, basic I/O primitives. Many
critical parts are now running in user space: The complete scheduler,
memory handling, file systems, and network stacks. Micro kernels were
invented as a reaction to traditional "monolithic" kernel design,
whereby all system functionality was put in a one static program running
in a special "system" mode of the processor. In the microkernel, only
the most fundamental of tasks are performed such as being able to access
some (not necessarily all) of the hardware, manage memory and
coordinate message passing between the processes. Some systems that use
micro kernels are QNX and the HURD. In the case of QNX and Hurd
user sessions can be entire snapshots of the system itself or views as
it is referred to. The very essence of the microkernel architecture
illustrates some of its advantages:
Easier to maintain
Patches can be tested in a separate instance, and then swapped in to take over a production instance.
Rapid development time and new software can be tested without having to reboot the kernel.
More persistence in general, if one instance goes haywire, it is often possible to substitute it with an operational mirror.
Most microkernels use a message passing system to handle requests from one server to another. The message passing system generally operates on a port
basis with the microkernel. As an example, if a request for more memory
is sent, a port is opened with the microkernel and the request sent
through. Once within the microkernel, the steps are similar to system
calls. The rationale was that it would bring modularity in the system
architecture, which would entail a cleaner system, easier to debug or
dynamically modify, customizable to users' needs, and more performing.
They are part of the operating systems like GNU Hurd, MINIX, MkLinux, QNX and Redox OS.
Although microkernels are very small by themselves, in combination
with all their required auxiliary code they are, in fact, often larger
than monolithic kernels. Advocates of monolithic kernels also point out
that the two-tiered structure of microkernel systems, in which most of
the operating system does not interact directly with the hardware,
creates a not-insignificant cost in terms of system efficiency. These
types of kernels normally provide only the minimal services such as
defining memory address spaces, inter-process communication (IPC) and
the process management. The other functions such as running the hardware
processes are not handled directly by microkernels. Proponents of micro
kernels point out those monolithic kernels have the disadvantage that
an error in the kernel can cause the entire system to crash. However,
with a microkernel, if a kernel process crashes, it is still possible to
prevent a crash of the system as a whole by merely restarting the
service that caused the error.
Other services provided by the kernel such as networking are implemented in user-space programs referred to as servers.
Servers allow the operating system to be modified by simply starting
and stopping programs. For a machine without networking support, for
instance, the networking server is not started. The task of moving in
and out of the kernel to move data between the various applications and
servers creates overhead which is detrimental to the efficiency of micro
kernels in comparison with monolithic kernels.
Disadvantages in the microkernel exist however. Some are:
More software for interfacing is required, there is a potential for performance loss.
Messaging bugs can be harder to fix due to the longer trip they have to take versus the one off copy in a monolithic kernel.
Process management in general can be very complicated.
The disadvantages for microkernels are extremely context-based. As an
example, they work well for small single-purpose (and critical) systems
because if not many processes need to run, then the complications of
process management are effectively mitigated.
A microkernel allows the implementation of the remaining part of
the operating system as a normal application program written in a high-level language,
and the use of different operating systems on top of the same unchanged
kernel. It is also possible to dynamically switch among operating
systems and to have more than one active simultaneously.
Monolithic kernels vs. microkernels
As the computer kernel grows, so grows the size and vulnerability of its trusted computing base; and, besides reducing security, there is the problem of enlarging the memory footprint. This is mitigated to some degree by perfecting the virtual memory system, but not all computer architectures have virtual memory support.
To reduce the kernel's footprint, extensive editing has to be performed
to carefully remove unneeded code, which can be very difficult with
non-obvious interdependencies between parts of a kernel with millions of
lines of code.
By the early 1990s, due to the various shortcomings of monolithic
kernels versus microkernels, monolithic kernels were considered
obsolete by virtually all operating system researchers. As a result, the design of Linux as a monolithic kernel rather than a microkernel was the topic of a famous debate between Linus Torvalds and Andrew Tanenbaum. There is merit on both sides of the argument presented in the Tanenbaum–Torvalds debate.
Performance
Monolithic kernels are designed to have all of their code in the same address space (kernel space), which some developers argue is necessary to increase the performance of the system. Some developers also maintain that monolithic systems are extremely efficient if well written. The monolithic model tends to be more efficient through the use of shared kernel memory, rather than the slower IPC system of microkernel designs, which is typically based on message passing.
The performance of microkernels was poor in both the 1980s and early 1990s.
However, studies that empirically measured the performance of these
microkernels did not analyze the reasons of such inefficiency.
The explanations of this data were left to "folklore", with the
assumption that they were due to the increased frequency of switches
from "kernel-mode" to "user-mode", to the increased frequency of inter-process communication and to the increased frequency of context switches.
In fact, as guessed in 1995, the reasons for the poor performance
of microkernels might as well have been: (1) an actual inefficiency of
the whole microkernel approach, (2) the particular concepts implemented in those microkernels, and (3) the particular implementation
of those concepts. Therefore it remained to be studied if the solution
to build an efficient microkernel was, unlike previous attempts, to
apply the correct construction techniques.
On the other end, the hierarchical protection domains architecture that leads to the design of a monolithic kernel
has a significant performance drawback each time there's an interaction
between different levels of protection (i.e., when a process has to
manipulate a data structure both in "user mode" and "supervisor mode"),
since this requires message copying by value.
The hybrid kernel approach combines the speed and simpler design of a monolithic kernel with the modularity and execution safety of a microkernel
Hybrid kernels are used in most commercial operating systems such as Microsoft Windows NT 3.1, NT 3.5, NT 3.51, NT 4.0, 2000, XP, Vista, 7, 8, 8.1 and 10. Apple's own macOS uses a hybrid kernel called XNU which is based upon code from OSF/1's Mach kernel (OSFMK 7.3) and FreeBSD's monolithic kernel.
They are similar to micro kernels, except they include some additional
code in kernel-space to increase performance. These kernels represent a
compromise that was implemented by some developers to accommodate the
major advantages of both monolithic and micro kernels. These types of
kernels are extensions of micro kernels with some properties of
monolithic kernels. Unlike monolithic kernels, these types of kernels
are unable to load modules at runtime on their own. Hybrid kernels are
micro kernels that have some "non-essential" code in kernel-space in
order for the code to run more quickly than it would were it to be in
user-space. Hybrid kernels are a compromise between the monolithic and
microkernel designs. This implies running some services (such as the network stack or the filesystem)
in kernel space to reduce the performance overhead of a traditional
microkernel, but still running kernel code (such as device drivers) as
servers in user space.
Many traditionally monolithic kernels are now at least adding (or
else using) the module capability. The most well known of these kernels
is the Linux kernel. The modular kernel essentially can have parts of
it that are built into the core kernel binary or binaries that load into
memory on demand. It is important to note that a code tainted module
has the potential to destabilize a running kernel. Many people become
confused on this point when discussing micro kernels. It is possible to
write a driver for a microkernel in a completely separate memory space
and test it before "going" live. When a kernel module is loaded, it
accesses the monolithic portion's memory space by adding to it what it
needs, therefore, opening the doorway to possible pollution. A few
advantages to the modular (or) Hybrid kernel are:
Faster development time for drivers that can operate from within
modules. No reboot required for testing (provided the kernel is not
destabilized).
On demand capability versus spending time recompiling a whole kernel for things like new drivers or subsystems.
Faster integration of third party technology (related to development but pertinent unto itself nonetheless).
Modules, generally, communicate with the kernel using a module
interface of some sort. The interface is generalized (although
particular to a given operating system) so it is not always possible to
use modules. Often the device drivers may need more flexibility than the
module interface affords. Essentially, it is two system calls and often
the safety checks that only have to be done once in the monolithic
kernel now may be done twice. Some of the disadvantages of the modular
approach are:
With more interfaces to pass through, the possibility of increased bugs exists (which implies more security holes).
Maintaining modules can be confusing for some administrators when dealing with problems like symbol differences.
A nanokernel delegates virtually all services – including even the most basic ones like interrupt controllers or the timer – to device drivers to make the kernel memory requirement even smaller than a traditional microkernel.
Exokernels are a still-experimental approach to operating system
design. They differ from other types of kernels in limiting their
functionality to the protection and multiplexing of the raw hardware,
providing no hardware abstractions on top of which to develop
applications. This separation of hardware protection from hardware
management enables application developers to determine how to make the
most efficient use of the available hardware for each specific program.
Exokernels in themselves are extremely small. However, they are accompanied by library operating systems (see also unikernel),
providing application developers with the functionalities of a
conventional operating system. This comes down to every user writing
their own rest-of-the kernel from near scratch, which is a very-risky,
complex and quite a daunting assignment - particularly in a
time-constrained production-oriented environment, which is why
exokernels have never caught on.[citation needed]
A major advantage of exokernel-based systems is that they can
incorporate multiple library operating systems, each exporting a
different API, for example one for high level UI development and one for real-time control.
A multikernel operating system treats a multi-core machine as a network of independent cores, as if it were a distributed system. It does not assume shared memory but rather implements inter-process communications as message passing. Barrelfish was the first operating system to be described as a multikernel.
Strictly speaking, an operating system (and thus, a kernel) is not required
to run a computer. Programs can be directly loaded and executed on the
"bare metal" machine, provided that the authors of those programs are
willing to work without any hardware abstraction or operating system
support. Most early computers operated this way during the 1950s and
early 1960s, which were reset and reloaded between the execution of
different programs. Eventually, small ancillary programs such as program loaders and debuggers were left in memory between runs, or loaded from ROM. As these were developed, they formed the basis of what became early operating system kernels. The "bare metal" approach is still used today on some video game consoles and embedded systems, but in general, newer computers use modern operating systems and kernels.
In 1969, the RC 4000 Multiprogramming System
introduced the system design philosophy of a small nucleus "upon which
operating systems for different purposes could be built in an orderly
manner",[46] what would be called the microkernel approach.
In the decade preceding Unix,
computers had grown enormously in power – to the point where computer
operators were looking for new ways to get people to use their spare
time on their machines. One of the major developments during this era
was time-sharing,
whereby a number of users would get small slices of computer time, at a
rate at which it appeared they were each connected to their own,
slower, machine.
The development of time-sharing systems led to a number of
problems. One was that users, particularly at universities where the
systems were being developed, seemed to want to hack the system to get more CPU time. For this reason, security and access control became a major focus of the Multics project in 1965.[48]
Another ongoing issue was properly handling computing resources: users
spent most of their time staring at the terminal and thinking about what
to input instead of actually using the resources of the computer, and a
time-sharing system should give the CPU time to an active user during
these periods. Finally, the systems typically offered a memory hierarchy several layers deep, and partitioning this expensive resource led to major developments in virtual memory systems.
The CommodoreAmiga
was released in 1985, and was among the first – and certainly most
successful – home computers to feature an advanced kernel
architecture. The AmigaOS kernel's executive component, exec.library, uses a microkernel message-passing design, but there are other kernel components, like graphics.library,
that have direct access to the hardware. There is no memory protection,
and the kernel is almost always running in user mode. Only special
actions are executed in kernel mode, and user-mode applications can ask
the operating system to execute their code in kernel mode.
For instance, printers
were represented as a "file" at a known location – when data was
copied to the file, it printed out. Other systems, to provide a similar
functionality, tended to virtualize devices at a lower level – that is,
both devices and files would be instances of some lower level concept. Virtualizing the system at the file level allowed users to manipulate the entire system using their existing file management
utilities and concepts, dramatically simplifying operation. As an
extension of the same paradigm, Unix allows programmers to manipulate
files using a series of small programs, using the concept of pipes,
which allowed users to complete operations in stages, feeding a file
through a chain of single-purpose tools. Although the end result was the
same, using smaller programs in this way dramatically increased
flexibility as well as ease of development and use, allowing the user to
modify their workflow by adding or removing a program from the chain.
In the Unix model, the operating system consists of two
parts: first, the huge collection of utility programs that drive most
operations; second, the kernel that runs the programs.
Under Unix, from a programming standpoint, the distinction between the
two is fairly thin; the kernel is a program, running in supervisor mode, that acts as a program loader and supervisor for the small utility programs making up the rest of the system, and to provide locking and I/O services for these programs; beyond that, the kernel didn't intervene at all in user space.
Over the years the computing model changed, and Unix's treatment of everything as a file or byte stream no longer was as universally applicable as it was before. Although a terminal could be treated as a file or a byte stream, which is printed to or read from, the same did not seem to be true for a graphical user interface. Networking
posed another problem. Even if network communication can be compared to
file access, the low-level packet-oriented architecture dealt with
discrete chunks of data and not with whole files. As the capability of
computers grew, Unix became increasingly cluttered with code. It is also
because the modularity of the Unix kernel is extensively scalable. While kernels might have had 100,000 lines of code in the seventies and eighties, kernels like Linux, of modern Unix successors like GNU, have more than 13 million lines.
Modern Unix-derivatives are generally based on module-loading monolithic kernels. Examples of this are the Linux kernel in the many distributions of GNU, IBM AIX, as well as the Berkeley Software Distribution variant kernels such as FreeBSD, DragonFly BSD, OpenBSD, NetBSD, and macOS. Apart from these alternatives, amateur developers maintain an active operating system development community,
populated by self-written hobby kernels which mostly end up sharing
many features with Linux, FreeBSD, DragonflyBSD, OpenBSD or NetBSD
kernels and/or being compatible with them.
Apple first launched its classic Mac OS in 1984, bundled with its Macintoshpersonal computer. Apple moved to a nanokernel design in Mac OS 8.6. Against this, the modern macOS (originally named Mac OS X) is based on Darwin, which uses a hybrid kernel called XNU, which was created by combining the 4.3BSD kernel and the Mach kernel.
Microsoft Windows was first released in 1985 as an add-on to MS-DOS. Because of its dependence on another operating system, initial releases of Windows, prior to Windows 95, were considered an operating environment (not to be confused with an operating system). This product line continued to evolve through the 1980s and 1990s, with the Windows 9x series adding 32-bit addressing and pre-emptive multitasking; but ended with the release of Windows Me in 2000.
Microsoft also developed Windows NT,
an operating system with a very similar interface, but intended for
high-end and business users. This line started with the release of Windows NT 3.1 in 1993, and was introduced to general users with the release of Windows XP in October 2001—replacing Windows 9x with a completely different, much more sophisticated operating system. This is the line that continues with Windows 11.
The architecture of Windows NT's
kernel is considered a hybrid kernel because the kernel itself contains
tasks such as the Window Manager and the IPC Managers, with a
client/server layered subsystem model. It was designed as a modified microkernel, as the Windows NT kernel was influenced by the Mach microkernel but does not meet all of the criteria of a pure microkernel.
Historically, this term was essentially associated with IBM's line of mainframe operating systems starting with OS/360. In other operating systems, the supervisor is generally called the kernel.
In the 1970s, IBM further abstracted the supervisor state from the hardware, resulting in a hypervisor that enabled full virtualization,
i.e. the capacity to run multiple operating systems on the same machine
totally independently from each other. Hence the first such system was
called Virtual Machine or VM.
Development of microkernels
Although Mach, developed by Richard Rashid at Carnegie Mellon University, is the best-known general-purpose microkernel, other microkernels have been developed with more specific aims. The L4 microkernel family (mainly the L3 and the L4 kernel) was created to demonstrate that microkernels are not necessarily slow. Newer implementations such as Fiasco and Pistachio are able to run Linux next to other L4 processes in separate address spaces.
