January 12, 2004 by Ray Kurzweil
Original link: http://www.kurzweilai.net/kurzweils-law-aka-the-law-of-accelerating-returns
Published on Edge.org and KurzweilAI.net Jan. 12, 2003
In an evolutionary process, positive feedback increases order exponentially. A correlate is that the “returns” of an evolutionary process (such as the speed, cost-effectiveness, or overall “power” of a process) increase exponentially over time — both for biology and technology. Ray Kurzweil submitted on essay based on that premise to Edge.org in response to John Brockman’s question: “What’s your law?”
Original link: http://www.kurzweilai.net/kurzweils-law-aka-the-law-of-accelerating-returns
Published on Edge.org and KurzweilAI.net Jan. 12, 2003
In an evolutionary process, positive feedback increases order exponentially. A correlate is that the “returns” of an evolutionary process (such as the speed, cost-effectiveness, or overall “power” of a process) increase exponentially over time — both for biology and technology. Ray Kurzweil submitted on essay based on that premise to Edge.org in response to John Brockman’s question: “What’s your law?”
Evolution applies positive feedback in that the more capable methods
resulting from one stage of evolutionary progress are used to create the
next stage. Each epoch of evolution has progressed more rapidly by
building on the products of the previous stage.
Evolution works through indirection: evolution created humans, humans created technology, humans are now working with increasingly advanced technology to create new generations of technology. As a result, the rate of progress of an evolutionary process increases exponentially over time.
Over time, the “order” of the information embedded in the evolutionary process (i.e., the measure of how well the information fits a purpose, which in evolution is survival) increases.
A comment on the nature of order. The concept of the “order” of information is important here, as it is not the same as the opposite of disorder. If disorder represents a random sequence of events, then the opposite of disorder should imply “not random.” Information is a sequence of data that is meaningful in a process, such as the DNA code of an organism, or the bits in a computer program. Noise, on the other hand, is a random sequence. Neither noise nor information is predictable. Noise is inherently unpredictable, but carries no information. Information, however, is also unpredictable. If we can predict future data from past data, then that future data stops being information. We might consider an alternating pattern (“0101010. . . .”) to be orderly, but it carries no information (beyond the first couple of bits).
Thus orderliness does not constitute order because order requires information. However, order goes beyond mere information. A recording of radiation levels from space represents information, but if we double the size of this data file, we have increased the amount of data, but we have not achieved a deeper level of order.
Order is information that fits a purpose. The measure of order is the measure of how well the information fits the purpose. In the evolution of life-forms, the purpose is to survive. In an evolutionary algorithm (a computer program that simulates evolution to solve a problem) applied to, say, investing in the stock market, the purpose is to make money. Simply having more information does not necessarily result in a better fit. A superior solution for a purpose may very well involve less data.
The concept of “complexity” is often used to describe the nature of the information created by an evolutionary process. Complexity is a close fit to the concept of order that I am describing, but is also not sufficient. Sometimes, a deeper order – a better fit to a purpose – is achieved through simplification rather than further increases in complexity. For example, a new theory that ties together apparently disparate ideas into one broader more coherent theory reduces complexity but nonetheless may increase the “order for a purpose” that I am describing. Indeed, achieving simpler theories is a driving force in science. Evolution has shown, however, that the general trend towards greater order does generally result in greater complexity.
Thus improving a solution to a problem – which may increase or decrease complexity – increases order. Now that just leaves the issue of defining the problem. Indeed, the key to an evolution algorithm (and to biological and technological evolution) is exactly this: defining the problem.
We may note that this aspect of “Kurzweil’s Law” (the law of accelerating returns) appears to contradict the Second Law of Thermodynamics, which implies that entropy (randomness in a closed system) cannot decrease, and, therefore, generally increases. However, the law of accelerating returns pertains to evolution, and evolution is not a closed system. It takes place amidst great chaos, and indeed depends on the disorder in its midst, from which it draws its options for diversity. And from these options, an evolutionary process continually prunes its choices to create ever greater order. Even a crisis, such as the periodic large asteroids that have crashed into the Earth, although increasing chaos temporarily, end up increasing – deepening – the order created by an evolutionary process.
A primary reason that evolution – of life-forms or of technology – speeds up is that it builds on its own increasing order, with ever more sophisticated means of recording and manipulating information. Innovations created by evolution encourage and enable faster evolution. In the case of the evolution of life forms, the most notable early example is DNA, which provides a recorded and protected transcription of life’s design from which to launch further experiments. In the case of the evolution of technology, ever improving human methods of recording information have fostered further technology. The first computers were designed on paper and assembled by hand. Today, they are designed on computer workstations with the computers themselves working out many details of the next generation’s design, and are then produced in fully-automated factories with human guidance but limited direct intervention.
The evolutionary process of technology seeks to improve capabilities in an exponential fashion. Innovators seek to improve things by multiples. Innovation is multiplicative, not additive. Technology, like any evolutionary process, builds on itself. This aspect will continue to accelerate when the technology itself takes full control of its own progression.
We can thus conclude the following with regard to the evolution of life-forms, and of technology: the law of accelerating returns as applied to an evolutionary process: An evolutionary process is not a closed system; therefore, evolution draws upon the chaos in the larger system in which it takes place for its options for diversity; and evolution builds on its own increasing order. Therefore, in an evolutionary process, order increases exponentially.
A correlate of the above observation is that the “returns” of an evolutionary process (e.g., the speed, cost-effectiveness, or overall “power” of a process) increase exponentially over time. We see this in Moore’s law, in which each new generation of computer chip (now spaced about two years apart) provides twice as many components, each of which operates substantially faster (because of the smaller distances required for the electrons to travel, and other innovations). This exponential growth in the power and price-performance of information-based technologies – now roughly doubling every year – is not limited to computers, but is true for a wide range of technologies, measured many different ways.
In another positive feedback loop, as a particular evolutionary process (e.g., computation) becomes more effective (e.g., cost effective), greater resources are deployed towards the further progress of that process. This results in a second level of exponential growth (i.e., the rate of exponential growth itself grows exponentially). For example, it took three years to double the price-performance of computation at the beginning of the twentieth century, two years around 1950, and is now doubling about once a year. Not only is each chip doubling in power each year for the same unit cost, but the number of chips being manufactured is growing exponentially.
Biological evolution is one such evolutionary process. Indeed it is the quintessential evolutionary process. It took place in a completely open system (as opposed to the artificial constraints in an evolutionary algorithm). Thus many levels of the system evolved at the same time.
Technological evolution is another such evolutionary process. Indeed, the emergence of the first technology-creating species resulted in the new evolutionary process of technology. Therefore, technological evolution is an outgrowth of – and a continuation of – biological evolution. Early stages of humanoid created technology were barely faster than the biological evolution that created our species. Homo sapiens evolved in a few hundred thousand years. Early stages of technology – the wheel, fire, stone tools – took tens of thousands of years to evolve and be widely deployed. A thousand years ago, a paradigm shift such as the printing press, took on the order of a century to be widely deployed. Today, major paradigm shifts, such as cell phones and the world wide web were widely adopted in only a few years time.
A specific paradigm (a method or approach to solving a problem, e.g., shrinking transistors on an integrated circuit as an approach to making more powerful computers) provides exponential growth until the method exhausts its potential. When this happens, a paradigm shift (a fundamental change in the approach) occurs, which enables exponential growth to continue.
Each paradigm follows an “S-curve,” which consists of slow growth (the early phase of exponential growth), followed by rapid growth (the late, explosive phase of exponential growth), followed by a leveling off as the particular paradigm matures.
During this third or maturing phase in the life cycle of a paradigm, pressure builds for the next paradigm shift, and research dollars are invested to create the next paradigm. We can see this in the enormous investments being made today in the next computing paradigm – three-dimensional molecular computing – despite the fact that we still have at least a decade left for the paradigm of shrinking transistors on a flat integrated circuit using photolithography (Moore’s Law). Generally, by the time a paradigm approaches its asymptote (limit) in price-performance, the next technical paradigm is already working in niche applications. For example, engineers were shrinking vacuum tubes in the 1950s to provide greater price-performance for computers, and reached a point where it was no longer feasible to shrink tubes and maintain a vacuum. At this point, around 1960, transistors had already achieved a strong niche market in portable radios.
When a paradigm shift occurs for a particular type of technology, the process begins a new S-curve.
Thus the acceleration of the overall evolutionary process proceeds as a sequence of S-curves, and the overall exponential growth consists of this cascade of S-curves.
The resources underlying the exponential growth of an evolutionary process are relatively unbounded.
One resource is the (ever-growing) order of the evolutionary process itself. Each stage of evolution provides more powerful tools for the next. In biological evolution, the advent of DNA allowed more powerful and faster evolutionary “experiments.” Later, setting the “designs” of animal body plans during the Cambrian explosion allowed rapid evolutionary development of other body organs, such as the brain. Or to take a more recent example, the advent of computer-assisted design tools allows rapid development of the next generation of computers.
The other required resource is the “chaos” of the environment in which the evolutionary process takes place and which provides the options for further diversity. In biological evolution, diversity enters the process in the form of mutations and ever- changing environmental conditions. In technological evolution, human ingenuity combined with ever-changing market conditions keep the process of innovation going.
If we apply these principles at the highest level of evolution on Earth, the first step, the creation of cells, introduced the paradigm of biology. The subsequent emergence of DNA provided a digital method to record the results of evolutionary experiments. Then, the evolution of a species that combined rational thought with an opposable appendage (the thumb) caused a fundamental paradigm shift from biology to technology. The upcoming primary paradigm shift will be from biological thinking to a hybrid combining biological and nonbiological thinking. This hybrid will include “biologically inspired” processes resulting from the reverse engineering of biological brains.
If we examine the timing of these steps, we see that the process has continuously accelerated. The evolution of life forms required billions of years for the first steps (e.g., primitive cells); later on progress accelerated. During the Cambrian explosion, major paradigm shifts took only tens of millions of years. Later on, Humanoids developed over a period of millions of years, and Homo sapiens over a period of only hundreds of thousands of years.
With the advent of a technology-creating species, the exponential pace became too fast for evolution through DNA-guided protein synthesis and moved on to human-created technology. Technology goes beyond mere tool making; it is a process of creating ever more powerful technology using the tools from the previous round of innovation, and is, thereby, an evolutionary process. As I noted, the first technological took tens of thousands of years. For people living in this era, there was little noticeable technological change in even a thousand years. By 1000 AD, progress was much faster and a paradigm shift required only a century or two. In the nineteenth century, we saw more technological change than in the nine centuries preceding it. Then in the first twenty years of the twentieth century, we saw more advancement than in all of the nineteenth century. Now, paradigm shifts occur in only a few years time.
The paradigm shift rate (i.e., the overall rate of technical progress) is currently doubling (approximately) every decade; that is, paradigm shift times are halving every decade (and the rate of acceleration is itself growing exponentially). So, the technological progress in the twenty-first century will be equivalent to what would require (in the linear view) on the order of 200 centuries. In contrast, the twentieth century saw only about 20 years of progress (again at today’s rate of progress) since we have been speeding up to current rates. So the twenty-first century will see about a thousand times greater technological change than its predecessor.
Evolution works through indirection: evolution created humans, humans created technology, humans are now working with increasingly advanced technology to create new generations of technology. As a result, the rate of progress of an evolutionary process increases exponentially over time.
Over time, the “order” of the information embedded in the evolutionary process (i.e., the measure of how well the information fits a purpose, which in evolution is survival) increases.
A comment on the nature of order. The concept of the “order” of information is important here, as it is not the same as the opposite of disorder. If disorder represents a random sequence of events, then the opposite of disorder should imply “not random.” Information is a sequence of data that is meaningful in a process, such as the DNA code of an organism, or the bits in a computer program. Noise, on the other hand, is a random sequence. Neither noise nor information is predictable. Noise is inherently unpredictable, but carries no information. Information, however, is also unpredictable. If we can predict future data from past data, then that future data stops being information. We might consider an alternating pattern (“0101010. . . .”) to be orderly, but it carries no information (beyond the first couple of bits).
Thus orderliness does not constitute order because order requires information. However, order goes beyond mere information. A recording of radiation levels from space represents information, but if we double the size of this data file, we have increased the amount of data, but we have not achieved a deeper level of order.
Order is information that fits a purpose. The measure of order is the measure of how well the information fits the purpose. In the evolution of life-forms, the purpose is to survive. In an evolutionary algorithm (a computer program that simulates evolution to solve a problem) applied to, say, investing in the stock market, the purpose is to make money. Simply having more information does not necessarily result in a better fit. A superior solution for a purpose may very well involve less data.
The concept of “complexity” is often used to describe the nature of the information created by an evolutionary process. Complexity is a close fit to the concept of order that I am describing, but is also not sufficient. Sometimes, a deeper order – a better fit to a purpose – is achieved through simplification rather than further increases in complexity. For example, a new theory that ties together apparently disparate ideas into one broader more coherent theory reduces complexity but nonetheless may increase the “order for a purpose” that I am describing. Indeed, achieving simpler theories is a driving force in science. Evolution has shown, however, that the general trend towards greater order does generally result in greater complexity.
Thus improving a solution to a problem – which may increase or decrease complexity – increases order. Now that just leaves the issue of defining the problem. Indeed, the key to an evolution algorithm (and to biological and technological evolution) is exactly this: defining the problem.
We may note that this aspect of “Kurzweil’s Law” (the law of accelerating returns) appears to contradict the Second Law of Thermodynamics, which implies that entropy (randomness in a closed system) cannot decrease, and, therefore, generally increases. However, the law of accelerating returns pertains to evolution, and evolution is not a closed system. It takes place amidst great chaos, and indeed depends on the disorder in its midst, from which it draws its options for diversity. And from these options, an evolutionary process continually prunes its choices to create ever greater order. Even a crisis, such as the periodic large asteroids that have crashed into the Earth, although increasing chaos temporarily, end up increasing – deepening – the order created by an evolutionary process.
A primary reason that evolution – of life-forms or of technology – speeds up is that it builds on its own increasing order, with ever more sophisticated means of recording and manipulating information. Innovations created by evolution encourage and enable faster evolution. In the case of the evolution of life forms, the most notable early example is DNA, which provides a recorded and protected transcription of life’s design from which to launch further experiments. In the case of the evolution of technology, ever improving human methods of recording information have fostered further technology. The first computers were designed on paper and assembled by hand. Today, they are designed on computer workstations with the computers themselves working out many details of the next generation’s design, and are then produced in fully-automated factories with human guidance but limited direct intervention.
The evolutionary process of technology seeks to improve capabilities in an exponential fashion. Innovators seek to improve things by multiples. Innovation is multiplicative, not additive. Technology, like any evolutionary process, builds on itself. This aspect will continue to accelerate when the technology itself takes full control of its own progression.
We can thus conclude the following with regard to the evolution of life-forms, and of technology: the law of accelerating returns as applied to an evolutionary process: An evolutionary process is not a closed system; therefore, evolution draws upon the chaos in the larger system in which it takes place for its options for diversity; and evolution builds on its own increasing order. Therefore, in an evolutionary process, order increases exponentially.
A correlate of the above observation is that the “returns” of an evolutionary process (e.g., the speed, cost-effectiveness, or overall “power” of a process) increase exponentially over time. We see this in Moore’s law, in which each new generation of computer chip (now spaced about two years apart) provides twice as many components, each of which operates substantially faster (because of the smaller distances required for the electrons to travel, and other innovations). This exponential growth in the power and price-performance of information-based technologies – now roughly doubling every year – is not limited to computers, but is true for a wide range of technologies, measured many different ways.
In another positive feedback loop, as a particular evolutionary process (e.g., computation) becomes more effective (e.g., cost effective), greater resources are deployed towards the further progress of that process. This results in a second level of exponential growth (i.e., the rate of exponential growth itself grows exponentially). For example, it took three years to double the price-performance of computation at the beginning of the twentieth century, two years around 1950, and is now doubling about once a year. Not only is each chip doubling in power each year for the same unit cost, but the number of chips being manufactured is growing exponentially.
Biological evolution is one such evolutionary process. Indeed it is the quintessential evolutionary process. It took place in a completely open system (as opposed to the artificial constraints in an evolutionary algorithm). Thus many levels of the system evolved at the same time.
Technological evolution is another such evolutionary process. Indeed, the emergence of the first technology-creating species resulted in the new evolutionary process of technology. Therefore, technological evolution is an outgrowth of – and a continuation of – biological evolution. Early stages of humanoid created technology were barely faster than the biological evolution that created our species. Homo sapiens evolved in a few hundred thousand years. Early stages of technology – the wheel, fire, stone tools – took tens of thousands of years to evolve and be widely deployed. A thousand years ago, a paradigm shift such as the printing press, took on the order of a century to be widely deployed. Today, major paradigm shifts, such as cell phones and the world wide web were widely adopted in only a few years time.
A specific paradigm (a method or approach to solving a problem, e.g., shrinking transistors on an integrated circuit as an approach to making more powerful computers) provides exponential growth until the method exhausts its potential. When this happens, a paradigm shift (a fundamental change in the approach) occurs, which enables exponential growth to continue.
Each paradigm follows an “S-curve,” which consists of slow growth (the early phase of exponential growth), followed by rapid growth (the late, explosive phase of exponential growth), followed by a leveling off as the particular paradigm matures.
During this third or maturing phase in the life cycle of a paradigm, pressure builds for the next paradigm shift, and research dollars are invested to create the next paradigm. We can see this in the enormous investments being made today in the next computing paradigm – three-dimensional molecular computing – despite the fact that we still have at least a decade left for the paradigm of shrinking transistors on a flat integrated circuit using photolithography (Moore’s Law). Generally, by the time a paradigm approaches its asymptote (limit) in price-performance, the next technical paradigm is already working in niche applications. For example, engineers were shrinking vacuum tubes in the 1950s to provide greater price-performance for computers, and reached a point where it was no longer feasible to shrink tubes and maintain a vacuum. At this point, around 1960, transistors had already achieved a strong niche market in portable radios.
When a paradigm shift occurs for a particular type of technology, the process begins a new S-curve.
Thus the acceleration of the overall evolutionary process proceeds as a sequence of S-curves, and the overall exponential growth consists of this cascade of S-curves.
The resources underlying the exponential growth of an evolutionary process are relatively unbounded.
One resource is the (ever-growing) order of the evolutionary process itself. Each stage of evolution provides more powerful tools for the next. In biological evolution, the advent of DNA allowed more powerful and faster evolutionary “experiments.” Later, setting the “designs” of animal body plans during the Cambrian explosion allowed rapid evolutionary development of other body organs, such as the brain. Or to take a more recent example, the advent of computer-assisted design tools allows rapid development of the next generation of computers.
The other required resource is the “chaos” of the environment in which the evolutionary process takes place and which provides the options for further diversity. In biological evolution, diversity enters the process in the form of mutations and ever- changing environmental conditions. In technological evolution, human ingenuity combined with ever-changing market conditions keep the process of innovation going.
If we apply these principles at the highest level of evolution on Earth, the first step, the creation of cells, introduced the paradigm of biology. The subsequent emergence of DNA provided a digital method to record the results of evolutionary experiments. Then, the evolution of a species that combined rational thought with an opposable appendage (the thumb) caused a fundamental paradigm shift from biology to technology. The upcoming primary paradigm shift will be from biological thinking to a hybrid combining biological and nonbiological thinking. This hybrid will include “biologically inspired” processes resulting from the reverse engineering of biological brains.
If we examine the timing of these steps, we see that the process has continuously accelerated. The evolution of life forms required billions of years for the first steps (e.g., primitive cells); later on progress accelerated. During the Cambrian explosion, major paradigm shifts took only tens of millions of years. Later on, Humanoids developed over a period of millions of years, and Homo sapiens over a period of only hundreds of thousands of years.
With the advent of a technology-creating species, the exponential pace became too fast for evolution through DNA-guided protein synthesis and moved on to human-created technology. Technology goes beyond mere tool making; it is a process of creating ever more powerful technology using the tools from the previous round of innovation, and is, thereby, an evolutionary process. As I noted, the first technological took tens of thousands of years. For people living in this era, there was little noticeable technological change in even a thousand years. By 1000 AD, progress was much faster and a paradigm shift required only a century or two. In the nineteenth century, we saw more technological change than in the nine centuries preceding it. Then in the first twenty years of the twentieth century, we saw more advancement than in all of the nineteenth century. Now, paradigm shifts occur in only a few years time.
The paradigm shift rate (i.e., the overall rate of technical progress) is currently doubling (approximately) every decade; that is, paradigm shift times are halving every decade (and the rate of acceleration is itself growing exponentially). So, the technological progress in the twenty-first century will be equivalent to what would require (in the linear view) on the order of 200 centuries. In contrast, the twentieth century saw only about 20 years of progress (again at today’s rate of progress) since we have been speeding up to current rates. So the twenty-first century will see about a thousand times greater technological change than its predecessor.
