Erosion

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Erosion 

An actively eroding rill on an intensively-farmed field in eastern Germany

Erosion is the action of surface processes (such as water flow or wind) that removes soil, rock, or dissolved material from one location on the Earth's crust, and then transports it to another location where it is deposited. Erosion is distinct from weathering which involves no movement. Removal of rock or soil as clastic sediment is referred to as physical or mechanical erosion; this contrasts with chemical erosion, where soil or rock material is removed from an area by dissolution. Eroded sediment or solutes may be transported just a few millimetres, or for thousands of kilometres.

Agents of erosion include rainfall; bedrock wear in rivers; coastal erosion by the sea and waves; glacial plucking, abrasion, and scour; areal flooding; wind abrasion; groundwater processes; and mass movement processes in steep landscapes like landslides and debris flows. The rates at which such processes act control how fast a surface is eroded. Typically, physical erosion proceeds fastest on steeply sloping surfaces, and rates may also be sensitive to some climatically-controlled properties including amounts of water supplied (e.g., by rain), storminess, wind speed, wave fetch, or atmospheric temperature (especially for some ice-related processes). Feedbacks are also possible between rates of erosion and the amount of eroded material that is already carried by, for example, a river or glacier. The transport of eroded materials from their original location is followed by deposition, which is arrival and emplacement of material at a new location.

While erosion is a natural process, human activities have increased by 10-40 times the rate at which erosion is occurring globally. At agriculture sites in the Appalachian Mountains, intensive farming practices have caused erosion at up to 100 times the natural rate of erosion in the region. Excessive (or accelerated) erosion causes both "on-site" and "off-site" problems. On-site impacts include decreases in agricultural productivity and (on natural landscapes) ecological collapse, both because of loss of the nutrient-rich upper soil layers. In some cases, this leads to desertification. Off-site effects include sedimentation of waterways and eutrophication of water bodies, as well as sediment-related damage to roads and houses. Water and wind erosion are the two primary causes of land degradation; combined, they are responsible for about 84% of the global extent of degraded land, making excessive erosion one of the most significant environmental problems worldwide.

Intensive agriculture, deforestation, roads, anthropogenic climate change and urban sprawl are amongst the most significant human activities in regard to their effect on stimulating erosion. However, there are many prevention and remediation practices that can curtail or limit erosion of vulnerable soils.

A natural arch produced by the wind erosion of differentially weathered rock in Jebel Kharaz, Jordan

 

A wave-like sea cliff produced by coastal erosion, in Jinshitan Coastal National Geopark, Dalian, Liaoning Province, China

Physical processes

Rainfall and surface runoff

Soil and water being splashed by the impact of a single raindrop

Rainfall, and the surface runoff which may result from rainfall, produces four main types of soil erosion: splash erosion, sheet erosion, rill erosion, and gully erosion. Splash erosion is generally seen as the first and least severe stage in the soil erosion process, which is followed by sheet erosion, then rill erosion and finally gully erosion (the most severe of the four).

In splash erosion, the impact of a falling raindrop creates a small crater in the soil, ejecting soil particles. The distance these soil particles travel can be as much as 0.6 m (two feet) vertically and 1.5 m (five feet) horizontally on level ground.

If the soil is saturated, or if the rainfall rate is greater than the rate at which water can infiltrate into the soil, surface runoff occurs. If the runoff has sufficient flow energy, it will transport loosened soil particles (sediment) down the slope. Sheet erosion is the transport of loosened soil particles by overland flow.

A spoil tip covered in rills and gullies due to erosion processes caused by rainfall: Rummu, Estonia

Rill erosion refers to the development of small, ephemeral concentrated flow paths which function as both sediment source and sediment delivery systems for erosion on hillslopes. Generally, where water erosion rates on disturbed upland areas are greatest, rills are active. Flow depths in rills are typically of the order of a few centimetres (about an inch) or less and along-channel slopes may be quite steep. This means that rills exhibit hydraulic physics very different from water flowing through the deeper, wider channels of streams and rivers.

Gully erosion occurs when runoff water accumulates and rapidly flows in narrow channels during or immediately after heavy rains or melting snow, removing soil to a considerable depth. A gully is distinguished from a rill based on a critical cross-sectional area of at least one square foot, i.e. the size of a channel that can no longer be erased via normal tillage operations.

Extreme gully erosion can progress to formation of badlands. These form under conditions of high relief on easily eroded bedrock in climates favorable to erosion. Conditions or disturbances that limit the growth of protective vegetation (rhexistasy) are a key element of badland formation.

Rivers and streams

Further information on water's erosive ability: Hydraulic action

 

Dobbingstone Burn, Scotland, showing two different types of erosion affecting the same place. Valley erosion is occurring due to the flow of the stream, and the boulders and stones (and much of the soil) that are lying on the stream's banks are glacial till that was left behind as ice age glaciers flowed over the terrain.

 

Layers of chalk exposed by a river eroding through them

Valley or stream erosion occurs with continued water flow along a linear feature. The erosion is both downward, deepening the valley, and headward, extending the valley into the hillside, creating head cuts and steep banks. In the earliest stage of stream erosion, the erosive activity is dominantly vertical, the valleys have a typical V-shaped cross-section and the stream gradient is relatively steep. When some base level is reached, the erosive activity switches to lateral erosion, which widens the valley floor and creates a narrow floodplain. The stream gradient becomes nearly flat, and lateral deposition of sediments becomes important as the stream meanders across the valley floor. In all stages of stream erosion, by far the most erosion occurs during times of flood when more and faster-moving water is available to carry a larger sediment load. In such processes, it is not the water alone that erodes: suspended abrasive particles, pebbles, and boulders can also act erosively as they traverse a surface, in a process known as traction.

Bank erosion is the wearing away of the banks of a stream or river. This is distinguished from changes on the bed of the watercourse, which is referred to as scour. Erosion and changes in the form of river banks may be measured by inserting metal rods into the bank and marking the position of the bank surface along the rods at different times.

Thermal erosion is the result of melting and weakening permafrost due to moving water. It can occur both along rivers and at the coast. Rapid river channel migration observed in the Lena River of Siberia is due to thermal erosion, as these portions of the banks are composed of permafrost-cemented non-cohesive materials. Much of this erosion occurs as the weakened banks fail in large slumps. Thermal erosion also affects the Arctic coast, where wave action and near-shore temperatures combine to undercut permafrost bluffs along the shoreline and cause them to fail. Annual erosion rates along a 100-kilometre (62-mile) segment of the Beaufort Sea shoreline averaged 5.6 metres (18 feet) per year from 1955 to 2002.

Most river erosion happens nearer to the mouth of a river. On a river bend, the longest least sharp side has slower moving water. Here deposits build up. On the narrowest sharpest side of the bend, there is faster moving water so this side tends to erode away mostly.

Rapid erosion by a large river can remove enough sediments to produce a river anticline, as isostatic rebound raises rock beds unburdened by erosion of overlying beds.

Coastal erosion

Main article: Coastal erosion

See also: Beach evolution

 

Wave cut platform caused by erosion of cliffs by the sea, at Southerndown in South Wales

 

Erosion of the boulder clay (of Pleistocene age) along cliffs of Filey Bay, Yorkshire, England

Shoreline erosion, which occurs on both exposed and sheltered coasts, primarily occurs through the action of currents and waves but sea level (tidal) change can also play a role.

Hydraulic action takes place when the air in a joint is suddenly compressed by a wave closing the entrance of the joint. This then cracks it. Wave pounding is when the sheer energy of the wave hitting the cliff or rock breaks pieces off. Abrasion or corrasion is caused by waves launching sea load at the cliff. It is the most effective and rapid form of shoreline erosion (not to be confused with corrosion). Corrosion is the dissolving of rock by carbonic acid in sea water. Limestone cliffs are particularly vulnerable to this kind of erosion. Attrition is where particles/sea load carried by the waves are worn down as they hit each other and the cliffs. This then makes the material easier to wash away. The material ends up as shingle and sand. Another significant source of erosion, particularly on carbonate coastlines, is boring, scraping and grinding of organisms, a process termed bioerosion.

Sediment is transported along the coast in the direction of the prevailing current (longshore drift). When the upcurrent supply of sediment is less than the amount being carried away, erosion occurs. When the upcurrent amount of sediment is greater, sand or gravel banks will tend to form as a result of deposition. These banks may slowly migrate along the coast in the direction of the longshore drift, alternately protecting and exposing parts of the coastline. Where there is a bend in the coastline, quite often a buildup of eroded material occurs forming a long narrow bank (a spit). Armoured beaches and submerged offshore sandbanks may also protect parts of a coastline from erosion. Over the years, as the shoals gradually shift, the erosion may be redirected to attack different parts of the shore.

Erosion of a coastal surface, followed by a fall in sea level, can produce a distinctive landform called a raised beach.

Chemical erosion

See also: Karst topography

Chemical erosion is the loss of matter in a landscape in the form of solutes. Chemical erosion is usually calculated from the solutes found in streams. Anders Rapp pioneered the study of chemical erosion in his work about Kärkevagge published in 1960.

Formation of sinkholes and other features of karst topography is an example of extreme chemical erosion.

Glaciers

The Devil's Nest (Pirunpesä), the deepest ground erosion in Europe, located in Jalasjärvi, Kurikka, Finland

 

Glacial moraines above Lake Louise, in Alberta, Canada

Glaciers erode predominantly by three different processes: abrasion/scouring, plucking, and ice thrusting. In an abrasion process, debris in the basal ice scrapes along the bed, polishing and gouging the underlying rocks, similar to sandpaper on wood. Scientists have shown that, in addition to the role of temperature played in valley-deepening, other glaciological processes, such as erosion also control cross-valley variations. In a homogeneous bedrock erosion pattern, curved channel cross-section beneath the ice is created. Though the glacier continues to incise vertically, the shape of the channel beneath the ice eventually remain constant, reaching a U-shaped parabolic steady-state shape as we now see in glaciated valleys. Scientists also provide a numerical estimate of the time required for the ultimate formation of a steady-shaped U-shaped valley—approximately 100,000 years. In a weak bedrock (containing material more erodible than the surrounding rocks) erosion pattern, on the contrary, the amount of over deepening is limited because ice velocities and erosion rates are reduced.

Glaciers can also cause pieces of bedrock to crack off in the process of plucking. In ice thrusting, the glacier freezes to its bed, then as it surges forward, it moves large sheets of frozen sediment at the base along with the glacier. This method produced some of the many thousands of lake basins that dot the edge of the Canadian Shield. Differences in the height of mountain ranges are not only being the result tectonic forces, such as rock uplift, but also local climate variations. Scientists use global analysis of topography to show that glacial erosion controls the maximum height of mountains, as the relief between mountain peaks and the snow line are generally confined to altitudes less than 1500 m. The erosion caused by glaciers worldwide erodes mountains so effectively that the term glacial buzzsaw has become widely used, which describes the limiting effect of glaciers on the height of mountain ranges. As mountains grow higher, they generally allow for more glacial activity (especially in the accumulation zone above the glacial equilibrium line altitude), which causes increased rates of erosion of the mountain, decreasing mass faster than isostatic rebound can add to the mountain. This provides a good example of a negative feedback loop. Ongoing research is showing that while glaciers tend to decrease mountain size, in some areas, glaciers can actually reduce the rate of erosion, acting as a glacial armor. Ice can not only erode mountains but also protect them from erosion. Depending on glacier regime, even steep alpine lands can be preserved through time with the help of ice. Scientists have proved this theory by sampling eight summits of northwestern Svalbard using Be10 and Al26, showing that northwestern Svalbard transformed from a glacier-erosion state under relatively mild glacial maxima temperature, to a glacier-armor state occupied by cold-based, protective ice during much colder glacial maxima temperatures as the Quaternary ice age progressed.

These processes, combined with erosion and transport by the water network beneath the glacier, leave behind glacial landforms such as moraines, drumlins, ground moraine (till), kames, kame deltas, moulins, and glacial erratics in their wake, typically at the terminus or during glacier retreat.

The best-developed glacial valley morphology appears to be restricted to landscapes with low rock uplift rates (less than or equal to 2mm per year) and high relief, leading to long-turnover times. Where rock uplift rates exceed 2mm per year, glacial valley morphology has generally been significantly modified in postglacial time. Interplay of glacial erosion and tectonic forcing governs the morphologic impact of glaciations on active orogens, by both influencing their height, and by altering the patterns of erosion during subsequent glacial periods via a link between rock uplift and valley cross-sectional shape.

Floods

The mouth of the River Seaton in Cornwall after heavy rainfall caused flooding in the area and cause a significant amount of the beach to erode; leaving behind a tall sand bank in its place

At extremely high flows, kolks, or vortices are formed by large volumes of rapidly rushing water. Kolks cause extreme local erosion, plucking bedrock and creating pothole-type geographical features called rock-cut basins. Examples can be seen in the flood regions result from glacial Lake Missoula, which created the channeled scablands in the Columbia Basin region of eastern Washington.

Wind erosion

Árbol de Piedra, a rock formation in the Altiplano, Bolivia sculpted by wind erosion

 

Main article: Aeolian processes

Wind erosion is a major geomorphological force, especially in arid and semi-arid regions. It is also a major source of land degradation, evaporation, desertification, harmful airborne dust, and crop damage—especially after being increased far above natural rates by human activities such as deforestation, urbanization, and agriculture.

Wind erosion is of two primary varieties: deflation, where the wind picks up and carries away loose particles; and abrasion, where surfaces are worn down as they are struck by airborne particles carried by wind. Deflation is divided into three categories: (1) surface creep, where larger, heavier particles slide or roll along the ground; (2) saltation, where particles are lifted a short height into the air, and bounce and saltate across the surface of the soil; and (3) suspension, where very small and light particles are lifted into the air by the wind, and are often carried for long distances. Saltation is responsible for the majority (50-70%) of wind erosion, followed by suspension (30-40%), and then surface creep (5-25%).

Wind erosion is much more severe in arid areas and during times of drought. For example, in the Great Plains, it is estimated that soil loss due to wind erosion can be as much as 6100 times greater in drought years than in wet years.

Mass wasting

A wadi in Makhtesh Ramon, Israel, showing gravity collapse erosion on its banks

 

Main article: Mass wasting

Mass wasting or mass movement is the downward and outward movement of rock and sediments on a sloped surface, mainly due to the force of gravity.

Mass wasting is an important part of the erosional process and is often the first stage in the breakdown and transport of weathered materials in mountainous areas. It moves material from higher elevations to lower elevations where other eroding agents such as streams and glaciers can then pick up the material and move it to even lower elevations. Mass-wasting processes are always occurring continuously on all slopes; some mass-wasting processes act very slowly; others occur very suddenly, often with disastrous results. Any perceptible down-slope movement of rock or sediment is often referred to in general terms as a landslide. However, landslides can be classified in a much more detailed way that reflects the mechanisms responsible for the movement and the velocity at which the movement occurs. One of the visible topographical manifestations of a very slow form of such activity is a scree slope.

Slumping happens on steep hillsides, occurring along distinct fracture zones, often within materials like clay that, once released, may move quite rapidly downhill. They will often show a spoon-shaped isostatic depression, in which the material has begun to slide downhill. In some cases, the slump is caused by water beneath the slope weakening it. In many cases it is simply the result of poor engineering along highways where it is a regular occurrence.

Surface creep is the slow movement of soil and rock debris by gravity which is usually not perceptible except through extended observation. However, the term can also describe the rolling of dislodged soil particles 0.5 to 1.0 mm (0.02 to 0.04 in) in diameter by wind along the soil surface.

Submarine sediment gravity flows

Bathymetry of submarine canyons in the continental slope off the coast of New York and New Jersey

On the continental slope, erosion of the ocean floor to create channels and submarine canyons can result from the rapid downslope flow of sediment gravity flows, bodies of sediment-laden water that move rapidly downslope as turbidity currents. Where erosion by turbidity currents creates oversteepened slopes it can also trigger underwater landslides and debris flows. Turbidity currents can erode channels and canyons into substrates ranging from recently deposited unconsolidated sediments to hard crystalline bedrock. Almost all continental slopes and deep ocean basins display such channels and canyons resulting from sediment gravity flows and submarine canyons act as conduits for the transfer of sediment from the continents and shallow marine environments to the deep sea. Turbidites, which are the sedimentary deposits resulting from turbidity currents, comprise some of the thickest and largest sedimentary sequences on Earth, indicating that the associated erosional processes must also have played a prominent role in Earth's history.

Factors affecting erosion rates

Climate

See also: Climatic geomorphology

The amount and intensity of precipitation is the main climatic factor governing soil erosion by water. The relationship is particularly strong if heavy rainfall occurs at times when, or in locations where, the soil's surface is not well protected by vegetation. This might be during periods when agricultural activities leave the soil bare, or in semi-arid regions where vegetation is naturally sparse. Wind erosion requires strong winds, particularly during times of drought when vegetation is sparse and soil is dry (and so is more erodible). Other climatic factors such as average temperature and temperature range may also affect erosion, via their effects on vegetation and soil properties. In general, given similar vegetation and ecosystems, areas with more precipitation (especially high-intensity rainfall), more wind, or more storms are expected to have more erosion.

In some areas of the world (e.g. the mid-western USA), rainfall intensity is the primary determinant of erosivity with higher intensity rainfall generally resulting in more soil erosion by water. The size and velocity of rain drops is also an important factor. Larger and higher-velocity rain drops have greater kinetic energy, and thus their impact will displace soil particles by larger distances than smaller, slower-moving rain drops.

In other regions of the world (e.g. western Europe), runoff and erosion result from relatively low intensities of stratiform rainfall falling onto the previously saturated soil. In such situations, rainfall amount rather than intensity is the main factor determining the severity of soil erosion by water. According to the climate change projections, erosivity will increase significantly in Europe and soil erosion may increase by 13-22.5% by 2050 

In Taiwan, where typhoon frequency increased significantly in the 21st century, a strong link has been drawn between the increase in storm frequency with an increase in sediment load in rivers and reservoirs, highlighting the impacts climate change can have on erosion.

Vegetative cover

See also: Vegetation and slope stability

Vegetation acts as an interface between the atmosphere and the soil. It increases the permeability of the soil to rainwater, thus decreasing runoff. It shelters the soil from winds, which results in decreased wind erosion, as well as advantageous changes in microclimate. The roots of the plants bind the soil together, and interweave with other roots, forming a more solid mass that is less susceptible to both water and wind erosion. The removal of vegetation increases the rate of surface erosion.

Topography

The topography of the land determines the velocity at which surface runoff will flow, which in turn determines the erosivity of the runoff. Longer, steeper slopes (especially those without adequate vegetative cover) are more susceptible to very high rates of erosion during heavy rains than shorter, less steep slopes. Steeper terrain is also more prone to mudslides, landslides, and other forms of gravitational erosion processes.

Tectonics

Main article: Erosion and tectonics

Tectonic processes control rates and distributions of erosion at the Earth's surface. If the tectonic action causes part of the Earth's surface (e.g., a mountain range) to be raised or lowered relative to surrounding areas, this must necessarily change the gradient of the land surface. Because erosion rates are almost always sensitive to the local slope (see above), this will change the rates of erosion in the uplifted area. Active tectonics also brings fresh, unweathered rock towards the surface, where it is exposed to the action of erosion.

However, erosion can also affect tectonic processes. The removal by erosion of large amounts of rock from a particular region, and its deposition elsewhere, can result in a lightening of the load on the lower crust and mantle. Because tectonic processes are driven by gradients in the stress field developed in the crust, this unloading can in turn cause tectonic or isostatic uplift in the region. In some cases, it has been hypothesised that these twin feedbacks can act to localize and enhance zones of very rapid exhumation of deep crustal rocks beneath places on the Earth's surface with extremely high erosion rates, for example, beneath the extremely steep terrain of Nanga Parbat in the western Himalayas. Such a place has been called a "tectonic aneurysm".

Development

Human land development, in forms including agricultural and urban development, is considered a significant factor in erosion and sediment transport, which aggravate food insecurity. In Taiwan, increases in sediment load in the northern, central, and southern regions of the island can be tracked with the timeline of development for each region throughout the 20th century. The intentional removal of soil and rock by humans is a form of erosion that has been named lisasion.

Erosion at various scales

Mountain ranges

See also: denudation and planation

Mountain ranges are known to take many millions of years to erode to the degree they effectively cease to exist. Scholars Pitman and Golovchenko estimate that it takes probably more than 450 million years to erode a mountain mass similar to the Himalaya into an almost-flat peneplain if there are no major sea-level changes. Erosion of mountains massifs can create a pattern of equally high summits called summit accordance. It has been argued that extension during post-orogenic collapse is a more effective mechanism of lowering the height of orogenic mountains than erosion.

Examples of heavily eroded mountain ranges include the Timanides of Northern Russia. Erosion of this orogen has produced sediments that are now found in the East European Platform, including the Cambrian Sablya Formation near Lake Ladoga. Studies of these sediments indicate that it is likely that the erosion of the orogen began in the Cambrian and then intensified in the Ordovician.

Soils

Further information: soil erosion and pedogenesis

If the rate of erosion is higher than the rate of soil formation the soils are being destroyed by erosion. Where soil is not destroyed by erosion, erosion can in some cases prevent the formation of soil features that form slowly. Inceptisols are common soils that form in areas of fast erosion.

While erosion of soils is a natural process, human activities have increased by 10-40 times the rate at which erosion is occurring globally. Excessive (or accelerated) erosion causes both "on-site" and "off-site" problems. On-site impacts include decreases in agricultural productivity and (on natural landscapes) ecological collapse, both because of loss of the nutrient-rich upper soil layers. In some cases, the eventual end result is desertification. Off-site effects include sedimentation of waterways and eutrophication of water bodies, as well as sediment-related damage to roads and houses. Water and wind erosion are the two primary causes of land degradation; combined, they are responsible for about 84% of the global extent of degraded land, making excessive erosion one of the most significant environmental problems.

In the United States, farmers cultivating highly erodible land must comply with a conservation plan to be eligible for certain forms of agricultural assistance.

Consequences of human-made soil erosion

Main articles: Human impact on the environment, Environmental impact of agriculture, Soil retrogression and degradation, and Land degradation

Chemical kinetics

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Chemical_kinetics 

Chemical kinetics, also known as reaction kinetics, is the branch of physical chemistry that is concerned with understanding the rates of chemical reactions. It is to be contrasted with chemical thermodynamics, which deals with the direction in which a reaction occurs but in itself tells nothing about its rate. Chemical kinetics includes investigations of how experimental conditions influence the speed of a chemical reaction and yield information about the reaction's mechanism and transition states, as well as the construction of mathematical models that also can describe the characteristics of a chemical reaction.

History

In 1864, Peter Waage and Cato Guldberg pioneered the development of chemical kinetics by formulating the law of mass action, which states that the speed of a chemical reaction is proportional to the quantity of the reacting substances.

Van 't Hoff studied chemical dynamics and in 1884 published his famous "Études de dynamique chimique". In 1901 he was awarded by the first Nobel Prize in Chemistry "in recognition of the extraordinary services he has rendered by the discovery of the laws of chemical dynamics and osmotic pressure in solutions". After van 't Hoff, chemical kinetics deals with the experimental determination of reaction rates from which rate laws and rate constants are derived. Relatively simple rate laws exist for zero order reactions (for which reaction rates are independent of concentration), first order reactions, and second order reactions, and can be derived for others. Elementary reactions follow the law of mass action, but the rate law of stepwise reactions has to be derived by combining the rate laws of the various elementary steps, and can become rather complex. In consecutive reactions, the rate-determining step often determines the kinetics. In consecutive first order reactions, a steady state approximation can simplify the rate law. The activation energy for a reaction is experimentally determined through the Arrhenius equation and the Eyring equation. The main factors that influence the reaction rate include: the physical state of the reactants, the concentrations of the reactants, the temperature at which the reaction occurs, and whether or not any catalysts are present in the reaction.

Gorban and Yablonsky have suggested that the history of chemical dynamics can be divided into three eras. The first is the van 't Hoff wave searching for the general laws of chemical reactions and relating kinetics to thermodynamics. The second may be called the Semenov--Hinshelwood wave with emphasis on reaction mechanisms, especially for chain reactions. The third is associated with Aris and the detailed mathematical description of chemical reaction networks.

Factors affecting reaction rate

Nature of the reactants

The reaction rate varies depending upon what substances are reacting. Acid/base reactions, the formation of salts, and ion exchange are usually fast reactions. When covalent bond formation takes place between the molecules and when large molecules are formed, the reactions tend to be slower.

The nature and strength of bonds in reactant molecules greatly influence the rate of their transformation into products.

Physical state

The physical state (solid, liquid, or gas) of a reactant is also an important factor of the rate of change. When reactants are in the same phase, as in aqueous solution, thermal motion brings them into contact. However, when they are in separate phases, the reaction is limited to the interface between the reactants. Reaction can occur only at their area of contact; in the case of a liquid and a gas, at the surface of the liquid. Vigorous shaking and stirring may be needed to bring the reaction to completion. This means that the more finely divided a solid or liquid reactant the greater its surface area per unit volume and the more contact it with the other reactant, thus the faster the reaction. To make an analogy, for example, when one starts a fire, one uses wood chips and small branches — one does not start with large logs right away. In organic chemistry, on water reactions are the exception to the rule that homogeneous reactions take place faster than heterogeneous reactions ( are those reactions in which solute and solvent not mix properly)

Surface area of solid state

In a solid, only those particles that are at the surface can be involved in a reaction. Crushing a solid into smaller parts means that more particles are present at the surface, and the frequency of collisions between these and reactant particles increases, and so reaction occurs more rapidly. For example, Sherbet (powder) is a mixture of very fine powder of malic acid (a weak organic acid) and sodium hydrogen carbonate. On contact with the saliva in the mouth, these chemicals quickly dissolve and react, releasing carbon dioxide and providing for the fizzy sensation. Also, fireworks manufacturers modify the surface area of solid reactants to control the rate at which the fuels in fireworks are oxidised, using this to create diverse effects. For example, finely divided aluminium confined in a shell explodes violently. If larger pieces of aluminium are used, the reaction is slower and sparks are seen as pieces of burning metal are ejected.

Concentration

Main article: Rate equation

The reactions are due to collisions of reactant species. The frequency with which the molecules or ions collide depends upon their concentrations. The more crowded the molecules are, the more likely they are to collide and react with one another. Thus, an increase in the concentrations of the reactants will usually result in the corresponding increase in the reaction rate, while a decrease in the concentrations will usually have a reverse effect. For example, combustion will occur more rapidly in pure oxygen than in air (21% oxygen).

The rate equation shows the detailed dependence of the reaction rate on the concentrations of reactants and other species present. The mathematical forms depend on the reaction mechanism. The actual rate equation for a given reaction is determined experimentally and provides information about the reaction mechanism. The mathematical expression of the rate equation is often given by

${\displaystyle v={\frac {\mathrm {d} c}{\mathrm {d} t}}=k\prod _{i}c_{i}^{m_{i}}}$

Here ${\displaystyle k}$ is the reaction rate constant, ${\displaystyle c_{i}}$ is the molar concentration of reactant i and ${\displaystyle m_{i}}$ is the partial order of reaction for this reactant. The partial order for a reactant can only be determined experimentally and is often not indicated by its stoichiometric coefficient.

Temperature

Main article: Arrhenius equation

Temperature usually has a major effect on the rate of a chemical reaction. Molecules at a higher temperature have more thermal energy. Although collision frequency is greater at higher temperatures, this alone contributes only a very small proportion to the increase in rate of reaction. Much more important is the fact that the proportion of reactant molecules with sufficient energy to react (energy greater than activation energy: E > E_a) is significantly higher and is explained in detail by the Maxwell–Boltzmann distribution of molecular energies.

The effect of temperature on the reaction rate constant usually obeys the Arrhenius equation ${\displaystyle k=Ae^{-E_{\rm {a}}/(RT)}}$, where A is the pre-exponential factor or A-factor, E_a is the activation energy, R is the molar gas constant and T is the absolute temperature.

At a given temperature, the chemical rate of a reaction depends on the value of the A-factor, the magnitude of the activation energy, and the concentrations of the reactants. Usually, rapid reactions require relatively small activation energies.

The 'rule of thumb' that the rate of chemical reactions doubles for every 10 °C temperature rise is a common misconception. This may have been generalized from the special case of biological systems, where the α (temperature coefficient) is often between 1.5 and 2.5.

The kinetics of rapid reactions can be studied with the temperature jump method. This involves using a sharp rise in temperature and observing the relaxation time of the return to equilibrium. A particularly useful form of temperature jump apparatus is a shock tube, which can rapidly increase a gas's temperature by more than 1000 degrees.

Catalysts

Main article: Catalysis

 

Generic potential energy diagram showing the effect of a catalyst in a hypothetical endothermic chemical reaction. The presence of the catalyst opens a new reaction pathway (shown in red) with a lower activation energy. The final result and the overall thermodynamics are the same.

A catalyst is a substance that alters the rate of a chemical reaction but it remains chemically unchanged afterwards. The catalyst increases the rate of the reaction by providing a new reaction mechanism to occur with in a lower activation energy. In autocatalysis a reaction product is itself a catalyst for that reaction leading to positive feedback. Proteins that act as catalysts in biochemical reactions are called enzymes. Michaelis–Menten kinetics describe the rate of enzyme mediated reactions. A catalyst does not affect the position of the equilibrium, as the catalyst speeds up the backward and forward reactions equally.

In certain organic molecules, specific substituents can have an influence on reaction rate in neighbouring group participation.

Pressure

Increasing the pressure in a gaseous reaction will increase the number of collisions between reactants, increasing the rate of reaction. This is because the activity of a gas is directly proportional to the partial pressure of the gas. This is similar to the effect of increasing the concentration of a solution.

In addition to this straightforward mass-action effect, the rate coefficients themselves can change due to pressure. The rate coefficients and products of many high-temperature gas-phase reactions change if an inert gas is added to the mixture; variations on this effect are called fall-off and chemical activation. These phenomena are due to exothermic or endothermic reactions occurring faster than heat transfer, causing the reacting molecules to have non-thermal energy distributions (non-Boltzmann distribution). Increasing the pressure increases the heat transfer rate between the reacting molecules and the rest of the system, reducing this effect.

Condensed-phase rate coefficients can also be affected by pressure, although rather high pressures are required for a measurable effect because ions and molecules are not very compressible. This effect is often studied using diamond anvils.

A reaction's kinetics can also be studied with a pressure jump approach. This involves making fast changes in pressure and observing the relaxation time of the return to equilibrium.

Absorption of light

The activation energy for a chemical reaction can be provided when one reactant molecule absorbs light of suitable wavelength and is promoted to an excited state. The study of reactions initiated by light is photochemistry, one prominent example being photosynthesis.

Experimental methods

The Spinco Division Model 260 Reaction Kinetics System measured the precise rate constants of molecular reactions.

The experimental determination of reaction rates involves measuring how the concentrations of reactants or products change over time. For example, the concentration of a reactant can be measured by spectrophotometry at a wavelength where no other reactant or product in the system absorbs light.

For reactions which take at least several minutes, it is possible to start the observations after the reactants have been mixed at the temperature of interest.

Fast reactions

For faster reactions, the time required to mix the reactants and bring them to a specified temperature may be comparable or longer than the half-life of the reaction. Special methods to start fast reactions without slow mixing step include

• Stopped flow methods, which can reduce the mixing time to the order of a millisecond. The stopped flow methods have limitation, for example, we need to consider the time it takes to mix gases or solutions and are not suitable if the half-life is less than about a hundredth of a second.
• Chemical relaxation methods such as temperature jump and pressure jump, in which a pre-mixed system initially at equilibrium is perturbed by rapid heating or depressurization so that it is no longer at equilibrium, and the relaxation back to equilibrium is observed. For example, this method has been used to study the neutralization H₃O⁺ + OH⁻ with a half-life of 1 μs or less under ordinary conditions.
• Flash photolysis, in which a laser pulse produces highly excited species such as free radicals, whose reactions are then studied.

Equilibrium

While chemical kinetics is concerned with the rate of a chemical reaction, thermodynamics determines the extent to which reactions occur. In a reversible reaction, chemical equilibrium is reached when the rates of the forward and reverse reactions are equal (the principle of dynamic equilibrium) and the concentrations of the reactants and products no longer change. This is demonstrated by, for example, the Haber–Bosch process for combining nitrogen and hydrogen to produce ammonia. Chemical clock reactions such as the Belousov–Zhabotinsky reaction demonstrate that component concentrations can oscillate for a long time before finally attaining the equilibrium.

Free energy

In general terms, the free energy change (ΔG) of a reaction determines whether a chemical change will take place, but kinetics describes how fast the reaction is. A reaction can be very exothermic and have a very positive entropy change but will not happen in practice if the reaction is too slow. If a reactant can produce two products, the thermodynamically most stable one will form in general, except in special circumstances when the reaction is said to be under kinetic reaction control. The Curtin–Hammett principle applies when determining the product ratio for two reactants interconverting rapidly, each going to a distinct product. It is possible to make predictions about reaction rate constants for a reaction from free-energy relationships.

The kinetic isotope effect is the difference in the rate of a chemical reaction when an atom in one of the reactants is replaced by one of its isotopes.

Chemical kinetics provides information on residence time and heat transfer in a chemical reactor in chemical engineering and the molar mass distribution in polymer chemistry. It is also provides information in corrosion engineering.

Applications and models

The mathematical models that describe chemical reaction kinetics provide chemists and chemical engineers with tools to better understand and describe chemical processes such as food decomposition, microorganism growth, stratospheric ozone decomposition, and the chemistry of biological systems. These models can also be used in the design or modification of chemical reactors to optimize product yield, more efficiently separate products, and eliminate environmentally harmful by-products. When performing catalytic cracking of heavy hydrocarbons into gasoline and light gas, for example, kinetic models can be used to find the temperature and pressure at which the highest yield of heavy hydrocarbons into gasoline will occur.

Chemical Kinetics is frequently validated and explored through modeling in specialized packages as a function of ordinary differential equation-solving (ODE-solving) and curve-fitting.

Numerical methods

In some cases, equations are unsolvable analytically, but can be solved using numerical methods if data values are given. There are two different ways to do this, by either using software programmes or mathematical methods such as the Euler method. Examples of software for chemical kinetics are i) Tenua, a Java app which simulates chemical reactions numerically and allows comparison of the simulation to real data, ii) Python coding for calculations and estimates and iii) the Kintecus software compiler to model, regress, fit and optimize reactions.

-Numerical integration: for a 1st order reaction A → B

The differential equation of the reactant A is:

${\displaystyle d[A]/dt=-k[A]\qquad \qquad }$

It can also be expressed as:

${\displaystyle d[A]/dt=f(t,[A])\qquad \qquad }$ which is the same as ${\displaystyle y'=f(y,x)\qquad \qquad }$

To solve the differential equations with Euler and Runge-Kutta methods we need to have the initial values.

• Euler method → simple but inaccurate.

At any point ${\displaystyle y'=f(y,x)\qquad \qquad }$ is the same as;

${\displaystyle y'=dy/dx\qquad \qquad }$

We can approximate the differentials as discrete increases:

${\displaystyle y'=dy/dx\qquad \qquad }$ ≃ ∆y/∆x = [y(x+∆x)-y(x)]/∆x

The unknown part of the equation is y(x+Δx), which can be found if we have the data for the initial values.

• Runge-Kutta methods → it is more accurate than the Euler method.

In this method, an initial condition is required: y=y₀ at x=x₀. The problem is to find the value of y when x=x₀ + h, where h is a given constant.

It can be shown analytically that the ordinate at that moment to the curve through (x₀, y₀) is given by the third-order Runge-Kutta formula.

In first-order ordinary equations, the Runge-Kutta method uses a mathematical model that represents the relationship between the temperature and the rate of reaction. It is worth it to calculate the rate of reaction at different temperatures for different concentrations. The equation obtained is: ${\displaystyle dr/dt=R/T+r\Delta H^{\circ }/RT^{2}}$

• Stochastic methods → probabilities of the differential rate laws and the kinetic constants.

In an equilibrioum reaction with direct and inverse rate constants, it is easier to transform from A to B rather than B to A.

As for probability computations, at each time it choose a random number to be compared with a threshold to know if the reaction runs from A to B or the other way around.

File format

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/File_format 

A file format is a standard way that information is encoded for storage in a computer file. It specifies how bits are used to encode information in a digital storage medium. File formats may be either proprietary or free.

Some file formats are designed for very particular types of data: PNG files, for example, store bitmapped images using lossless data compression. Other file formats, however, are designed for storage of several different types of data: the Ogg format can act as a container for different types of multimedia including any combination of audio and video, with or without text (such as subtitles), and metadata. A text file can contain any stream of characters, including possible control characters, and is encoded in one of various character encoding schemes. Some file formats, such as HTML, scalable vector graphics, and the source code of computer software are text files with defined syntaxes that allow them to be used for specific purposes.

Specifications

File formats often have a published specification describing the encoding method and enabling testing of program intended functionality. Not all formats have freely available specification documents, partly because some developers view their specification documents as trade secrets, and partly because other developers never author a formal specification document, letting precedent set by other already existing programs that use the format define the format via how these existing programs use it.

If the developer of a format doesn't publish free specifications, another developer looking to utilize that kind of file must either reverse engineer the file to find out how to read it or acquire the specification document from the format's developers for a fee and by signing a non-disclosure agreement. The latter approach is possible only when a formal specification document exists. Both strategies require significant time, money, or both; therefore, file formats with publicly available specifications tend to be supported by more programs.

Patents

Patent law, rather than copyright, is more often used to protect a file format. Although patents for file formats are not directly permitted under US law, some formats encode data using patented algorithms. For example, using compression with the GIF file format requires the use of a patented algorithm, and though the patent owner did not initially enforce their patent, they later began collecting royalty fees. This has resulted in a significant decrease in the use of GIFs, and is partly responsible for the development of the alternative PNG format. However, the GIF patent expired in the US in mid-2003, and worldwide in mid-2004.

Identifying file type

Different operating systems have traditionally taken different approaches to determining a particular file's format, with each approach having its own advantages and disadvantages. Most modern operating systems and individual applications need to use all of the following approaches to read "foreign" file formats, if not work with them completely.

Filename extension

Main article: Filename extension

One popular method used by many operating systems, including Windows, macOS, CP/M, DOS, VMS, and VM/CMS, is to determine the format of a file based on the end of its name, more specifically the letters following the final period. This portion of the filename is known as the filename extension. For example, HTML documents are identified by names that end with .html (or .htm), and GIF images by .gif. In the original FAT file system, file names were limited to an eight-character identifier and a three-character extension, known as an 8.3 filename. There are a limited number of three-letter extensions, which can cause a given extension to be used by more than one program. Many formats still use three-character extensions even though modern operating systems and application programs no longer have this limitation. Since there is no standard list of extensions, more than one format can use the same extension, which can confuse both the operating system and users.

One artifact of this approach is that the system can easily be tricked into treating a file as a different format simply by renaming it — an HTML file can, for instance, be easily treated as plain text by renaming it from filename.html to filename.txt. Although this strategy was useful to expert users who could easily understand and manipulate this information, it was often confusing to less technical users, who could accidentally make a file unusable (or "lose" it) by renaming it incorrectly.

This led most versions of Windows and Mac OS to hide the extension when listing files. This prevents the user from accidentally changing the file type, and allows expert users to turn this feature off and display the extensions.

Hiding the extension, however, can create the appearance of two or more identical filenames in the same folder. For example, a company logo may be needed both in .eps format (for publishing) and .png format (for web sites). With the extensions visible, these would appear as the unique filenames: "CompanyLogo.eps" and "CompanyLogo.png". On the other hand, hiding the extensions would make both appear as "CompanyLogo", which can lead to confusion.

Hiding extensions can also pose a security risk. For example, a malicious user could create an executable program with an innocent name such as "Holiday photo.jpg.exe". The ".exe" would be hidden and an unsuspecting user would see "Holiday photo.jpg", which would appear to be a JPEG image, usually unable to harm the machine. However, the operating system would still see the ".exe" extension and run the program, which would then be able to cause harm to the computer. The same is true with files with only one extension: as it is not shown to the user, no information about the file can be deduced without explicitly investigating the file. To further trick users, it is possible to store an icon inside the program, in which case some operating systems' icon assignment for the executable file (.exe) would be overridden with an icon commonly used to represent JPEG images, making the program look like an image. Extensions can also be spoofed: some Microsoft Word macro viruses create a Word file in template format and save it with a .doc extension. Since Word generally ignores extensions and looks at the format of the file, these would open as templates, execute, and spread the virus. This represents a practical problem for Windows systems where extension-hiding is turned on by default.

Internal metadata

A second way to identify a file format is to use information regarding the format stored inside the file itself, either information meant for this purpose or binary strings that happen to always be in specific locations in files of some formats. Since the easiest place to locate them is at the beginning, such area is usually called a file header when it is greater than a few bytes, or a magic number if it is just a few bytes long.

File header

The metadata contained in a file header are usually stored at the start of the file, but might be present in other areas too, often including the end, depending on the file format or the type of data contained. Character-based (text) files usually have character-based headers, whereas binary formats usually have binary headers, although this is not a rule. Text-based file headers usually take up more space, but being human-readable, they can easily be examined by using simple software such as a text editor or a hexadecimal editor.

As well as identifying the file format, file headers may contain metadata about the file and its contents. For example, most image files store information about image format, size, resolution and color space, and optionally authoring information such as who made the image, when and where it was made, what camera model and photographic settings were used (Exif), and so on. Such metadata may be used by software reading or interpreting the file during the loading process and afterwards.

File headers may be used by an operating system to quickly gather information about a file without loading it all into memory, but doing so uses more of a computer's resources than reading directly from the directory information. For instance, when a graphic file manager has to display the contents of a folder, it must read the headers of many files before it can display the appropriate icons, but these will be located in different places on the storage medium thus taking longer to access. A folder containing many files with complex metadata such as thumbnail information may require considerable time before it can be displayed.

If a header is binary hard-coded such that the header itself needs complex interpretation in order to be recognized, especially for metadata content protection's sake, there is a risk that the file format can be misinterpreted. It may even have been badly written at the source. This can result in corrupt metadata which, in extremely bad cases, might even render the file unreadable.

A more complex example of file headers are those used for wrapper (or container) file formats.

Magic number

See also: Magic number (programming) § In files

One way to incorporate file type metadata, often associated with Unix and its derivatives, is to just store a "magic number" inside the file itself. Originally, this term was used for a specific set of 2-byte identifiers at the beginnings of files, but since any binary sequence can be regarded as a number, any feature of a file format which uniquely distinguishes it can be used for identification. GIF images, for instance, always begin with the ASCII representation of either GIF87a or GIF89a, depending upon the standard to which they adhere. Many file types, especially plain-text files, are harder to spot by this method. HTML files, for example, might begin with the string <html> (which is not case sensitive), or an appropriate document type definition that starts with <!DOCTYPE HTML>, or, for XHTML, the XML identifier, which begins with <?xml. The files can also begin with HTML comments, random text, or several empty lines, but still be usable HTML.

The magic number approach offers better guarantees that the format will be identified correctly, and can often determine more precise information about the file. Since reasonably reliable "magic number" tests can be fairly complex, and each file must effectively be tested against every possibility in the magic database, this approach is relatively inefficient, especially for displaying large lists of files (in contrast, file name and metadata-based methods need to check only one piece of data, and match it against a sorted index). Also, data must be read from the file itself, increasing latency as opposed to metadata stored in the directory. Where file types don't lend themselves to recognition in this way, the system must fall back to metadata. It is, however, the best way for a program to check if the file it has been told to process is of the correct format: while the file's name or metadata may be altered independently of its content, failing a well-designed magic number test is a pretty sure sign that the file is either corrupt or of the wrong type. On the other hand, a valid magic number does not guarantee that the file is not corrupt or is of a correct type.

So-called shebang lines in script files are a special case of magic numbers. Here, the magic number is human-readable text that identifies a specific command interpreter and options to be passed to the command interpreter.

Another operating system using magic numbers is AmigaOS, where magic numbers were called "Magic Cookies" and were adopted as a standard system to recognize executables in Hunk executable file format and also to let single programs, tools and utilities deal automatically with their saved data files, or any other kind of file types when saving and loading data. This system was then enhanced with the Amiga standard Datatype recognition system. Another method was the FourCC method, originating in OSType on Macintosh, later adapted by Interchange File Format (IFF) and derivatives.

External metadata

A final way of storing the format of a file is to explicitly store information about the format in the file system, rather than within the file itself.

This approach keeps the metadata separate from both the main data and the name, but is also less portable than either filename extensions or "magic numbers", since the format has to be converted from filesystem to filesystem. While this is also true to an extent with filename extensions—for instance, for compatibility with MS-DOS's three character limit—most forms of storage have a roughly equivalent definition of a file's data and name, but may have varying or no representation of further metadata.

Note that zip files or archive files solve the problem of handling metadata. A utility program collects multiple files together along with metadata about each file and the folders/directories they came from all within one new file (e.g. a zip file with extension .zip). The new file is also compressed and possibly encrypted, but now is transmissible as a single file across operating systems by FTP transmissions or sent by email as an attachment. At the destination, the single file received has to be unzipped by a compatible utility to be useful. The problems of handling metadata are solved this way using zip files or archive files.

Mac OS type-codes

The Mac OS' Hierarchical File System stores codes for creator and type as part of the directory entry for each file. These codes are referred to as OSTypes. These codes could be any 4-byte sequence, but were often selected so that the ASCII representation formed a sequence of meaningful characters, such as an abbreviation of the application's name or the developer's initials. For instance a HyperCard "stack" file has a creator of WILD (from Hypercard's previous name, "WildCard") and a type of STAK. The BBEdit text editor has a creator code of R*ch referring to its original programmer, Rich Siegel. The type code specifies the format of the file, while the creator code specifies the default program to open it with when double-clicked by the user. For example, the user could have several text files all with the type code of TEXT, but which each open in a different program, due to having differing creator codes. This feature was intended so that, for example, human-readable plain-text files could be opened in a general purpose text editor, while programming or HTML code files would open in a specialized editor or IDE. However, this feature was often the source of user confusion, as which program would launch when the files were double-clicked was often unpredictable. RISC OS uses a similar system, consisting of a 12-bit number which can be looked up in a table of descriptions—e.g. the hexadecimal number FF5 is "aliased" to PoScript, representing a PostScript file.

Mac OS X uniform type identifiers (UTIs)

Main article: Uniform Type Identifier

A Uniform Type Identifier (UTI) is a method used in macOS for uniquely identifying "typed" classes of entity, such as file formats. It was developed by Apple as a replacement for OSType (type & creator codes).

The UTI is a Core Foundation string, which uses a reverse-DNS string. Some common and standard types use a domain called public (e.g. public.png for a Portable Network Graphics image), while other domains can be used for third-party types (e.g. com.adobe.pdf for Portable Document Format). UTIs can be defined within a hierarchical structure, known as a conformance hierarchy. Thus, public.png conforms to a supertype of public.image, which itself conforms to a supertype of public.data. A UTI can exist in multiple hierarchies, which provides great flexibility.

In addition to file formats, UTIs can also be used for other entities which can exist in macOS, including:

• Pasteboard data
• Folders (directories)
• Translatable types (as handled by the Translation Manager)
• Bundles
• Frameworks
• Streaming data
• Aliases and symlinks

OS/2 extended attributes

The HPFS, FAT12 and FAT16 (but not FAT32) filesystems allow the storage of "extended attributes" with files. These comprise an arbitrary set of triplets with a name, a coded type for the value and a value, where the names are unique and values can be up to 64 KB long. There are standardized meanings for certain types and names (under OS/2). One such is that the ".TYPE" extended attribute is used to determine the file type. Its value comprises a list of one or more file types associated with the file, each of which is a string, such as "Plain Text" or "HTML document". Thus a file may have several types.

The NTFS filesystem also allows storage of OS/2 extended attributes, as one of the file forks, but this feature is merely present to support the OS/2 subsystem (not present in XP), so the Win32 subsystem treats this information as an opaque block of data and does not use it. Instead, it relies on other file forks to store meta-information in Win32-specific formats. OS/2 extended attributes can still be read and written by Win32 programs, but the data must be entirely parsed by applications.

POSIX extended attributes

On Unix and Unix-like systems, the ext2, ext3, ext4, ReiserFS version 3, XFS, JFS, FFS, and HFS+ filesystems allow the storage of extended attributes with files. These include an arbitrary list of "name=value" strings, where the names are unique and a value can be accessed through its related name.

PRONOM unique identifiers (PUIDs)

The PRONOM Persistent Unique Identifier (PUID) is an extensible scheme of persistent, unique and unambiguous identifiers for file formats, which has been developed by The National Archives of the UK as part of its PRONOM technical registry service. PUIDs can be expressed as Uniform Resource Identifiers using the info:pronom/ namespace. Although not yet widely used outside of UK government and some digital preservation programmes, the PUID scheme does provide greater granularity than most alternative schemes.

MIME types

MIME types are widely used in many Internet-related applications, and increasingly elsewhere, although their usage for on-disc type information is rare. These consist of a standardised system of identifiers (managed by IANA) consisting of a type and a sub-type, separated by a slash—for instance, text/html or image/gif. These were originally intended as a way of identifying what type of file was attached to an e-mail, independent of the source and target operating systems. MIME types identify files on BeOS, AmigaOS 4.0 and MorphOS, as well as store unique application signatures for application launching. In AmigaOS and MorphOS the Mime type system works in parallel with Amiga specific Datatype system.

There are problems with the MIME types though; several organisations and people have created their own MIME types without registering them properly with IANA, which makes the use of this standard awkward in some cases.

File format identifiers (FFIDs)

File format identifiers is another, not widely used way to identify file formats according to their origin and their file category. It was created for the Description Explorer suite of software. It is composed of several digits of the form NNNNNNNNN-XX-YYYYYYY. The first part indicates the organisation origin/maintainer (this number represents a value in a company/standards organisation database), the 2 following digits categorize the type of file in hexadecimal. The final part is composed of the usual filename extension of the file or the international standard number of the file, padded left with zeros. For example, the PNG file specification has the FFID of 000000001-31-0015948 where 31 indicates an image file, 0015948 is the standard number and 000000001 indicates the International Organization for Standardization (ISO).

File content based format identification

Another but less popular way to identify the file format is to examine the file contents for distinguishable patterns among file types. The contents of a file are a sequence of bytes and a byte has 256 unique permutations (0–255). Thus, counting the occurrence of byte patterns that is often referred as byte frequency distribution gives distinguishable patterns to identify file types. There are many content-based file type identification schemes that use byte frequency distribution to build the representative models for file type and use any statistical and data mining techniques to identify file types.

File structure

There are several types of ways to structure data in a file. The most usual ones are described below.

Unstructured formats (raw memory dumps)

Earlier file formats used raw data formats that consisted of directly dumping the memory images of one or more structures into the file.

This has several drawbacks. Unless the memory images also have reserved spaces for future extensions, extending and improving this type of structured file is very difficult. It also creates files that might be specific to one platform or programming language (for example a structure containing a Pascal string is not recognized as such in C). On the other hand, developing tools for reading and writing these types of files is very simple.

The limitations of the unstructured formats led to the development of other types of file formats that could be easily extended and be backward compatible at the same time.

Chunk-based formats

In this kind of file structure, each piece of data is embedded in a container that somehow identifies the data. The container's scope can be identified by start- and end-markers of some kind, by an explicit length field somewhere, or by fixed requirements of the file format's definition.

Throughout the 1970s, many programs used formats of this general kind. For example, word-processors such as troff, Script, and Scribe, and database export files such as CSV. Electronic Arts and Commodore-Amiga also used this type of file format in 1985, with their IFF (Interchange File Format) file format.

A container is sometimes called a "chunk", although "chunk" may also imply that each piece is small, and/or that chunks do not contain other chunks; many formats do not impose those requirements.

The information that identifies a particular "chunk" may be called many different things, often terms including "field name", "identifier", "label", or "tag". The identifiers are often human-readable, and classify parts of the data: for example, as a "surname", "address", "rectangle", "font name", etc. These are not the same thing as identifiers in the sense of a database key or serial number (although an identifier may well identify its associated data as such a key).

With this type of file structure, tools that do not know certain chunk identifiers simply skip those that they do not understand. Depending on the actual meaning of the skipped data, this may or may not be useful (CSS explicitly defines such behavior).

This concept has been used again and again by RIFF (Microsoft-IBM equivalent of IFF), PNG, JPEG storage, DER (Distinguished Encoding Rules) encoded streams and files (which were originally described in CCITT X.409:1984 and therefore predate IFF), and Structured Data Exchange Format (SDXF).

Indeed, any data format must somehow identify the significance of its component parts, and embedded boundary-markers are an obvious way to do so:

• MIME headers do this with a colon-separated label at the start of each logical line. MIME headers cannot contain other MIME headers, though the data content of some headers has sub-parts that can be extracted by other conventions.
• CSV and similar files often do this using a header records with field names, and with commas to mark the field boundaries. Like MIME, CSV has no provision for structures with more than one level.
• XML and its kin can be loosely considered a kind of chunk-based format, since data elements are identified by markup that is akin to chunk identifiers. However, it has formal advantages such as schemas and validation, as well as the ability to represent more complex structures such as trees, DAGs, and charts. If XML is considered a "chunk" format, then SGML and its predecessor IBM GML are among the earliest examples of such formats.
• JSON is similar to XML without schemas, cross-references, or a definition for the meaning of repeated field-names, and is often convenient for programmers.
• YAML is similar to JSON, but use indentation to separate data chunks and aim to be more human-readable than JSON or XML.
• Protocol Buffers are in turn similar to JSON, notably replacing boundary-markers in the data with field numbers, which are mapped to/from names by some external mechanism.

Directory-based formats

This is another extensible format, that closely resembles a file system (OLE Documents are actual filesystems), where the file is composed of 'directory entries' that contain the location of the data within the file itself as well as its signatures (and in certain cases its type). Good examples of these types of file structures are disk images, OLE documents TIFF, libraries. ODT and DOCX, being PKZIP-based are chunked and also carry a directory.