Algorithms for calculating variance

From Wikipedia, the free encyclopedia
https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance

Algorithms for calculating variance play a major role in computational statistics. A key difficulty in the design of good algorithms for this problem is that formulas for the variance may involve sums of squares, which can lead to numerical instability as well as to arithmetic overflow when dealing with large values.

Naïve algorithm

A formula for calculating the variance of an entire population of size N is:

${\displaystyle {\sigma }^2=\overline{(x^2)}-{\overline{x}}^2=\frac{\sum _{i=1}^N{x}_i^2-(\sum _{i=1}^N{x}_i{)}^2/N}{N}.}$

Using Bessel's correction to calculate an unbiased estimate of the population variance from a finite sample of n observations, the formula is:

${\displaystyle s^2=\left(\frac{\sum _{i=1}^n{x}_i^2}{n}-{\left(\frac{\sum _{i=1}^n{x}_i}{n}\right)}^2\right)\cdot \frac{n}{n-1}.}$

Therefore, a naïve algorithm to calculate the estimated variance is given by the following:

• Let n ← 0, Sum ← 0, SumSq ← 0
• For each datum x:
  • n ← n + 1
  • Sum ← Sum + x
  • SumSq ← SumSq + x × x
• Var = (SumSq − (Sum × Sum) / n) / (n − 1)

This algorithm can easily be adapted to compute the variance of a finite population: simply divide by n instead of n − 1 on the last line.

Because SumSq and (Sum×Sum)/n can be very similar numbers, cancellation can lead to the precision of the result to be much less than the inherent precision of the floating-point arithmetic used to perform the computation. Thus this algorithm should not be used in practice, and several alternate, numerically stable, algorithms have been proposed. This is particularly bad if the standard deviation is small relative to the mean.

Computing shifted data

The variance is invariant with respect to changes in a location parameter, a property which can be used to avoid the catastrophic cancellation in this formula.

${\displaystyle \mathrm{Var}(X-K)=\mathrm{Var}(X).}$

with ${\displaystyle K}$ any constant, which leads to the new formula

${\displaystyle {\sigma }^2=\frac{\sum _{i=1}^n(x_i-K{)}^2-(\sum _{i=1}^n(x_i-K){)}^2/n}{n-1}.}$

the closer ${\displaystyle K}$ is to the mean value the more accurate the result will be, but just choosing a value inside the samples range will guarantee the desired stability. If the values ${\displaystyle (x_i-K)}$ are small then there are no problems with the sum of its squares, on the contrary, if they are large it necessarily means that the variance is large as well. In any case the second term in the formula is always smaller than the first one therefore no cancellation may occur.

If just the first sample is taken as ${\displaystyle K}$ the algorithm can be written in Python programming language as

def shifted_data_variance(data):
    if len(data) < 2:
        return 0.0
    K = data[0]
    n = Ex = Ex2 = 0.0
    for x in data:
        n += 1
        Ex += x - K
        Ex2 += (x - K) ** 2
    variance = (Ex2 - Ex**2 / n) / (n - 1)
    # use n instead of (n-1) if want to compute the exact variance of the given data
    # use (n-1) if data are samples of a larger population
    return variance

This formula also facilitates the incremental computation that can be expressed as

K = Ex = Ex2 = 0.0
n = 0


def add_variable(x):
    global K, n, Ex, Ex2
    if n == 0:
        K = x
    n += 1
    Ex += x - K
    Ex2 += (x - K) ** 2

def remove_variable(x):
    global K, n, Ex, Ex2
    n -= 1
    Ex -= x - K
    Ex2 -= (x - K) ** 2

def get_mean():
    global K, n, Ex
    return K + Ex / n

def get_variance():
    global n, Ex, Ex2
    return (Ex2 - Ex**2 / n) / (n - 1)

Two-pass algorithm

An alternative approach, using a different formula for the variance, first computes the sample mean,

${\displaystyle \overline{x}=\frac{\sum _{j=1}^n{x}_j}{n},}$

and then computes the sum of the squares of the differences from the mean,

${\displaystyle \text{sample variance}=s^2={\displaystyle \frac{\sum _{i=1}^n(x_i-\overline{x}{)}^2}{n-1}},}$

where s is the standard deviation. This is given by the following code:

def two_pass_variance(data):
    n = len(data)
    mean = sum(data) / n
    variance = sum((x - mean) ** 2 for x in data) / (n - 1)
    return variance

This algorithm is numerically stable if n is small. However, the results of both of these simple algorithms ("naïve" and "two-pass") can depend inordinately on the ordering of the data and can give poor results for very large data sets due to repeated roundoff error in the accumulation of the sums. Techniques such as compensated summation can be used to combat this error to a degree.

Welford's online algorithm

It is often useful to be able to compute the variance in a single pass, inspecting each value ${\displaystyle x_i}$ only once; for example, when the data is being collected without enough storage to keep all the values, or when costs of memory access dominate those of computation. For such an online algorithm, a recurrence relation is required between quantities from which the required statistics can be calculated in a numerically stable fashion.

The following formulas can be used to update the mean and (estimated) variance of the sequence, for an additional element x_n. Here, ${\overline{x}}_n=\frac{1}{n}\sum _{i=1}^n{x}_i$ denotes the sample mean of the first n samples ${\displaystyle (x_1,\dots ,x_n)}$, ${\sigma }_n^2=\frac{1}{n}\sum _{i=1}^n{\left(x_i-{\overline{x}}_n\right)}^2$ their biased sample variance, and $s_n^2=\frac{1}{n-1}\sum _{i=1}^n{\left(x_i-{\overline{x}}_n\right)}^2$ their unbiased sample variance.

${\displaystyle {\overline{x}}_n=\frac{(n-1){\overline{x}}_{n-1}+x_n}{n}={\overline{x}}_{n-1}+\frac{x_n-{\overline{x}}_{n-1}}{n}}$

${\displaystyle {\sigma }_n^2=\frac{(n-1){\sigma }_{n-1}^2+(x_n-{\overline{x}}_{n-1})(x_n-{\overline{x}}_n)}{n}={\sigma }_{n-1}^2+\frac{(x_n-{\overline{x}}_{n-1})(x_n-{\overline{x}}_n)-{\sigma }_{n-1}^2}{n}.}$

${\displaystyle s_n^2=\frac{n-2}{n-1}{s}_{n-1}^2+\frac{(x_n-{\overline{x}}_{n-1}{)}^2}{n}=s_{n-1}^2+\frac{(x_n-{\overline{x}}_{n-1}{)}^2}{n}-\frac{s_{n-1}^2}{n-1},n>1}$

These formulas suffer from numerical instability, as they repeatedly subtract a small number from a big number which scales with n. A better quantity for updating is the sum of squares of differences from the current mean, $\sum _{i=1}^n(x_i-{\overline{x}}_n{)}^2$, here denoted ${\displaystyle M_{2,n}}$:

${\displaystyle \begin{array}{rl}{M}_{2,n}& =M_{2,n-1}+(x_n-{\overline{x}}_{n-1})(x_n-{\overline{x}}_n)\\ {\sigma }_n^2& =\frac{M_{2,n}}{n}\\ s_n^2& =\frac{M_{2,n}}{n-1}\end{array}}$

This algorithm was found by Welford, and it has been thoroughly analyzed. It is also common to denote ${\displaystyle M_k={\overline{x}}_k}$ and ${\displaystyle S_k=M_{2,k}}$.

An example Python implementation for Welford's algorithm is given below.

# For a new value new_value, compute the new count, new mean, the new M2.
# mean accumulates the mean of the entire dataset
# M2 aggregates the squared distance from the mean
# count aggregates the number of samples seen so far
def update(existing_aggregate, new_value):
    (count, mean, M2) = existing_aggregate
    count += 1
    delta = new_value - mean
    mean += delta / count
    delta2 = new_value - mean
    M2 += delta * delta2
    return (count, mean, M2)

# Retrieve the mean, variance and sample variance from an aggregate
def finalize(existing_aggregate):
    (count, mean, M2) = existing_aggregate
    if count < 2:
        return float("nan")
    else:
        (mean, variance, sample_variance) = (mean, M2 / count, M2 / (count - 1))
        return (mean, variance, sample_variance)

This algorithm is much less prone to loss of precision due to catastrophic cancellation, but might not be as efficient because of the division operation inside the loop. For a particularly robust two-pass algorithm for computing the variance, one can first compute and subtract an estimate of the mean, and then use this algorithm on the residuals.

The parallel algorithm below illustrates how to merge multiple sets of statistics calculated online.

Weighted incremental algorithm

The algorithm can be extended to handle unequal sample weights, replacing the simple counter n with the sum of weights seen so far. West (1979) suggests this incremental algorithm:

def weighted_incremental_variance(data_weight_pairs):
    w_sum = w_sum2 = mean = S = 0

    for x, w in data_weight_pairs:
        w_sum = w_sum + w
        w_sum2 = w_sum2 + w**2
        mean_old = mean
        mean = mean_old + (w / w_sum) * (x - mean_old)
        S = S + w * (x - mean_old) * (x - mean)

    population_variance = S / w_sum
    # Bessel's correction for weighted samples
    # Frequency weights
    sample_frequency_variance = S / (w_sum - 1)
 	
    # Reliability weights
    sample_reliability_variance = S / (1 - w_sum2 / (w_sum**2))

Further information: Weighted arithmetic mean § Weighted sample variance

Parallel algorithm

Chan et al. note that Welford's online algorithm detailed above is a special case of an algorithm that works for combining arbitrary sets ${\displaystyle A}$ and ${\displaystyle B}$:

${\displaystyle \begin{array}{rl}{n}_{AB}& =n_A+n_B\\ \delta & ={\overline{x}}_B-{\overline{x}}_A\\ {\overline{x}}_{AB}& ={\overline{x}}_A+\delta \cdot \frac{n_B}{n_{AB}}\\ M_{2,AB}& =M_{2,A}+M_{2,B}+{\delta }^2\cdot \frac{n_A{n}_B}{n_{AB}}\end{array}}$.

This may be useful when, for example, multiple processing units may be assigned to discrete parts of the input.

Chan's method for estimating the mean is numerically unstable when ${\displaystyle n_A\approx n_B}$ and both are large, because the numerical error in ${\displaystyle \delta ={\overline{x}}_B-{\overline{x}}_A}$ is not scaled down in the way that it is in the ${\displaystyle n_B=1}$ case. In such cases, prefer ${\overline{x}}_{AB}=\frac{n_A{\overline{x}}_A+n_B{\overline{x}}_B}{n_{AB}}$.

def parallel_variance(n_a, avg_a, M2_a, n_b, avg_b, M2_b):
    n = n_a + n_b
    delta = avg_b - avg_a
    M2 = M2_a + M2_b + delta**2 * n_a * n_b / n
    var_ab = M2 / (n - 1)
    return var_ab

This can be generalized to allow parallelization with AVX, with GPUs, and computer clusters, and to covariance.

Example

Assume that all floating point operations use standard IEEE 754 double-precision arithmetic. Consider the sample (4, 7, 13, 16) from an infinite population. Based on this sample, the estimated population mean is 10, and the unbiased estimate of population variance is 30. Both the naïve algorithm and two-pass algorithm compute these values correctly.

Next consider the sample (10⁸ + 4, 10⁸ + 7, 10⁸ + 13, 10⁸ + 16), which gives rise to the same estimated variance as the first sample. The two-pass algorithm computes this variance estimate correctly, but the naïve algorithm returns 29.333333333333332 instead of 30.

While this loss of precision may be tolerable and viewed as a minor flaw of the naïve algorithm, further increasing the offset makes the error catastrophic. Consider the sample (10⁹ + 4, 10⁹ + 7, 10⁹ + 13, 10⁹ + 16). Again the estimated population variance of 30 is computed correctly by the two-pass algorithm, but the naïve algorithm now computes it as −170.66666666666666. This is a serious problem with naïve algorithm and is due to catastrophic cancellation in the subtraction of two similar numbers at the final stage of the algorithm.

Higher-order statistics

Terriberry extends Chan's formulae to calculating the third and fourth central moments, needed for example when estimating skewness and kurtosis:

${\displaystyle \begin{array}{rl}{M}_{3,X}=M_{3,A}+M_{3,B}& +{\delta }^3\frac{n_A{n}_B(n_A-n_B)}{n_X^2}+3\delta \frac{n_A{M}_{2,B}-n_B{M}_{2,A}}{n_X}\\ M_{4,X}=M_{4,A}+M_{4,B}& +{\delta }^4\frac{n_A{n}_B\left(n_A^2-n_A{n}_B+n_B^2\right)}{n_X^3}\\ & +6{\delta }^2\frac{n_A^2{M}_{2,B}+n_B^2{M}_{2,A}}{n_X^2}+4\delta \frac{n_A{M}_{3,B}-n_B{M}_{3,A}}{n_X}\end{array}}$

Here the ${\displaystyle M_k}$ are again the sums of powers of differences from the mean $\sum (x-\overline{x}{)}^k$, giving

${\displaystyle \begin{array}{rl}& \text{skewness}=g_1=\frac{\sqrt{n}{M}_3}{M_2^{3/2}},\\ & \text{kurtosis}=g_2=\frac{n{M}_4}{M_2^2}-3.\end{array}}$

For the incremental case (i.e., ${\displaystyle B=\{x\}}$), this simplifies to:

${\displaystyle \begin{array}{rl}\delta & =x-m\\ m^{\prime }& =m+\frac{\delta }{n}\\ M_2^{\prime }& =M_2+{\delta }^2\frac{n-1}{n}\\ M_3^{\prime }& =M_3+{\delta }^3\frac{(n-1)(n-2)}{n^2}-\frac{3\delta M_2}{n}\\ M_4^{\prime }& =M_4+\frac{{\delta }^4(n-1)(n^2-3n+3)}{n^3}+\frac{6{\delta }^2{M}_2}{n^2}-\frac{4\delta M_3}{n}\end{array}}$

By preserving the value ${\displaystyle \delta /n}$, only one division operation is needed and the higher-order statistics can thus be calculated for little incremental cost.

An example of the online algorithm for kurtosis implemented as described is:

def online_kurtosis(data):
    n = mean = M2 = M3 = M4 = 0

    for x in data:
        n1 = n
        n = n + 1
        delta = x - mean
        delta_n = delta / n
        delta_n2 = delta_n**2
        term1 = delta * delta_n * n1
        mean = mean + delta_n
        M4 = M4 + term1 * delta_n2 * (n**2 - 3*n + 3) + 6 * delta_n2 * M2 - 4 * delta_n * M3
        M3 = M3 + term1 * delta_n * (n - 2) - 3 * delta_n * M2
        M2 = M2 + term1

    # Note, you may also calculate variance using M2, and skewness using M3
    # Caution: If all the inputs are the same, M2 will be 0, resulting in a division by 0.
    kurtosis = (n * M4) / (M2**2) - 3
    return kurtosis

Pébaÿ further extends these results to arbitrary-order central moments, for the incremental and the pairwise cases, and subsequently Pébaÿ et al. for weighted and compound moments. One can also find there similar formulas for covariance.

Choi and Sweetman offer two alternative methods to compute the skewness and kurtosis, each of which can save substantial computer memory requirements and CPU time in certain applications. The first approach is to compute the statistical moments by separating the data into bins and then computing the moments from the geometry of the resulting histogram, which effectively becomes a one-pass algorithm for higher moments. One benefit is that the statistical moment calculations can be carried out to arbitrary accuracy such that the computations can be tuned to the precision of, e.g., the data storage format or the original measurement hardware. A relative histogram of a random variable can be constructed in the conventional way: the range of potential values is divided into bins and the number of occurrences within each bin are counted and plotted such that the area of each rectangle equals the portion of the sample values within that bin:

${\displaystyle H(x_k)=\frac{h(x_k)}{A}}$

where ${\displaystyle h(x_k)}$ and ${\displaystyle H(x_k)}$ represent the frequency and the relative frequency at bin ${\displaystyle x_k}$ and $A=\sum _{k=1}^{K}h(x_k)\mathrm{\Delta }{x}_k$ is the total area of the histogram. After this normalization, the ${\displaystyle n}$ raw moments and central moments of ${\displaystyle x(t)}$ can be calculated from the relative histogram:

${\displaystyle m_n^{(h)}=\sum _{k=1}^K{x}_k^{n}H(x_k)\mathrm{\Delta }{x}_k=\frac{1}{A}\sum _{k=1}^K{x}_k^{n}h(x_k)\mathrm{\Delta }{x}_k}$

${\displaystyle {\theta }_n^{(h)}=\sum _{k=1}^K(x_k-m_1^{(h)}{)}^{n}H(x_k)\mathrm{\Delta }{x}_k=\frac{1}{A}\sum _{k=1}^K(x_k-m_1^{(h)}{)}^{n}h(x_k)\mathrm{\Delta }{x}_k}$

where the superscript ${\displaystyle {}^{(h)}}$ indicates the moments are calculated from the histogram. For constant bin width ${\displaystyle \mathrm{\Delta }{x}_k=\mathrm{\Delta }x}$ these two expressions can be simplified using ${\displaystyle I=A/\mathrm{\Delta }x}$:

${\displaystyle m_n^{(h)}=\frac{1}{I}\sum _{k=1}^K{x}_k^{n}h(x_k)}$

${\displaystyle {\theta }_n^{(h)}=\frac{1}{I}\sum _{k=1}^K(x_k-m_1^{(h)}{)}^{n}h(x_k)}$

The second approach from Choi and Sweetman is an analytical methodology to combine statistical moments from individual segments of a time-history such that the resulting overall moments are those of the complete time-history. This methodology could be used for parallel computation of statistical moments with subsequent combination of those moments, or for combination of statistical moments computed at sequential times.

If ${\displaystyle Q}$ sets of statistical moments are known: ${\displaystyle ({\gamma }_{0,q},{\mu }_q,{\sigma }_q^2,{\alpha }_{3,q},{\alpha }_{4,q})}$ for ${\displaystyle q=1,2,\dots ,Q}$, then each ${\displaystyle {\gamma }_n}$ can be expressed in terms of the equivalent ${\displaystyle n}$ raw moments:

${\displaystyle {\gamma }_{n,q}=m_{n,q}{\gamma }_{0,q}\text{for}n=1,2,3,4\text{ and }q=1,2,\dots ,Q}$

where ${\displaystyle {\gamma }_{0,q}}$ is generally taken to be the duration of the ${\displaystyle q^{th}}$ time-history, or the number of points if ${\displaystyle \mathrm{\Delta }t}$ is constant.

The benefit of expressing the statistical moments in terms of ${\displaystyle \gamma }$ is that the ${\displaystyle Q}$ sets can be combined by addition, and there is no upper limit on the value of ${\displaystyle Q}$.

${\displaystyle {\gamma }_{n,c}=\sum _{q=1}^Q{\gamma }_{n,q}\text{for }n=0,1,2,3,4}$

where the subscript ${\displaystyle {}_c}$ represents the concatenated time-history or combined ${\displaystyle \gamma }$. These combined values of ${\displaystyle \gamma }$ can then be inversely transformed into raw moments representing the complete concatenated time-history

${\displaystyle m_{n,c}=\frac{{\gamma }_{n,c}}{{\gamma }_{0,c}}\text{for }n=1,2,3,4}$

Known relationships between the raw moments (${\displaystyle m_n}$) and the central moments (${\displaystyle {\theta }_n=\mathrm{E}[(x-\mu {)}^n])}$) are then used to compute the central moments of the concatenated time-history. Finally, the statistical moments of the concatenated history are computed from the central moments:

${\displaystyle {\mu }_c=m_{1,c}{\sigma }_c^2={\theta }_{2,c}{\alpha }_{3,c}=\frac{{\theta }_{3,c}}{{\sigma }_c^3}{\alpha }_{4,c}=\frac{{\theta }_{4,c}}{{\sigma }_c^4}-3}$

Covariance

Very similar algorithms can be used to compute the covariance.

Naïve algorithm

The naïve algorithm is

${\displaystyle \mathrm{Cov}(X,Y)=\frac{\sum _{i=1}^n{x}_i{y}_i-(\sum _{i=1}^n{x}_i)(\sum _{i=1}^n{y}_i)/n}{n}.}$

For the algorithm above, one could use the following Python code:

def naive_covariance(data1, data2):
    n = len(data1)
    sum1 = sum(data1)
    sum2 = sum(data2)
    sum12 = sum([i1 * i2 for i1, i2 in zip(data1, data2)])

    covariance = (sum12 - sum1 * sum2 / n) / n
    return covariance

With estimate of the mean

As for the variance, the covariance of two random variables is also shift-invariant, so given any two constant values ${\displaystyle k_x}$ and ${\displaystyle k_y,}$ it can be written:

${\displaystyle \mathrm{Cov}(X,Y)=\mathrm{Cov}(X-k_x,Y-k_y)={\displaystyle \frac{\sum _{i=1}^n(x_i-k_x)(y_i-k_y)-(\sum _{i=1}^n(x_i-k_x))(\sum _{i=1}^n(y_i-k_y))/n}{n}}.}$

and again choosing a value inside the range of values will stabilize the formula against catastrophic cancellation as well as make it more robust against big sums. Taking the first value of each data set, the algorithm can be written as:

def shifted_data_covariance(data_x, data_y):
    n = len(data_x)
    if n < 2:
        return 0
    kx = data_x[0]
    ky = data_y[0]
    Ex = Ey = Exy = 0
    for ix, iy in zip(data_x, data_y):
        Ex += ix - kx
        Ey += iy - ky
        Exy += (ix - kx) * (iy - ky)
    return (Exy - Ex * Ey / n) / n

Two-pass

The two-pass algorithm first computes the sample means, and then the covariance:

${\displaystyle \overline{x}=\sum _{i=1}^n{x}_i/n}$

${\displaystyle \overline{y}=\sum _{i=1}^n{y}_i/n}$

${\displaystyle \mathrm{Cov}(X,Y)=\frac{\sum _{i=1}^n(x_i-\overline{x})(y_i-\overline{y})}{n}.}$

The two-pass algorithm may be written as:

def two_pass_covariance(data1, data2):
    n = len(data1)
    mean1 = sum(data1) / n
    mean2 = sum(data2) / n

    covariance = 0
    for i1, i2 in zip(data1, data2):
        a = i1 - mean1
        b = i2 - mean2
        covariance += a * b / n
    return covariance

A slightly more accurate compensated version performs the full naive algorithm on the residuals. The final sums $\sum _i{x}_i$ and $\sum _i{y}_i$ should be zero, but the second pass compensates for any small error.

Online

A stable one-pass algorithm exists, similar to the online algorithm for computing the variance, that computes co-moment $C_n=\sum _{i=1}^n(x_i-{\overline{x}}_n)(y_i-{\overline{y}}_n)$:

${\displaystyle \begin{array}{rlrl}{\overline{x}}_n& ={\overline{x}}_{n-1}& +& \frac{x_n-{\overline{x}}_{n-1}}{n}\\ {\overline{y}}_n& ={\overline{y}}_{n-1}& +& \frac{y_n-{\overline{y}}_{n-1}}{n}\\ C_n& =C_{n-1}& +& (x_n-{\overline{x}}_n)(y_n-{\overline{y}}_{n-1})\\ & =C_{n-1}& +& (x_n-{\overline{x}}_{n-1})(y_n-{\overline{y}}_n)\end{array}}$

The apparent asymmetry in that last equation is due to the fact that $(x_n-{\overline{x}}_n)=\frac{n-1}{n}(x_n-{\overline{x}}_{n-1})$, so both update terms are equal to $\frac{n-1}{n}(x_n-{\overline{x}}_{n-1})(y_n-{\overline{y}}_{n-1})$. Even greater accuracy can be achieved by first computing the means, then using the stable one-pass algorithm on the residuals.

Thus the covariance can be computed as

${\displaystyle \begin{array}{rl}{\mathrm{Cov}}_N(X,Y)=\frac{C_N}{N}& =\frac{{\mathrm{Cov}}_{N-1}(X,Y)\cdot (N-1)+(x_n-{\overline{x}}_n)(y_n-{\overline{y}}_{n-1})}{N}\\ & =\frac{{\mathrm{Cov}}_{N-1}(X,Y)\cdot (N-1)+(x_n-{\overline{x}}_{n-1})(y_n-{\overline{y}}_n)}{N}\\ & =\frac{{\mathrm{Cov}}_{N-1}(X,Y)\cdot (N-1)+\frac{N-1}{N}(x_n-{\overline{x}}_{n-1})(y_n-{\overline{y}}_{n-1})}{N}\\ & =\frac{{\mathrm{Cov}}_{N-1}(X,Y)\cdot (N-1)+\frac{N}{N-1}(x_n-{\overline{x}}_n)(y_n-{\overline{y}}_n)}{N}.\end{array}}$

def online_covariance(data1, data2):
    meanx = meany = C = n = 0
    for x, y in zip(data1, data2):
        n += 1
        dx = x - meanx
        meanx += dx / n
        meany += (y - meany) / n
        C += dx * (y - meany)

    population_covar = C / n
    # Bessel's correction for sample variance
    sample_covar = C / (n - 1)

A small modification can also be made to compute the weighted covariance:

def online_weighted_covariance(data1, data2, data3):
    meanx = meany = 0
    wsum = wsum2 = 0
    C = 0
    for x, y, w in zip(data1, data2, data3):
        wsum += w
        wsum2 += w * w
        dx = x - meanx
        meanx += (w / wsum) * dx
        meany += (w / wsum) * (y - meany)
        C += w * dx * (y - meany)

    population_covar = C / wsum
    # Bessel's correction for sample variance
    # Frequency weights
    sample_frequency_covar = C / (wsum - 1)
    # Reliability weights
    sample_reliability_covar = C / (wsum - wsum2 / wsum)

Likewise, there is a formula for combining the covariances of two sets that can be used to parallelize the computation:

${\displaystyle C_X=C_A+C_B+({\overline{x}}_A-{\overline{x}}_B)({\overline{y}}_A-{\overline{y}}_B)\cdot \frac{n_A{n}_B}{n_X}.}$

Weighted batched version

A version of the weighted online algorithm that does batched updated also exists: let ${\displaystyle w_1,\dots w_N}$ denote the weights, and write

${\displaystyle \begin{array}{rlrl}{\overline{x}}_{n+k}& ={\overline{x}}_n& +& \frac{\sum _{i=n+1}^{n+k}{w}_i(x_i-{\overline{x}}_n)}{\sum _{i=1}^{n+k}{w}_i}\\ {\overline{y}}_{n+k}& ={\overline{y}}_n& +& \frac{\sum _{i=n+1}^{n+k}{w}_i(y_i-{\overline{y}}_n)}{\sum _{i=1}^{n+k}{w}_i}\\ C_{n+k}& =C_n& +& \sum _{i=n+1}^{n+k}{w}_i(x_i-{\overline{x}}_{n+k})(y_i-{\overline{y}}_n)\\ & =C_n& +& \sum _{i=n+1}^{n+k}{w}_i(x_i-{\overline{x}}_n)(y_i-{\overline{y}}_{n+k})\end{array}}$

The covariance can then be computed as

${\displaystyle {\mathrm{Cov}}_N(X,Y)=\frac{C_N}{\sum _{i=1}^N{w}_i}}$

Genocide Convention

From Wikipedia, the free encyclopedia
https://en.wikipedia.org/wiki/Genocide_Convention

Genocide Convention

Convention on the Prevention and Punishment of the Crime of Genocide
Signed	9 December 1948
Location	Palais de Chaillot, Paris, France
Effective	12 January 1951
Signatories	39
Parties	153
Depositary	Secretary-General of the United Nations

The Convention on the Prevention and Punishment of the Crime of Genocide (CPPCG), or the Genocide Convention, is an international treaty that criminalizes genocide and obligates state parties to pursue the enforcement of its prohibition. It was the first legal instrument to codify genocide as a crime, and the first human rights treaty unanimously adopted by the United Nations General Assembly, on 9 December 1948, during the third session of the United Nations General Assembly. The Convention entered into force on 12 January 1951 and has 153 state parties as of June 2024.

The Genocide Convention was conceived largely in response to World War II, which saw atrocities such as the Holocaust that lacked an adequate description or legal definition. Polish-Jewish lawyer Raphael Lemkin, who had coined the term genocide in 1944 to describe Nazi policies in occupied Europe and the Armenian genocide, campaigned for its recognition as a crime under international law. Lemkin also linked colonialism with genocide, mentioning colonial genocides outside of Europe in his writings. In a 1946 resolution, the General Assembly recognized genocide as an international crime and called for the creation of a binding treaty to prevent and punish its perpetration. Subsequent discussions and negotiations among UN member states resulted in the CPPCG.

The Convention defines genocide as any of five "acts committed with intent to destroy, in whole or in part, a national, ethnic, racial or religious group." These five acts include killing members of the group, causing them serious bodily or mental harm, imposing living conditions intended to destroy the group, preventing births, and forcibly transferring children out of the group. Victims are targeted because of their real or perceived membership of a group, not randomly. The convention further criminalizes "complicity, attempt, or incitement of its commission." Member states are prohibited from engaging in genocide and obligated to pursue the enforcement of this prohibition. All perpetrators are to be tried regardless of whether they are private individuals, public officials, or political leaders with sovereign immunity.

The CPPCG has influenced law at both the national and international level. Its definition of genocide has been adopted by international and hybrid tribunals, such as the International Criminal Court, and incorporated into the domestic law of several countries. Its provisions are widely considered to be reflective of customary law and therefore binding on all nations whether or not they are parties. The International Court of Justice (ICJ) has likewise ruled that the principles underlying the Convention represent a peremptory norm against genocide that no government can derogate. The Genocide Convention authorizes the mandatory jurisdiction of the ICJ to adjudicate disputes, leading to international litigation such as the Rohingya genocide case and dispute over the 2022 Russian invasion of Ukraine.

Definition of genocide

Article 2 of the Convention defines genocide as:

... any of the following acts committed with intent to destroy, in whole or in part, a national, ethnical, racial or religious group, as such:

(a) Killing members of the group;

(b) Causing serious bodily or mental harm to members of the group;

(c) Deliberately inflicting on the group conditions of life calculated to bring about its physical destruction in whole or in part;

(d) Imposing measures intended to prevent births within the group;

(e) Forcibly transferring children of the group to another group.

— Convention on the Prevention and Punishment of the Crime of Genocide, Article 2

Article 3 defines the crimes that can be punished under the convention:

(a) Genocide;

(b) Conspiracy to commit genocide;

(c) Direct and public incitement to commit genocide;

(d) Attempt to commit genocide;

(e) Complicity in genocide.

— Convention on the Prevention and Punishment of the Crime of Genocide, Article 3

The convention was passed to outlaw actions similar to the Armenian genocide and the Holocaust.

The first draft of the Convention included political killing, but the USSR along with some other nations would not accept that actions against groups identified as holding similar political opinions or social status would constitute genocide, so these stipulations were subsequently removed in a political and diplomatic compromise.

Early drafts also included acts of cultural destruction in the concept of genocide, but these were opposed by former European colonial powers and some settler countries. Such acts, which Lemkin saw as part and parcel of the concept of genocide, have since often been discussed as cultural genocide (a term also not enshrined in international law). In June 2021, the International Criminal Court issued new guidelines for how cultural destruction, when occurring alongside other recognized acts of genocide, can potentially be corroborating evidence for the intent of the crime of genocide.

The Genocide Convention establishes five prohibited acts that, when committed with the requisite intent, amount to genocide. Genocide is not just defined as wide scale massacre-style killings that are visible and well-documented. International law recognizes a broad range of forms of violence in which the crime of genocide can be enacted.

Killing members of the group Article II(a)

While mass killing is not necessary for genocide to have been committed, it has been present in almost all recognized genocides. In certain instances, men and adolescent boys are singled out for murder in the early stages, such as in the genocide of the Yazidis by Daesh, the Ottoman Turks' attack on the Armenians, and the Burmese security forces' attacks on the Rohingya. Men and boys are typically subject to "fast" killings, such as by gunshot. Women and girls are more likely to die slower deaths by slashing, burning, or as a result of sexual violence. The jurisprudence of the International Criminal Tribunal for Rwanda (ICTR), among others, shows that both the initial executions and those that quickly follow other acts of extreme violence, such as rape and torture, are recognized as falling under the first prohibited act.

A less settled discussion is whether deaths that are further removed from the initial acts of violence can be addressed under this provision of the Genocide Convention. Legal scholars have posited, for example, that deaths resulting from other genocidal acts including causing serious bodily or mental harm or the successful deliberate infliction of conditions of life calculated to bring about physical destruction should be considered genocidal killings.

Causing serious bodily or mental harm to members of the group Article II(b)

This second prohibited act can encompass a wide range of non-fatal genocidal acts. The ICTR and International Criminal Tribunal for the former Yugoslavia (ICTY) have held that rape and sexual violence may constitute the second prohibited act of genocide by causing both physical and mental harm. In its landmark Akayesu decision, the ICTR held that rapes and sexual violence resulted in "physical and psychological destruction". Sexual violence is a hallmark of genocidal violence, with most genocidal campaigns explicitly or implicitly sanctioning it. It is estimated that 250,000 to 500,000 women were raped in the three months of the Rwandan genocide, many of whom were subjected to multiple rapes or gang rape. In Darfur, a systemic campaign of rape and often sexual mutilation was carried out and in Burma public mass rapes and gang rapes were inflicted on the Rohingya by Burmese security forces. Sexual slavery was documented in the Armenian genocide by the Ottoman Turks and Daesh's genocide of the Yazidi.

Torture and other cruel, inhuman, or degrading treatment or punishment, when committed with the requisite intent, are also genocide by causing serious bodily or mental harm to members of the group. The ICTY found that both experiencing a failed execution and watching the murder of one's family members may constitute torture. The Syrian Commission of Inquiry (COI) also found that enslavement, removal of one's children into indoctrination or sexual slavery, and acts of physical and sexual violence rise to the level of torture, as well. While it was subject to some debate, the ICTY and, later, the Syrian COI held that under some circumstances deportation and forcible transfer may also cause serious bodily or mental harm.

Deliberately inflicting on the group conditions of life calculated to bring about its physical destruction Article II(c)

During the Indian wars, the U.S. federal government promoted bison hunting for various reasons, including as a way of destroying the means of survival of Plains Indians to pressure them to remain on Indian reservations. This has been cited by experts as an example of genocide that involves removing the means of survival.

The third prohibited act is distinguished from the genocidal act of killing because the deaths are not immediate (or may not even come to pass), but rather create circumstances that do not support prolonged life. Due to the longer period of time before the actual destruction would be achieved, the ICTR held that courts must consider the duration of time the conditions are imposed as an element of the act. In the 19th century the United States federal government supported the extermination of bison, which Native Americans in the Great Plains relied on as a source of food. This was done for various reasons, primarily to pressure them onto reservations during times of conflict. Some genocide experts describe this as an example of genocide that involves removing the means of survival.

The ICTR provided guidance into what constitutes a violation of the third act. In Akayesu, it identified "subjecting a group of people to a subsistence diet, systematic expulsion from homes and the reduction of essential medical services below minimum requirement" as rising to genocide. In Kayishema and Ruzindana, it extended the list to include: "lack of proper housing, clothing, hygiene and medical care or excessive work or physical exertion" among the conditions. It further noted that, in addition to deprivation of necessary resources, rape could also fit within this prohibited act. In August 2023, founding chief prosecutor of the International Criminal Court (ICC) Luis Moreno Ocampo published a report presenting evidence that Azerbaijan was committing genocide against the ethnic Armenians of Artsakh Nagorno-Karabakh under Article II(c) of the Genocide Convention by placing their historic land under a comprehensive blockade, cutting all access to food, medical supplies, electricity, gas, internet, and stopping all movement of people to and from Armenia.

Imposing measures intended to prevent births within the group Article II(d)

See also: Compulsory sterilization

The fourth prohibited act is aimed at preventing the protected group from regenerating through reproduction. It encompasses acts affecting reproduction and intimate relationships, such as involuntary sterilization, forced abortion, the prohibition of marriage, and long-term separation of men and women intended to prevent procreation. Rape has been found to violate the fourth prohibited act on two bases: where the rape was committed with the intent to impregnate a woman and thereby force her to carry a child of another group (in societies where group identity is determined by patrilineal identity) and where the person raped subsequently refuses to procreate as a result of the trauma. Accordingly, it can take into account both physical and mental measures imposed by the perpetrators.

Forcibly transferring children of the group to another group Article II(e)

See also: Forced assimilation

The final prohibited act is the only prohibited act that does not lead to physical or biological destruction, but rather to destruction of the group as a cultural and social unit. It occurs when children of the protected group are transferred to the perpetrator group. Boys are typically taken into the group by changing their names to those common of the perpetrator group, converting their religion, and using them for labor or as soldiers. Girls who are transferred are not generally converted to the perpetrator group, but instead treated as chattel, as played out in both the Yazidi and Armenian genocides.

Parties

Main article: List of parties to the Genocide Convention

Participation in the Genocide Convention

  Signed and ratified

  Acceded or succeeded

  Only signed

As of June 2024, there are 153 state parties to the Genocide Convention—representing the vast majority of sovereign nations—with the most recent being Zambia in April 2022; one state, the Dominican Republic, has signed but not ratified the treaty. Forty-four states have neither signed nor ratified the convention.

Despite its delegates playing a key role in drafting the convention, the United States did not become a party until 1988—a full forty years after it was opened for signature—and did so only with reservations precluding punishment of the country if it were ever accused of genocide. These were due to traditional American suspicion of any international authority that could override US law. U.S. ratification of the convention was owed in large part to campaigning by Senator William Proxmire, who addressed the Senate in support of the treaty every day it was in session between 1967 and 1986.

Reservations

Immunity from prosecutions

Several parties conditioned their ratification of the Convention on reservations that grant immunity from prosecution for genocide without the consent of the national government:

Parties making reservations from prosecution	Note
Bahrain
Bangladesh
China
India
Malaysia	Opposed by Netherlands, United Kingdom
Morocco
Myanmar
Singapore	Opposed by Netherlands, United Kingdom
United Arab Emirates
United States of America	Opposed by Denmark, Estonia, Finland, Greece, Ireland, Italy, Mexico, Netherlands, Norway, Spain, Sweden, Turkey and United Kingdom
Venezuela
Vietnam	Opposed by United Kingdom
Yemen	Opposed by United Kingdom

Application to non-self-governing territories

Any Contracting Party may at any time, by notification addressed to the Secretary-General of the United Nations, extend the application of the present Convention to all or any of the territories for the conduct of whose foreign relations that Contracting Party is responsible

— Convention on the Prevention and Punishment of the Crime of Genocide, Article 12

Several countries opposed this article, considering that the convention automatically also should apply to Non-Self-Governing Territories:

• Albania
• Belarus
• Bulgaria
• Hungary
• Mongolia
• Myanmar
• Poland
• Romania
• Russian Federation
• Ukraine

The opposition of those countries were in turn opposed by:

• Australia
• Belgium
• Brazil
• Ecuador
• China
• Netherlands
• Sri Lanka
• United Kingdom

(However, exceptionally, Australia did make such a notification at the same time as the ratification of the convention for Australia proper, 8 July 1949, with the effect that the convention did apply also to all territories under Australian control simultaneously, as the USSR et alii had demanded. The European colonial powers in general did not then make such notifications.)

Litigation

United States

Main article: We Charge Genocide

One of the first accusations of genocide submitted to the UN after the Convention entered into force concerned the treatment of Black Americans. The Civil Rights Congress drafted a 237-page petition arguing that even after 1945, the United States had been responsible for hundreds of wrongful deaths, both legal and extra-legal, as well as numerous other supposedly genocidal abuses. Leaders from the Black community and left activists William Patterson, Paul Robeson, and W. E. B. Du Bois presented this petition to the UN in December 1951. It was rejected as a misuse of the intent of the treaty. Charges under We Charge Genocide entailed the lynching of more than 10,000 African Americans with an average of more than 100 per year, with the full number being unconfirmed at the time due to unreported murder cases.

Yugoslavia

The first state and parties to be found in breach of the Genocide Convention were Serbia and Montenegro, and numerous Bosnian Serb leaders. In Bosnia and Herzegovina v. Serbia and Montenegro, the International Court of Justice presented its judgment on 26 February 2007. It cleared Serbia of direct involvement in genocide during the Bosnian war. International Tribunal findings have addressed two allegations of genocidal events, including the 1992 ethnic cleansing campaign in municipalities throughout Bosnia, as well as the convictions found in regards to the Srebrenica massacre of 1995 in which the tribunal found, "Bosnian Serb forces committed genocide, they targeted for extinction, the 40,000 Bosnian Muslims of Srebrenica ... the trial chamber refers to the crimes by their appropriate name, genocide ..." However, individual convictions applicable to the 1992 ethnic cleansings have not been secured. A number of domestic courts and legislatures have found these events to have met the criteria of genocide, and the ICTY found the acts of, and intent to destroy to have been satisfied, the "dolus specialis" still in question and before the MICT, a UN war crimes court, but ruled that Belgrade did breach international law by failing to prevent the 1995 Srebrenica genocide, and for failing to try or transfer the persons accused of genocide to the ICTY, in order to comply with its obligations under Articles I and VI of the Genocide Convention, in particular in respect of General Ratko Mladić.

Myanmar

Main article: Rohingya genocide case

Myanmar has been accused of genocide against its Rohingya community in Rakhine State after around 800,000 Rohingya fled at gunpoint to neighbouring Bangladesh in 2016 and 2017, while their home villages were systematically burned. The International Court of Justice has given its first circular in 2018 asking Myanmar to protect its Rohingya from genocide. Myanmar's civilian government was overthrown by the military on 1 February 2021; since the military is widely seen as the main culprit of the genocide, the coup presents a further challenge to the ICJ.

Russia

Main article: Ukraine v. Russian Federation (2022)

Russian accusations of genocide by Ukraine

Main article: Accusations of genocide in Donbas

In February 2022, Russia invaded Ukraine, claiming that it acted, among other reasons, in order to protect Russian-speaking Ukrainians from genocide. This unfounded and false Russian charge has been widely condemned, and has been called by genocide experts accusation in a mirror, a powerful, historically recurring, form of incitement to genocide.

Russian atrocities in Ukraine

Main article: Allegations of genocide of Ukrainians in the Russian invasion of Ukraine

Russian forces committed numerous atrocities and war crimes in Ukraine, including all five of the potentially genocidal acts listed in the Genocide Convention. Canada, Czechia, Estonia, Ireland, Latvia, Lithuania, Poland, and Ukraine have accused Russia of genocide. In April 2022 Genocide Watch issued a genocide alert for Ukraine. A May 2022 report by 35 legal and genocide experts concluded that Russia has violated the Genocide Convention by the direct and public incitement to commit genocide, and that a pattern of Russian atrocities implies the intent to destroy the Ukrainian national group, and the consequent serious risk of genocide triggers the obligation to prevent it on signatory states.

Israel

Main articles: Gaza genocide and South Africa's genocide case against Israel

In December 2023 South Africa formally accused Israel of violating the Genocide Convention, filing the case South Africa v. Israel (Genocide Convention), due to Israel's actions during the Israel-Hamas War. In addition to starting the litigation process, South Africa also asked the International Court of Justice to demand that Israel cease their military operations in the Gaza Strip as a provisional measure.

Circular reporting

From Wikipedia, the free encyclopedia
https://en.wikipedia.org/wiki/Circular_reporting

Two types of false confirmation. Dashed lines indicate sourcing invisible to a reviewer. In each case, a source (top) appears to a reviewer (bottom) as two independent sources.

Circular reporting, or false confirmation, is a situation in source criticism where a piece of information appears to come from multiple independent sources, but in reality comes from only one source. In many cases, the problem happens mistakenly through sloppy reporting or intelligence-gathering. However, the situation can also be intentionally contrived by the source or reporter as a way of reinforcing the widespread belief in its information.

Circular reporting occurs in a variety of fields, including intelligence gathering, journalism, and scholarly research. It is of particular concern in military intelligence because the original source has a higher likelihood of wanting to pass on misinformation, and because the chain of reporting is more prone to being obscured. It is also a problem in journalism and the development of conspiracy theories, in which the primary goal of a source spreading unlikely or hard-to-believe information is to make it appear to be widely known.

The case of the 2002 Niger uranium forgeries was a classic instance of circular reporting by intelligence agencies.

Examples of circular reporting

1976 novel Roots

Author Alex Haley grew up hearing the oral history that his family's first ancestor to enter the United States was a young man named Kunta Kinte, who lived near the Kamby Bolongo, or Gambia River, and was kidnapped into slavery when out gathering wood. As an adult, Haley researched his family genealogy for what would become the 1976 novel Roots: The Saga of an American Family, and he traveled to the Gambia in an attempt to confirm the family history of Kinte. Haley told the story of Kinte to a seminar of Gambian tribal experts, who searched for a griot—an oral historian—who might be able to confirm the story. Ultimately, Haley met a man named Kebba Fofana in the town of Juffure who was able to relate a story of Kunta Kinte that was strikingly similar to Haley's lifelong family history, an apparent confirmation that grounded Haley's novel (as well as the landmark 1977 miniseries adapted from the novel). After publication, however, it was discovered that griot oral histories were not reliable for dates before the 19th century, that Fofana was not a true griot, and that Fofana's confirmation of Haley's history was ultimately a retelling of the story Haley himself told Gambian experts.

Iraq War

In 2001, the Niger uranium forgeries, documents initially released by SISMI (the former military intelligence agency of Italy), seemed to depict an attempt made by Saddam Hussein in Iraq to purchase yellowcake uranium powder from Niger during the Iraq disarmament crisis. They were referenced by other intelligence agencies to convince their governments or public that such a purchase had taken place.

In 2004, the Chairman of the US Senate Report on Pre-war Intelligence on Iraq told NBC's Tim Russert that a single informant, 'Curveball' "had really provided 98 percent of the assessment as to whether or not the Iraqis had a biological weapon." This was despite the fact that "nobody inside the U.S. government had ever actually spoken to the informant—except [for a single] Pentagon analyst, who concluded the man was an alcoholic and utterly useless as a source."

Other examples

In early 2012, a TV Tropes user named Tunafish claimed that a bug existed in Civilization that caused Gandhi to be much more aggressive. Tunafish did not provide any proof. The repetition of this false information led to the "Nuclear Gandhi" internet meme.

In 2018, Shehroze Chaudhry was identified as an active member of the Islamic State who participated in the killing of several individuals, through reporting involving a New York Times podcast, among others. The podcast and other outlets referenced blog posts authored by Chaudhry starting in 2016. The podcast was taken by government officials and others as evidence of the crime; however, the original posts were unverified and later renounced by the author.

Circular reporting on Wikipedia

See also: Wikipedia:List of citogenesis incidents

Wikipedia is sometimes criticized for being used as a source of circular reporting, particularly a variant where an unsourced claim in a Wikipedia article is repeated by a reliable source, often without citing the article; which is then added as a source to the claim on Wikipedia.

History of citogenesis

The xkcd comic strip that coined the term citogenesis

The first recorded use of the term citogenesis to describe this phenomenon was in November 2011, when Randall Munroe used it in an xkcd comic strip. The neologism is attributed as being a homophonic wordplay on 'cytogenesis', the formation, development and variation of biological cells.

An article in the magazine Slate referenced the four-step process described in the comic, to raise awareness about citogenesis as facilitated by Wikipedia. This type of circular reporting has been described as particularly hard-to-catch because of the speed of revisions of modern webpages, and the lack of "as of" timestamps in citations and "last updated" timestamps on pages online.

Inspired by the comic, Wikipedia editors have since maintained an internal list of citogenesis incidents to monitor its prevalence.

Wikipedia advises researchers and journalists to be wary of, and generally avoid, using Wikipedia as a direct source, and to focus instead on verifiable information found in an article's cited references. Researchers and Wikipedians alike are advised to note the retrieved-on date of any web citation, to support identification of the earliest source of a claim.

Examples on Wikipedia

Circular reporting by Wikipedia and the press

Prominent examples of false claims that were propagated on Wikipedia and in news sources because of circular reporting:

• 2007: Wikipedia and The Independent propagated the false information that comedian Sacha Baron Cohen had worked at Goldman Sachs.
• 2008: A student arbitrarily added, "also known as....Brazilian Aardvarks" to the article on the coati, leading to subsequent commentary on the mammal that mentioned this nickname. Outlets repeating the nickname included The Independent, the Daily Express, the Metro, The Daily Telegraph, the Daily Mail, a book published by the University of Chicago, and a scholarly work published by the University of Cambridge.
• 2009: The middle name "Wilhelm" was falsely added into Karl-Theodor zu Guttenberg's name. This was propagated by a raft of publications, including German and international press.
• 2009: An incorrect release year of 1991 was added to the Wikipedia article of the Casio F-91W watch. The BBC repeated this in a 2011 article. Communication with primary sources repeatedly confirmed a 1989 release year, but because the BBC is usually a reliable secondary source, their use of 1991 made the misinformation difficult to remove. In 2019, KSNV cited this incident as another example of citogenesis. The correct year was only restored after that review, with the KSNV article becoming cited in the article to support restoring the 1989 release date.
• 2014: A statement was anonymously added to the Wikipedia page on UK comedian Dave Gorman stating that he had "taken a career break for a sponsored hitchhike around the Pacific Rim countries". When this was questioned, an article published at a later date (September 2014) in The Northern Echo, a daily regional newspaper in North East England was cited as evidence. Gorman repudiated the claim in episode 3, season 4 of his UK television show Modern Life Is Goodish (first broadcast 22 November 2016).
• 2022: A hoax article about Alan MacMasters, purported inventor of the toaster, was discovered to have been picked up in news media later used as citations.

Blood substitute

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Blood_substitute

A blood substitute (also called artificial blood or blood surrogate) is a substance used to mimic and fulfill some functions of biological blood. It aims to provide an alternative to blood transfusion, which is transferring blood or blood-based products from one person into another. Thus far, there are no well-accepted oxygen-carrying blood substitutes, which is the typical objective of a red blood cell transfusion; however, there are widely available non-blood volume expanders for cases where only volume restoration is required. These are helping doctors and surgeons avoid the risks of disease transmission and immune suppression, address the chronic blood donor shortage, and address the concerns of Jehovah's Witnesses and others who have religious objections to receiving transfused blood.^{[citation needed]}

The main categories of "oxygen-carrying" blood substitutes being pursued are hemoglobin-based oxygen carriers (HBOC) and perfluorocarbon emulsions. Oxygen therapeutics are in clinical trials in the U.S. and European Union, and Hemopure is available in South Africa.

History

After William Harvey discovered blood pathways in 1616, many people tried to use fluids such as beer, urine, milk, and non-human animal blood as blood substitute. Sir Christopher Wren suggested wine and opium as blood substitute.

At the beginning of the 20th century, the development of modern transfusion medicine initiated through the work of Landsteiner and co-authors opened the possibility to understanding the general principle of blood group serology. Simultaneously, significant progress was made in the fields of heart and circulation physiology as well as in the understanding of the mechanism of oxygen transport and tissue oxygenation.

Restrictions in applied transfusion medicine, especially in disaster situations such as World War II, laid the grounds for accelerated research in the field of blood substitutes. Early attempts and optimism in developing blood substitutes were very quickly confronted with significant side effects, which could not be promptly eliminated due to the level of knowledge and technology available at that time. The emergence of HIV in the 1980s renewed impetus for development of infection-safe blood substitutes. Public concern about the safety of the blood supply was raised further by mad cow disease. The continuous decline of blood donation combined with the increased demand for blood transfusion (increased ageing of population, increased incidence of invasive diagnostic, chemotherapy and extensive surgical interventions, terror attacks, international military conflicts) and positive estimation of investors in biotechnology branch made for a positive environment for further development of blood substitutes.

Efforts to develop blood substitutes have been driven by a desire to replace blood transfusion in emergency situations, in places where infectious disease is endemic and the risk of contaminated blood products is high, where refrigeration to preserve blood may be lacking, and where it might not be possible or convenient to find blood type matches.

In 2023, DARPA announced funding twelve universities and labs for synthetic blood research. Human trials would be expected to happen between 2028-2030.

Approaches

Efforts have focused on molecules that can carry oxygen, and most work has focused on recombinant hemoglobin, which normally carries oxygen, and perfluorocarbons (PFC), chemical compounds which can carry and release oxygen.

The first approved oxygen-carrying blood substitute was a perfluorocarbon-based product called Fluosol-DA-20, manufactured by Green Cross of Japan. It was approved by the Food and Drug Administration (FDA) in 1989. Because of limited success, complexity of use and side effects, it was withdrawn in 1994. However, Fluosol-DA remains the only oxygen therapeutic ever fully approved by the FDA. As of 2017 no hemoglobin-based product had been approved.

Perfluorocarbon based

Perfluorochemicals are not water soluble and will not mix with blood, therefore emulsions must be made by dispersing small drops of PFC in water. This liquid is then mixed with antibiotics, vitamins, nutrients and salts, producing a mixture that contains about 80 different components, and performs many of the vital functions of natural blood. PFC particles are about ⁠1/40⁠ the size of the diameter of a red blood cell (RBC). This small size can enable PFC particles to traverse capillaries through which no RBCs are flowing. In theory this can benefit damaged, blood-starved tissue, which conventional red cells cannot reach. PFC solutions can carry oxygen so well that mammals, including humans, can survive breathing liquid PFC solution, called liquid breathing.

Perfluorocarbon-based blood substitutes are completely man-made; this provides advantages over blood substitutes that rely on modified hemoglobin, such as unlimited manufacturing capabilities, ability to be heat-sterilized, and PFCs' efficient oxygen delivery and carbon dioxide removal. PFCs in solution act as an intravascular oxygen carrier to temporarily augment oxygen delivery to tissues. PFCs are removed from the bloodstream within 48 hours by the body's normal clearance procedure for particles in the blood – exhalation. PFC particles in solution can carry several times more oxygen per cubic centimeter (cc) than blood, while being 40 to 50 times smaller than hemoglobin.

Fluosol was made mostly of perfluorodecalin or perfluorotributylamine suspended in an albumin emulsion. It was developed in Japan and first tested in the United States in November 1979. In order to "load" sufficient amounts of oxygen into it, people who had been given it had to breathe pure oxygen by mask or in a hyperbaric chamber. It was approved by the FDA in 1989, and was approved in eight other countries. Its use was associated with a reduction in ischemic complications and with an increase in pulmonary edema and congestive heart failure. Due to difficulty with the emulsion storage of Fluosol use (frozen storage and rewarming), its popularity declined and its production ended in 1994.

Name	Sponsor	Description
Oxycyte	Oxygen Biotherapeutics	Tested in a Phase II-b Trials in the United States. Targeted as an oxygen therapeutic rather than a blood substitute, with successful small-scale open label human trials treating traumatic brain injury at Virginia Commonwealth University. The trial was later terminated.
PHER-O 2	Sanguine Corp	In research
Perftoran	Russia	Contains perfluorodecalin and perfluoro-N-(4-methylcyclohexyl)-piperidine along with a surfactant, Proxanol-268. It was developed in Russia and as of 2005 was marketed there.
NVX-108	NuvOx Pharma	In a Phase Ib/II clinical trial where it raises tumor oxygen levels prior to radiation therapy in order to radiosensitize them.

Oxygent was a second-generation, lecithin-stabilized emulsion of a PFC that was under development by Alliance Pharmaceuticals. In 2002 a Phase III study was halted early due an increase in incidences of strokes in the study arm.

Haemoglobin based

Haemoglobin is the main component of red blood cells, comprising about 33% of the cell mass. Haemoglobin-based products are called haemoglobin-based oxygen carriers (HBOCs).

Unmodified cell-free haemoglobin is not useful as a blood substitute because its oxygen affinity is too high for effective tissue oxygenation, the half-life within the intravascular space that is too short to be clinically useful, it has a tendency to undergo dissociation in dimers with resultant kidney damage and toxicity, and because free haemoglobin tends to take up nitric oxide, causing vasoconstriction.

Efforts to overcome this toxicity have included making genetically engineered versions, cross-linking, polymerization, and encapsulation.

HemAssist, a diaspirin cross-linked haemoglobin (DCLHb) was developed by Baxter Healthcare; it was the most widely studied of the haemoglobin-based blood substitutes, used in more than a dozen animal and clinical studies. It reached Phase III clinical trials, in which it failed due to increased mortality in the trial arm, mostly due to severe vasoconstriction complications. The results were published in 1999.

Hemolink (Hemosol Inc., Mississauga, Canada) was a haemoglobin solution that contained cross-linked an o-rafinose polymerised human haemoglobin. Hemosol struggled after Phase II trials were halted in 2003 on safety concerns and declared bankruptcy in 2005.

Hemopure was developed by Biopure Corp and was a chemically stabilized, cross-linked bovine (cow) haemoglobin in a salt solution intended for human use; the company developed the same product under the trade name Oxyglobin for veterinary use in dogs. Oxyglobin was approved in the US and Europe and was introduced to veterinary clinics and hospitals in March 1998. Hemopure was approved in South Africa and Russia. Biopure filed for bankruptcy protection in 2009. Its assets were subsequently purchased by HbO2 Therapeutics in 2014.

PolyHeme was developed over 20 years by Northfield Laboratories and began as a military project following the Vietnam War. It is human haemoglobin, extracted from red blood cells, then polymerized, then incorporated into an electrolyte solution. In April 2009, the FDA rejected Northfield's Biologic License Application and in June 2009, Northfield filed for bankruptcy.

Dextran-Haemoglobin was developed by Dextro-Sang Corp as a veterinary product, and was a conjugate of the polymer dextran with human haemoglobin.

Hemotech was developed by HemoBiotech and was a chemically modified haemoglobin.

Somatogen developed a genetically engineered and crosslinked tetramer it called Optro. It failed in a phase II trial and development was halted.

A pyridoxylated Hb conjugated with polyoxyethylene was created by scientists at Ajinomoto and eventually developed by Apex Biosciences, a subsidiary of Curacyte AG; it was called "PHP" and failed in a Phase III trial published in 2014, due to increased mortality in the control arm, which led to Curacyte shutting down.

Similarly, Hemospan was developed by Sangart, and was a pegylated haemoglobin provided in a powdered form. While early trials were promising Sangart ran out of funding and closed down.

Stem cells

Stem cells offer a possible means of producing transfusable blood. A study performed by Giarratana et al. describes a large-scale ex-vivo production of mature human blood cells using hematopoietic stem cells. The cultured cells possessed the same haemoglobin content and morphology as native red blood cells. The authors contend that the cells had a near-normal lifespan, when compared to natural red blood cells.

Scientists from the experimental arm of the United States Department of Defense began creating artificial blood for use in remote areas and transfuse blood to wounded soldiers more quickly in 2010. The blood is made from the hematopoietic stem cells removed from the umbilical cord between human mother and newborn using a method called blood pharming. Pharming has been used in the past on animals and plants to create medical substances in large quantities. Each cord can produce approximately 20 units of blood. The blood is being produced for the Defense Advanced Research Projects Agency by Arteriocyte. The Food and Drug Administration has examined and approved the safety of this blood from previously submitted O-negative blood. Using this particular artificial blood will reduce the costs per unit of blood from $5,000 to equal or less than $1,000. This blood will also serve as a blood donor to all common blood types.