Data mining

From Wikipedia, the free encyclopedia

Data mining is the process of discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems. Data mining is an interdisciplinary subfield of computer science with an overall goal to extract information (with intelligent methods) from a data set and transform the information into a comprehensible structure for further use. Data mining is the analysis step of the "knowledge discovery in databases" process, or KDD. Aside from the raw analysis step, it also involves database and data management aspects, data pre-processing, model and inference considerations, interestingness metrics, complexity considerations, post-processing of discovered structures, visualization, and online updating.

The term "data mining" is in fact a misnomer, because the goal is the extraction of patterns and knowledge from large amounts of data, not the extraction (mining) of data itself. It also is a buzzword and is frequently applied to any form of large-scale data or information processing (collection, extraction, warehousing, analysis, and statistics) as well as any application of computer decision support system, including artificial intelligence (e.g., machine learning) and business intelligence. The book Data mining: Practical machine learning tools and techniques with Java (which covers mostly machine learning material) was originally to be named just Practical machine learning, and the term data mining was only added for marketing reasons. Often the more general terms (large scale) data analysis and analytics – or, when referring to actual methods, artificial intelligence and machine learning – are more appropriate.

The actual data mining task is the semi-automatic or automatic analysis of large quantities of data to extract previously unknown, interesting patterns such as groups of data records (cluster analysis), unusual records (anomaly detection), and dependencies (association rule mining, sequential pattern mining). This usually involves using database techniques such as spatial indices. These patterns can then be seen as a kind of summary of the input data, and may be used in further analysis or, for example, in machine learning and predictive analytics. For example, the data mining step might identify multiple groups in the data, which can then be used to obtain more accurate prediction results by a decision support system. Neither the data collection, data preparation, nor result interpretation and reporting is part of the data mining step, but do belong to the overall KDD process as additional steps.

The related terms data dredging, data fishing, and data snooping refer to the use of data mining methods to sample parts of a larger population data set that are (or may be) too small for reliable statistical inferences to be made about the validity of any patterns discovered. These methods can, however, be used in creating new hypotheses to test against the larger data populations.

Etymology

In the 1960s, statisticians and economists used terms like data fishing or data dredging to refer to what they considered the bad practice of analyzing data without an a-priori hypothesis. The term "data mining" was used in a similarly critical way by economist Michael Lovell in an article published in the Review of Economic Studies 1983. Lovell indicates that the practice "masquerades under a variety of aliases, ranging from "experimentation" (positive) to "fishing" or "snooping" (negative).

The term data mining appeared around 1990 in the database community, generally with positive connotations. For a short time in 1980s, a phrase "database mining"™, was used, but since it was trademarked by HNC, a San Diego-based company, to pitch their Database Mining Workstation; researchers consequently turned to data mining. Other terms used include data archaeology, information harvesting, information discovery, knowledge extraction, etc. Gregory Piatetsky-Shapiro coined the term "knowledge discovery in databases" for the first workshop on the same topic (KDD-1989) and this term became more popular in AI and machine learning community. However, the term data mining became more popular in the business and press communities. Currently, the terms data mining and knowledge discovery are used interchangeably.

In the academic community, the major forums for research started in 1995 when the First International Conference on Data Mining and Knowledge Discovery (KDD-95) was started in Montreal under AAAI sponsorship. It was co-chaired by Usama Fayyad and Ramasamy Uthurusamy. A year later, in 1996, Usama Fayyad launched the journal by Kluwer called Data Mining and Knowledge Discovery as its founding editor-in-chief. Later he started the SIGKDDD Newsletter SIGKDD Explorations. The KDD International conference became the primary highest quality conference in data mining with an acceptance rate of research paper submissions below 18%. The journal Data Mining and Knowledge Discovery is the primary research journal of the field.

Background

The manual extraction of patterns from data has occurred for centuries. Early methods of identifying patterns in data include Bayes' theorem (1700s) and regression analysis (1800s). The proliferation, ubiquity and increasing power of computer technology has dramatically increased data collection, storage, and manipulation ability. As data sets have grown in size and complexity, direct "hands-on" data analysis has increasingly been augmented with indirect, automated data processing, aided by other discoveries in computer science, such as neural networks, cluster analysis, genetic algorithms (1950s), decision trees and decision rules (1960s), and support vector machines (1990s). Data mining is the process of applying these methods with the intention of uncovering hidden patterns in large data sets. It bridges the gap from applied statistics and artificial intelligence (which usually provide the mathematical background) to database management by exploiting the way data is stored and indexed in databases to execute the actual learning and discovery algorithms more efficiently, allowing such methods to be applied to ever larger data sets.

Process

The knowledge discovery in databases (KDD) process is commonly defined with the stages:

1. Selection
2. Pre-processing
3. Transformation
4. Data mining
5. Interpretation/evaluation.

It exists, however, in many variations on this theme, such as the Cross Industry Standard Process for Data Mining (CRISP-DM) which defines six phases:

1. Business understanding
2. Data understanding
3. Data preparation
4. Modeling
5. Evaluation
6. Deployment

or a simplified process such as (1) Pre-processing, (2) Data Mining, and (3) Results Validation.
Polls conducted in 2002, 2004, 2007 and 2014 show that the CRISP-DM methodology is the leading methodology used by data miners. The only other data mining standard named in these polls was SEMMA. However, 3–4 times as many people reported using CRISP-DM. Several teams of researchers have published reviews of data mining process models, and Azevedo and Santos conducted a comparison of CRISP-DM and SEMMA in 2008.

Pre-processing

Before data mining algorithms can be used, a target data set must be assembled. As data mining can only uncover patterns actually present in the data, the target data set must be large enough to contain these patterns while remaining concise enough to be mined within an acceptable time limit. A common source for data is a data mart or data warehouse. Pre-processing is essential to analyze the multivariate data sets before data mining. The target set is then cleaned. Data cleaning removes the observations containing noise and those with missing data.

Data mining

Data mining involves six common classes of tasks:

• Anomaly detection (outlier/change/deviation detection) – The identification of unusual data records, that might be interesting or data errors that require further investigation.
• Association rule learning (dependency modelling) – Searches for relationships between variables. For example, a supermarket might gather data on customer purchasing habits. Using association rule learning, the supermarket can determine which products are frequently bought together and use this information for marketing purposes. This is sometimes referred to as market basket analysis.
• Clustering – is the task of discovering groups and structures in the data that are in some way or another "similar", without using known structures in the data.
• Classification – is the task of generalizing known structure to apply to new data. For example, an e-mail program might attempt to classify an e-mail as "legitimate" or as "spam".
• Regression – attempts to find a function which models the data with the least error that is, for estimating the relationships among data or datasets.
• Summarization – providing a more compact representation of the data set, including visualization and report generation.

Results validation

An example of data produced by data dredging through a bot operated by statistician Tyler Vigen, apparently showing a close link between the best word winning a spelling bee competition and the number of people in the United States killed by venomous spiders. The similarity in trends is obviously a coincidence.

Data mining can unintentionally be misused, and can then produce results which appear to be significant; but which do not actually predict future behaviour and cannot be reproduced on a new sample of data and bear little use. Often this results from investigating too many hypotheses and not performing proper statistical hypothesis testing. A simple version of this problem in machine learning is known as overfitting, but the same problem can arise at different phases of the process and thus a train/test split - when applicable at all - may not be sufficient to prevent this from happening.

The final step of knowledge discovery from data is to verify that the patterns produced by the data mining algorithms occur in the wider data set. Not all patterns found by the data mining algorithms are necessarily valid. It is common for the data mining algorithms to find patterns in the training set which are not present in the general data set. This is called overfitting. To overcome this, the evaluation uses a test set of data on which the data mining algorithm was not trained. The learned patterns are applied to this test set, and the resulting output is compared to the desired output. For example, a data mining algorithm trying to distinguish "spam" from "legitimate" emails would be trained on a training set of sample e-mails. Once trained, the learned patterns would be applied to the test set of e-mails on which it had not been trained. The accuracy of the patterns can then be measured from how many e-mails they correctly classify. A number of statistical methods may be used to evaluate the algorithm, such as ROC curves.

If the learned patterns do not meet the desired standards, subsequently it is necessary to re-evaluate and change the pre-processing and data mining steps. If the learned patterns do meet the desired standards, then the final step is to interpret the learned patterns and turn them into knowledge.

Research

The premier professional body in the field is the Association for Computing Machinery's (ACM) Special Interest Group (SIG) on Knowledge Discovery and Data Mining (SIGKDD). Since 1989, this ACM SIG has hosted an annual international conference and published its proceedings, and since 1999 it has published a biannual academic journal titled "SIGKDD Explorations".

Computer science conferences on data mining include:

• CIKM Conference – ACM Conference on Information and Knowledge Management
• European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases
• KDD Conference – ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Data mining topics are also present on many data management/database conferences such as the ICDE Conference, SIGMOD Conference and International Conference on Very Large Data Bases

Standards

There have been some efforts to define standards for the data mining process, for example the 1999 European Cross Industry Standard Process for Data Mining (CRISP-DM 1.0) and the 2004 Java Data Mining standard (JDM 1.0). Development on successors to these processes (CRISP-DM 2.0 and JDM 2.0) was active in 2006, but has stalled since. JDM 2.0 was withdrawn without reaching a final draft.

For exchanging the extracted models – in particular for use in predictive analytics – the key standard is the Predictive Model Markup Language (PMML), which is an XML-based language developed by the Data Mining Group (DMG) and supported as exchange format by many data mining applications. As the name suggests, it only covers prediction models, a particular data mining task of high importance to business applications. However, extensions to cover (for example) subspace clustering have been proposed independently of the DMG.

Notable uses

Data mining is used wherever there is digital data available today. Notable examples of data mining can be found throughout business, medicine, science, and surveillance.

Privacy concerns and ethics

While the term "data mining" itself may have no ethical implications, it is often associated with the mining of information in relation to peoples' behavior (ethical and otherwise).

The ways in which data mining can be used can in some cases and contexts raise questions regarding privacy, legality, and ethics. In particular, data mining government or commercial data sets for national security or law enforcement purposes, such as in the Total Information Awareness Program or in ADVISE, has raised privacy concerns.

Data mining requires data preparation which can uncover information or patterns which may compromise confidentiality and privacy obligations. A common way for this to occur is through data aggregation. Data aggregation involves combining data together (possibly from various sources) in a way that facilitates analysis (but that also might make identification of private, individual-level data deducible or otherwise apparent). This is not data mining per se, but a result of the preparation of data before – and for the purposes of – the analysis. The threat to an individual's privacy comes into play when the data, once compiled, cause the data miner, or anyone who has access to the newly compiled data set, to be able to identify specific individuals, especially when the data were originally anonymous.

It is recommended that an individual is made aware of the following before data are collected:

• the purpose of the data collection and any (known) data mining projects;
• how the data will be used;
• who will be able to mine the data and use the data and their derivatives;
• the status of security surrounding access to the data;
• how collected data can be updated.

Data may also be modified so as to become anonymous, so that individuals may not readily be identified. However, even "de-identified"/"anonymized" data sets can potentially contain enough information to allow identification of individuals, as occurred when journalists were able to find several individuals based on a set of search histories that were inadvertently released by AOL.

The inadvertent revelation of personally identifiable information leading to the provider violates Fair Information Practices. This indiscretion can cause financial, emotional, or bodily harm to the indicated individual. In one instance of privacy violation, the patrons of Walgreens filed a lawsuit against the company in 2011 for selling prescription information to data mining companies who in turn provided the data to pharmaceutical companies.

Situation in Europe

Europe has rather strong privacy laws, and efforts are underway to further strengthen the rights of the consumers. However, the U.S.-E.U. Safe Harbor Principles currently effectively expose European users to privacy exploitation by U.S. companies. As a consequence of Edward Snowden's global surveillance disclosure, there has been increased discussion to revoke this agreement, as in particular the data will be fully exposed to the National Security Agency, and attempts to reach an agreement have failed.

Situation in the United States

In the United States, privacy concerns have been addressed by the US Congress via the passage of regulatory controls such as the Health Insurance Portability and Accountability Act (HIPAA). The HIPAA requires individuals to give their "informed consent" regarding information they provide and its intended present and future uses. According to an article in Biotech Business Week, "'[i]n practice, HIPAA may not offer any greater protection than the longstanding regulations in the research arena,' says the AAHC. More importantly, the rule's goal of protection through informed consent is approach a level of incomprehensibility to average individuals." This underscores the necessity for data anonymity in data aggregation and mining practices.

U.S. information privacy legislation such as HIPAA and the Family Educational Rights and Privacy Act (FERPA) applies only to the specific areas that each such law addresses. Use of data mining by the majority of businesses in the U.S. is not controlled by any legislation.

Copyright law

Situation in Europe

Due to a lack of flexibilities in European copyright and database law, the mining of in-copyright works such as web mining without the permission of the copyright owner is not legal. Where a database is pure data in Europe there is likely to be no copyright, but database rights may exist so data mining becomes subject to regulations by the Database Directive. On the recommendation of the Hargreaves review this led to the UK government to amend its copyright law in 2014 to allow content mining as a limitation and exception. Only the second country in the world to do so after Japan, which introduced an exception in 2009 for data mining. However, due to the restriction of the Copyright Directive, the UK exception only allows content mining for non-commercial purposes. UK copyright law also does not allow this provision to be overridden by contractual terms and conditions. The European Commission facilitated stakeholder discussion on text and data mining in 2013, under the title of Licences for Europe. The focus on the solution to this legal issue being licences and not limitations and exceptions led to representatives of universities, researchers, libraries, civil society groups and open access publishers to leave the stakeholder dialogue in May 2013.

Situation in the United States

By contrast to Europe, the flexible nature of US copyright law, and in particular fair use means that content mining in America, as well as other fair use countries such as Israel, Taiwan and South Korea is viewed as being legal. As content mining is transformative, that is it does not supplant the original work, it is viewed as being lawful under fair use. For example, as part of the Google Book settlement the presiding judge on the case ruled that Google's digitisation project of in-copyright books was lawful, in part because of the transformative uses that the digitisation project displayed - one being text and data mining.

Software

Free open-source data mining software and applications

The following applications are available under free/open source licenses. Public access to application source code is also available.

• Carrot2: Text and search results clustering framework.
• Chemicalize.org: A chemical structure miner and web search engine.
• ELKI: A university research project with advanced cluster analysis and outlier detection methods written in the Java language.
• GATE: a natural language processing and language engineering tool.
• KNIME: The Konstanz Information Miner, a user friendly and comprehensive data analytics framework.
• Massive Online Analysis (MOA): a real-time big data stream mining with concept drift tool in the Java programming language.
• MEPX - cross platform tool for regression and classification problems based on a Genetic Programming variant.
• ML-Flex: A software package that enables users to integrate with third-party machine-learning packages written in any programming language, execute classification analyses in parallel across multiple computing nodes, and produce HTML reports of classification results.
• MLPACK library: a collection of ready-to-use machine learning algorithms written in the C++ language.
• NLTK (Natural Language Toolkit): A suite of libraries and programs for symbolic and statistical natural language processing (NLP) for the Python language.
• OpenNN: Open neural networks library.
• Orange: A component-based data mining and machine learning software suite written in the Python language.
• R: A programming language and software environment for statistical computing, data mining, and graphics. It is part of the GNU Project.
• scikit-learn is an open source machine learning library for the Python programming language
• Torch: An open source deep learning library for the Lua programming language and scientific computing framework with wide support for machine learning algorithms.
• UIMA: The UIMA (Unstructured Information Management Architecture) is a component framework for analyzing unstructured content such as text, audio and video – originally developed by IBM.
• Weka: A suite of machine learning software applications written in the Java programming language.

Proprietary data-mining software and applications

The following applications are available under proprietary licenses.

• Angoss KnowledgeSTUDIO: data mining tool
• Clarabridge: text analytics product.
• KXEN Modeler: data mining tool provided by KXEN Inc..
• LIONsolver: an integrated software application for data mining, business intelligence, and modeling that implements the Learning and Intelligent OptimizatioN (LION) approach.
• Megaputer Intelligence: data and text mining software is called PolyAnalyst.
• Microsoft Analysis Services: data mining software provided by Microsoft.
• NetOwl: suite of multilingual text and entity analytics products that enable data mining.
• OpenText Big Data Analytics: Visual Data Mining & Predictive Analysis by Open Text Corporation
• Oracle Data Mining: data mining software by Oracle Corporation.
• PSeven: platform for automation of engineering simulation and analysis, multidisciplinary optimization and data mining provided by DATADVANCE.
• Qlucore Omics Explorer: data mining software.
• RapidMiner: An environment for machine learning and data mining experiments.
• SAS Enterprise Miner: data mining software provided by the SAS Institute.
• SPSS Modeler: data mining software provided by IBM.
• STATISTICA Data Miner: data mining software provided by StatSoft.
• Tanagra: Visualisation-oriented data mining software, also for teaching.
• Vertica: data mining software provided by Hewlett-Packard.

Marketplace surveys

Several researchers and organizations have conducted reviews of data mining tools and surveys of data miners. These identify some of the strengths and weaknesses of the software packages. They also provide an overview of the behaviors, preferences and views of data miners. Some of these reports include:

• Hurwitz Victory Index: Report for Advanced Analytics as a market research assessment tool, it highlights both the diverse uses for advanced analytics technology and the vendors who make those applications possible.Recent-research
• Rexer Analytics Data Miner Surveys (2007–2015)
• 2011 Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery
• Forrester Research 2010 Predictive Analytics and Data Mining Solutions report
• Gartner 2008 "Magic Quadrant" report
• Robert A. Nisbet's 2006 Three Part Series of articles "Data Mining Tools: Which One is Best For CRM?"
• Haughton et al.'s 2003 Review of Data Mining Software Packages in The American Statistician
• Goebel & Gruenwald 1999 "A Survey of Data Mining a Knowledge Discovery Software Tools" in SIGKDD Explorations

The American Economy Is Rigged

And what we can do about it.

By Joseph E. Stiglitz | Scientific American November 2018 Issue

Original link:  https://www.scientificamerican.com/article/the-american-economy-is-rigged/?fbclid=IwAR3-x_YWa2gUrH0-VtvgeG6iRCs_z35w3ARBjW8U-1oaFtGn_jO2VlOOFK8

Credit: Andrea Ucini

Americans are used to thinking that their nation is special. In many ways, it is: the U.S. has by far the most Nobel Prize winners, the largest defense expenditures (almost equal to the next 10 or so countries put together) and the most billionaires (twice as many as China, the closest competitor). But some examples of American Exceptionalism should not make us proud. By most accounts, the U.S. has the highest level of economic inequality among developed countries. It has the world's greatest per capita health expenditures yet the lowest life expectancy among comparable countries. It is also one of a few developed countries jostling for the dubious distinction of having the lowest measures of equality of opportunity.

The notion of the American Dream—that, unlike old Europe, we are a land of opportunity—is part of our essence. Yet the numbers say otherwise. The life prospects of a young American depend more on the income and education of his or her parents than in almost any other advanced country. When poor-boy-makes-good anecdotes get passed around in the media, that is precisely because such stories are so rare.

Things appear to be getting worse, partly as a result of forces, such as technology and globalization, that seem beyond our control, but most disturbingly because of those within our command. It is not the laws of nature that have led to this dire situation: it is the laws of humankind. Markets do not exist in a vacuum: they are shaped by rules and regulations, which can be designed to favor one group over another. President Donald Trump was right in saying that the system is rigged—by those in the inherited plutocracy of which he himself is a member. And he is making it much, much worse.

America has long outdone others in its level of inequality, but in the past 40 years it has reached new heights. Whereas the income share of the top 0.1 percent has more than quadrupled and that of the top 1 percent has almost doubled, that of the bottom 90 percent has declined. Wages at the bottom, adjusted for inflation, are about the same as they were some 60 years ago! In fact, for those with a high school education or less, incomes have fallen over recent decades. Males have been particularly hard hit, as the U.S. has moved away from manufacturing industries into an economy based on services.

Deaths of Despair

Wealth is even less equally distributed, with just three Americans having as much as the bottom 50 percent—testimony to how much money there is at the top and how little there is at the bottom. Families in the bottom 50 percent hardly have the cash reserves to meet an emergency. Newspapers are replete with stories of those for whom the breakdown of a car or an illness starts a downward spiral from which they never recover.

In significant part because of high inequality [see “The Health-Wealth Gap,” by Robert M. Sapolsky], U.S. life expectancy, exceptionally low to begin with, is experiencing sustained declines. This in spite of the marvels of medical science, many advances of which occur right here in America and which are made readily available to the rich. Economist Ann Case and 2015 Nobel laureate in economics Angus Deaton describe one of the main causes of rising morbidity—the increase in alcoholism, drug overdoses and suicides—as “deaths of despair” by those who have given up hope.

Credit: Jen Christiansen; Sources: “The Fading American Dream: Trends in Absolute Income Mobility Since 1940,” by Raj Chetty et al., in Science, Vol. 356; April 28, 2017 (child-parent wealth comparison); World Inequality database (90% versus 1% wealth trend data)

Defenders of America's inequality have a pat explanation. They refer to the workings of a competitive market, where the laws of supply and demand determine wages, prices and even interest rates—a mechanical system, much like that describing the physical universe. Those with scarce assets or skills are amply rewarded, they argue, because of the larger contributions they make to the economy. What they get merely represents what they have contributed. Often they take out less than they contributed, so what is left over for the rest is that much more.

This fictional narrative may at one time have assuaged the guilt of those at the top and persuaded everyone else to accept this sorry state of affairs. Perhaps the defining moment exposing the lie was the 2008 financial crisis, when the bankers who brought the global economy to the brink of ruin with predatory lending, market manipulation and various other antisocial practices walked away with millions of dollars in bonuses just as millions of Americans lost their jobs and homes and tens of millions more worldwide suffered on their account. Virtually none of these bankers were ever held to account for their misdeeds.

I became aware of the fantastical nature of this narrative as a schoolboy, when I thought of the wealth of the plantation owners, built on the backs of slaves. At the time of the Civil War, the market value of the slaves in the South was approximately half of the region's total wealth, including the value of the land and the physical capital—the factories and equipment. The wealth of at least this part of this nation was not based on industry, innovation and commerce but rather on exploitation. Today we have replaced this open exploitation with more insidious forms, which have intensified since the Reagan-Thatcher revolution of the 1980s. This exploitation, I will argue, is largely to blame for the escalating inequality in the U.S.

After the New Deal of the 1930s, American inequality went into decline. By the 1950s inequality had receded to such an extent that another Nobel laureate in economics, Simon Kuznets, formulated what came to be called Kuznets's law. In the early stages of development, as some parts of a country seize new opportunities, inequalities grow, he postulated; in the later stages, they shrink. The theory long fit the data—but then, around the early 1980s, the trend abruptly reversed.

Explaining Inequality

Economists have put forward a range of explanations for why inequality has in fact been increasing in many developed countries. Some argue that advances in technology have spurred the demand for skilled labor relative to unskilled labor, thereby depressing the wages of the latter. Yet that alone cannot explain why even skilled labor has done so poorly over the past two decades, why average wages have done so badly and why matters are so much worse in the U.S. than in other developed nations. Changes in technology are global and should affect all advanced economies in the same way. Other economists blame globalization itself, which has weakened the power of workers. Firms can and do move abroad unless demands for higher wages are curtailed. But again, globalization has been integral to all advanced economies. Why is its impact so much worse in the U.S.?

The shift from a manufacturing to a service-based economy is partly to blame. At its extreme—a firm of one person—the service economy is a winner-takes-all system. A movie star makes millions, for example, whereas most actors make a pittance. Overall, wages are likely to be far more widely dispersed in a service economy than in one based on manufacturing, so the transition contributes to greater inequality. This fact does not explain, however, why the average wage has not improved for decades. Moreover, the shift to the service sector is happening in most other advanced countries: Why are matters so much worse in the U.S.?

Again, because services are often provided locally, firms have more market power: the ability to raise prices above what would prevail in a competitive market. A small town in rural America may have only one authorized Toyota repair shop, which virtually every Toyota owner is forced to patronize. The providers of these local services can raise prices over costs, increasing their profits and the share of income going to owners and managers. This, too, increases inequality. But again, why is U.S. inequality practically unique?

In his celebrated 2013 treatise Capital in the Twenty-First Century, French economist Thomas Piketty shifts the gaze to capitalists. He suggests that the few who own much of a country's capital save so much that, given the stable and high return to capital (relative to the growth rate of the economy), their share of the national income has been increasing. His theory has, however, been questioned on many grounds. For instance, the savings rate of even the rich in the U.S. is so low, compared with the rich in other countries, that the increase in inequality should be lower here, not greater.

An alternative theory is far more consonant with the facts. Since the mid-1970s the rules of the economic game have been rewritten, both globally and nationally, in ways that advantage the rich and disadvantage the rest. And they have been rewritten further in this perverse direction in the U.S. than in other developed countries—even though the rules in the U.S. were already less favorable to workers. From this perspective, increasing inequality is a matter of choice: a consequence of our policies, laws and regulations.

In the U.S., the market power of large corporations, which was greater than in most other advanced countries to begin with, has increased even more than elsewhere. On the other hand, the market power of workers, which started out less than in most other advanced countries, has fallen further than elsewhere. This is not only because of the shift to a service-sector economy—it is because of the rigged rules of the game, rules set in a political system that is itself rigged through gerrymandering, voter suppression and the influence of money. A vicious spiral has formed: economic inequality translates into political inequality, which leads to rules that favor the wealthy, which in turn reinforces economic inequality.

Feedback Loop

Political scientists have documented the ways in which money influences politics in certain political systems, converting higher economic inequality into greater political inequality. Political inequality, in its turn, gives rise to more economic inequality as the rich use their political power to shape the rules of the game in ways that favor them—for instance, by softening antitrust laws and weakening unions. Using mathematical models, economists such as myself have shown that this two-way feedback loop between money and regulations leads to at least two stable points. If an economy starts out with lower inequality, the political system generates rules that sustain it, leading to one equilibrium situation. The American system is the other equilibrium—and will continue to be unless there is a democratic political awakening.

An account of how the rules have been shaped must begin with antitrust laws, first enacted 128 years ago in the U.S. to prevent the agglomeration of market power. Their enforcement has weakened—at a time when, if anything, the laws themselves should have been strengthened. Technological changes have concentrated market power in the hands of a few global players, in part because of so-called network effects: you are far more likely to join a particular social network or use a certain word processor if everyone you know is already using it. Once established, a firm such as Facebook or Microsoft is hard to dislodge. Moreover, fixed costs, such as that of developing a piece of software, have increased as compared with marginal costs—that of duplicating the software. A new entrant has to bear all these fixed costs up front, and if it does enter, the rich incumbent can respond by lowering prices drastically. The cost of making an additional e-book or photo-editing program is essentially zero.

In short, entry is hard and risky, which gives established firms with deep war chests enormous power to crush competitors and ultimately raise prices. Making matters worse, U.S. firms have been innovative not only in the products they make but in thinking of ways to extend and amplify their market power. The European Commission has imposed fines of billions of dollars on Microsoft and Google and ordered them to stop their anticompetitive practices (such as Google privileging its own comparison shopping service). In the U.S., we have done too little to control concentrations of market power, so it is not a surprise that it has increased in many sectors.

Credit: Jen Christiansen; Sources: Economic Report of the President. January 2017; World Inequality database

Rigged rules also explain why the impact of globalization may have been worse in the U.S. A concerted attack on unions has almost halved the fraction of unionized workers in the nation, to about 11 percent. (In Scandinavia, it is roughly 70 percent.) Weaker unions provide workers less protection against the efforts of firms to drive down wages or worsen working conditions. Moreover, U.S. investment treaties such as the North Atlantic Free Trade Agreement—treaties that were sold as a way of preventing foreign countries from discriminating against American firms—also protect investors against a tightening of environmental and health regulations abroad. For instance, they enable corporations to sue nations in private international arbitration panels for passing laws that protect citizens and the environment but threaten the multinational company's bottom line. Firms like these provisions, which enhance the credibility of a company's threat to move abroad if workers do not temper their demands. In short, these investment agreements weaken U.S. workers' bargaining power even further.

Liberated Finance

Many other changes to our norms, laws, rules and regulations have contributed to inequality. Weak corporate governance laws have allowed chief executives in the U.S. to compensate themselves 361 times more than the average worker, far more than in other developed countries. Financial liberalization—the stripping away of regulations designed to prevent the financial sector from imposing harms, such as the 2008 economic crisis, on the rest of society—has enabled the finance industry to grow in size and profitability and has increased its opportunities to exploit everyone else. Banks routinely indulge in practices that are legal but should not be, such as imposing usurious interest rates on borrowers or exorbitant fees on merchants for credit and debit cards and creating securities that are designed to fail. They also frequently do things that are illegal, including market manipulation and insider trading. In all of this, the financial sector has moved money away from ordinary Americans to rich bankers and the banks' shareholders. This redistribution of wealth is an important contributor to American inequality.

Other means of so-called rent extraction—the withdrawal of income from the national pie that is incommensurate with societal contribution—abound. For example, a legal provision enacted in 2003 prohibited the government from negotiating drug prices for Medicare—a gift of some $50 billion a year or more to the pharmaceutical industry. Special favors, such as extractive industries' obtaining public resources such as oil at below fair-market value or banks' getting funds from the Federal Reserve at near-zero interest rates (which they relend at high interest rates), also amount to rent extraction. Further exacerbating inequality is favorable tax treatment for the rich. In the U.S., those at the top pay a smaller fraction of their income in taxes than those who are much poorer—a form of largesse that the Trump administration has just worsened with the 2017 tax bill.

Some economists have argued that we can lessen inequality only by giving up on growth and efficiency. But recent research, such as work done by Jonathan Ostry and others at the International Monetary Fund, suggests that economies with greater equality perform better, with higher growth, better average standards of living and greater stability. Inequality in the extremes observed in the U.S. and in the manner generated there actually damages the economy. The exploitation of market power and the variety of other distortions I have described, for instance, makes markets less efficient, leading to underproduction of valuable goods such as basic research and overproduction of others, such as exploitative financial products.

Credit: Jen Christiansen; Sources: World Inequality Report 2018. World Inequality Lab, 2017; Branko Milanovic

Moreover, because the rich typically spend a smaller fraction of their income on consumption than the poor, total or “aggregate” demand in countries with higher inequality is weaker. Societies could make up for this gap by increasing government spending—on infrastructure, education and health, for instance, all of which are investments necessary for long-term growth. But the politics of unequal societies typically puts the burden on monetary policy: interest rates are lowered to stimulate spending. Artificially low interest rates, especially if coupled with inadequate financial market regulation, often give rise to bubbles, which is what happened with the 2008 housing crisis.
It is no surprise that, on average, people living in unequal societies have less equality of opportunity: those at the bottom never get the education that would enable them to live up to their potential. This fact, in turn, exacerbates inequality while wasting the country's most valuable resource: Americans themselves.

Restoring Justice

Morale is lower in unequal societies, especially when inequality is seen as unjust, and the feeling of being used or cheated leads to lower productivity. When those who run gambling casinos or bankers suffering from moral turpitude make a zillion times more than the scientists and inventors who brought us lasers, transistors and an understanding of DNA, it is clear that something is wrong. Then again, the children of the rich come to think of themselves as a class apart, entitled to their good fortune, and accordingly more likely to break the rules necessary for making society function. All of this contributes to a breakdown of trust, with its attendant impact on social cohesion and economic performance.

There is no magic bullet to remedy a problem as deep-rooted as America's inequality. Its origins are largely political, so it is hard to imagine meaningful change without a concerted effort to take money out of politics—through, for instance, campaign finance reform. Blocking the revolving doors by which regulators and other government officials come from and return to the same industries they regulate and work with is also essential.

Credit: Jen Christiansen; Sources: Raising America’s Pay: Why It’s Our Central Economic Policy Challenge, by Josh Bivens et al. Economic Policy Institute, June 4, 2014; The State of Working America, by Lawrence Mishel, Josh Bivens, Elise Gould and Heidi Shierholz. 12th Edition. ILR Press, 2012

Beyond that, we need more progressive taxation and high-quality federally funded public education, including affordable access to universities for all, no ruinous loans required. We need modern competition laws to deal with the problems posed by 21st-century market power and stronger enforcement of the laws we do have. We need labor laws that protect workers and their rights to unionize. We need corporate governance laws that curb exorbitant salaries bestowed on chief executives, and we need stronger financial regulations that will prevent banks from engaging in the exploitative practices that have become their hallmark. We need better enforcement of antidiscrimination laws: it is unconscionable that women and minorities get paid a mere fraction of what their white male counterparts receive. We also need more sensible inheritance laws that will reduce the intergenerational transmission of advantage and disadvantage.

The basic perquisites of a middle-class life, including a secure old age, are no longer attainable for most Americans. We need to guarantee access to health care. We need to strengthen and reform retirement programs, which have put an increasing burden of risk management on workers (who are expected to manage their portfolios to guard simultaneously against the risks of inflation and market collapse) and opened them up to exploitation by our financial sector (which sells them products designed to maximize bank fees rather than retirement security). Our mortgage system was our Achilles' heel, and we have not really fixed it. With such a large fraction of Americans living in cities, we have to have urban housing policies that ensure affordable housing for all.

It is a long agenda—but a doable one. When skeptics say it is nice but not affordable, I reply: We cannot afford to not do these things. We are already paying a high price for inequality, but it is just a down payment on what we will have to pay if we do not do something—and quickly. It is not just our economy that is at stake; we are risking our democracy.

As more of our citizens come to understand why the fruits of economic progress have been so unequally shared, there is a real danger that they will become open to a demagogue blaming the country's problems on others and making false promises of rectifying “a rigged system.” We are already experiencing a foretaste of what might happen. It could get much worse.

This article was originally published with the title "A Rigged Economy"

MORE TO EXPLORE

The Price of Inequality: How Today's Divided Society Endangers Our Future. Joseph E. Stiglitz. W. W. Norton, 2012.

The Great Divide: Unequal Societies and What We Can Do about Them. Joseph E. Stiglitz. W. W. Norton, 2015.

Rewriting the Rules of the American Economy: An Agenda for Growth and Shared Prosperity. Joseph E. Stiglitz. W. W. Norton, 2015.

Globalization and Its Discontents Revisited: Anti-globalization in the Era of Trump. Joseph E. Stiglitz. W. W. Norton, 2017.

ABOUT THE AUTHOR(S)

Joseph E. Stiglitz

Joseph E. Stiglitz is a University Professor at Columbia University and Chief Economist at the Roosevelt Institute. He received the Nobel prize in economics in 2001. Stiglitz chaired the Council of Economic Advisers from 1995–1997, during the Clinton administration, and served as the chief economist and senior vice president of the World Bank from 1997–2000. He chaired the United Nations commission on reforms of the international financial system in 2008–2009. His latest authored book is Globalization and Its Discontents Revisited (2017).

Credit: Nick Higgins

Data analysis

From Wikipedia, the free encyclopedia

Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. Data analysis has multiple facets and approaches, encompassing diverse techniques under a variety of names, while being used in different business, science, and social science domains.

Data mining is a particular data analysis technique that focuses on modeling and knowledge discovery for predictive rather than purely descriptive purposes, while business intelligence covers data analysis that relies heavily on aggregation, focusing mainly on business information. In statistical applications, data analysis can be divided into descriptive statistics, exploratory data analysis (EDA), and confirmatory data analysis (CDA). EDA focuses on discovering new features in the data while CDA focuses on confirming or falsifying existing hypotheses. Predictive analytics focuses on application of statistical models for predictive forecasting or classification, while text analytics applies statistical, linguistic, and structural techniques to extract and classify information from textual sources, a species of unstructured data. All of the above are varieties of data analysis.

Data integration is a precursor to data analysis, and data analysis is closely linked to data visualization and data dissemination. The term data analysis is sometimes used as a synonym for data modeling.

The process of data analysis

Data science process flowchart from "Doing Data Science", Cathy O'Neil and Rachel Schutt, 2013

Analysis refers to breaking a whole into its separate components for individual examination. Data analysis is a process for obtaining raw data and converting it into information useful for decision-making by users. Data is collected and analyzed to answer questions, test hypotheses or disprove theories.

Statistician John Tukey defined data analysis in 1961 as: "Procedures for analyzing data, techniques for interpreting the results of such procedures, ways of planning the gathering of data to make its analysis easier, more precise or more accurate, and all the machinery and results of (mathematical) statistics which apply to analyzing data."

There are several phases that can be distinguished, described below. The phases are iterative, in that feedback from later phases may result in additional work in earlier phases.

Data requirements

The data is necessary as inputs to the analysis, which is specified based upon the requirements of those directing the analysis or customers (who will use the finished product of the analysis). The general type of entity upon which the data will be collected is referred to as an experimental unit (e.g., a person or population of people). Specific variables regarding a population (e.g., age and income) may be specified and obtained. Data may be numerical or categorical (i.e., a text label for numbers).

Data collection

Data is collected from a variety of sources. The requirements may be communicated by analysts to custodians of the data, such as information technology personnel within an organization. The data may also be collected from sensors in the environment, such as traffic cameras, satellites, recording devices, etc. It may also be obtained through interviews, downloads from online sources, or reading documentation.

Data processing

The phases of the intelligence cycle used to convert raw information into actionable intelligence or knowledge are conceptually similar to the phases in data analysis.

Data initially obtained must be processed or organised for analysis. For instance, these may involve placing data into rows and columns in a table format (i.e., structured data) for further analysis, such as within a spreadsheet or statistical software.

Data cleaning

Once processed and organised, the data may be incomplete, contain duplicates, or contain errors. The need for data cleaning will arise from problems in the way that data is entered and stored. Data cleaning is the process of preventing and correcting these errors. Common tasks include record matching, identifying inaccuracy of data, overall quality of existing data, deduplication, and column segmentation. Such data problems can also be identified through a variety of analytical techniques. For example, with financial information, the totals for particular variables may be compared against separately published numbers believed to be reliable. Unusual amounts above or below pre-determined thresholds may also be reviewed. There are several types of data cleaning that depend on the type of data such as phone numbers, email addresses, employers etc. Quantitative data methods for outlier detection can be used to get rid of likely incorrectly entered data. Textual data spell checkers can be used to lessen the amount of mistyped words, but it is harder to tell if the words themselves are correct.

Exploratory data analysis

Once the data is cleaned, it can be analyzed. Analysts may apply a variety of techniques referred to as exploratory data analysis to begin understanding the messages contained in the data. The process of exploration may result in additional data cleaning or additional requests for data, so these activities may be iterative in nature. Descriptive statistics, such as the average or median, may be generated to help understand the data. Data visualization may also be used to examine the data in graphical format, to obtain additional insight regarding the messages within the data.

Modeling and algorithms

Mathematical formulas or models called algorithms may be applied to the data to identify relationships among the variables, such as correlation or causation. In general terms, models may be developed to evaluate a particular variable in the data based on other variable(s) in the data, with some residual error depending on model accuracy (i.e., Data = Model + Error).

Inferential statistics includes techniques to measure relationships between particular variables. For example, regression analysis may be used to model whether a change in advertising (independent variable X) explains the variation in sales (dependent variable Y). In mathematical terms, Y (sales) is a function of X (advertising). It may be described as Y = aX + b + error, where the model is designed such that a and b minimize the error when the model predicts Y for a given range of values of X. Analysts may attempt to build models that are descriptive of the data to simplify analysis and communicate results.

Data product

A data product is a computer application that takes data inputs and generates outputs, feeding them back into the environment. It may be based on a model or algorithm. An example is an application that analyzes data about customer purchasing history and recommends other purchases the customer might enjoy.

Communication

Data visualization to understand the results of a data analysis.

Once the data is analyzed, it may be reported in many formats to the users of the analysis to support their requirements. The users may have feedback, which results in additional analysis. As such, much of the analytical cycle is iterative.

When determining how to communicate the results, the analyst may consider data visualization techniques to help clearly and efficiently communicate the message to the audience. Data visualization uses information displays (such as tables and charts) to help communicate key messages contained in the data. Tables are helpful to a user who might lookup specific numbers, while charts (e.g., bar charts or line charts) may help explain the quantitative messages contained in the data.

Quantitative messages

A time series illustrated with a line chart demonstrating trends in U.S. federal spending and revenue over time.

 

A scatterplot illustrating correlation between two variables (inflation and unemployment) measured at points in time.

Stephen Few described eight types of quantitative messages that users may attempt to understand or communicate from a set of data and the associated graphs used to help communicate the message. Customers specifying requirements and analysts performing the data analysis may consider these messages during the course of the process.

1. Time-series: A single variable is captured over a period of time, such as the unemployment rate over a 10-year period. A line chart may be used to demonstrate the trend;
2. Ranking: Categorical subdivisions are ranked in ascending or descending order, such as a ranking of sales performance (the measure) by sales persons (the category, with each sales person a categorical subdivision) during a single period. A bar chart may be used to show the comparison across the sales persons;
3. Part-to-whole: Categorical subdivisions are measured as a ratio to the whole (i.e., a percentage out of 100%). A pie chart or bar chart can show the comparison of ratios, such as the market share represented by competitors in a market;
4. Deviation: Categorical subdivisions are compared against a reference, such as a comparison of actual vs. budget expenses for several departments of a business for a given time period. A bar chart can show comparison of the actual versus the reference amount;
5. Frequency distribution: Shows the number of observations of a particular variable for given interval, such as the number of years in which the stock market return is between intervals such as 0–10%, 11–20%, etc. A histogram, a type of bar chart, may be used for this analysis;
6. Correlation: Comparison between observations represented by two variables (X,Y) to determine if they tend to move in the same or opposite directions. For example, plotting unemployment (X) and inflation (Y) for a sample of months. A scatter plot is typically used for this message;
7. Nominal comparison: Comparing categorical subdivisions in no particular order, such as the sales volume by product code. A bar chart may be used for this comparison;
8. Geographic or geospatial: Comparison of a variable across a map or layout, such as the unemployment rate by state or the number of persons on the various floors of a building. A cartogram is a typical graphic used.

Techniques for analyzing quantitative data

Author Jonathan Koomey has recommended a series of best practices for understanding quantitative data. These include:

• Check raw data for anomalies prior to performing your analysis;
• Re-perform important calculations, such as verifying columns of data that are formula driven;
• Confirm main totals are the sum of subtotals;
• Check relationships between numbers that should be related in a predictable way, such as ratios over time;
• Normalize numbers to make comparisons easier, such as analyzing amounts per person or relative to GDP or as an index value relative to a base year;
• Break problems into component parts by analyzing factors that led to the results, such as DuPont analysis of return on equity.

For the variables under examination, analysts typically obtain descriptive statistics for them, such as the mean (average), median, and standard deviation. They may also analyze the distribution of the key variables to see how the individual values cluster around the mean.

An illustration of the MECE principle used for data analysis.

The consultants at McKinsey and Company named a technique for breaking a quantitative problem down into its component parts called the MECE principle. Each layer can be broken down into its components; each of the sub-components must be mutually exclusive of each other and collectively add up to the layer above them. The relationship is referred to as "Mutually Exclusive and Collectively Exhaustive" or MECE. For example, profit by definition can be broken down into total revenue and total cost. In turn, total revenue can be analyzed by its components, such as revenue of divisions A, B, and C (which are mutually exclusive of each other) and should add to the total revenue (collectively exhaustive).

Analysts may use robust statistical measurements to solve certain analytical problems. Hypothesis testing is used when a particular hypothesis about the true state of affairs is made by the analyst and data is gathered to determine whether that state of affairs is true or false. For example, the hypothesis might be that "Unemployment has no effect on inflation", which relates to an economics concept called the Phillips Curve. Hypothesis testing involves considering the likelihood of Type I and type II errors, which relate to whether the data supports accepting or rejecting the hypothesis.

Regression analysis may be used when the analyst is trying to determine the extent to which independent variable X affects dependent variable Y (e.g., "To what extent do changes in the unemployment rate (X) affect the inflation rate (Y)?"). This is an attempt to model or fit an equation line or curve to the data, such that Y is a function of X.

Necessary condition analysis (NCA) may be used when the analyst is trying to determine the extent to which independent variable X allows variable Y (e.g., "To what extent is a certain unemployment rate (X) necessary for a certain inflation rate (Y)?"). Whereas (multiple) regression analysis uses additive logic where each X-variable can produce the outcome and the X's can compensate for each other (they are sufficient but not necessary), necessary condition analysis (NCA) uses necessity logic, where one or more X-variables allow the outcome to exist, but may not produce it (they are necessary but not sufficient). Each single necessary condition must be present and compensation is not possible.

Analytical activities of data users

Users may have particular data points of interest within a data set, as opposed to general messaging outlined above. Such low-level user analytic activities are presented in the following table. The taxonomy can also be organized by three poles of activities: retrieving values, finding data points, and arranging data points.

#	Task	General Description	Pro Forma Abstract	Examples
1	Retrieve Value	Given a set of specific cases, find attributes of those cases.	What are the values of attributes {X, Y, Z, ...} in the data cases {A, B, C, ...}?	- What is the mileage per gallon of the Ford Mondeo? - How long is the movie Gone with the Wind?
2	Filter	Given some concrete conditions on attribute values, find data cases satisfying those conditions.	Which data cases satisfy conditions {A, B, C...}?	- What Kellogg's cereals have high fiber? - What comedies have won awards? - Which funds underperformed the SP-500?
3	Compute Derived Value	Given a set of data cases, compute an aggregate numeric representation of those data cases.	What is the value of aggregation function F over a given set S of data cases?	- What is the average calorie content of Post cereals? - What is the gross income of all stores combined? - How many manufacturers of cars are there?
4	Find Extremum	Find data cases possessing an extreme value of an attribute over its range within the data set.	What are the top/bottom N data cases with respect to attribute A?	- What is the car with the highest MPG? - What director/film has won the most awards? - What Marvel Studios film has the most recent release date?
5	Sort	Given a set of data cases, rank them according to some ordinal metric.	What is the sorted order of a set S of data cases according to their value of attribute A?	- Order the cars by weight. - Rank the cereals by calories.
6	Determine Range	Given a set of data cases and an attribute of interest, find the span of values within the set.	What is the range of values of attribute A in a set S of data cases?	- What is the range of film lengths? - What is the range of car horsepowers? - What actresses are in the data set?
7	Characterize Distribution	Given a set of data cases and a quantitative attribute of interest, characterize the distribution of that attribute’s values over the set.	What is the distribution of values of attribute A in a set S of data cases?	- What is the distribution of carbohydrates in cereals? - What is the age distribution of shoppers?
8	Find Anomalies	Identify any anomalies within a given set of data cases with respect to a given relationship or expectation, e.g. statistical outliers.	Which data cases in a set S of data cases have unexpected/exceptional values?	- Are there exceptions to the relationship between horsepower and acceleration? - Are there any outliers in protein?
9	Cluster	Given a set of data cases, find clusters of similar attribute values.	Which data cases in a set S of data cases are similar in value for attributes {X, Y, Z, ...}?	- Are there groups of cereals w/ similar fat/calories/sugar? - Is there a cluster of typical film lengths?
10	Correlate	Given a set of data cases and two attributes, determine useful relationships between the values of those attributes.	What is the correlation between attributes X and Y over a given set S of data cases?	- Is there a correlation between carbohydrates and fat? - Is there a correlation between country of origin and MPG? - Do different genders have a preferred payment method? - Is there a trend of increasing film length over the years?
11	Contextualization	Given a set of data cases, find contextual relevancy of the data to the users.	Which data cases in a set S of data cases are relevant to the current users' context?	- Are there groups of restaurants that have foods based on my current caloric intake?

Barriers to effective analysis

Barriers to effective analysis may exist among the analysts performing the data analysis or among the audience. Distinguishing fact from opinion, cognitive biases, and innumeracy are all challenges to sound data analysis.

Confusing fact and opinion

You are entitled to your own opinion, but you are not entitled to your own facts.

 

Daniel Patrick Moynihan

Effective analysis requires obtaining relevant facts to answer questions, support a conclusion or formal opinion, or test hypotheses. Facts by definition are irrefutable, meaning that any person involved in the analysis should be able to agree upon them. For example, in August 2010, the Congressional Budget Office (CBO) estimated that extending the Bush tax cuts of 2001 and 2003 for the 2011–2020 time period would add approximately $3.3 trillion to the national debt. Everyone should be able to agree that indeed this is what CBO reported; they can all examine the report. This makes it a fact. Whether persons agree or disagree with the CBO is their own opinion.

As another example, the auditor of a public company must arrive at a formal opinion on whether financial statements of publicly traded corporations are "fairly stated, in all material respects." This requires extensive analysis of factual data and evidence to support their opinion. When making the leap from facts to opinions, there is always the possibility that the opinion is erroneous.

Cognitive biases

There are a variety of cognitive biases that can adversely affect analysis. For example, confirmation bias is the tendency to search for or interpret information in a way that confirms one's preconceptions. In addition, individuals may discredit information that does not support their views.

Analysts may be trained specifically to be aware of these biases and how to overcome them. In his book Psychology of Intelligence Analysis, retired CIA analyst Richards Heuer wrote that analysts should clearly delineate their assumptions and chains of inference and specify the degree and source of the uncertainty involved in the conclusions. He emphasized procedures to help surface and debate alternative points of view.

Innumeracy

Effective analysts are generally adept with a variety of numerical techniques. However, audiences may not have such literacy with numbers or numeracy; they are said to be innumerate. Persons communicating the data may also be attempting to mislead or misinform, deliberately using bad numerical techniques.

For example, whether a number is rising or falling may not be the key factor. More important may be the number relative to another number, such as the size of government revenue or spending relative to the size of the economy (GDP) or the amount of cost relative to revenue in corporate financial statements. This numerical technique is referred to as normalization or common-sizing. There are many such techniques employed by analysts, whether adjusting for inflation (i.e., comparing real vs. nominal data) or considering population increases, demographics, etc. Analysts apply a variety of techniques to address the various quantitative messages described in the section above.

Analysts may also analyze data under different assumptions or scenarios. For example, when analysts perform financial statement analysis, they will often recast the financial statements under different assumptions to help arrive at an estimate of future cash flow, which they then discount to present value based on some interest rate, to determine the valuation of the company or its stock. Similarly, the CBO analyzes the effects of various policy options on the government's revenue, outlays and deficits, creating alternative future scenarios for key measures.

Other topics

Smart buildings

A data analytics approach can be used in order to predict energy consumption in buildings. The different steps of the data analysis process are carried out in order to realise smart buildings, where the building management and control operations including heating, ventilation, air conditioning, lighting and security are realised automatically by miming the needs of the building users and optimising resources like energy and time.

Analytics and business intelligence

Analytics is the "extensive use of data, statistical and quantitative analysis, explanatory and predictive models, and fact-based management to drive decisions and actions." It is a subset of business intelligence, which is a set of technologies and processes that use data to understand and analyze business performance.

Education

Analytic activities of data visualization users

In education, most educators have access to a data system for the purpose of analyzing student data. These data systems present data to educators in an over-the-counter data format (embedding labels, supplemental documentation, and a help system and making key package/display and content decisions) to improve the accuracy of educators’ data analyses.

Practitioner notes

Initial data analysis

The most important distinction between the initial data analysis phase and the main analysis phase, is that during initial data analysis one refrains from any analysis that is aimed at answering the original research question. The initial data analysis phase is guided by the following four questions:

Quality of data

The quality of the data should be checked as early as possible. Data quality can be assessed in several ways, using different types of analysis: frequency counts, descriptive statistics (mean, standard deviation, median), normality (skewness, kurtosis, frequency histograms, n: variables are compared with coding schemes of variables external to the data set, and possibly corrected if coding schemes are not comparable.

• Test for common-method variance.

The choice of analyses to assess the data quality during the initial data analysis phase depends on the analyses that will be conducted in the main analysis phase.

Quality of measurements

The quality of the measurement instruments should only be checked during the initial data analysis phase when this is not the focus or research question of the study. One should check whether structure of measurement instruments corresponds to structure reported in the literature.

There are two ways to assess measurement: [NOTE: only one way seems to be listed]

• Analysis of homogeneity (internal consistency), which gives an indication of the reliability of a measurement instrument. During this analysis, one inspects the variances of the items and the scales, the Cronbach's α of the scales, and the change in the Cronbach's alpha when an item would be deleted from a scale.

Initial transformations

After assessing the quality of the data and of the measurements, one might decide to impute missing data, or to perform initial transformations of one or more variables, although this can also be done during the main analysis phase.

Possible transformations of variables are:

• Square root transformation (if the distribution differs moderately from normal);
• Log-transformation (if the distribution differs substantially from normal);
• Inverse transformation (if the distribution differs severely from normal);
• Make categorical (ordinal / dichotomous) (if the distribution differs severely from normal, and no transformations help).

Did the implementation of the study fulfill the intentions of the research design?

One should check the success of the randomization procedure, for instance by checking whether background and substantive variables are equally distributed within and across groups.
If the study did not need or use a randomization procedure, one should check the success of the non-random sampling, for instance by checking whether all subgroups of the population of interest are represented in sample.

Other possible data distortions that should be checked are:

• dropout (this should be identified durin. the initial data analysis phase);
• Item nonresponse (whether this is random or not should be assessed during the initial data analysis phase);
• Treatment quality (using manipulation checks).

Characteristics of data sample

In any report or article, the structure of the sample must be accurately described. It is especially important to exactly determine the structure of the sample (and specifically the size of the subgroups) when subgroup analyses will be performed during the main analysis phase.

The characteristics of the data sample can be assessed by looking at:

• Basic statistics of important variables;
• Scatter plots;
• Correlations and associations;
• Cross-tabulations;

Final stage of the initial data analysis

During the final stage, the findings of the initial data analysis are documented, and necessary, preferable, and possible corrective actions are taken.

Also, the original plan for the main data analyses can and should be specified in more detail or rewritten.

In order to do this, several decisions about the main data analyses can and should be made:

• In the case of non-normals: should one transform variables; make variables categorical (ordinal/dichotomous); adapt the analysis method?
• In the case of missing data: should one neglect or impute the missing data; which imputation technique should be used?
• In the case of outliers: should one use robust analysis techniques?
• In case items do not fit the scale: should one adapt the measurement instrument by omitting items, or rather ensure comparability with other (uses of the) measurement instrument(s)?
• In the case of (too) small subgroups: should one drop the hypothesis about inter-group differences, or use small sample techniques, like exact tests or bootstrapping?
• In case the randomization procedure seems to be defective: can and should one calculate propensity scores and include them as covariates in the main analyses?

Analysis

Several analyses can be used during the initial data analysis phase:

• Univariate statistics (single variable);
• Bivariate associations (correlations);
• Graphical techniques (scatter plots).

It is important to take the measurement levels of the variables into account for the analyses, as special statistical techniques are available for each level:

• Nominal and ordinal variables
  • Frequency counts (numbers and percentages);
  • Associations
    • circumambulations (crosstabulations);
    • hierarchical loglinear analysis (restricted to a maximum of 8 variables);
    • loglinear analysis (to identify relevant/important variables and possible confounders);
  • Exact tests or bootstrapping (in case subgroups are small);
  • Computation of new variables;
• Continuous variables
  • Distribution;
    • Statistics (M, SD, variance, skewness, kurtosis);
    • Stem-and-leaf displays;
    • Box plots;

Nonlinear analysis

Nonlinear analysis will be necessary when the data is recorded from a nonlinear system. Nonlinear systems can exhibit complex dynamic effects including bifurcations, chaos, harmonics and subharmonics that cannot be analyzed using simple linear methods. Nonlinear data analysis is closely related to nonlinear system identification.

Main data analysis

In the main analysis phase analyses aimed at answering the research question are performed as well as any other relevant analysis needed to write the first draft of the research report.

Exploratory and confirmatory approaches

In the main analysis phase either an exploratory or confirmatory approach can be adopted. Usually the approach is decided before data is collected. In an exploratory analysis no clear hypothesis is stated before analysing the data, and the data is searched for models that describe the data well. In a confirmatory analysis clear hypotheses about the data are tested.

Exploratory data analysis should be interpreted carefully. When testing multiple models at once there is a high chance on finding at least one of them to be significant, but this can be due to a type 1 error. It is important to always adjust the significance level when testing multiple models with, for example, a Bonferroni correction. Also, one should not follow up an exploratory analysis with a confirmatory analysis in the same dataset. An exploratory analysis is used to find ideas for a theory, but not to test that theory as well. When a model is found exploratory in a dataset, then following up that analysis with a confirmatory analysis in the same dataset could simply mean that the results of the confirmatory analysis are due to the same type 1 error that resulted in the exploratory model in the first place. The confirmatory analysis therefore will not be more informative than the original exploratory analysis.

Stability of results

It is important to obtain some indication about how generalizable the results are. While this is hard to check, one can look at the stability of the results. Are the results reliable and reproducible? There are two main ways of doing this:

• Cross-validation: By splitting the data in multiple parts we can check if an analysis (like a fitted model) based on one part of the data generalizes to another part of the data as well;
• Sensitivity analysis: A procedure to study the behavior of a system or model when global parameters are (systematically) varied. One way to do this is with bootstrapping.

Statistical methods

Many statistical methods have been used for statistical analyses. A very brief list of four of the more popular methods is:

• General linear model: A widely used model on which various methods are based (e.g. t test, ANOVA, ANCOVA, MANOVA). Usable for assessing the effect of several predictors on one or more continuous dependent variables;
• Generalized linear model: An extension of the general linear model for discrete dependent variables;
• Structural equation modelling: Usable for assessing latent structures from measured manifest variables;
• Item response theory: Models for (mostly) assessing one latent variable from several binary measured variables (e.g. an exam).

Free software for data analysis

• DevInfo – a database system endorsed by the United Nations Development Group for monitoring and analyzing human development;
• ELKI – data mining framework in Java with data mining oriented visualization functions;
• KNIME – the Konstanz Information Miner, a user friendly and comprehensive data analytics framework;
• Orange – A visual programming tool featuring interactive data visualization and methods for statistical data analysis, data mining, and machine learning;
• PAST – free software for scientific data analysis;
• PAW – FORTRAN/C data analysis framework developed at CERN;
• R – a programming language and software environment for statistical computing and graphics;
• ROOT – C++ data analysis framework developed at CERN;
• SciPy and Pandas – Python libraries for data analysis.

International data analysis contests

Different companies or organizations hold a data analysis contests to encourage researchers utilize their data or to solve a particular question using data analysis. A few examples of well-known international data analysis contests are as follows.

• Kaggle competition held by Kaggle;
• LTPP data analysis contest held by FHWA and ASCE.