Month: April 2016

One of the bases of the quncertain platform is that probabilities are not measures of physical quantities. Rather, they are measures of the limits of your knowledge.

From this point of view, making educated guesses about the future is very similar to making educated guesses about the past. Of course there are a couple of differences.

For one, it is possible to have complete knowledge about certain aspects of the past. This is not true of the future.

Also, even if we have not memorized facts about the past completely, we still have ‘contact knowledge’ about the past. We may have heard about a heat wave in the eastern part of the US, or about a crash in the price of commodities, or a sports teams’ winning streak. These incomplete pieces of knowledge help inform us about specific events in the past.

But, unless you have encyclopedic access to past events, predicting events in the future is the same as predicting events in the past. With an extra dose of uncertainty.

Quncertain uses this equivalence to give immediate feedback on past predictions, allowing users to calibrate the plausibility they lend to their predictions.

 

 

Here are some common investing assumptions. How much plausibility do you lend to each one?

1. Though the stock market will fluctuate, it tends to go higher. An index fund which tracks the US stock market will be worth more in a year. 5 years? 10 years? 20 years?
2. Real Estate is a good investment. A house purchased now will be worth more in a year. 5 years? 10 years? 20 years? Portland market? Detroit market?
3. Apple is the best company in the world, and investing in AAPL will give above average returns.

I hope to supplement this post with an interactive question set for each of the assumptions above. There are times in the past where each of the above statements turned out to be true, and times when each of them turned out to be false. The financial questions on quncertain.com will currently let you do something close — predict the performance of different sectors of the US economy, both in the past and in the future.

You will find that (unless you cheat), you will ‘predict’ the past imperfectly. And predicting the future is even more uncertain. This is not a revelation, but most people feel they have some insight into the market at some point of time. Maybe using this app will help you verify that feeling.

People have taken different strategies in reaction to efficient markets that are almost impossible to predict:

1. Invest passively in index funds
2. Opt out and keep cash
3. Be a contrarian and fade the market when it makes large moves in either direction
4. Be a momentum trader and pile in and out of stocks in proportion to market movements
5. Use computers to model and predict the market
6. Trade other people’s money, and take your guaranteed profit from salary, commissions, and other incentives (in other words, take your own skin out of the game)

Each of these strategies has assumptions, and we should realize that these assumptions may not be true. I hope using this site will help you understand that there is uncertainty in all forward-looking statements. It is wise to prepare for your assumptions to be incorrect. This is called risk management, an area I will write more about. Probabilities are just one of the inputs needed in handling risk optimally.

Quncertain is a platform for making many predictions, in domains you are very knowledgeable and domains where you’re very ignorant. As you make enough predictions you come to realize a couple things:

First, you realize that even in domains where you are very knowledgeable, and about topics where you feel very confident, you will sometimes be wrong.

Second, you realize that even in domains where you are very ignorant, you usually are able to answer at a higher rate than a completely naive guess (assuming you’re not arbitrarily selecting answers and are actually trying to be correct).

In the quncertain platform, the first scenario will manifest in making wrong predictions about things you assign a high plausibility.

The second scenario will manifest in topics where you continually pick an answer, but pick it at only 50% plausibility. Giving with an answer with 50% plausibility is like saying “I don’t know” but being forced to pick an answer anyway. You’ll find that over time you get a higher rate than 50% of these correct.

This is because you are tangentially aware of some knowledge in most domains. You don’t follow the stock market, but recognize certain company names. You know cities in the southern US are generally warmer than cities in the north. You have heard of certain books or authors or athletes, which makes them more likely to be higher on lists than more obscure options.

People might start overconfident, incorrectly answering items they rated with a high plausibility. The solution is to realize it and be more cautious in the amount of plausibility you assign to your answers. Or, they might start underconfident, and see that they are getting answers right at a higher rate than the 50% or 60% plausibilities they’re assigning. You adjust by consciously raising the plausibility you assign to answers you were previously conservative about.

If you push yourself both ways, using the calibration charts for feedback, you’ll have a better idea of how different levels of plausibility ‘feel’.

 
And, importantly, you will know to have contingency plans when you make decisions with real life consequences. Even in things where you’re very certain, prepare a little to be wrong. And even if you’re completely unsure about some outcome, listen for, and have a little faith in your gut feeling.

 

The quncertain results graph used to look like this,

and now it looks like this:

The data visualization is now more interactive, and gives the audience the ability to drill down to specific pieces of data.

For context, the above chart is a tool for users of quncertain.com to predict future events. They assign a numeric plausibility to the likelihood of a future event, and then observe how often it actually occurs. In time, they can calibrate a 70% plausibility of a future event, to an actual 70% frequency of occurrence in reality. The chart above is from a calibration step of “predicting” events that happened in the past.

But this post is specifically about charts. And about authors who cite data, audiences that scrutinize the data, and how presenting large amounts of non-abstracted data increases the trust of the audience in the author’s conclusions. Data presented without a large amount of context should make an audience suspicious; the possibility that the author is doctoring the data becomes more plausible.

Charts can misrepresent the underlying facts in many ways. Here are three common misrepresentations

 

1. Averaging and other abstractions

Abstractions allow an audience to see higher-level ideas faster and more easily than inspecting a table of numeric values. Showing the average high temperature for a particular month expresses a range of related measurements in a single value. Data visualizations like time-series charts, bar graphs, area charts, etc, are also designed to explore high-level ideas built on lower-level data. But all these abstractions can be misleading, even the widely-used concept of average.

Because we are taught averages when we’re young and because use them every day, we might forget that averages are misleading if the data being averaged is not symmetric and closely grouped. If the data is spread out, or has a skew, averages mask important information. This is why whenever an average (or median, or any single number summary) is provided, the author should place next to it a histogram showing the distribution of the points being averaged. Current technology allows us to make this enhancement easily, even in journal articles and news articles.

Metrics like “Customer Lifetime Value” have averages built into them. But a histogram will usually show the lifetime value of your customers varies widely, and probably has long tails. Looking at this distribution is probably much more revealing than looking at an average. Only data that looks like it has a close-to-central (or close-to-gaussian, close-to-normal) distribution should be represented as just a number. However even in the case of centrally-distributed data, the audience has no assurance this is the case without being able to see the distribution themselves. So the distribution should be shown in all cases.

Examples:

American mean net worth is $301,000. American median net worth is $45,000.

US Family mean income is $89,000, median income is $67,000 (https://projects.quncertain.com/us_income/income_data_2014.html)

 

2. Misclassification

Authors will sometimes split data into groups and then summarize each group. Conclusions are drawn by comparing the summaries of each group. Obviously, these conclusions are only as valid as the accuracy of the classification. And yet usually when an author presents a chart like this, there is no mechanism for an audience to inspect the classifications. Because of this, its common for data to be misclassified, both intentionally and unintentionally.

Authors have the ability to inspect raw data, and manipulate classification rules before drawing a line between (for example) high-value customers and low-value customers. Audiences do not have this ability. Maybe changing the classification rule slightly has a large impact on the resulting metrics, and makes the author’s conclusions suspect. Even when classification rules are sound, there is value from the ability to inspect group membership.

The only solution to this problem is to allow the audience to inspect the individual members that comprise the groups. Audience members can spot-check individual data points that they’re knowledgeable about. If all their spot-checks prove properly classified, they will rightly gain confidence that the author’s conclusions are sound.

Examples:

Arguments about official vs real unemployment rates

Bill Clinton’s “I did not have sex with that woman”

 

3. Undocumented Data Filtering / Outlier Removal

Data filtering is a subset of classification. In data filtering, the author removes data they consider irrelevant or distorting. Hopefully that statement sounded backwards. Because the if we are judging the data and an author’s abstraction of the data, then unquestionably the data is the ‘realer’ thing. Abstractions (averages, charts, graphs) are the things that distort the data. Modifying the data because it distorts an abstraction is disingenuous, and should make an audience less certain of an author’s conclusions.

For handling things like typos and garbage data, data filtering may be necessary. But even in this case, the data should be included somewhere and should be inspectable by the audience. The author should describe why the data was removed. And, all data that was NOT removed should be handled with additional uncertainty, knowing that the input to the author’s abstraction is likely still susceptible to typos and other inaccuracies.

Giving the audience the ability to look at the data included and excluded allows them to spot-check outliers they are aware of and see that they are handled reasonably in the chart.

Examples:

Cherry-picking endpoints on time-series graphs to show a desired trend

Regression models that don’t strictly satisfy the rules for regression

 

 

All the issues I have discussed can be largely solved by authors providing access to raw and lower-level data with their metrics, charts, and graphs. Dashboards in analytics, CRM, and financial systems show the power and elegance of being able to drill down into data. Charts and figures in news articles, journal publications, and business and conference presentations have a long way to go.

I hope to write some more about data transparency in the future and work on tools that help with this effort. Please take a look at quncertain.com, and make some predictions!