Too much data can also pollute decision making. You need enough of the right data. I've got an essay coming out shortly on Infobesity talking about this.
Not analysis paralysis per se. But more data doesn’t always lead to better analysis and can often bias the data if you aren’t extrememly careful about how you are filtering it. Tons of data can give you patterns that distract.
There's definitely a lot of material just centered around analyzing data that we both need to write, I think. That subject won't get old any time soon.
My work in human genetics of complex disease put me in the field of "wide" data, that is, more predictors than samples available (DNA/RNA). My job was to apply machine learning to tackle this problem. Curse of dimensionality!
That's really cool! This piece sat in my drafts for a long time (pretty unusual for me) until I came across the Laplace incident a few weeks ago, and suddenly I knew how to finish it. I had a lot of experience with this sort of guessing, mainly via small business (but also, to a limited degree, with martial arts).
I am currently reading a book called "Noise" that goes in depth to why decision making and forecasting are so difficult. I am about halfway through it, and while a little dry in spots, it is enlightening.
In answer to your question about having you ever had to make decisions with incomplete data, my answer is "All the time!"😂
I worked in resource management, and I was once told, early in my career, that "Success in resource management is productivity in the absence of data." I found that to be true most of my career. I was forced to be creative in ways to make inferences from little or no data that were still credible, at times relying on things like allometric relationships or the comparative method. Often times we (I almost always worked in a team) had to use indirect methods, or had to take data collected for other purposes to make inferences about different resources.
I've worked in incredibly data rich fields, too...commercial fisheries harvest management is about as data rich as it gets...and while it has advantages, I can tell you that you never have enough data to answer all the questions you need to answer (in resource management, anyway)!😂
It's a weird spot to be in where you have to say, "I'd love to have about 10x as much data, but this decision has to be made today." You've had a lot of opportunities to do this over the years!
I believe in data driven decisions. You're assured of meticulous accuracy and keeping bias away from the decision table. But the more we have more data, the more confusion reigns. Over analysis paralysis, time wastage will obstruct decision making process. In my perspective, we need only 30 or less. When the rubber meets the road, feedback data will offer clarity and clarity is what is highly needed in data driven decisions.
Data driven decision making is a cult in product companies. Problem is when you have millions of usage datapoints, figuring out which ones matter. So you create KPIs (key performance indicators) that are often a combination of data points and drive to OKRs (objective->key result) that are data defined. Then you build ridiculous dashboards of your KPIs and OKRs that measure everything until you don’t even know what you’re looking at anymore.
Man, I make bold decisions based on zero data, like, all the time. I wouldn't call any of them "brilliant." I probably wouldn't even call many of them "legal" or "not a danger to everyone around me."
Too much data can also pollute decision making. You need enough of the right data. I've got an essay coming out shortly on Infobesity talking about this.
Analysis paralysis! Yeah, there is a balance for sure. Looking forward to the new essay.
Not analysis paralysis per se. But more data doesn’t always lead to better analysis and can often bias the data if you aren’t extrememly careful about how you are filtering it. Tons of data can give you patterns that distract.
Oh yes, I see. That too.
There's definitely a lot of material just centered around analyzing data that we both need to write, I think. That subject won't get old any time soon.
I want to do a TV show about the history of animation that would be like what Sagan did with "Cosmos".
How is your Saganvoice? Mine's pretty good, but only when I say "billions and billions."
"Billions and billions of dollars..." I can save that for talking about Pixar.
My work in human genetics of complex disease put me in the field of "wide" data, that is, more predictors than samples available (DNA/RNA). My job was to apply machine learning to tackle this problem. Curse of dimensionality!
That's really cool! This piece sat in my drafts for a long time (pretty unusual for me) until I came across the Laplace incident a few weeks ago, and suddenly I knew how to finish it. I had a lot of experience with this sort of guessing, mainly via small business (but also, to a limited degree, with martial arts).
I am currently reading a book called "Noise" that goes in depth to why decision making and forecasting are so difficult. I am about halfway through it, and while a little dry in spots, it is enlightening.
https://www.amazon.com/Noise-Human-Judgment-Daniel-Kahneman/dp/0316451401
Nice. Kehneman has a lot of insight in areas like this.
In answer to your question about having you ever had to make decisions with incomplete data, my answer is "All the time!"😂
I worked in resource management, and I was once told, early in my career, that "Success in resource management is productivity in the absence of data." I found that to be true most of my career. I was forced to be creative in ways to make inferences from little or no data that were still credible, at times relying on things like allometric relationships or the comparative method. Often times we (I almost always worked in a team) had to use indirect methods, or had to take data collected for other purposes to make inferences about different resources.
I've worked in incredibly data rich fields, too...commercial fisheries harvest management is about as data rich as it gets...and while it has advantages, I can tell you that you never have enough data to answer all the questions you need to answer (in resource management, anyway)!😂
It's a weird spot to be in where you have to say, "I'd love to have about 10x as much data, but this decision has to be made today." You've had a lot of opportunities to do this over the years!
Dayum Andrew. That video was brilliant! ❤️
The Sagan video? He was just absolutely amazing at explaining things and invoking that sense of wonder. Sagan made it real for us.
I need to watch more of him. 🥰
Every ten years or so, I think it's a good idea to rewatch the original Cosmos series.
I can predict you'd make a lot of money if you reprinted the USG Sub-only gorilla t-shirts.
I may or may not have heard this before.
I believe in data driven decisions. You're assured of meticulous accuracy and keeping bias away from the decision table. But the more we have more data, the more confusion reigns. Over analysis paralysis, time wastage will obstruct decision making process. In my perspective, we need only 30 or less. When the rubber meets the road, feedback data will offer clarity and clarity is what is highly needed in data driven decisions.
I agree - you need a lot less data than you probably would guess at first. Analysis paralysis is real!
Data driven decision making is a cult in product companies. Problem is when you have millions of usage datapoints, figuring out which ones matter. So you create KPIs (key performance indicators) that are often a combination of data points and drive to OKRs (objective->key result) that are data defined. Then you build ridiculous dashboards of your KPIs and OKRs that measure everything until you don’t even know what you’re looking at anymore.
I use KPIs in my current businesses! You have to be careful not to lose all meaning, though; that's 100% accurate.
Man, I make bold decisions based on zero data, like, all the time. I wouldn't call any of them "brilliant." I probably wouldn't even call many of them "legal" or "not a danger to everyone around me."
But I sure am fast at making them!
You're like Laplace, but faster. That's what I'm getting out of this comment.