Finding answers to complex questions on the Internet is often a challenge, as a simple search can sometimes lead you down a rabbit hole of impersonal data. In A Haystack Full of Needles: Cutting Through the Clutter of the Online World to Find a Place, Partner or President, Jim Hornthal explores groundbreaking new approaches to discovering the useful insights buried deep within our complex and noisy datasphere. Hornthal, a venture capitalist in Silicon Valley, introduces us to innovators who are pushing the edges of data science and data visualization by applying the principles of pattern recognition to isolate relevant signals in the noise. Their efforts will have enormous implications for the way we practice medicine, discover music and movies, and even identify our romantic partners.

Curious to hear more about the ideas he explores in his e-book, the TED Blog asked Hornthal a few questions over email.

So much content comes pouring at us every day and it’s getting trickier to find material that is useful. What gives you hope?

It comes from the efforts of great innovators and entrepreneurs who are investing their talents and creating technologies to manage the data, filter the noise, and provide powerful new tools to help users navigate the growing tsunami of content.  So while an online search today can often feel generic and not suited to your individual tastes, there are a lot of innovators who continue to push the limits of what’s possible. And there’s hope that we can stay ahead of this unimaginable content overflow.

In you book you talk about ‘discovery engines’ as promising great rewards in this area. What are those?

A discovery engine is a smart set of algorithms and techniques that help identify a potential list of useful answers for a user when there is more than one “right” answer. One of the key defining points of discovery engines is that they get smarter with each use. For example, Pandora uses sophisticated pattern recognition techniques to build custom playlists.  The default playlists are further refined by the billions of “thumbs up” and “thumbs down” that Pandora users have registered as they listen to the music Pandora recommends.

What is the difference between the capabilities of the old-fashioned search engines and this next-generation discovery engine?

Filtering out the junk is something that both approaches to data mining have to tackle.  However, next generation discovery engines are structured more like specialized “genomes” that reflect the complexity of a user’s interests and needs, and then generates a custom set of alternatives.  For example, Triporati‘s Destination Genome Project deals with 560 quadrillion potential unique combinations of attributes (interests and activities)  to present personalized “bucket lists” from over 2,500 indexed destinations for leisure travelers looking to discover their perfect vacation destination.

While data analysis programs can help us become smarter and more efficient consumers of goods, the ability to find the signal in the noise comes with added responsibilities and risks when protecting our personal data. What are they?

There is a trade off we all make between providing information providers with a clearer picture of who we are (our interest graph, demographic profile, etc.), and the expected increase in relevance that this disclosure implies.  What is not always so clear is how that data might be used, and what its ‘shelf life’ might be.  If information providers are transparent about their intent (and that does not mean hiding behind obtuse “Terms of Service” paragraphs that most people don’t read or can’t understand) and consumers willingly opt in for the benefits provided by the increased level of intimacy and sharing, then the system works.  When either part starts to break down, the potential for harm increases.  Be aware of who you are telling what to, and don’t be afraid to use “Incognito Windows” or “Private Browsing” settings on your browsers when you are surfing new sites that you don’t necessarily know or trust.

What will the search or discovery engine of 2015 look like? Where are we heading?

Siri is a leading indicator of sophisticated voice recognition and contextual search to deliver more relevant results.  With continued higher bandwidth and faster processing speeds, big data challenges will be more readily tamed by the pioneers in data science and data visualization, whose combined efforts should help consumers.

.
A Haystack Full of Needles is part of the TED Books series. It is available for the Kindle and Nook, as well as through the iBookstore. Or download the TED Books app for your iPad or iPhone.

Are we becoming blank-eyed cyberzombies, thanks to the internet and all the tech tools we obsess about every day? Instead of asking whether the Web is making us stupid, Howard Rheingold turns that lens around and asks how digital media could actually improve our intelligence. In his new TED Book, Mind Amplifier: Can Our Digital Tools Make Us Smarter?, he examines the origins of digital mind-extending tools, and lays out the foundations for a smarter future.

So how could this play out? We asked Rheingold to explain his theory.

We often hear that technology numbs us out and makes us dumber by degree. You argue the opposite. How?

The alphabet and the printing press were technologies that enabled more people to know more, to contribute their knowledge to others, and for more people to do more complex things together. The personal computer as we know it was originally imagined to be a tool to “augment human intellect” and “raise the collective IQ,” according to pioneer Doug Engelbart. Mind Amplifier acquaints people with the original intention of digital tools and introduces both the mental and technological components of tools that can amplify human thought and communication and suggests ways that future digital media can be designed.

How does that work?

I introduce the idea of cultural evolution — the add-ons we humans have invented to extend our physical and mental capabilities, and which we teach each other. When speech or writing, alphabets or Internets are invented, the people who learn to make use of them are capable of doing things that humans were not able to do before. When entire populations gain these literacies, civilizations become more powerful and complex. As Marshall McLuhan said: we shape our tools, and then they shape us. It’s time to be more deliberate, mindful and systematic about this process of inventing tools that change those who learn to use them.

What have we done wrong so far?

The design and production of digital tools has been driven by market forces.

We’ve seen both positive and negative effects of the widespread use of digital media, and are only beginning to see the emergence of an interdisciplinary study of mind-amplifier design. So what we have done so far has been less than mindful. Although personal computers were invented to help people solve problems, and they certainly do that, the engineering and marketing of these tools has not been mindful of the more humanistic contexts of empathy, compassion and systems thinking.

You write that humans are ‘natural-born cyborgs.’ How so?

That’s a phrase I took from philosopher Andy Clark, who wrote a book with that phrase as its title. He points out that the human mind has been partially outside our brains for a long time. If you doubt it, try to multiply two four-digit numbers without reaching for a pencil or a calculator. Although inventions such as writing and mathematics are cultural inventions, the human brain is optimized for socializing, for making tools and for social learning.

The Internet has delivered an explosion of learning opportunities for today’s students, creating an abundance of information, knowledge, and teachers as well as a starkly different landscape from the one in which our ideas about school were born. Traditional educators, classrooms, and brick-and-mortar schools are no longer necessary to access information. Instead, things like blogs and wikis, as well as remote collaborations and an emphasis on critical thinking skills are the coins of the realm in this new kingdom. Yet the national dialogue on education reform focuses on using technology to update the traditional education model, failing to reassess the fundamental model on which it is built.

In Why School? How Education Must Change When Learning and Information Are Everywhere, educator, parent and blogger Will Richardson challenges traditional thinking about education— questioning whether it still holds value in its current form. How can schools adjust to this new age? Or students? Or parents? In this provocative read, Richardson provides an in-depth look at how connected educators are beginning to change their classroom practice. Ultimately, Why School? serves as a starting point for the important conversations around real school reforms that must ensue, offering a bold plan for rethinking how we teach our kids, and the consequences if we don’t.

Why must schools change how they teach? What’s at stake?
Schools were built upon the fundamental premise that teachers and knowledge and information were scarce. That is no longer the reality. Now, as so many more of us gain faster and broader access to the Web, all of those things are suddenly abundant. That means that the traditional role of school, to deliver an education, is quickly becoming less and less relevant. If we continue to see schools as the place where our children go to master a narrow list of content, knowledge and skills that were originally defined almost 150 years ago, we risk putting those kids out into the world with little idea of how to take advantage of the explosion of learning opportunities that now exist. The problem, however, is that most “reform” efforts are aimed at simply doing what we’ve been doing better, almost exclusively in the form of raising test scores. But doing “better” on measures that don’t account for this huge shift we’re in the midst of is the absolute wrong emphasis. Instead, we need to think very differently about the experiences, outcomes, skills and literacies we desire for our kids when they come to school.

Every generation seems to think its students are different. How are today’s youth different in terms of how they gather and absorb information?

Students in the K-12 system have never known a world without the Internet. No question, some kids have had more access than others, and that digital divide is something that we must address with more focus. But for the vast majority who have access, information and answers are a Google search away. They expect to use their technology to get their answers…except in school. In school, we ask them all sorts of questions that they could answer with their phones or laptops, but we don’t let them. So, I think the biggest difference is that our children are connected to people and to knowledge in ways that no other generation before them has been. We have not fully realized all of the ramifications of that, and in large measure, those who oversee our education systems have not yet begun to understand that this is a much different time for learning.

With so much information out there, it seems that finding information is easy but assessing it is tricky. How important are critical thinking skills?
Critical thinking skills around information have never been more important. For all of the value that comes with individuals being able to publish freely and widely to the Web, the huge potential downside is that we haven’t developed the literacies that are required to make sense of all that unedited content that’s out there now. In the scarce world, almost everything we consumed was edited or checked by someone else. Now, each one of us has to have the dispositions and skills to edit the world as it comes to us. Again, this is a huge problem for school systems that were designed for a different time, and it’s an even greater challenge since few if any assessments that we give kids ask them to make sense of an abundance of unedited media and information.

What can schools do to implement some of your ideas?
It’s a difficult moment for schools and the administrators and teachers who  in large measure care deeply about kids but haven’t fully understood or acclimated to this moment of abundance we find ourselves in. Most policy makers and businesspeople are focused on finding more and more efficiencies in the system, and they see technology as a way to “deliver” that traditional education to get “better” results needing fewer and fewer teachers while making greater and greater profits in the process. The next 10 years are going to be exceedingly difficult for schools to navigate the gap between maintaining the traditional curriculum that reformers want and providing the learning opportunities and literacies that kids desperately need today, opportunities that few outside of education are asking for. I think the first step is that educators have to reexamine their own learning practice and move toward becoming more networked and connected themselves. It’s hard to have meaningful conversations around change in a 21st Century sense if you’re coming at it from a 20th (or even 19th) Century lens.

The educational process is pretty slow-moving and sclerotic. Do you have hope that these changes will be made?
I have hope because I see more and more individual classrooms that are beginning to understand what abundance means, places where teachers and kids are getting connected, doing real, meaningful, beautiful work for real audiences that help students become true modern learners in the process. I have hope because every one of us knows that amazing relationships and amazing learning happens in those real life places we call school, that they are an important part of our communities and histories. And I have hope because, at the end of the day, just as we’ve seen with many other institutions, old thinking simply cannot prevail. This isn’t optional. The fact is that schools are not going to go away in the near term for a host of reasons. But what we do in schools, the way we answer the “Why School?” question will change. It has to. The more that each one of us begins to get involved in the process of answering that question, the sooner and more effectively we’ll make the changes our kids are waiting for us to make.

Why School? How Education Must Change When Learning and Information Are Everywhere is part of the TED Books series.

How long do you want to live, and why? These are the questions that bestselling author and science writer David Ewing Duncan asks in the new TED ebook, When I’m 164: The New Science of Radical Life Extension, and What Happens If It Succeeds. Duncan examines the increasingly legitimate science of radical life extension — from genetics and regeneration to machine solutions — and considers the costs and benefits of living to 164, or beyond. The book also considers the impact of extended life on cities, services, and the cost of living as well as what happens to love, curiosity, and general health. We recently spoke with Duncan to ask him a few questions about the ideas underlying the book.

In terms of life extension, what is reasonable to expect in the next few decades? How many years can we add to the average life?

To answer this, let’s first consider that human lifespan at least in the west has nearly doubled since the late nineteenth century, from under 40 years old to nearly 80 years old. This is due primarily to better hygiene and nutrition, but also to a more than a century of extraordinary advances in bioscience and medtech — everything from antibiotics and heart bypass surgery to new targeted drugs for cancer. According to the United Nations, within a century the average life expectancy in the west will jump to nearly 100 years.

Added to this steady upward tilt in aging is a range of new technologies – genetics, stem cell therapies that regenerate tissue, and bionics – that may provide an even bigger boost more quickly. How big a boost is open to debate, with serious scientists giving ranges from a few years to a few decades. When these lifespan-boosts will be available is another question that gets a wide range of answers, with some saying within 10 to 20 years and more saying more like 40-50 years. But few doubt that one of the new technologies or more will succeed.

What breakthroughs are most important to these developments?

The book describes four main areas — healthy living and predictive and preventive medicine; genetics; regeneration; and machine solutions.

Healthy living already has increased lifespans and prevented death for literally billions of people over the past 150 years, but we could still do more, especially to combat lifestyle conditions and diseases like obesity and diabetes, which prematurely kill millions of people a year. The science of predictive medicine is moving fast to help set up profiles for individuals and populations that can assess a person’s DNA and a wealth of other biomarkers that can be used to determine future risk factors that may, in many cases, be acted on to prevent or mitigate disease.

For genetics, mainstream scientists for 30 years have been studying and trying to better understand the process of aging at the genetic and cellular level, as well as in entire organisms. They have succeeded in manipulating genes and proteins that seem to regulate lifespan in worms, flies, mice and other critters — sometime upping lifespan by many times. More importantly perhaps, they slow the aging process by delaying or preventing diseases of aging like heart disease and diabetes. Several drug companies are developing drugs for conditions like diabetes and inflammation that activate enzymes linked to increased lifespans in mice and other animals, and may work in bumping up lifespans for humans, too. At least one of these pills, a compound that treats inflammation being tested by GlaxoSmithKline, is in Phase II human testing (out of 3 phases for FDA approved drugs). If successful, it could be on the market in under 5 years.

For regeneration, scientists have succeeded in using stem cells – the special cells that replace dying cells in different organs – to regrow or repair hearts, livers and other tissue in animals. They have some success in regenerating tissue in humans, but only for simple organs like a bladder or bone marrow. Using stem cells to regrow, say, cells in the heart or brain remain many years in the future, say scientists.

For machine solutions, humans have long fused machines or engineered devices and tools to their biology to improve or treat conditions or maladies. These include everything from eyeglasses to pacemakers and joint replacements. More recently, inventions and breakthroughs are already linking devices to the brain to help patients with Parkinson’s Disease control tremors and to help some people who are deaf to hear again. Other experimental machine-brain interfaces may soon allow the paralyzed to operate computers using thought.

In your research you discovered nearly universal rejection of immortality. Surprising; most people don’t want to live forever?

For the past 3 or 4 years I’ve been asking a question at the start of most of my talks: How long do you wan to live? I’ve kept track of the show of hands and over time have polled some 30,000 people. I asked people to vote on one of four answers: 80 years; 120 years; 150 years; or forever. It was a surprise to me that 60 percent of the people want to live the current average life expectancy in the West – about 80 years. Other results: 30 percent want to live to age 120; less than 10 percent to age 150; and less than one percent forever. I told people not to assume there was a scientific “fix” allowing them to live longer – although they were free to imagine there was such a fix – or not.

I encourage people to vote for their preferred lifespan on the book’s website: whenim164.com. A chart keeps the percentages voting for each “age preference” up to date.

You’ve done a lot of first-person reporting on life extension techniques. What’s the oddest test you ever took in your research?
The most intriguing was a test by the drug company Sirtris (now owned by GSK) that tested the levels of an enzyme inside me that is associated with longer lives and better health in a number of animals, and may work to extend lifespan in humans. Called SIRT1, this enzyme only works if activated – probably by times of stress when people can’t get enough to eat (this is called caloric restriction – one hypothesis is that SIRT1 is activated during famine to keep cells and organisms in good health to get them through the crisis so they can live to reproduce). It turns out that I “over-express” this enzyme by almost 1,000 times compared to others tested. This may explain the longevity of my family. It also might deliver a greater than average bump should I take the drug Sirtris is testing in human trials right now – which acts by activating SIRT1.

What are the upsides and downsides to living much longer? 

For the book I asked hundreds of people why they voted the way they did in the “how long do you want to live” survey. Here are the eight primary reasons people voted to live to the ages of 80 or 120, and not longer:

• Fear of prolonged frailty
• Money: how to pay for an extended life
• Life is hard
• Wars, plagues and poverty
• Overpopulation, resource depletion and the environment
• Love and relationships
• Boredom
• We would cease to be human

As for those respondents who said they want to live to 150 years or beyond, their stated reasons for wanting to live radically longer than current human life expectancy fell into five broad categories:

• More time with loved ones
• Geniuses would still be alive
• Want to know the future
• More to do and accomplish in life
• Avoiding the frailty of old age

As you can see, prolonging life can be both a blessing and a burden. But few doubt that we are at the cusp of an age that will see humans able to live much longer lives.

When I’m 164: The New Science of Radical Life Extension, and What Happens If It Succeeds is part of the TED Books series.

The massive layer of ice covering Greenland melted at a faster rate in July than at any other time in recorded history, with 97% of the entire ice sheet showing signs of significant thaw. The unprecedented rapid melting, which was captured in images taken from several satellites and released by NASA, alarmed scientists and deepened fears about the immediate consequences of climate change. Water once safely anchored in glacial ice is now surging into the sea at an alarming rate. Within a just a few decades, the flow could become a deluge, with millions of people living near coastlines in danger of inundation.

But scientists don’t yet know how fast this ice will melt, or how high our seas could rise as a result. In an effort to find out, a team of renowned and quirky geologists took a 4,000-mile road trip across Western Australia. They collected fossils and rocks from ancient shorelines and accumulated new evidence that ancient sea levels were frighteningly high during epochs when average global temperatures were barely higher than today.

In the new TED Book Deep Water: As Polar Ice Melts, Scientists Debate How High Our Oceans Will Rise, veteran environmental journalist, radio producer and documentary filmmaker Daniel Grossman explores the new and fascinating science — and scientists — of sea-level rise. His investigation turns up both startling and worrisome evidence that humans are upsetting a delicate natural equilibrium. If knocked off balance, it could hastily melt the planet’s ice and send sea level soaring.

We talked with Grossman about what inspired his new book.

The recent news of the Greenland ice melt is very troubling. Polar ice seems to be melting at an increasing rate. How worried should we be?

Very worried. There is good evidence that two of the three polar ice sheets — the Greenland and West Antarctic ice sheets — are beginning to shrink. While the amount they lose each year is small compared to the volume of the ocean, nobody knows how much the loss could accelerate in the future. Researchers of past climates have discovered natural incidents where sea level rose abruptly. Could the unprecedented warming and rate of warming of the 20th and 21st century catalyze a catastrophic collapse of the ice sheets? Nobody can say yet. According to some researchers I’ve spoken to, it could be that such a collapse is already under way and we just can’t see it yet.

Your book details the debate, even among scientists, of the rate of the melt. Why is it so hard to predict this precisely?

It is very difficult to figure out what’s happening with something as big as the Greenland or Antarctic ice sheets. If they were a country, the East and West Antarctic ice sheets would together be second in size only to Russia. They occupy the most forbidding and distant part of the planet. Also, despite the importance of studying polar ice, the scientific community is not provided with the resources it needs. Just to take one example: NASA doesn’t have a functioning icebreaker to clear a path for fuel and supplies to McMurdo Station, the largest and most important base in Antarctica. One of the two NASA breakers that used to service McMurdo has been mothballed. The other is dry-docked for repairs. Several years ago, I sailed on one of the world’s most powerful icebreakers, the Swedish ship Oden. The US has been leasing such ships for several years as a stopgap.

Scientists can observe what’s happening at the poles by satellite. But the satellite record only goes back several decades. Moreover, some important satellites have failed or retired early and haven’t been replaced. For instance, NASA’s ICESAT satellite monitored the ice sheets between 2003 and 2010, before failing. It had made more than a billion measurements of the elevation of the ice sheets. Due to budgetary restrictions, a replacement isn’t scheduled to be lofted until 2016. In the meantime, NASA flies a specially equipped aircraft in passes over Greenland and the Antarctic. (See Grossman’s  interview on Living on Earth about Operation Ice Bridge.)

Scientists try to use computer models of ice sheets, informed by estimates of future warming, to predict melting. But not enough is known about how ice sheets behave to anticipate with certainty what will happen in the future. Today’s warming is like a global experimental trial that will only be run once. Some scientists believe that the best way to anticipate the results is to look back in time to see how Earth’s various parts responded to warming in the past. The approach makes sense, but it’s imperfect

Why is there so much difference in opinion amongst the scientists in this field?

Trying to understand what will happen to the ice sheets in the future is a difficult, cutting-edge scientific and technical challenge. As with any question on the ragged edge of scientific understanding, there are a variety of opinions and hypotheses. In contrast to many areas of scientific inquiry, the answer to this question is of grave consequence to the United States and the rest of the world. Unfortunately, the research is not funded as if it were an urgent national priority rather than simply an interesting curiosity. If Americans appreciated the seriousness of the potential that polar ice sheets could melt dramatically in the future, I believe they would demand that appropriate scientific studies be given higher priority.

Is the game already over? Can we slow polar ice melt?

The most worrisome indication that we might have started circling the drain concerns not ice sheets, but the floating sea ice in the Arctic. The sea ice there responds faster than glaciers to warming conditions, because it is never more than about 15 feet thick, and generally only 3 to 6 feet thick. Each year the sea ice expands to fill most of the Arctic Ocean in the winter and then contracts to about half its area in the summer. (Here’s a cool NASA visualization of the annual process.) The minimum size of the ice cap has been declining precipitously in the last 40 years. The rate increased substantially around 2000. In 2007, the minimum size reached the smallest ever observed, 37 percent smaller than the average size between 1979 and 2000. The size of the Arctic sea ice has no impact on sea level, since it is floating. (For the same reason, a full glass of soda doesn’t overflow when its ice cubes melt.) But brilliant white ice reflects much more solar energy than dark ocean water. Researchers say it is likely that within a matter of decades the Arctic will be ice-free in summers. That will vastly increase the amount of heat absorbed in the Arctic. The Greenland Ice Sheet nearby will warm faster, accelerating its loss of ice to the sea.

For all anyone knows, the insulating blanket of carbon dioxide that humans beings have spewed into the air has already set in motion an unstoppable process that will unleash the polar ice sheets, causing catastrophic sea level rise. But we might not know for decades. In the meantime, the sensible course of action would be to act as if it is not too late do do anything.

Your new book is not only a scientific story, it’s an adventure story. What’s the strangest place you’ve ever been while reporting on climate change?

I’ve been to many unusual places. In 2009, I reported on concern about decreased rainfall in parts of South Africa and Mozambique. Among the places I visited was the small kingdom of the Balobedu people of South Africa. There the Rain Queen, or Modjadji, is supposed to bless the region with abundant precipitation in otherwise-dry Limpopo Province. The most recent queen had recently died and nobody had yet succeeded her. However, an ancient attendant received me in a royal courtyard, a circle of heavily trodden earth surrounded by a tall stick fence. In Mozambique I visited a bat guano mine, which a project of CARE was reviving as a way to help increase agricultural productivity.

My most fun trip I’ve taken was to Palmer Station, the smallest US base in Antarctica, on the Antarctic Peninsula. I spent a month there with a small crew of quirky researchers and support staff. While the scientific research there was serious — I reported on research showing that the dramatic decline of Adélie penguins was caused by melting sea ice — the base sometimes reminded me of a summer camp for adults. The inhabitants organized activities like tie-dying to keep from going stir crazy. They played practical jokes on each other and sometimes jumped into their inflatable Zodiac boats to get closeup looks at whales that happened by.

Deep Water: As Polar Ice Melts, Scientists Debate How High Our Oceans Will Rise is part of the TED Books series.