Tech Forecast 2017

I have been writing these forecasts, largely with an emphasis on the data space, since the late 1990s. I do these less to proclaim my own prognostication scores (which run about 70% or so - not bad but not great) and more as an exercise to determine where I personally will be focusing on.

In the last eighteen years, I have never found a year that was as filled with uncertainty as 2016 has been (2000 was perhaps the next worst, followed by 2007). These are inflection points in the economy, where trends usually break. For 2017, the political climate will have an impact in a way that is unprecedented on IT, and likely not for the good. However, this has been the season for unexpected results.

As is typically the case, I will start with my own landscape of data science and expand outward from there.

#1. Semantics/Graph Goes Mainstream

I started covering semantics, data modeling and ontologies about 2011, and I have watched it go from a cultural IT backwater to become one of the most significant arenas of enterprise data management, especially as more and more companies and organizations struggle with Big Data initiatives and find the issue of managing conceptual data dissonance to be a far harder problem than settting up a Haddoop cluster or data lake. Master data management has generally shifted to a semantic representation, management of controlled vocabularies and knowledge systems are going that way, and most taxonomy management tools are now semantic under the hood (with rights management and provenance rapidly going that way).

There are several Spark/RDF Triple Store hybrid efforts underway, as the need for intelligent and flexible data modeling has become more and more obvious to most data architects. I suspect the next major breakout player in the semantic space will be doing it on a Spark stack probably on GPUs optimized for graph operations. Indeed, graph databases in general are becoming the smart tech play, with graph database backgrounds outweighing JSON/BSON based structured NoSQL stores backgrounds as required for people playing in the Machine Learning space.

While on the topic, Machine Learning skills are replacing demand for data scientists with analytics backgrounds as the most sought after developers. Not surprisingly, hybrid flexible graph stores are the foundation of such systems and these types of systems, capable of building and storing inferences and determining categorizations (this is a car, this is a bridge, this is a person) are increasingly found in everything from medical diagnostic systems to self-driving vehicles.

Domain specific knowledge in major ontologies is becoming a core proficiency for business analysts. Every bank that I've talked to are exploring FIBO - the financial businesses ontology, health care, health insurance and life-sciences are slowly lurching towards consolidated ontologies, and the automation of both data governance and data provenance management seems to be the hot areas of discussion into 2017. Finally, I've been involved in three different projects where open annotations built upon a semantic basis (the W3C Web Annotations specification) formed a critical part of that, and have flagged this area in particular as one I think will see interest grow dramatically in the next year.

#2. Hadoop is dead, long live Spark

No, Hadoop is not really dead, but it has clearly lost much of its luster. Too many data lakes were created that didn't really seem to have clearly defined goals, and instead became expensive boondoggles, In many respects this was foreseeable - Hadoop started out as a way of doing distributed map/reduce, and in that role met some very critical needs, including the core one of building and maintaining data indexes.

However, as massive investment money poured into Hadoop, everyone saw databases as being the most clearly monetizable aspect of the language, despite the fact that for Hadoop to take that role meant that all of the things that most modern databases (relational and non-) took as a matter of course could not easily fit into the Hadoop paradigm - ACID transactional support, role based security, multi-tenancy, data flexibility and so forth. These features were jettisoned (or never developed in the first case) in favor of scaling, but the reality was that data scaling has long been a myth that database vendors have used to sell multiple licenses, and many Hadoop projects that were launched could have been done just as effectively (and at lesser cost) with more traditional data systems.

Spark is a rethink of the database from the ground up, rather than a data management system built on a map/reduce system. Spark containers are generalized data stores, and are increasingly seen as data virtualization environments. Many of the features that were bolted onto Hadoop are now incorporated natively in Spark, and as flexible data storage (not just relational databases) become scalable and cloud-based, they scale well with the rise of virtualized computing in all dimensions.

#3. Javascript ES6 Matures While NoSQL Splinters ...

Much of the news in the Javascript space in 2016 has been the continued rollout of EcmaScript 6 features in various environments. ES6 is now supported natively in Firefox, Chrome, Node, Edge and the newcomer in the browser space, Vivaldi, as well as platforms such as MarkLogic's server 9.0 Early Access and Amazon's Lambda stack (with Node 4.3.2 supported - see more on Lambda below).

This has resulted in a shift through the last year, with Babel support for ES6 and Livescript giving way to much faster native code. The use of classes, arrow functions, template strings, generators and the like is also changing the way that code is developed, one consequence being the profusion of libraries and frameworks in the Javascript space is giving way to smaller native applications that dramatically reduce dependencies and complex Javascript bloat.

Coupled with changes in CSS 3 that are finally becoming consistent across all platforms without the use of polyfills, the widespread adoption of SVG on most browser spaces, and the growing reliance on immutable data structures, web client design and development seems to be moving away from framework hell even as Javascript solidifies its place as a full stack language.

The one area that I think is overdue in the arena is the realization on the part of ECMA that a common query language for JSON stores is sorely needed. There are several different candidates for performing queries against large stores, with JSONiq being a personal favorite, but including MarkLogic's JSONiq-like query language, N1QL by CouchDB, Amazon's DynamoDB QL, the semantics friendly GraphQL, Mongo's Query Language and others. Each have their strengths and their weaknesses, but the biggest problem that the NoSQL space faces right now is that there is no consistent standard for queries, meaning that the market is fragmented as badly as relational databases were in the 1980s before SQL became standardized. My suspicion is that consolidation in this space (which has become crowded) and the demands of enterprise customers for consistent standard APIs will eventually force a standardization attempt within 2017 or 2018. Until then, this will weigh down the NoSQL category dramatically.

#4 ... and Amazon Goes Lambda

This one wasn't on my radar until recently, but I believe its going to have a revolutionary aspect upon application development. Amazon recently debuted what they are calling the first "serverless" application. Yes, there's still a server on the back end, but with Amazon Lambda, you don't even need to know abut the server. You simply write code (Javascript, Java and Python at this stage, with more coming), work with a Lambda API and a back end dataset of a specific type, and specify your output channels. No complex provisioning or scaling code needed, where you pay only for processing time and storage used.

This to me looks very much like the future of computing. You can put Lambda on a mobile device (a phone, watch, or router), on a desktop, on a server farm. You reduce your operations to pipelines of operation. The operating systems are still there, but the distinctions of "tiers" disappears. DevOps requirements drop dramatically. It's my expectation that others (most notably Microsoft with Azure and Google) will follow suit - most of their offerings have been moving in this direction for a while, if not necessarily to this level of transparency.

#5. Data Scientist In A Box

One of the most notable employment occurrences in the last few years has been the rise of the data scientist - more properly, the data analyst - within the IT umbrella. This very quickly gave rise to data science teams, usually consisting of a person concerned with data quality, a data synthesist (typically a Hadoop developer), a data analyst who did the statistical analysis, a data visualizer, and a data manager.

Not surprisingly, when you suddenly need at least six people to perform seemingly critical functions, the desire to automate much of that on the part of the bean counters grows dramatically. This is beginning to happen now as software vendors spot a market opportunity, and attempt to apply a Parieto 80/20 division - end to end data analytics solutions that let companies do the 80 percent then rely upon technical expertise to get that remaining twenty percent expertise, slowly phasing that out over time with Machine Learning systems. My suspicion is that by 2020, data scientist will end up going the route of the desktop content specialist before most of those jobs rolled up in software.

One area that I see potentially ripe for automation is a generalized charts API model. Most charts and graphs follow consistent patterns - labeling of axes, identifying data framesets, legends and so forth. A consistent data visualization model can reduce the headaches of data configuration, and would make interchange of charts - one of the most common types of information graphics - feasible. Again, I think this will happen over the next three to four years (look to Seattle based Tableau as a possible source for that).

#6. IoT Falls Apart, Drones Emerge

There for awhile, you couldn't get through a day on LinkedIn without someone posting an article on The Internet of Things article. Yet as I predicted, there are signs that the IoT market is stuttering, and may collapse altogether. The reason for this is simple - you cannot have an Internet of Things without an HTML of Things. I contend now (as I have repeatedly) that a true IoT will not emerge until the industry standardizes, and so far there are no signs that any one standard has emerged as truly dominant.

This means standardization of security systems, authentication, generalized interface function and so forth. The closest I see to this is in the consumer electronics division, where you can now use a smartPhone as a universal remote, but even there you have too much configuration necessary for all but the most hard-core entertainment buff to take the time to get all the configurations worked out correctly.

The one area where I think standardization may actually happen more quickly is in the drone arena. Most drones have 85% of the same functionality, involved primarily with flight or mobility, which reduces the complexity of establishing a standard. The remaining 15% I think will take care of itself when a drone manufacturer figures out that they can create a cradle for a smart-phone or two on the chassis of the drone itself which connects through the USB port or Bluetooth (this may already have occurred to some enterprising individual). The smartphone contains the sensor bundle, transmitters and processors, the drone the motive power, secondary processors and additional peripherals.

Once this happens, you replace specialized hardware with a software emulation layer run as a background app, making standardization in this space much easier, as you'll need to abstract this information anyway to communicate with various smartphone models.

Standardization of drones will also need to involve standardization of airspace networking. One problem that has emerged in the last couple of years with drones is their inability to communicate with one another in order to orchestrate actions - such as the competition between news drones and emergency response zones in wildfires or hurricane zones. This is an issue that the FAA or similar agency will need to address sooner rather than later.

On the topic of drones - this year has been the year where drones have really emerged to a level that they affect social discourse. In July, Army veteran Micah Johnson, who had shot several bystanders, was killed by a wheeled drone that exploded when it got close enough to him, after a stand-off with the shooter failed. This was the first documented case where a weaponized drone was used on US soil to cause the death of an American citizen.

Then, in December, the Chinese captured a US drone submarine in international waters, both creating an embarassing incident and raising awareness of the possibility of drone submarines armed with nuclear missiles and launch systems being emplaced within coastal areas globally. Unlike manned submarines, they could remain quiet and nearly dark indefinitely, would be almost impossible to track except by accident (reports indicate this was originally picked up by a fishing trawler), and would be far cheaper and faster to build and deploy.

#7. Autonomous Vehicles Gain Traction

An autonomous, or self-driving, automotive vehicle (AAV) is a drone with a large scale chassis and a more powerful AI. In 2016, many of these went from interesting concept to field test, and by the end of 2017 you should see the first wide scale commercial production of these vehicles. In all likelihood, the first recipients of these cars will be delivery services and rental fleets, as they solve the fundamental problem of getting cars to the people who need them (and get them back) that has always been a factor keeping rental companies from becoming more than an ancillary industry. Not surprisingly Uber has been working its way into a middleman relationship, to the extent that they may end up phasing out their driver services by the early 2020s.

Tesla's run with autonomous vehicle technology hit a pothole when a driver using the AAV was killed when the system became confused by a large truck passing by that confused the spatial recognition system. Even given that, the accident rate for Tesla (and Google with their own efforts) is still well below that for human drivers.

Uber has been doing the same in the trucking industry with its Otto spin-off, which in October delivered its first load (beer). Currently, the trucking industry is claiming that such systems are intended to address a shortfall of about 40,000 drivers, though truck driver associations are claiming that the long term goal is to eliminate (or at least greatly reduce) the number of drivers required dramatically.

Additionally, in July 2015, a semi-autonomous Jeep was hacked and crashed, highlighting the need for additional security, something that is a critical problem now in the data space. Again, this supports the previous made contention that consistent (and proven) security measures and standardization of these will accelerate the overall standardization of drones, including AAVs.

#8. The Coming Blockchain/Fintech Revolution

Bitcoin was the major story of 2015, but its underlying technology - blockchain - is becoming the bigger story for 2017. Blockchain can be thought of as "ledger-tech", a distributed block system containing transactional information that both provides replication and surety of provenance. Not surprisingly, a number of larger banks, brokerage services and currency forex players are investing heavily in blockchain skunkworks projects right now, which should start seeing quiet rollout in 2017 and beyond.

Chances are pretty good that blockchains will be largely invisible - and that's the whole idea. Blockchains provide surety in the back-end of financial services, as well as any place where unique identifiers are used to perform transactions uniquely and identifiably while potentially incorporating a layer of anonymity. Not surprisingly, government regulatory agencies worldwide are also studying blockchain technology carefully, as it has the potential to create black money pools that are completely opaque to taxing authorities and criminal/terrorism investigation agencies.

Beyond this, one of the other major innovations in Fintech is the growing use of Legal Entity Identifiers (LEIs). LEIs are unique identifiers for specific corporate entities and financial vehicles. Managing LEIs make it possible to identify fraudulent business activity or unravel deep shell companies intended to hide criminal activity, but they also help lending institutions determine the ownership of collateral assets both to determine net worth and to resolve disputes about ownership of orphaned assets (such as mortgages that were locked into ownership limbo in the aftermath of the housing meltdown of 2008).

Finally, the bank branch office is also fading as online bank transactions continue subsuming more roles. Many banks are now allowing the scanning or photographing of checks for use with OCR systems via phone cameras. Longer term, these and similar innovations is also making it easier for exclusively online banks to function, to the extent that many larger banks are now moving out of the commercial banking sector altogether. Ironically, as interest rates may finally be starting to climb again, this could lead to both a credit crisis and an opportunity for virtual banks to establish themselves in the near future.

#9. 3D Printing Gets Real (and Biological)

A growing shift away from globalization in many parts of the world will likely prove a windfall for another emerging technology - 3D printing. Companies that benefitted from telecommunications improvements in the late 1990s and 2000s in order to outsource both work and production are now in a situation where they are in danger of losing control over their core competencies as expertise goes elsewhere - especially for large companies with extended supply chains.

The rise of additive 3D printing is making it possible to create many low-volume items without the need to add significant production lines, which, coupled with smarter automatons, is causing companies to shift production back to more local facilities. Note that these are not adding significant new jobs (I'll be writing a tech impact upon society piece as a conjunction to this report that covers these issues in more detail) but are reducing the amount of wastage that is now generated using traditional production techniques.

Ironically, the biggest beneficiaries in the 3D space currently are in the field of medical prostheses and organ replacement. Bone and cartilage can be scanned and then recreated in 3D with hard plastic, ceramic or enhanced ceramic/metallic mixes, fit specifically to the bone shape in question. Additionally, trials with mice have shown how to seed cow bone tissue with stem cells taken from the test animal to create a replaceable scaffolding. This not only makes it possible to recreate the bone, but over time the bone begins to regain marrow functionality. These will likely enter human trials next year.

Similar activity is happening with softer organs, and by 2018 you should see the implanting of the first human 3D scaffolded heart. This gets around the rejection problems that many transplants face. It's also spurring the development of test tube beef, grown in a vat rather than taken from cows. By the mid-2020s grown organ replacement should be routine, and it's likely that you will start seeing cosmetic reconstructions (double-jointedness, elongated flippers or reduced feet, second thumbs and so forth) by the late 2020s.

This is also occurring in conjunction with cybernetic (or bionic) interfaces, with a lot of that work being done at the laboratory stage in 2016 (papers here and here). These experiments are intended to build a high capacity communication channel between neurochemical systems (such as the brain/muscle combinations) and electronic prosthetics.

Indeed, the combination of servo-motors / actuators and intelligent machine/learning systems that can be embedded within these directly mirrors what is happening in the drone space. It's very likely that this will be the dominant trend throughout at least the next decade.

#10. Energy Interfaces

The last half decade has seen a radical explosion in the development of alternative fuel sources, especially in solar photo-voltaic (SPV) cells, wind turbines and storage batteries. The total cost of producing energy from solar in particular has dropped dramatically, to the extent that it is now cost-competitive (TCO) with oil/gasoline production.

Perhaps the most intriguing work here has been done by Elon Musk. Late in 2016, his company rolled out roofing tiles with embedded SPVs built into the substrate, solving an oft unstated but common problem with SPVs - they provided poor protection from the elements and were visually unattractive. Demand for the new tiles, by all indications, is high and growing.

A second Musk piece has been the introduction of wall storage batteries - Lithium ion based batteries (produced at Musk's Gigafactory) capable of storing energy for long periods of time that can be inserted in the air space between walls. By themselves, on the grid, they can provide power in the event of an outage, but when combined with PV they essentially take the house of the grid altogether, even making such houses net producers of energy.

A recent referendum in Florida launched by power utilities to discourage solar usage (while appearing to promote it) was defeated at the polls in November. The referendum was brought primarily to deal with the fact that as people move off the grid, demand for electricity has dropped, which in turn has reduced profits for these utilities (many of them privatized during the deregulation efforts from 1980 on). As solar gets mixed with storage cells, the inherent costs should drop even more, even with a likely attempt at defunding much of this by the Trump administration.

Turbines have also been improved, both from a performance standpoint and from a design standpoint. One factor that has raised environmental concerns is the degree to which many older turbines are impacting (in some cases literally) bird populations in the region. A second comes from the overall size of the blades. Recent innovations in this space have led to bladeless turbines that, without the need for large circulating blades, cut down both the cost and the footprint of such turbines. Again, storage cells help complete this, as power can be produced asynchronously and stored, then transported to sites that need backup power in particular.

You're seeing similar arrangements being utilized in latest generation electric cars. By mounting thin PVs on the roofs of cars, the cars can self-charge either from the sun, from converted rotational to magnetic energy in the flywheel or from charging stations. It's likely that specialized thin-film PVs may be mounted on windows or similar surfaces by early 2018, letting windows become augmentative power sources.

On To Social Tech

This is a survey of primary technical trends that I see having the greatest immediate impact in 2017. I've tried as much as possible to keep the impact of these technical trends separate from the trends themselves, and will be exploring that in a follow-on report, looking at how the above trends are impacting society. Look for it soon.

Kurt Cagle is the founder of Semantical LLC, and has been writing about technology and societal impacts of same since the late 1990s.