1000 Tuesdays

I joined BT Labs on 1st December 1997, and I'll be leaving on Friday 29th November 2019. That's 22 years, +/- a weekend. Yesterday I had to deliver the last major outcomes of the work that I've been doing, the half year reviews for the AI program, and I was thinking about other Tuesdays at the labs - and it struck me that there had been more that 1000 of them. That's a staggering number, it doesn't feel right, but arithmetic tells me it is, so I've been made me reflect on what's happened with all those Tuesdays to date.

The first few hundred Tuesdays at BT Labs were almost certainly spent elbow deep in Java code writing Intelligent Agents. Initially I worked on a project called AEW - Agent Enhanced Workflow, we were trying to develop applications of the kinds of intelligent agents that were developed in the award winning ADEPT project that ran at BT before I joined. The clue was in the name - the feedback that the team had received was that it was unrealistic to implement novel middleware, but instead an overlay providing enhanced functionality was what was needed. We worked to implement this for a workflow management system that ran correspondence and billing services for BT, the intelligence layer used machine learning (a support vector machine) to predict demand and to smooth the work provisioning requests between the correspondence handling queues - basically it was an intelligent anti-bullwhip system.

That project matured and moved on and I worked on SAMBA - which aimed to integrate workflow and personal agent functionality (booking sequences of appointments and tasks into people diaries and co-ordinating teams), and then JITABUG which in hindsight was an unbelievably visionary project which aimed to provide an enterprise management system using agent based simulation - think https://en.wikipedia.org/wiki/Project_Cybersyn but without the Starship Enterprise chairs and with an agent based simulator of BT under the hood. Of course, we couldn't make it work!

At that time the Zeus team at BT had developed a toolkit for building intelligent agents, and everyone else was being encouraged to use it in their projects. I got involved by trying to use it and by writing some demos to go into the distribution. Zeus was the first open source project BT had done (as far as I know) and got released on sourceforge in about the year 2000. The team got headhunted in the dot.com bubble and I was left as the only person who had experience of writing the code in the core of the project (at least, the only person who was still around who would admit this) so I became the maintainer. Working on Agents and Zeus led to a fruitful collaboration withe University of Southampton where I was given a visiting fellowship, and this led to some brilliant joint projects with Cardiff and Aberdeen developing agents with notions of trust and sophisticated coalition formation capabilities. Around this time BT won a bid as a partner in an project called Agentcities, and my job became to provide an interoperation layer within Zeus to the FIPA agent communication standards to support the heterogeneous Agent Platform that Agentcities wanted to build.

During Agentcities I concluded that Intelligent Agents as we thought of them in the 2000's were a bust. The problem was that KQML and FIPA (the communication languages) were based on speech-acts and modal logics, which meant that they were "competent" in some approximation of a theoretical sense. When we implemented real systems and then incentivized developers to use them (I ran a project in Agentcities to give out grants to developers across Europe to develop services for the platform, so we paid them €10k to build something like a museum guide or city guide service using JADE or FIPA-OS or Zeus) the result was that the developers simply did not (could not?) use the communication layer as intended and the ones that tried found that programming an intelligent agent to program (cause to act) another agent by proxy what just too complex. Another fundamental challenge was that most of the agent intelligence I developed relied (at its heart) on AI Planning, and at the time the SAT based planners that could make this kind of reasoner feasible were in their infancy. Luckily I was a machine learning guy and there were a bunch of other problems at BT that needed some ML skills, so I refocused on that. You can still find an opensource version of Zeus if you are interested though (https://github.com/sgt101/zeus)

At some point I got invited to take part in a technical due diligence visit to SRI to validate a deal on mobile technologies that was under discussion. While I was there I was very lucky to get to speak with the Agent team there and to hear about their proposed CALO project. BT couldn't participate because of DARPA rules, but I was asked about what the time line to business value/production of the technology was - I said 10 years or so, but in fact the project turned into SIRI and came to market 7 years later. So - I was wrong, but also kind of right.

A by-product of working on intelligent middleware was that I got involved in the world of web-services. We tried to use WSDL and BPEL and a host of other XML standards to enable access to services, and built composition systems using AI planning based on preconditions and postconditions published in UDDI. This effort was hampered by the staggering complexity of the webservice standards landscape and the fundamental difficulty of dealing with semantics - as discovered by the rest of the Semantic Web community. The good thing for me was that because I was an "expert" on web service standards I got seconded into the Web-21C project at BT which was a massive mainstream project to create an all new top to bottom service system. This gave me a big opportunity understand the engineering and networks side of the business, and established a raft of contacts that served me well for many years.

Web-21C was doomed from the beginning - I should have known it when I saw the first presentation. The fella in charge drew a cloud, which he labelled "web services" and then drew a set of arrows labelled "problems" and one big arrow to a box that contained the word "solution" and described this as the project in a nutshell. The fundamental web service that BT was able to produce was voice calling services; eventually the beta test user community discovered that they could algorithmically arbitrage the web-21c API with other voice API's, and a very significant bill arrived which effectively ended the effort, but by then I was working on some new projects.

So I guess that the first 500 Tuesdays or so were devoted to intelligent agents in some way or other.

My refocusing led me to get involved in the world of customer support communities. These were a big opportunity for providing service to customers cheaply, and also a big bane because customers are very vocal in their complaints - putting other people off. So we developed a series of solutions that provided smart interfaces to these communities by indexing and filtering the customer issues to get the best people in the BT support eco-system to respond in the most appropriate and effective ways as fast as possible. This meant better outcomes for the customers, more information available to help them, and a better image for BT's services. We used simple NLP and clustering techniques to achieve these outcomes, and we started to have to handle a lot of data.

The handing a lot of data created another interest for me - Big Data. Our sad little mySQL box (later Oracle to meet enterprise architecture requirements - which created another side interest and skill for me..) was fine for most of the operational queries and updates that were needed to run a contact system, although when we implemented into production with the brave folks at the Enniskillen contact centre we had to do a lot of hard work to make it efficient enough to provide a good experience. But when we wanted to do queries that required many table scans - for example to create graphs of related posts or matrices of similar terms to support recommendations we were totally stuffed. So, Hadoop became an interesting technology. Other people in the Labs were looking at it as well and this led to first, a pitch to the Director of Research for funding for a research cluster, and then after a proof of value project a joint pitch with Engineering and Architecture for a BAU cluster and a transformation program to drive a data technology and architecture change pan-BT.

On the other side of that effort I took over the Customer Experience Practice in BT Labs, I rebranded it as Big Data and Customer Experience, but the big change for me was that I had to get my arms around the other groups working on CX in BT Labs. The two new communities were the qualitative/social science researchers and the folks working on access network diagnostics. Taking over a larger organisation was hugely enriching for me, I'd run a team for a decade, but running teams of teams is a different thing. Teams of teams mean providing strategic input and delivering support and enablers. Rarely you have to get tactical to get something done, but life proceeds well when you support your organisation to deliver and make sure that it is delivering in the right direction. It also means representing them and advocating and advertising their work. You've got to be an ambassador for your organisation, and understanding the breadth of the work that is going on. It also means that you have to make changes and manage the evolution of the organisation, rather than just evolving yourself.

The implementation of VDSL into the BT broadband network created a huge challenge, the fault rate shot up as a more complex and technically demanding product (it leverages higher frequencies to transmit more information and is therefore natively less tolerant of faults, and initially BT did not use vectoring to reduce sensitivity to noise). This challenge and the need to build national super and ultra-fast networks produced a raft of high profile challenges and opportunities for the use of big data, AI and ML in the company. The breakthrough in ML technology in 2011(ish) and subsequently created a big opportunity to do more with ML and big data. and eventually that enabled a resurrection of a separate program on AI research in BT that I was asked to lead. We've been able to forge strong links to work at MIT in the Systems that Learn initiative and into the CISR team in Sloan as well, and to develop and deepen links into UK academia as well. The focus has been on applying AI into BT's core business, with projects that look at managing the flows of energy and money in the company as well as the flows of bits, and the new opportunities of radio networks but a core of novel technology work is growing rapidly. This year has seen the development of work on deep reinforcement learning, adversarial networks and graph networks as well as a thriving strand of work developing AI standards and management and governance infrastructures.

So, overall that's how I've spent 1000 Tuesdays, but the thing that strikes me more than anything else is not the details of the projects, the patents, the conferences or the papers, but instead the people. I've had the luck to spend all that time working with some remarkable people, I can't list them, but they were the font of the work and ideas and outcomes that I've listed, and the root of the happiness that I've had during my time with BT. I've travelled the world, had experiences and interactions that I had no right to, met incredible people who've lived lives totally different from mine, worked with organisations of all shapes and sizes and learned things that I'm pretty sure that no one else knows.

I would not be the person that I am today if I hadn't joined BT. I would not be able to do the things I can do, and I would think totally differently. Big companies are often dismissed by commentators as less glamorous or somehow less likely to provide a great career. BT has a real purpose, and massive opportunities and resources for anyone who wants to lean into it.

It's been great, but 1000 Tuesdays is a few Tuesdays too many, and if I stay any longer I don't think I'll ever be able to leave and be happy afterwards, and leave you must - in the end. Which is why when I got the offer of a new job, building a new organisation and working in new sectors I decided that it would be good to make the jump on my terms.

So, this Tuesday was the 1000th and last, and I start a new job on Monday. I reckon I've got at least another 1000 Tuesdays in me, I'll probably write something about them as well at some point, perhaps it'll be shorter, but perhaps not! I'll probably know more after the first one.