Session Information:

Managing the AI process: putting humans (back) in the loop

AI today is all over the news, yet it has not transformed the workplace as promised. We will argue that the reason is that we forgot humans, both as developers of AI (lacking organizational tools in AI development) and as consumers of AI (lacking transparent interfaces).

I will show how we deal with these issues at Huawei’s Noah’s Ark lab where we (AI researchers) work closely with systems engineers (our AI consumers and collaborators).

Key takeaways:

1. The AI development ecosystem, actors, hurdles, work processes.
2. Our long term goal: autopilots for engineering systems.
3. Case study: how human and organizational constraints determine the research and development in our lab.

Session Transcript:

AI leader, in mind, Chief Data Scientist, for Y Way, For allies, Tagle with us, but ...

is a machine learning research veteran, with over 25 years of experience, within that, an impressive publication list, both in AI and in physics. After his PHD, he worked in Canada as a professor.

Then in France, as an AI researchers, in a physics lab, where he gained experience in applying AI to other disciplines, he got interested in the human and organizational aspects of AI. When directing the ... Center for Data Science, between 2014 and 2019.

He crossed from academia to industry two years ago, and today he leads a team of 15 researchers at Weiwei Spot, Paris, Noah's Ark, AI, Research Lab.

So ..., what a pleasure to have you with us directly from France to the world. We're very, very excited to have you here and a very much look forward. Looking forward to your presentation.

OK, thank you and thank you for inviting me.

So I just switched on the slides, which helped us to be.

It looks good.

OK, so thank you, everyone, for having me, and thank you for inviting for this talk.

As I said, I'm a, I'm originally an AI researcher, so I thought that I could give you a view of all this automation within businesses. From the researchers point of view of the data scientists once. So, who am I? And so they said, I have been doing research for 25 years, on AI started.

When I was seventy five in ninety five, the community was like, 500 people in the world. Now, we are like 500,000.

So it's the big, big change in the last 10 years.

I crossed over to industrial from academic research recently, but even before that I was at ..., the French Public Research Institute.

I became interested in them in the human processes behind AI because I was the Director of the Center for Data Science or our mission was to connect to AI technologies with other Sciences, so put machine learning pipelines into astrophysics, economy, chemistry, or a lot of other domains.

And I figured out that most of the time the problem was not technological but organizational.

At the same time we started to consult Say I saw that in industry the problems are very similar to what we had in academic research. The more the bottlenecks were, organizational management.

So in 20 19, I decided to join the hallway because I wanted to get my hands dirty and try all the things that I discovered during those five years in the real industrial setting.

So, right now, I'm, I'm leading a team of 50 researchers in hallways ..., because the AI level four way, and I'm the head of the group in Paris.

So it's essentially, research scientists, research engineers, PHD students. And we are partly doing AI research, but we are facing BU, so we are solving the problems for BU.

So I'm getting into this interface with the We use.

The main research objective we have is to build AI L two pilots for engineering systems.

So my group is on the network boo of Huawei.

So we are not working on the phone, which means that most of the things we deal with our engineering systems. This can be software that runs the mobile network in the mobile antenna. This can be centers. That's where we are controlling the cooling system or it can even be the Wi-Fi network. So there are parameters to set. So the connections are the best.

And we want to do AI for improving these systems, making them better, cheaper, more reliable, safer, or more energy efficient, which is a big part of what do you do?

If so, I will basically talk about how to put AI in engineering systems with the digital tour, into more general question of how to organize the data science ecosystem in the big company.

So I will be going back and forth between these two sections.

So first, a lot of you might have known this big cruise out in 20 17. I think we're off a go.

So Google DeepMind Design AlphaGo, which beat the Lisa, though, was that were champions of goal.

That happened actually 20 years after Kasparov was beaten by Deep Blue.

And it was a short period of time, actually, the, what are the predictions, said that Google will be sold very, very late.

And the reason it was soft so fast is because we figured out how to put AI into the system, so deep, blue, the Chess champion did not use any machine learning tools, the rule based system. And AlphaGo was completely designed by learning, because learning from gave them was learning from playing against itself.

This was a big breakthrough, because it showed the first big results on a sub area of artificial intelligence, which is called reinforcement learning, it's essentially putting AI into dynamic systems, putting AI into control.

So, the natural question you can ask, OK, if we can solve such a complex problem, is go, why aren't these advances are already in engineering disciplines?

Whereas, engineering systems, you could imagine them as less, much less complicated game. The AI systems should play.

The question is even more interesting give him that everything is engineering systems. It's it's 10 trillions of dollars business per year. So, if we can just improve 1% of the efficiency of the system, that's huge money, so there should be a big incentive to improve the systems, yet, we don't really see these algorithms in engine steps. So, why?

So, let's have, like, an abstract. You want an engineering system.

Uh, you have a system that produces system states, observables that are observed by the engineer who is, controlling the system the pilots.

So, you can imagine that play as the system and the pilot as engineer.

The pilot also observes the performance indicators, like speed, how far am I from the, where I'm going or how high my sword indicators that tell you how well, I'm doing my job of controlling the system.

As an engineer, I'm also tuning some of the parameters time to time. And the goal is to optimize the performance indicators, or at least, keep them with the limit.

OK, so this I want you to have this abstract view of an Nginx system in mind, for the rest of the talk.

Yeah.

So we have a lot of automation already.

If, but if they exist, they are based on deep understanding of the physics of the system, not on learning the system from data, sometimes it goes wrong. So this is a system that you might know.

It's the control room of chair will do just before explore you.

There are a lot of automation in the system, and there are human operators.

And you see that even when there are human operators who make decisions, things can go very, very wrong. But most of the time, it works.

Like a lot of manufacturing?

Pipelines are automated.

It just does what the automation is based on knowing the system and programming the system. So it does it automatically, without human interaction, but it doesn't learn even mistakes or doesn't learn in terms of getting better at what it's doing.

So what is artificial intelligence in this context? Because you can talk about AI in, in a lot of different domains, but in controlling engineering systems, what you want to do with AI.

You learn the system behavior based on historical data, and they use this knowledge for improving the control performance indicators.

So this was the introduction of what engineering systems, what do you want to do with AI engine systems.

So now let us go in a little detour, into more general subject, which is how do you put data science into your ecosystem? How, what is the data science engineering ecosystem?

What are the roles, the tasks, the work process?

So, we designed this, where we had this picture, when I was the head of the Center for Data Science.

So, this is, this diagram is, describing the scientific ecosystem, the To button. On the bottom, you have to build those engineers.

So, this can be, You can say, this is IT in a company, and the Data Domains, which is scientific domain, is different than what we call domain Scientists, Astrophysicist doctors. In the business is visited BU.

So, that's like the what you see in the bottom are the classical set up in a business. You have software engineers in IT, you have domain experts in the BU.

So, once you start doing data driven stuff in the company, the top will appear. So, we will have data scientists, and you will have people at the interfaces. So, between data science and software engineering, you have the sort of core data engineer. This is a little bit better known than the right-hand side.

These are the people who are symptoms, software engineers, and data scientists, so they know how to put the solutions, the data scientists design in the idea of the company. I will not talk about the left-hand side of this diagram.

What I'm talking about is the right hand side, because it's for more organizational point of view, That's the more interesting thing.

That's where the value is created from beta.

So you see three.

Uh, Rose, they're the domain expert to BU.

The data scientist and between them, are all what they call the data value architects.

This, like the, probably, the most obscure role, and we will be talking about it, and at the end.

So first, let's meet the data scientists.

So, if you're coming from a BU, The closest role to this may be, business is the business intelligence, but what you should know. And what I would like you to understand is how we work. So I am a data scientist slash AI researchers. I can do both roles. And if you want to integrate these people in the flow, in the workflow, you have to understand them a little bit how they work.

So, a lot of people say, why we call data science, Why did the science, Uh, in my mind, actually, this is one of the, row, is where is the word science has the.

The most role to be there, why?

Because data, scientists actually iterate the scientific method Dailey the Formulate hypotheses seek to collect data.

Or they look at the data they have.

They run a lot of experiments to validate their hypothesis.

And then the theory, this, it's a lot of trial and error to be with the AI systems. So we are actually writing the scientific methods. We are like, doing the science, like four times a day.

If you look at physics, they do do it like every 10, 20 years, the big experiment. Data scientists does a lot of times a day.

By the way, the person who see on the right hand side is George Stevens on, who is the inventor of the locomotive and debts.

For me, it's a good metaphor how you should imagine that the saint is today.

This is, uh, industry, one, perhaps, where we didn't have pipelines.

Things were created in the workshop, 1 by 1, with specialized tools.

And people who wore what I call a full stack, so there were little specialization. They built the locomotive from the beginning to the end.

Funny thing is that George Stephenson here works on the on the model train. and what we call what we work on is that the scientists are what we call model predictive models to try to predict what's going to happen.

We work alone or in small teams, mainly because the in the work process, there is not much specialization and the interfaces are not very well figured out yet.

So, most of the time, it's small teams, because everybody does everything A A lot of times, it's a single person to the other scientists, sometimes 2 3.

The last two points are very important. We build our own sandbox.

So there are a lot of tools, software tools for the fans, but at the end of the day, every data scientist have their idiosyncratic workshop and we hate operational constraints. There are little scientist who like to work in their workshops. We don't like standardized tools.

If you want to change the way your Docker Scientist, the best way to do it is to give them graphical user interface tools, because we hate them.

But just to give you an idea, how our work workshop looks like when we see we are sitting in front of the computer.

Just wait for the slide to change.

OK, so this is what you see on the right-hand side, you see what we call a model.

The model is just a piece of subject codes, with a special KPI. and on the left-hand side, the black screen is an experiment that's running some metrics that we are trying to optimize, OK? And this is very, very far from what even the person in Business Intelligence works with digital tools, like excellent, the Graphical User Interfaces. So this is a typical screen of a data scientist.

It's very, very, very variable, but this is what, Ally, because it's very, very flexible to scenes, AI is a, an experiment. The early phase, science will have evolved to be flexible.

So this is another thing, just now, let us see the, the domain expert, which is in our kids, into always the systems engineer.

Is the pilot. It can be a pilot of an airplane, or the way doesn't work on airplanes, but it can be a pilot to drive like a beat the fire, the data center cooling system.

Or somebody who's sitting in front of a computer and choose the mobile software, so the users, the client are served well.

So the main responsibility of systems engineers, the safe operation, not performance. So, that's important to see. Performance is important, but it's secondary.

The first thing is not to crash the system, and it's true for a lot of a lot of engineering systems.

And the third point, which is important from the point of view of getting AI in these systems, is that the systems engineers has very little incentive of collecting data.

And lead the data scientist, experiment. And the data scientist wants to experiment because AI is about learning from data, and it has to go through trial and error.

So, the challenge is the following.

The systems engineer in this context is not only the consumer of the AI. So, the systems engineer will be the person who will work with the AI, once the AI is implemented in either as a decision support tool or maybe semi automation.

So, then, for sure, there should be interface to build between the human engineering they are, but why is building the system the systems engineers also the collaborator of the scientist? So, this is very important to understand, from an organizational point of view, because if you don't get it, then you will not be able to develop your AI in your company.

The question is, can they work together? Because, they have to work together not only when the system is developed, but also in the development loop.

And to see you, the difficulties just show you.

not so imaginary conversation between a data scientist and the systems engineer.

So, systems engineers comes to us, Data scientists, or research, scientists?

The requests I want, today, I heard about that. It's so good to be able to be the go world, champion, I want AI my system, because I wanted to be better at it, and they want to, actually, as a ...

cell, the hardware to the, to, to our client, with AI, insight, OK, so, I want to, I really want you to implement AI, my engineering system.

So, then I asked the systems engineer as another fantasy, OK, can I access your system with the logarithm that I've been developing, which will take control of the system.

So, it will, you will, you can look at it, but this is the algorithm that controls the system, and why learning possibly breaking the system.

This is the basic point.

This is where a system engineer says, over my dead body, so, that, it just doesn't work.

Second question, OK, I cannot learn in the system, but maybe you can give me a stimulator of the system, which or which I can learn. We can experimentally the algorithm. It doesn't matter.

If I crash the Simulator and, for example, for ..., there are good simulators or which we trade, not aid, but human pilots.

But most of the system, this is a very hard, because this simulate this should build on, should be built on first principles, which are sometimes unknown, are very complicated to understand.

So this is why we want AI, because we want to learn them from data.

So the answer we get most of the time, yeah, we will have assimilate that. We're working on it.

But in any case, it's going to be some simplified things, which will never be good enough to be trusted.

So even if I show it at my quarter or Google Alerts from a simulator, I will be reluctant to put it on the real system.

So, that's the revenue, which can be followed, but there are bumps on it.

And the third one, we should be actually following is the is the solution is aware.

The idea is to experiment with the system button very carefully.

So, I get some data from the systems engineer.

I learned some control algorithm. I give the control algorithm back to the systems engineer. But they keep the control their systems, and you can decide whether follow the algorithm or not.

So, the control, this day of the systems engineer, but we can still experiment with the system very, very carefully.

And the key here is that they will add the systems engineer to login to the system variables, anytime there is something new he does, or she, that's, so I can learn from.

So, this is the avenue of we are following.

So, from, uh, abstracts, sort of like analysis point of view, what just happened there was the following.

Normally, I'm impact tech transfer, R&D is not new. We've been doing it for 100 years, at least, but most of the time, that the way it happens is that the systems engineer specifies a problem, gives it to the scientists. Scientists: Source, it delivers the technology, and the systems engineering implemented, whereas, data science is an iterative process, because it's data driven. So, there's an iteration of where the system vendors specifies the problem.

They are already very happy with this problem, is well designed. Well described, but, let us say it is, then, sent. It describes the data simulator, or the system that she needs.

They design tools to collect and annotate the data, and the interfaces to the AI algorithm together.

And the data scientists, the designers, the algorithms, the pipeline, the AI experiments, the metrics with which he works, then iterate because most of the time, it doesn't work the first time.

Right?

So, who manages this?

It turns out, that.

The best projects I saw, there was somebody, which I call the Data Value architect, to manage it. So that the person who designs the problem, in the first place, with the BU under data scientists together. It's a person who understands not only the business needs, and the business processes, but also somehow the assembled, the data science process.

And that's things like breaking the vicious cycle of ... data without value. low value, without data data, is expensive. So we invest in good data collection, We have to show some value to the business unit.

But to get this value, we need the data, so that it's a vicious cycle that is usually broken by doing small pilot, some of them bigger and bigger pilots of classical risk management.

The other big task of the data value architect is change management, which is not a technical role at all.

It's more like an organizational role, because it turns out that most of the time, to adapt, to accept AI in your process, you need to change how people work.

OK, so for the last couple of slides, we will go back to the engineering system, and I'll show you the big picture of how we are implementing these principles to put AI into engineering systems. So, what you see on the left is the loop show I showed in the beginning.

In the system of Systems Engineer, the red arrows are in the computer. So these are APIs, and the purple arrows are the project management steps taken by the humans. So, on the API side, what we need to add is a way to log the system.

As I said in the beginning, sometimes, it happens for, for legal or security reasons that the system variables are logged. But these data is usually not designed for feeding it into AI.

So we have to design data collection on the system.

Once this dataset is created, typically contains time-stamps of control actions that were taken by the engineer.

The system observables that describe the state of the system and the performance indicators. So these are all, all of them are several columns, and it's usually just an excess.

So this dataset is going to communicate it to the data scientist, who will basically the data set design a controller, which should it be safe, and should be better.

Then, the clinical policy that the systems engineers did for collecting the data and the first this, so this is where we do research, how to do this controller, why this controller is done.

We give it back to the systems engineer, who we repeat the cycle, and the systems engineer will keep the control of the system.

So if the artificial controller we gave them suggests control actions that are, that the systems that just thinks it's bad, they can overwrite it. So, that's very important.

Part for safety. Alice. We can do it on a simulation.

So, to complicate the picture, and this is actually sort of like the state of the art diagram we have is a bit complicated a little bit.

The side of the data scientists, what we design is not only a controller, but also what we call a stochastic simulator. So, we will actually learn simulators of the system which what do they do?

They predict the anticipate how the system we react with certain elections.

And the reason we do this is two things.

First will have a simulator. So on that simulator, we can safely try any control algorithm you want. And second, the simulator itself can be an output of this project.

We can give it to the systems engineers. They will be usually happy with it.

It's like designing an airplane, simulate them just by observing the how their brain reacts to pilot sections.

As I said, most of the times, these systems do not have a first principle based physics based simulator, so these data driven simulators are all supposed to output of the projects.

So these are the two reasons why this, sort of, algorithmic setup or, modular ization is good.

What we found very, very interesting, and this is where my researcher sites kicks in, is, what I'm really interested in, is to drive research so that the output of the research can be inserted in, with integrating into the work process.

So, the algorithms we develop, have to observe, have satisfied the operational constraints in which they will operate. And so we were very, very happy to come up with this diagram.

Because this diagram, I believe, can be implemented within the work processes of the company, and can drive research in terms of like, we have to fill those boxes out how to make the controller how to make the stochastic simulator. So, this is what my research team is actually doing after figuring out, you know, how lucky we can be useful for systems.

So, just to really dive in to the kinds of things we do, what you see on the right hand side is, one of the system observables, it can be a KPI, or it can be justice system, state. The red part is the bus.

Step 18 does the present.

The black one is the real future. So these are either simulators or observed data, and our goal is to stop at 18 and try to predict what's going to happen. So the orange curve that you see, is the output of our algorithms, that tries to predict the future.

So, you can also think about forecasting like methodology, where you want to predict the different temperature boss, one day, see, they won't leak ahead.

So, once we have these, then we have a simulator and we can design our control algorithms.

So, this is the big picture of what we do and how we implement the principles of PI organization between this framework.

And so, this is my last slide, so the, we are going back a little bit to, the apps are clever.

This would be my recommendations for any company, wants to, the dev in AI and two wants to put AI into our process.

So first, you need this talk them down Monday, It's usually C D O C C DSO. Positions are not very strong.

Usually put close to the IT.

Whereas, in my mind, that person has to be strong because we have to impose changes on all the BU works and has to impose costs change on a data scientist.

So without this position, either inside a company, or by talking to consultants with this will not work.

The second recommendation is to form commandoes.

So if you talk about, talk to consulting companies that do this, most of the time, 70% of the project is not clinically change management, because AI will have to be adopted by people work, so it's like to work tool. To 20 10 is that the science of 30%? Where 20 is like data cleaning? And only 10 percent is what we call a purely that the same.

So that's that's an interesting thing to note, because the commanders have to have people who are not technical but organizational experts. And the third one is to build trust through iterative pilots.

This is not something that AI invented. We have to start small, because we have to show value.

So we can have, you know, funds for data collection, and building the pipeline, which is an engineering problem.

Uh, the fourth one is design a data science process.

So, the idea of moving it to sort of standardization what the big ...

does, you will see a lot of resistance from the scientists because as I said, we like to be very, very free. Battling the longer, and especially if you have water one pipeline to implement. It's a good idea to start having a memory and standardized tools that the data scientists use, and they have a blog post on this, if you're interested.

And the last one is the continuation of the fourth point is that you need to start building standard operational tools for data science, which, what can we have basic, very bare bone low-level tools but not operational, too? So what I mean here, like pipelines, things that are standardized.

So we are actually using an experimental software library for this within the team that I'm trying to push also to the to the be used, because the idea is that we use the same tools for doing the experiments. And the BU uses the same tools for production analyzing the pipelines.

And, so, with this, finished, so, thank you for listening to me.

I'm opening the floor for questions.

So, we'll build the slides and give the mic.

Thank you, very much, buys, great coverage of what?

Artificial intelligence, specifically on the applications, on engineering systems, I think that's quite interesting.

I've had the privilege of leading a very large transformation for an organization with 12,000 engineers, interdisciplinary, and very aware of the applications that you're talking about there, and the challenges. I always told people, it's a place field of ideas and 12,000 different opinions about anything that does Years.

They'll sit there and talk about, know, defend their point of view on a certain aspect of an idea for a very long period of time and that it can be it's both very exciting, but can be quite challenging. So, I'm glad you're focusing on that. The first question that has come up here, it's a bit of a more general question. Since you have such an expertise in artificial intelligence on, if you look back in the last, you know, I think most people now have caught up with what happened with goal and the Deep mind.

The Deep Mind work that has taken place, if you look back at the last 12 months, 18 months, has there are any being, has there been other advances in artificial intelligence that you think are not noteworthy, that we should be keeping? Amazon?

Yeah.

So my domain.

Good enforcement, learning control.

It's a slow moving domain.

We have big breakthroughs. But right now, it's a lot of, like, small steps to move towards some something that's robust.

So most of the algorithms we are working on can we work very well? Sometimes they do beat the championship, but sometimes they do very dumb moves. And robustness is a set of says more important than performance on the average.

But if you, if I go out of this, this sub domain of AI, the biggest breakthroughs are in language processing.

So right now, the big IT companies, like Google, Facebook, Amazon, they have the computational infrastructure, they have the data to feed, basically all internet into language models.

So the key word, if you want to look for the GPA is three, which was developed by Google, not feasible, but by Open AI.

This was A which is a company that was founded by, or funded by Elon Musk, uh, and they did this.

So it's a lot of the Berga, a lot of computer cycles, but at the end they have a language model, which is very, very tricky.

Sometimes it writes an essay that sounds, like, comes from a philosopher.

But sometimes they still do very dangerous things, because they don't live in the world, they live in language. That's a very interesting advance.

It feeds into the bot industry.

So a lot of companies now can interface with GPL V three and design their own language bots.

But they wouldn't trust it with things that I need safety like, or understanding the world, and the basic like level it in the sixties, or child on this too.

So it's a very interesting thing, because sometimes it's very, very sophisticated, but sometimes it just makes dumb thing.

But for me, that's the biggest sort of MTV biggest breakthrough in AI in the last two years.

Very good, very good. Marla, Vanessa, Zuniga has a question for you.

She, she is asking, I know why way, of course has developed applications, probably on the, on the consumer segment, and industrial segment of its technology development, if you will.

How does that work for a research group like to end with a company like Y Way? Or In terms of the collaborations for Develop, You know, artificial intelligence algorithms are some of this consumer applications, or industrial applications helpful, and useful for you in the research work that you do, or you're tapping.

Maybe other external sources for that development.

So we are working at least half of our time, but even more for BU. And Noah's Ark, which is a lab of about 5 to 500 researchers. We cover all the space of worldwide product. So there is a big, big part of Noah's ark who works for the phone. And the most successful AI applications are in pipelines of air.

The pipeline is well defined, like, we know what we want, And we know how to measure what you want.

And so, for example, who is a leader in photo enhancement for the phone, or three-d., or, you know, low light?

photo enhancement because they are, we know what we want. We have like Lowlight Photo, we take another photo with a better camera, which is a highlight.

And so, we just want to mention it to so and this D in these pipelines, you can have them, you can develop the research cycle that delivers very rapid results, because this is what researchers are trained to.

There are other pipelines, like this also on the, on the on the network side, where the problem is well defined, like, I don't know, in the mobile tower. We want to know which packet is coming from, which type of applications. So if it's video, if it's a video phone, or if it's just somebody browsing, we need the, which we can set the best parameters for that kind of application.

And that's, again, very nicely designed pipelines where we know how to measure performance.

So the way we work is sort of like any that the same effects going on.

A big company is trying to go, go for the pipelines where these elements are already there, was the output you want from the input.

Do you have data somebody annotated data? And I can measure how well it works, So I can do the cycling of improvement. The trouble comes when this is not.

or even if this is hard. So it's not the law because somebody didn't do it, but to genuinely heart. And, in a certain sense, that's what I'm most interested in, how to improve this process and get these two domains, where figuring out where the value will come from and how to measure that is hard.

So I don't know if I asked the question, but these are my thoughts on around this.

Now, you provided some very interesting sites there.

The question was, was about advances in technology and maybe how they cross-pollinate from consumer applications, maybe two more industrial or engineering applications.

And as a matter of fact, there's a number of that questions around this theme and I'm gonna have a little bit of fun with it. I'm gonna ask you this question sci-fi question as well That that people are focusing on the On the chat right now, and the question is about General AI, right? And the Why? Of course, a lot of this, and this is interesting on the context that you explain where a lot of engineering systems type of innovations and improvements, they require very domain specific knowledge, Which AI could be useful for that, But they also require a lot of interdisciplinary, multi domain knowledge, which is ... quite challenging, because we don't have a general AI model yet. What is your assessment from a research standpoint? Is, is general AI and inevitable progression that we're gonna have to, or it's really not like that, it's more of a discontinuity on the curve.

and you're gonna have to have, it's not a natural progression of the advances in AI we have today. You will have to be a breakthrough of a different sort. So curious from a researcher's standpoint, how you see the development of AI and specifically moving towards a generalized AI model.

I don't think anybody knows what General AI is, because you don't know what intelligence is.

And either it's something very reductionist, where we get into a domain and where we can measure the performance. So we know that it's like image recognition, or speech recognition, language understanding, where we design those metrics, And we have the pipeline. And we call it AI.

And it particular afforded pipeline.

Or we're getting to almost philosophical questions about what AI is.
40:54
So I think the question is a little bit overblown.

At least, I'm not that much interested in the question.

I think, if we get to something like, like generic generally, I really need at least 2, 3 big breakthroughs as because we have in the present tense or deep learning, if, you know, the topic.

That does my must come from the area of the research areas, which are completely under the radar today, so it's hard to predict them because, as deep learning was hard to predict in 2000, and everybody was left a believer in it. I think the next big, through the hour or so unpredictable and it's, it's, it's very likely that it will not come from the domain of deep learning or neural nets, but from somewhere else.

I have a vision about which direction to go, and it's related to what we're actually doing on systems. We actually need to design systems that live in the real world.

Matt.

On the Internet.

I d hours discussing. So, let me move on to the next one. Let's talk about the one of the things that has come up. A lot of people in this, the audience here, have over 2000 registrations here.

And the business transformation leaders, operational excellence leaders, and a lot of engineers and scientists are listening to us as well, and one of the themes that has emerged as a final wrap up question here, is the big data versus small data around, you know, And, I'll give you a very specific one. I was leading at one point. I was leading a major innovation project for a very large energy organization, and we're looking into finding ways that we can remove mercury emissions, more effective ways. And we had a very big, complex machine that was designed to do so. Sulfur dioxide removal. And we're trying to make adaptations and to remove mercury as well. In any case, it's a very complex system. And then we asked for some data to be collected and we got 18,000 data points on this machine.

But after looking at the data points, and we know this is not AI, this is just like multivariate regression analysis, just really trying to understand the key drivers. And, after looking at it, we realized that, you know, there is, it's good, but we, we don't need 18,000 data points. We would be better off with eight data points, but the right data points, data points in operating conditions.

They normally don't run Bitcoin because, you know, of course, when they are running the machine that want to keep it very tight control. So, there's this challenge that if you learn by observing operations, you're not going to have a variation of the inputs that will teach anything very valuable about how they affect the output.

So, the question is, Big data versus small data. There's a lot of talk about using Big Data Free, I can you tell us in the minner. So what's going on with small?

Is small data, high quality data in AI? Is there an area that we can use AI or not really?

Then, you just touch the question we are brainstorming right now.

So, it says, it's very interesting, and all the data we have here is spoiled.

Because engineering systems are getting faster Time, y'know, computer systems are, you know, CPUs, GPUs. They get fast.

There's the Moodle, but the physical systems that we are studying on the same clock.

So the data we collect is always the same size, and usually small.

The other question I form, it's even more interesting is this control questions.

That all the data we have come from systems when the engineers tried to be very good, can robust.

So they always do the same thing in the same state. So it's very hard from learn to learn from this data, because they never changed their control policy.

So one of the big questions we are studying is how to give algorithms that explore the space carefully.

So we sort of learn what the engineer does and around that, a little bit of randomization so we can discover the regions where we can improve the algorithms.

And this is one of the biggest questions for us if you want to apply this.

And one of the questions, this is, it, this study, the entities, because I'm not saying it's completely nuts that did, but it's not the focal point of AI research.

The big part of the AI research is on big data.

It's getting bigger and bigger data, putting on bigger, bigger machines, and klum pipelines that are known, like image recognition of NLP, but this question of, like, having methods for heterogeneous systems, where I can just take the method, put it on the system, and it learns by itself.

We are very far from it.

And so it's a very good question, I don't have an answer to this, but this is what we are studying.

Yeah, Fantastic, ... these are, those are some of the real challenges, I think the research going forward, the kind of, combination of, design of experiments with AI development, the syntactic. listen. We're, we're out of time. Unfortunately, We could talk about this for hours. I'm very, very grateful for you to take the time of your busy schedule from Paris, France, to the world. Share your insights about the developments of AI In the in the engineering contacts and industrial context. Which are quite interesting for our audience, so thank you so much for doing that. We had a lot of fun just learning with you today.

OK, and thank you very much for inviting me. I was a pleasure to talk to you guys and I'm looking forward to keep, you know, you can contact me on LinkedIn or find my mail and we'd be happy to share insights.

Thank you again.

Thank you. Ladies and gentlemen, that was ... research leader for Y way, directly from Paris, France, and the tremendous insights on how artificial intelligence is being used in the challenges of its usage in the engineering systems in industrial settings. Terrific, terrific insights. Our next presentation at the top of the hour will come from Donald Cook, who is rapidly transforming RP enterprises for RPA and intelligent automation for this virtual ... economy. Donald Cook is a global leader on and Senior Exec Executive Advisor for many, many leading global organizations on this topic on how you build intelligent automation.

On, on social and industrial practices. So, you do not want to miss that. He is a speaker at the World Economic Forum, and just tremendous insights with decades of experience and application of high technology. So we're going to see you back at the top of the hour, with Donald Cook on the rapidly transforming enterprise through RPA and intelligent automation, Susan?