Tech Leader Pro John Collins

In the early days of the Internet, we thought it would bring enlightenment via free knowledge, instead we have hate, division, toxicity.


Notes:



In the early days of the Internet, we thought it would bring enlightenment via free knowledge, "the greatest library ever known".
Instead we have hate, division, toxicity...where did it all go wrong?
The optimists need to regroup.
I still remember the first time I dialed onto the Internet, using an old PC with a modem in a public library.
It was so exciting, I had all of that information at my finger tips!
In the early days in the 1990s, we we genuinely excited about the "information superhighway", but now we have the "disinformation swamp".
There used to be a technical barrier to entry to publish something online: you needed to be able to write some basic HTML code, and upload it to a server somewhere.
Now, with the advent of user-generated platforms like X, Facebook, and TikTok, ANYONE can publish content online from their smartphones with no effort required.
As a result, we are swamped in poor-quality content.
Imagine a library where each book is the random stream of consciousness of anonymous Internet users, shouting abuse at each other. That is the library we have built, and it's terrible.
I no longer believe it is a net good.
The most perverse trend of all is the growth of technology skepticism online, with neo-Luddites using technology to spread their dislike of technology, without any sense of hypocrisy.
We are a few news cycles away from angry righteous mobs attacking data centres, encouraged to do so via their smartphones.
What I am working on this week:


Behind on dev projects this week.

Media I am enjoying this week:


The Revenge of the Rose by Michael Moorcock. https://centipedepress.com/fantasy/elric5u.html



Notes and subscription links are here: https://techleader.pro/a/645-Tech-Leader-Pro-podcast-2024-week-21,-the-Internet-has-failed

As an engineering leader, I'd like to share some harsh truths about managing engineers.


Notes:



As an engineering leader, I'd like to share some harsh truths about managing engineers.
If you are a young engineering leader listening this this, you may not be ready to agree with all of these yet, but it will challenge you to think about things differently.
Firstly, the engineers in your team are not your customers.
Your customers are your customers, please don't lose sight of that.
The engineers in your team are exchanging their time and skills for money and career growth. That's it.
The relationship between you and your team is transactional, and that relationship can end at any time when:


They quit because they found a better opportunity.
They quit because they are frustrated.
They criticize your publicly or privately.
They work to undermine you.
They ask to be transferred from your team.
You are forced to fire them, for cost cutting or performance reasons.

Personally over the years, I have had to deal with all of those situations.
They are not your friends, your family, or your customers. Your job is to give direction and support, not to make them happy at any cost.
In the beginning of my leadership career, I would have proudly described myself as a servant leader, but many disappointments made me realize I was naive.
I had to learn the hard way, and I hope by sharing my experience here, that's others can learn the easier way which is via other people's learned experienced.
Many engineers think them are smarter than you, and may even call you an idiot privately.
They have no experience in leadership, politics, and setting business priorities, but think writing code is harder than all of that.
The best ones get it, but they are rare.
Finally, loyalty is overrated and rarely goes both ways.
It's better to accept that professional relationships are transactional, and be honest about that.
What I am working on this week:


Search indexer improvements for greppr.org.

Media I am enjoying this week:


Shogun on FX.



Notes and subscription links are here: https://techleader.pro/a/644-Tech-Leader-Pro-podcast-2024-week-19,-Engineers-are-not-your-customers

The new knowledge acquisition bottleneck is accessing silos of private content that can be used to train modern AI models.


Notes:



Decades ago when I was working on expert systems, which were an early kind of AI, one of the main issues facing us was the "knowledge acquisition bottleneck".
Expert systems were rule-based engines, and those rules where defined based upon input from domain experts.
The challenge was getting those rules from the heads of those experts, and into the software as repeatable rules, hence this was labeled the "knowledge acquisition bottleneck".
According to Wikipedia: "Knowledge acquisition is the process used to define the rules and ontologies required for a knowledge-based system. The phrase was first used in conjunction with expert systems to describe the initial tasks associated with developing an expert system, namely finding and interviewing domain experts and capturing their knowledge via rules, objects, and frame-based ontologies.
Expert systems were one of the first successful applications of artificial intelligence technology to real world business problems. Researchers at Stanford and other AI laboratories worked with doctors and other highly skilled experts to develop systems that could automate complex tasks such as medical diagnosis.". Ref: https://en.wikipedia.org/wiki/Knowledge_acquisition
So in simple terms, if you wanted to build an AI that could represent an expert like a doctor, you would begin by interviewing many doctors on how they do they work and then capture that as rules in the system.
It was laborious, time-consuming, error-prone, and often the experts in question would simply refuse to cooperate.
Modern AI systems are no longer knowledge-based expert systems, but instead are neural networks that aim to mimic the behavior of the brain.
Such systems can learn via large quantities of clean data being loaded into their model, which is their representation of the World.
Interviewing experts is no longer required, but vast quantities of data is required in order to train the AI.
In a previous episode, I shared my view that the web is already being mined for data for such models, but that may eventually be exhausted.
I believe a new market will emerge that will sell access to private, offline repos of data, which will become invaluable for training AI.
This is the new oil.
According to an excellent article on this topic from 2018 entitled "Did We Just Replace the ‘Knowledge Bottleneck’ With a ‘Data Bottleneck’?", the author mentions that "Some studies indicate that data scientists spend almost 80% of their time on preparing data, and even after that tedious and time consuming process is done, unexpected results are usually blamed by the data ‘scientist’ on the inadequacy of the data, and another long iteration of data collection, data cleaning, transformation, massaging, etc. goes on.". Ref: https://cacm.acm.org/blogcacm/did-we-just-replace-the-knowledge-bottleneck-with-a-data-bottleneck/
Apart from the quality of the data being ingested, we also need to worry about the reliability of the source: bad data passed to a model on purpose can act as a kind of supply-chain attack on an AI, if not detected and filtered out early enough.
In comparison, the expert systems of old were easier to trust as the models were build by human hands, not by bulk training.
In reality this space is not solved yet, but we are getting closer with each new generation of the technology.
Just this week, the FT has sold access to their content to OpenAI to help train their ChatGPT AI, OpenAI said in their press release "Through the partnership, ChatGPT users will be able to see select attributed summaries, quotes and rich links to FT journalism in response to relevant queries.". Ref: https://openai.com/index/content-partnership-with-financial-times
Weirdly since I read this, OpenAI have apparently removed the announcement from their site.
I expect many more such deals to happen.
What I am w

By asking why, repeating back, and sticking to first principles: you can drive your team towards a truthful outcome.


Notes:



During the week I posted the following on X:
"How to get to the truth:


Ask why.
Repeat back what they just said, so they hear how stupid it sounded.
Stick to first principles."

Some of that post was borne out of recent frustrations in my day job (hence the use of stupid), but there are three serious points there which are worth exploring further.
So let's start with asking why:


This is a great way to challenge statements made as facts. Why do we have to do that?
This is also a great way to learn when asked with genuine intent. Why is that so?

Now let's look at repeating back:


You can use this to hold a mirror up to bad ideas. "So what you said is..."
You can use it to show you understood the point. "So what I got is..."
Finally, you can use this to summarize the discussion and get people aligned around your summary. "So what I am taking away is..."

Finally we have first principles. A first principle is a basic assumption that cannot be deduced any further, is a fundamental unit that everyone accepts within a given domain.
When we reduce the discussion to first principles, namely those things that cannot be misunderstood, it will allow us to get closer to further basic truths.
In any organization, we have to deal with large volumes of noisy data, and human interpretations of that which are biased. The truth can become very elusive.
By asking why, repeating back, and sticking to first principles: you can drive your team towards a truthful outcome and cut through the noise.
What I am working on this week:


Search indexer improvements for greppr.org.

Media I am enjoying this week:


Martian Chronicles by Ray Bradbury.



Notes and subscription links are here: https://techleader.pro/a/642-Tech-Leader-Pro-podcast-2024-week-16,-getting-to-the-truth

After twenty plus years in the tech industry, you start to spot patterns. One such situation is when a new leader joins an organization.


Notes:



After twenty plus years in the tech industry, you start to spot patterns.
Some of those are positive, but many are pointless or even harmful.
For the negative ones, tech people like to use the term anti-pattern, which according to Wikipedia is defined as:
"An anti-pattern in software engineering, project management, and business processes is a common response to a recurring problem that is usually ineffective and risks being highly counterproductive.", source - https://en.m.wikipedia.org/wiki/Anti-pattern
One such situation is when a new leader joins an organization, and I must admit to being guilty of these myself in the past.
There are a few classic moves that I see new leaders in an organization repeat over and over in their first six months:


Team reorg - this can include firing existing people, reducing overall headcount, or bringing in their own trusted people from their past.
UX refresh - this is a classic response: blame the product for the current problems, and immediately design a new UX to address this.
Process re-engineering - again a classic response: blame the current processes for the problems, and bring in new processes to address this.

After a few decades in an industry, these patterns become dull and predictable.
I have come to the conclusion that most issues are cultural issues, and that takes time to fix.
Sadly however, a new leader may only have a few quarters to make an impact, especially in a public company, so it can be tempting to reach for these easier levers to pull upon.
The higher up an organization you climb, the less time you should be spending on detail, and the more time you should dedicate to culture and strategy.
An inexperienced leader will tend to do the opposite, and get lost in the details.
An experienced leader entering a new organization will know it will take them 1-2 years to make a meaningful impact on culture and strategy, and should negotiate such a timeline during their interview process.
Patience is the key, and expectations need to be set accordingly from the very beginning.
What I am working on this week:


Fighting against a spam bot attack on greppr.org

Media I am enjoying this week:


Diaspora by Greg Egan.



Notes and subscription links are here: https://techleader.pro/a/641-Tech-Leader-Pro-podcast-2024-week-15,-new-leader-anti-patterns

The amount of authority figures you defer to in your life should be a very short list indeed.


Notes:



Recently I seen Dune part two, and it got me thinking about deference again.
According to the dictionary:
"deference - respectful submission or yielding to the judgement, opinion, will, etc., of another." Ref: https://www.dictionary.com/browse/deference
The Dune story acts as a warning for us not to blindly follow a charismatic leader.
In the story, a tribal leader Stilgar suffers from a profound religious deference towards a young leader, named Paul Atreides, who he believes is a messianic prophet that will lead his people to freedom.
Stilgar places so much deference in Paul, it almost becomes comical in the latest version of the movie. He is portrayed as a zealot.
In spite of Paul's discomfort with this, he finds himself surrounded by fanatical followers who will follow him blindly into a holy war.
An extreme fictional example perhaps, but our history is full similar real examples.
Polite deference to a senior colleague, a teacher, or a parent is welcomed as a mark of respect.
Blind deference to a superior can be dangerous however.
The amount of authority figures you defer to in your life should be a very short list indeed. Everyone else should be questioned.
Deference is intrinsically related to trust, and once trust broken, so is deference.
Deference placed upon the shoulders of a young leader is a heavy burden, a theme explored wonderfully in Dune. Paul resents it.
The movies are great, but please read the Dune books by Frank Herbert!
What I am working on this week:


Ongoing testing of the internet search indexer for the Alpha Framework.

Media I am enjoying this week:


Diaspora by Greg Egan.



Notes and subscription links are here: https://techleader.pro/a/640-Tech-Leader-Pro-podcast-2024-week-14,-be-careful-where-you-place-your-deference