A tag called Line1.Power might be obvious to the engineer who created it. It is less obvious to a scheduling system trying to calculate the energy cost of a production order, or to someone comparing the same machine across five sites.

To answer those questions, one needs a model of the factory. ISA-95 is a good place to start.

What ISA-95 gives you​

ISA-95, also published internationally as IEC 62264, is a set of standards for integrating manufacturing control systems with business systems. It defines common terminology, system boundaries, activities, and information models.

The familiar ISA-95 levels are:

• Level 0: the physical production process.
• Level 1: sensing and manipulating the process.
• Level 2: monitoring and supervisory control.
• Level 3: manufacturing operations management.
• Level 4: business planning and logistics.

The standard mainly concerns the boundary between manufacturing operations at Level 3 and business systems at Level 4. This is where operations requests, schedules, resource requirements, and performance results have to move between systems without everyone inventing a different meaning for them.

The levels are useful, but they are not the main reason I use ISA-95 when building a factory model. The value is the vocabulary underneath them: equipment, materials, personnel, segments, requests, schedules, and performance.

ISA-95 gives us a shared language. It does not give us a PostgreSQL schema that should be copied table for table.

Start with the questions​

It is tempting to begin by modeling the entire standard. This will produce a large and impressive schema before it produces anything useful.

I prefer starting with the questions the model needs to answer:

• Where is this machine in the factory?
• What material or utility flows through it?
• Which measurement belongs to which machine or flow?
• What process is supposed to happen here?
• Which production order was running at 10:30?
• How much electricity did that order use?
• Which input lots ended up in this finished product?
• Why did the line miss its target?

These questions quickly expose the concepts and relationships that are actually required. They also provide tests for the model. If the schema contains 40 tables but still cannot answer which machine consumed the most steam last Tuesday, the model is not finished.

A factory is more than one graph​

I use “factory graph” as a description of the relationships, not as a requirement to use a graph database. A relational database handles most of this well.

The important part is recognizing that a factory contains several overlapping graphs:

They are related, but they are not interchangeable.

The equipment hierarchy​

The equipment hierarchy describes containment:

Enterprise
└── Site
    └── Area
        └── Work Center
            └── Work Unit

This is the backbone of the model. A self-referencing equipment table is usually enough to represent it:

CREATE TYPE equipment_level AS ENUM (
    'Enterprise',
    'Site',
    'Area',
    'WorkCenter',
    'WorkUnit'
);

CREATE TABLE equipment (
    id                  BIGSERIAL PRIMARY KEY,
    ref_id              VARCHAR(64) UNIQUE NOT NULL,
    name                VARCHAR(255) NOT NULL,
    level               equipment_level NOT NULL,
    parent_equipment_id BIGINT REFERENCES equipment(id)
);

The application or additional database constraints should prevent cycles and invalid parent-child level combinations.

The hierarchy lets one aggregate measurements and production results from a machine to a line, area, site, or enterprise. It also gives every other object a stable location in the factory.

Do not confuse a logical equipment position with the physical asset currently installed there. “Filler 1” may be a permanent position in the process while the motor, meter, or complete machine occupying it changes over time. If replacement history matters, model physical assets separately and assign them to equipment with start and end timestamps.

The physical topology​

Containment does not describe flow.

A boiler may sit in a utilities area while supplying steam to three production lines in another area. A buffer may receive material from two machines and feed a third. None of this can be inferred from the equipment hierarchy.

Represent those relationships explicitly:

CREATE TABLE flow_connection (
    id                BIGSERIAL PRIMARY KEY,
    ref_id            VARCHAR(64) UNIQUE NOT NULL,
    from_equipment_id BIGINT NOT NULL REFERENCES equipment(id),
    to_equipment_id   BIGINT NOT NULL REFERENCES equipment(id),
    material_id       VARCHAR(64) REFERENCES material(id),
    CHECK (from_equipment_id <> to_equipment_id)
);

This graph answers questions about material, energy, water, steam, compressed air, waste, and emissions. It should not also be forced to represent process sequence. A pipe between two machines and the rule that operation B follows operation A are different facts, even when they happen to point in the same direction.

Process intent​

ISA-95 uses segments to describe reusable work and its resource requirements. In a production example, a segment can require:

• an equipment class or specific machine
• input and output materials
• personnel or a personnel class
• parameters such as temperature, pressure, speed, or target cycle time

A workflow or route connects segments in the required order. This is what should happen, independent of a specific order or Tuesday's production result.

Keeping process intent separate from physical topology makes the model much more useful. The same process step may run on one of four equivalent machines. The route stays the same while the equipment assignment changes.

Operations and production execution​

The 2026 edition of IEC 62264-2 uses generalized operations models rather than separate production-only schedule and performance models. This allows the same structure to describe production, maintenance, quality, and inventory work. I focus on production here, but the separation is the same:

Demand → Schedule → Dispatched work → Performance

For production, a request might say: make 5,000 units of product X by Friday. A schedule assigns time and resources. Dispatch turns scheduled work into executable job orders. Performance records actual start and end times, quantities produced, quality results, and the resources used.

This separation is necessary for plan-versus-actual reporting. If a schedule row is continuously edited until it resembles reality, the original plan is lost. Keep versions of the plan and record actual performance separately.

It also provides the data required for OEE: planned production time, runtime, ideal cycle time or rate, total count, and good count.

Materials and genealogy​

A material definition and a material lot are not the same thing.

“Steel grade X” is master data. “Lot S-2026-0814” is a particular quantity of that steel received or produced at a particular time. Recipes and process segments normally refer to material definitions. Traceability refers to lots.

Once execution results are linked to consumed and produced lots, genealogy becomes another directed graph:

Input lot A ─┐
             ├── Production run ── Output lot C
Input lot B ─┘

This supports recalls, quality investigations, and product-level cost or emission attribution. It is useful, but not every first version of a factory model needs it. Add lot genealogy when there is a real traceability question to answer.

Measurements​

A measurement point defines the meaning of a signal. A reading carries the observed value and timestamp, normally with a quality or status indicator.

The point should identify:

• what is measured
• the unit of measure
• whether it belongs to equipment or a flow connection
• the material or utility, where relevant
• whether it is physical or calculated
• which sensor or calculation provided it during a given time range

This distinction keeps the logical point stable when a meter is replaced. Historical readings remain attached to “Steam flow into Line 2” even though the physical device has changed.

I would normally keep this metadata in the relational model and store high-volume readings in a time-series database. At 1 Hz, 10,000 tags generate 864 million readings per day. The factory model should provide context for those readings without forcing every reading through the same transactional database used for equipment, recipes, and schedules.

Units and time are part of the model​

Units should not be stored as inconsistent free text across sensor and material tables. For gas volumes, m³ identifies the unit of volume but does not state the temperature and pressure conditions. A value expressed in Nm³ has been normalized to defined reference conditions. Those conditions must also be recorded.

A unit dictionary should include a stable code, symbol, quantity kind, quantity dimension, and, where valid, a deterministic conversion to a canonical unit. Every measured or calculated quantity should reference a defined unit. Cost and emission factors must also state their unit explicitly.

Time validity matters for the same reason. Electricity tariffs change. Emission factors change. Assets are replaced. A sensor is remapped. A machine moves to another line.

If the model overwrites these relationships, historical calculations can silently change. Store valid_from and valid_to for any fact whose value or assignment can change over time.

Build it in layers​

The first useful version does not need the whole standard. I would build it in three layers.

Layer 1: structure and measurements​

Start with:

• equipment and its hierarchy
• material and utility definitions
• physical flow connections
• measurement points and units

This is enough to browse the factory, map sensor data, and answer basic consumption questions by equipment, line, or site.

Layer 2: process and execution​

Add:

• segment definitions and workflows
• equipment, material, and personnel requirements
• operations requests and schedules
• dispatched work or job orders
• performance results

Now the model can connect what should happen to what did happen. Scheduling, OEE, throughput, and resource attribution become possible.

Layer 3: traceability and optimization​

Add these only where required:

• material lots and genealogy
• personnel capabilities
• asset assignment history
• time-valid cost and emission factors
• storage capacity and compatibility
• changeover costs and scheduling constraints

This is where a factory model becomes valuable for optimization, product costing, emissions accounting, and traceability. It is also where complexity increases quickly. Each extension should still be justified by a question or workflow.

Common mistakes​

Putting everything in one node and edge table​

A generic graph looks flexible at first. Eventually every query depends on a growing list of node types, edge types, and JSON properties. Database constraints become difficult and invalid relationships become easy.

Use explicit tables for concepts with different rules. An equipment hierarchy, process dependency, material genealogy edge, and steam connection can all be traversed as graphs without pretending they are the same relationship.

Copying ISA-95 literally​

ISA-95 is a conceptual standard designed to work across industries and systems. Your implementation serves a particular product and set of factories.

Keep the standard's semantics, but use names your team understands. Document the mapping. Do not add tables merely because an object exists somewhere in the standard.

Mixing definitions, plans, and actuals​

A recipe is not a production order. A production order is not a schedule. A schedule is not a production result.

Combining them makes the first demo faster and every later analysis harder. Keep intent, plan, dispatch, and actual performance separate.

Ignoring identity​

The same machine will have different IDs in ERP, MES, SCADA, maintenance, and the historian. Pick a stable internal identity and keep aliases per source system. Avoid making a display name or one vendor's identifier the primary key for the whole factory.

Modeling before asking​

The most dangerous version of a factory model is one that is technically elegant but operationally irrelevant.

Start with five to ten questions from engineers, planners, operators, and managers. Build the smallest model that answers them. Add the next concept when the next question requires it.

The model is the integration layer​

ISA-95 was created to make integration between manufacturing and business systems less ambiguous. That remains the useful idea.

A factory model should allow a production request from ERP, a machine state from a PLC, a meter reading from a historian, and a finished quantity from MES to refer to the same operational context.

The final schema will differ between a steel mill, a battery plant, and a food factory. The underlying questions are similar: where did this happen, what was supposed to happen, what actually happened, what went in, what came out, and what did it cost?

ISA-95 gives us a solid language for answering those questions. The work is choosing how much of that language the factory actually needs.

If building software for factories sounds interesting, take a look at the open roles at Juna AI.

A community driver existed, but installing it involved old dependencies and manual configuration. I wanted something simpler: open a normal macOS package, add the printer in System Settings and print.

I also wanted to connect the printer to my Unraid server so that every computer on the network could use it without moving a USB cable around.

I used Codex with GPT-5.6 Sol on high to do most of the work. I gave it the old driver and asked it to turn it into something I could install normally on my Mac, then added the Unraid setup. After a few rounds of testing and cleanup, I had a much smaller driver, a macOS installer and a Docker container.

A smaller macOS driver​

The printer expects pages in Ricoh's own format. The new driver lets macOS do the normal document conversion, then performs the final two jobs the printer needs: compressing the page with JBIG and wrapping it in the Ricoh printer commands.

This reduced the driver to one small CUPS filter and a printer description. It can now be installed with a standard .pkg file and appears as Ricoh SP 112 Gurich when adding a printer on macOS.

It supports both 600×600 and 1200×600 dpi. The closely related Ricoh SP 110 uses the same protocol and should work too.

Turning it into a network printer​

The same filter can run in a small CUPS container on Unraid. The printer stays connected to the server by USB, while CUPS shares it over the network using IPP. Macs normally discover it automatically through Bonjour, and other computers can connect directly to its IPP address.

Once the repository is on Unraid, starting it is one command:

docker compose -f docker/compose.unraid.yaml up -d --build

The container detects the USB printer, creates the queue and shares it. Client computers do not need the Ricoh driver because the server handles the final conversion.

Using it​

The source code, macOS installer and Unraid instructions are available in the ricoh-sp112-cups repository.

For a Mac connected directly over USB, download the package from the latest release, install it and add the printer through System Settings → Printers & Scanners. The package is not notarized, so macOS may ask you to approve it under System Settings → Privacy & Security → Open Anyway.

For Unraid, clone the repository into an appdata directory, connect the printer by USB and start the included Docker Compose configuration. The shared queue is then available at:

ipp://UNRAID-IP:631/printers/RICOH_SP_112

It is a satisfying way to keep a simple, reliable printer useful instead of replacing working hardware just because the original software stopped keeping up.

AI models can pretty much solve any trivial/medium and even "hard" task you throw at them. It's basically magic. Tests will pass, the functions will work, and the task will be "solved".

You see others shipping features at the speed of prompt and you want the same. You use the tools to your advantage, as you should.

Overtime, you're left with a strange feeling. You reflect and think, what did I learn these last months and nothing comes to mind. At least you shipped 100 new features, and rewrote your entire codebase. Improve, enhance, repeat.

How do you deal with this. Let's view a few scenarios, depending on how much you care.

The stages of caring​

You don't care​

You might look through the code on the surface and then say "looks good to me". Commit to main.

Job done. Let's go home.

Care a little bit​

You care a little bit, you read through the code. This is code you know well. There are some minor changes here and there that don't look too dangerous on the surface. You ask yourself, is this an improvement? Maybe. Upon closer inspection, some tradeoffs you spent many hours thinking about are ignored. You might catch the ignored tradeoff, you might not. Ultimately, the feature is shiny. You say, "how bad can it be?" and commit.

You care a lot​

You're different.

You care a lot. You take it slow. You look at your code in great detail. Inspect every change, hand-craft the architecture where needed.

Perfection achieved much faster than you could have done without these tools, but not quite at the speed of a prompt.

Teamwork​

Let's assume we're all in the "you care a lot" camp.

You take a day off. Someone wants to implement a new feature and vibe codes all your carefully crafted perfection away. They "fix" a bug on code they don't understand.

All those hours you spent thinking about the pros and cons of various decisions along the way get lost to history for the sake of speed. At least the tests pass.

The price of good work​

One used to have to work to produce something. Even something of poor quality. Now it is effectively free. Anyone can produce something and at first glance it looks acceptable. The barrier to entry is effectively zero. The barrier to reach the top is harder than before. The bar is higher and lower at the same time.

The people who can tell the difference between good and bad code are more powerful than before, just as a person who knows how to sew can identify what good stitching looks like on premium clothing.

The price of good work only increases as the need to filter out unlimited low quality output from someone takes more effort than they put into creating it.

It's not just coding​

There are analogies to other industries.

Building has become more efficient than ever, but housing remains as expensive. Clothing has become cheap to make, but quality has deteriorated and it gets thrown away.

Vice coding fits perfectly into society today of fast fashion. Your code will only last the season before falling apart. Sometimes it just goes straight to the bin.

It makes things more accessible, but increases the amount of energy wasted along the way.

The vybe chicken is born​

The flood of code has created the vybe chicken. Vybe chicken don't care. Vybe chicken ships 10 features a day.

Vybe chicken doesn't care about what thought others put into a bit of code. After all, the others probably didn't think, so why should it. Vybe chicken does what it's told, but no more.

After a while, vybe chicken spends its day only fixing bugs. Feature development slows down. Vybe chicken doesn't understand why.

This is the chicken's life. Vibe coding bugfixes for features which have been vibe coded. All to just ship it bro.

Where to​

This is not a new problem. Everything and timelines have just been amplified, and without discipline will only get worse.

Humans have developed tools to help them since the dawn of time. Our tools get more advanced, and each comes with additional skills and education required to master them. At university, we learned fundamentals before we learned the higher level stuff. Not because it was useful, but we understood it was necessary.

I've very rarely needed to use the direct fundamentals in real life, but they still help one use the more advanced tools.

Currently, these models are no different. Sure for simple stuff it doesn't matter, and it's addictive to feel productive. Eventually everyone will catch up with what it really means to be productive given the current tools, and a balance will be found.

Or it doesn't matter as the models get so good and all these issues become irrelevant. Nobody knows.

This was my first hackathon in many years and it was incredibly well organized by Factory Network and {Tech: Berlin}. The hackathon was sponsored by Mistral, Weaviate and LumaAI who provided free credits for their platforms.

I have experience with natural language processing in my previous role as a data science lead, but hadn't spent much time playing around with current LLMs. These days, I mostly use them as a coding assistant. It was fun to get to test out the latest LLM, image understanding and video generation models to discover what works well (and what doesn't).

We had written down a few ideas before the event but after a bit of brainstorming we settled on building a ghostwriter. The idea was inspired by Axel who had someone write a beautiful story about an event in his life, which he can fondly look back at forever. Most of us don't have the will or the time to write down our experiences. When we do write them down and read them months or years later they are a pleasure to have and bring one back into the memory. I have personally started writing down summaries of any trips I go on but unfortunately often forget to or am just lazy to do a proper job.

I usually make time to write about my day to friends or family, typically sending along a few pictures that highlight my experiences throughout the day. Messaging in short snippets is easier than writing a perfectly written story. While messaging, we don't let perfect get in the way of good enough so we are more likely to do it.

Building​

The aim, divide and conquer. Ferdi, Axel and myself broke our tasks down into separate chunks which we could stitch together. We broke the problem down into a picture summarizer, an interviewer, and a writer. As a bonus (and because we could use the API for free) we added a step to generate a 50-word description of a video to send to the LumaAI's Dream Machine model to generate a 5 second video.

Each stage builds upon the previous one, and the aim of each step is to capture information to help with the next task.

Here is a general overview of the flow:

We had some trouble getting the model to say when the interview should end, so to save time we just added an 'End conversation' button that the user could press when they got tired.

 The code is available in my GitHub if you wish to play around with it. To run it locally, you will need a Mistral API key, which you can get for free from their website.

The final product​

We ultimately deployed the final product to Streamlit . Feel free to give it a try and let me know if you have any feedback. Please note this is hackathon quality, and was made in 24 hours!

Here are some screenshots of the various steps:

The interview:

Pick your story adventure:

The output​

Here is an example story that I made up and is completely fictional, but based upon my made up answers. I used the Tesla marketing images for the Fremont factory. The pictures are automatically placed alongside the relevant part of the story.

Video​

Here are some sample videos generated by the Dream Machine. They were really a bit of a hit or a miss. The less realistic the video, the better it appeared to be.

Conclusion​

We made it to the final (Top 6 of 25 teams), and really impressed the judges in our first demonstration. Our project was one of the few that actually worked, and was developed from scratch during the hackathon.

The final pitch went well, but the judges had some reservations about the amount of hallucinations that the models could generate. This could be reduced with better prompting and having more information added in the interview. Overall we were happy to make it to the finals, but it was bitter-sweet to leave with nothing. Maybe we could have done with a product or marketing team mate who could have sold the idea better.

Overall, there is something that could be useful here, and probably needs more than 24 hours to solve. It needs to be made more convenient to use and to gather more information to limit any hallucinations.

Convenience leads down a big privacy path due to the nature of having 'someone' follow you around and ask questions about your experiences in your life. It would need to be built in a way that doesn't stream your entire life to an unknown server. Ignoring that, ideally it should be something that could read the information you are already capturing, built into your existing flow of communication. It could summarize messages, photos and possibly even your chats about a topic with various contacts, but only extract the relevant bits for the story you would like to tell.

I hope you enjoyed the read, and please let me know if you have any comments!

This post was inspired by DHH's “Merchants of Complexity”.

Over the years, I've seen many teams over-engineer their dashboards. Don't do this. What you want is something that meets your requirements, is as simple as possible and as complex as necessary.

In this post, I'll explain my approach to building performant dashboards that are easy to develop and maintain.

Introduction​

There are many tools available for creating dashboards. They range from easy to use code-free tools, to fully custom made "solutions" built with significant investment.

The easy-to-use tools are great for simple reports with limited interactivity. They also don't require a development team. For many teams these will be enough.

On the latter end of the scale, a typical approach looks like a custom backend that interfaces with various data sources and then uses a custom React frontend. This is great if you have infinite time and resources with relaxed deadlines. But let's be real. These solutions often don't live up to their initial goals, and any changes that need to be made require input from the entire development team.

I've personally settled on something in the middle for my own projects. Plotly Dash. It's a versatile and simple solution for creating highly interactive dashboards.

Problem​

In the real world, businesses typically need to see a report on something promptly. Time is money, after all, and not having visibility into a critical process costs you money each day. Businesses might also not know exactly which metrics they need to see. They just know that they need to see something, and something is better than nothing. The development process will have a lot of back and forth, with a lot of iteration until finally settling on a solution. The key point is providing value along the way and not waiting for once everything is complete.

Solution​

Plotly Dash allows one to develop reports quickly and present them to your users within a few days, if not hours. One can iterate to provide something useful within a very short time frame. Version control and multiple environments can be set up so that any changes can easily be rolled back. To sum it up, you can move quickly and break things.

In addition:

• Dash looks good enough with sensible defaults.
• Custom components enable your app to follow your companies design guidelines.
• Fast, especially with caching.

Dash allows you to build interactive dashboards quickly, at scale and provide benefits incrementally as you build.

Dash does require someone that can program Python and have some basic understanding of web development. However, it is easy to learn and get started with. It can be as complex as needed and doesn't require a huge amount of boilerplate to get started. If you are more experienced, you can still use all your web development skills if you happen to have them. In my opinion, it is a huge benefit when one can have something that satisfies both the beginner and the expert, with both feeling challenged and productive.

Typical project layout​

Let's get into how to structure your Dash project and go through a few common issues that you might encounter.

While a Plotly Dash project is simple to set up, it still requires some overhead. A typical (maybe slightly more advanced) layout might look something like this:

assets
components
db
config
pages
redis
  - Dockerfile
  - redis.conf
prometheus
schemas
tests
utils
.dockerignore
.gitignore
.pre-commit-config.yaml
.python-version
config.env
docker-compose.yml
Dockerfile
gunicorn.conf.py
main.py
odbcinst.ini
poetry.toml
poetry.lock
pyproject.toml
README.md
secrets.env (Don't add to git)

Configuration​

For all my projects, I follow the Twelve Factor App principles. Keep your config out of your code.

Application config and secrets​

For any Python application, the Pydantic Settings Management works wonderfully to load your config from environment variables and validate that your config is correct. This helps debug deployment issues and also helps document what should go into each environment variable.

Managing data sources and database connections​

Dash allows you to use data from any source. Databases, APIs, CSV files - it doesn't matter.

In some applications I've developed, it's been necessary to connect to many different databases. This can get quite overwhelming, but a simple pattern can help manage all the connections.

Firstly, the database connection details are defined in your Pydantic Config. After the config has been initialised you can set up the various connections using connection strings. In this case I'm using SQLAlchemy which allows connection to many databases with a common interface.

To standardize the connection, create a simple Pydantic model.

from pydantic import BaseModel

class SQLAlchemyBind(BaseModel):
    """SQLAlchemy bind config.
    This object is a generic representation of SQLAlchemy bind configs.
    """
    url: str
    isolation_level: str | None = None

In the config file, define all your database connection strings.

config = DashboardConfig(_env_file=["config.env", "secrets.env"], _env_file_encoding="utf-8") # type: ignore

config.sqlalchemy_binds = {
	"mysql_db": SQLAlchemyBind(
		url=f"mysql+pymysql://{config.mysql_user}:{config.mysql_pass.get_secret_value()}@{config.mysql_host}:{config.mysql_port}/{config.mysql_name}",
		isolation_level="READ UNCOMMITTED",
		),
	"clickhouse_db": SQLAlchemyBind(
		url=f"clickhouse://{config.clickhouse_username}:{config.clickhouse_secret.get_secret_value()}@{config.clickhouse_host}:{config.clickhouse_port}/{config.clickhouse_name}",
		isolation_level=None,
),

To create and collect all the database sessions, create an init_sessions function which will store all the database sessions in a dictionary as a singleton.

You can then access a particular database session using database_sessions[bind].

In db/__init__.py:

database_sessions: dict[str, scoped_session] = {}

@contextmanager
def query_db(
    query_path: str, bind: str, params: dict = {}
) -> Generator[list[dict[str, Any]], None, None]:
    """Executes the given query and returns the results.
    Parameters
    ----------
    query_path: str
        Path to the query file
    bind: str
        sqlalchemy bind id
    params: dict
        Parameters to render in query
    Yields
    ------
    list[Dict[str, Any]]: query results
    """

    try:
        with open(query_path, "r") as f:
            query = f.read()
    except FileNotFoundError as e:
        logger.error(f"Error opening query file {query_path}: {e}")
        raise
    except Exception as e:
        logger.error(f"An unexpected error occurred while opening query file {query_path}: {e}")
        raise

    db_session = database_sessions[bind]
    session = db_session()

    try:
        logger.debug(f"Query {bind}: {query_path} executed with params {params}")
        results = session.execute(text(query), params=params).mappings().all()
        # Convert SQLAlchemy Row to dict for compatibility
        yield [dict(row) for row in results]
    except Exception as e:
        logger.error(f"An error occurred while executing query {query_path}: {e}")
        raise
    finally:
        try:
            db_session.close()
        except Exception as e:
            logger.error(f"An error occurred while closing the database session: {e}")

def init_sessions():
    for bind in config.sqlalchemy_binds.keys():
        engine = create_engine(
            url=config.sqlalchemy_binds[bind].url,
            isolation_level=config.sqlalchemy_binds[bind].isolation_level,
            pool_recycle=3600,
            pool_pre_ping=True,
        )
        session_factory = sessionmaker(bind=engine)
        db_session = scoped_session(session_factory)
        database_sessions[bind] = db_session

I personally put all queries into .sql files structured according to the database name. The queries can be linted and formatted with SQLFluff.

Then you can use the query using the template:

db/clickhouse.py

@cache.memoize(config.cache_timeout_seconds)
def get_ultimate_question(
    parameter_one: str
) -> pl.DataFrame:

    with get_result(
        query_path="db/queries/clickhouse/some_amazing_query.sql",
        bind="clickhouse",
        params={
            "parameter_one": parameter_one,
        },
    ) as results:
        ultimate_question = pl.DataFrame(
            results,
            schema={
                "name": pl.Utf8,
                "date": pl.Date,
                "hour": pl.Int8,
                "answer": pl.Int16,
            },
            orient="row",
        )

		# Any other logic you need in Python...

    return ultimate_question

Caching and background callbacks​

Caching is important for any Plotly Dash app to ensure good performance. It limits recomputing things unnecessarily.

Because Plotly Dash is based on Flask, one can just use the excellent Flask Caching library.

After setting it up, function calls are cached with the @cache.memoize decorator. In practice, this usually means adding the decorator to any database query functions to cache the database queries but any function can be cached using this pattern.

The cached data can then be used in multiple callbacks, without having to make repeat database queries.

Caching goes hand in hand with using background callbacks to execute queries that take longer than a few seconds. Functions that query the database are called first in the background callback. Once the background callback is complete a subsequent callback can call that same function and get the result almost immediately.

This leads onto the next topic...

Background callback signalling​

Signalling is the process of letting one callback know that the long running background callback is complete.

The general pattern goes like this:

The general steps of the code below are:

1. Add a dcc.Store component to our layout.
2. Create a background callback that calls the expensive function and outputs a "signalling" value to the dcc.Store when it's complete.
3. Create another callback that uses the dcc.Store as an input so that it is only triggered once the expensive function is complete.

# Add to layout
data_signal_store = dcc.Store(id="data-signal-store")

@callback(
    Output("data-signal-store", "data"),
    Input("date-picker", "date"),
    background=True,
)
def fetch_data(datestr: str) -> str:
    """Fetches the data in the background
    """
    _date = datetime.strptime(datestr, DATE_FORMAT_STRING)

	# Takes a long time to execute
    data_df = get_clickhouse_data(date=_date)
    
    return f"signal-{start_datetime.strftime(DATE_FORMAT_STRING)}">)

@dash.callback(
    Output("output-one", "children"),
    Output("output-two", "children"),
    Input("data-signal-store", "data"),
    State("date-picker", "date"),
)
def update_widget(
    board_state_signal: str,
    datestr: str,
) -> tuple[str, str]:
    """Updates the widgets with the data
    """
    _date = datetime.strptime(datestr, DATE_FORMAT_STRING)

	# Gets fetched from cache
	data_df = get_clickhouse_data(date=_date)

	# Other logic

	return widget_one, widget_two
    

In this way, we can ensure the long-running tasks always execute before the main callbacks as Dash will only execute the update_widget callback once the background callback is complete.

Authentication​

As Dash is based upon Flask, you can just use Flask Login to manage your Auth.

Create routes for your particular use case such as /login, /unauthorized, /auth on the Flask server directly …

@server.route("/login", methods=["GET"])
def login():
    next_url = flask.request.args.get("next")

    # Validate that next is a valid url and has the same host as the server
    valid_next_url = validate_next_parameter(flask.request, next_url)
    if not valid_next_url:
        return "Invalid URL for redirect", 400

    flash("You have to be logged in to access this page.")

    logger.info("Logging user in...")
    # Custom login logic

    return redirect(f"{url}/?next={next_url}"

... then add authentication to routes by checking whether the user is logged in before every request. Here is an example:

@server.before_request
def before_request_func():
    # Allow the auth routes
    allowed_paths = [
        "/alive",
        "/ready",
        "/auth",
        "/unauthorized",
        "/login",
    ]

    if not current_user.is_authenticated and not any(
        request.path.startswith(path) for path in allowed_paths
    ):
        next_url = request.url
        valid_next_url = validate_next_parameter(flask.request, next_url)
        if not valid_next_url:
            return "Invalid URL for redirect", 400

        return redirect(
            f"/login?next={next_url}",
        )

Embedding your application in an iFrame​

I sometimes have the case where users would like to embed certain graphs in Confluence or some other tool. To achieve this, one needs to add the website to X-Frame-Options and Content-Security-Policy in the request headers. To read more about these policies see Content Security Policy and Same-origin policy.

@server.after_request
def add_headers(response):
    response.headers["X-Frame-Options"] = "ALLOW-FROM https://confluence.com/"
    response.headers[
        "Content-Security-Policy"
    ] = "frame-ancestors 'self' https://confluence.com/;"
    return response

Logging and metrics​

I like to use Loguru for logging and Prometheus for metrics. You can then use your companies usual monitoring tools such as Grafana or ELK to monitor the usage of your application.

Customizing components​

Many companies have their own design system and typically want any developed applications to match their design. With Plotly Dash, you can relatively easily convert your companies components (especially if they are React components) to Dash components. They even provide a template to start with.

Alternatively, if you just want something that looks good without too much customization Dash Bootstrap Components or Dash Mantine Components are good alternatives to the dash core components.

Deploying​

Deploying a Dash app is pretty easy. To include caching and background jobs, it requires just three containers. One for the Dash app itself, one for the background jobs and one for Redis.

Where to deploy is a matter of personal preference. I've always deployed to Kubernetes using a CI pipeline for seamless updates but docker compose would work just as well.

Local testing with Docker​

This can all be tested locally with docker.

docker-compose.yaml

services:
  myapp:
    platform: linux/arm64
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 8080:8080
      - 8050:8050
    volumes:
      - ./config.env:/app/config.env
      - ./secrets.env:/app/secrets.env
      - .:/app:delegated
      - /app/.venv
      - prometheus_data:/prometheus
    environment:
      - GIT_COMMIT=latest
      - GIT_BRANCH=latest
      - BUILD_TIMESTAMP=now
      - PYTHONUNBUFFERED=1
      - USE_REDIS=True
      - REDIS_DSN=redis
    command: ["gunicorn", "main:server", "--reload", "--workers=1"]
  dash-celery:
    platform: linux/arm64
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - 5555:5555
    volumes:
      - ./config.env:/app/config.env
      - ./secrets.env:/app/secrets.env
      - .:/app:delegated
      - /app/.venv
      - prometheus_data:/prometheus
    tmpfs:
      - /tmp:exec,size=100M
    entrypoint: ["celery"]
    command: ['-A', 'main:celery_app', 'worker', '--loglevel=debug', '--concurrency=2', '--without-mingle', '-n', 'worker1', "--prefetch-multiplier=1"]
    environment:
      - GIT_COMMIT=latest
      - GIT_BRANCH=latest
      - BUILD_TIMESTAMP=now
      - USE_REDIS=True
      - REDIS_DSN=redis
      - PROMETHEUS_MULTIPROC_DIR=./prometheus
  redis:
    build:
      context: ./redis/
      dockerfile: Dockerfile
    ports:
      - 6379:6379
volumes:
  prometheus_data:

Adding pages and creating direct links to reports​

Dash supports multiple pages, so the different pages go into the pages folder.

Once the initial overhead of setting up the database connections, deployments and creating a few sample pages even inexperienced developers and analysts can create new pages using the existing ones as a template.

It's as easy as copying an existing page from your pages folder and editing as needed.

Sharing views​

Sharing reports with others can be done using query parameters as long as you ensure all callback inputs are parametrized in the URL. This means you can share the exact view by copying the URL.

# Add to the header in your main layout
dcc.Location(id="url", refresh=False),

# In each page create the following callback
@dash.callback(
    Output("url", "search", allow_duplicate=True),
    Input("input-one", "value"),
    Input("input-two", "value"),
    prevent_initial_call="initial_duplicate",
)
def update_url_state(
    input_one: list[str],
    input_two: int,
) -> str:
    """
    Updates the URL query parameters based upon the input values
    """

    params = ""
    if input_one:
	    #Serialize any objects to json
        params += f"input_one={json.dumps(input_one, separators=(',', ':'))}"
    if input_two:
        params += f"&input_two={input_two}"

    if params != "":
        return f"?{params}"
    return ""


# Define the layout for the page with the parameters and inputs
def layout(
    input_one: str | None = None,
    input_two: int | None = None,
) -> dash.development.base_component.Component:

	# Deserialize the object from json in the query parameter
	_input_one = []
	if input_one:
		_input_one = json.loads(input_one)

	... other layout

Conclusion​

Dash is a great tool to create easy to update and maintainable reports. Once the initial overhead of configuration and deployment is done, adding and updating reports becomes trivial. If someone in your company is proposing to create over-engineered reporting infrastructure, pause for a moment and ask yourself whether it is really needed.

In many cases, something simpler might be all that's required.

You might find that all you need is Dash.

Please let me know if I have missed something or made any mistakes.

Introduction​

Across different industries there are many ways to measure productivity. In manufacturing, OEE is the metric most commonly used to know how productive your factory is.

OEE is a subset of the Total Effective Equipment Performance (TEEP) which is defined as the total amount of units a line could produce if it ran perfectly all the time.

There is a great resource online at oee.com to learn about OEE. I suggest you read it thoroughly. This post will act as a summary with practical examples to understand OEE.

Why do we measure OEE?​

Measuring OEE allows us to answer a few important questions when running a production line.

1. How well are we performing compared to the ideal case?
2. Why didn't we hit our production targets?
3. What can we do to increase the productivity of our line?

These three questions can be broken down further, depending on whether it is an engineer or manager asking the question.

The key outcome, is helping one figure out where to allocate resources to get the biggest increase in productivity. We want more product out for less money in.

OEE Definition​

OEE has the following mathematical definition:

OEE can be defined as the ratio of good units produced to the theoretical number of units that could have been produced in a given amount of working time. This means that OEE can never exceed 100% and is bound within the range of 0 to 100. In comparison to TEEP, it excludes any losses due to your scheduling and only considers planned production time.

Any reduction, from the ideal 100%, in OEE is caused for a particular reason. These are known as losses, or reasons why we couldn't produce at 100% of our capacity. The common losses are typically:

• Availability loss - Production lost due to the equipment not being able to produce.
• Performance loss - Production lost due to slow cycles.
• Quality loss - Production lost due to poor quality/scrap.

Graphically, we can visualize everything we have defined with the figure below:

Allocating losses​

OEE is often represented as the product of availability, performance, and quality.

Frequently, people confuse the two methods of calculating OEE as two separate definitions. The value of OEE is always the same with both methods of calculation. Specifically, the breakdown into A, P and Q is only used to allocate the reasons for an OEE reduction to a specific category so that you know what needs to be changed to improve OEE the most. If you get different values for OEE with the two calculation methods, you have made a mistake and are ignoring some unknown losses.

Availability, performance, and quality are defined as:

Multiplying all of these together, results in the initial OEE formula defined above.

Line vs equipment OEE and loss breakdown​

The OEE losses will be measured differently depending on whether you are measuring it at the equipment or line level.

For the equipment level measurement, the OEE losses allows you to determine how the specific equipment is performing and understand the reasons why. As an example, the machine might be operating perfectly, but it does not get enough parts coming in due to an upstream process issue. Equipment OEE enables you to understand the root causes for lost output on a specific piece of equipment.

For line level measurement, the OEE allows you to compare how the entire line, department, or factory is doing. Line OEE enables you to determine where the bottlenecks are that prevent you from increasing your output of the complete end product.

OEE example​

Let's work through an example. We are going to calculate OEE for something we are all familiar with… A road trip.

We are going on a road trip to our holiday destination 1000 km away.

To start, we define our TEEP for our car. We can define the limit of TEEP of our car as if we drove it 100% of the time at the speed limit with every kilometer driven taking us closer to our destination, with no problems, maintenance or need to stop for a charge. Obviously this is impossible, but theoretically a car is being wasted every time it is not used or drives a kilometre that doesn't need to be driven. All roads should point directly to our destination.

For our road trip, we plan to drive 1000 km in a single day and assume a 100 kmh average speed limit (To keep things simple) so we could drive 2400km a day if we drove 100% of the time.

Therefore, we can define our utilization or TEEP with ideal OEE as:

This means, that assuming our car works perfectly all the time on this trip, we will get a utilization of 0.42 for the day.

As a side-note:

Typically, a car will be used for less than 2 hours a day, and often driven slower than the ideal speed. Car sharing companies effectively hope to make money by increasing this utilization rate of a car. They make more use of each car in a given time (And pass on any losses while the car is being used to the customer).

If we want to calculate the OEE of our trip, we can define this as:

Now we want to figure out how we could have reached our destination sooner. We ended up taking an additional 3 hours to reach our destination, as if we could have driven at the speed limit without stopping the entire way.

To do this, we want to determine our losses. Why did we take longer to reach our destination than planned?

Quality loss of our road trip​

In our case, we can specify a good kilometre as a kilometre travelled that brings us closer to our destination. Extra kilometres needed to get to a charging point, stop off for snacks or do some sightseeing add kilometres to our trip and don't bring us closer to the destination.

Let's say we had to do an extra 50 km. There were a few reasons for this:

• There was a nice tourist attraction on the way that the family desperately wanted to see
• We took a wrong turn, driving in the wrong direction because the kids were playing with the navigation
• We took a slight detour to a get to a closer charging point because we misjudged how far we could travel on a charge
• The roads are not a straight line from our house to our destination

We can therefore define the quality of our trip as:

Availability loss of our road trip​

While driving, we needed to stop to supercharge a few times, change a tyre, eat, balance our fluid levels and spend time enjoying the view.

We needed 1.5 hours to charge and managed to fit our eating and fluid level balancing into this time. Unfortunately, we spent 30 minutes changing a burst tyre, because we forgot to check the tyre pressure before the trip.

As this is a road trip, we spent some time enjoying the view. Do you think we should allocate this loss to availability or performance? In this case we are calculating the OEE for the car and the car was available to drive. It is you, the operator, that spent some time to enjoy the view. We therefore will allocate it as a performance loss in the next section. The car was starved of its driver.

Taking all of this into account, we can define the availability of the trip as:

Performance loss of our trip​

As mentioned before, we spent some time enjoying the view, but we also encountered some other issues on the way.

We ended up driving some kilometres where we couldn't drive at the speed limit due to road works. We also hit some stop start traffic. We were blocked.

We will allocate all of these other losses as a loss of performance:

We can also calculate the performance using the known OEE, availability and quality of our trip.

How could we improve our trip?​

Now that we know where our losses are, we can focus on how to improve our trip and reach our destination faster next time. There are some things that we wouldn't change (It is a road trip, after all).

What would you focus on improving first? As a reminder, our OEE is:

It is quite clear that the availability is the main reason why our trip took longer. How could we improve this?

Well, it turns out we didn't start our trip with a full charge, so we spent more time charging than we needed. We also didn't do the required maintenance like checking tyre pressure before the trip, so when we hit a pothole, the tyre burst. If we did the required checks before going on our trip, we would have saved 30 minutes because of the tyre change and at least 20 minutes from one less charge. A saving of over 50 minutes.

This would bring our OEE up to:

A nice 5% improvement.

Because this is a road trip, we don't really want to spend less time enjoying the view but would rather try to increase our performance some other way.

How could we do this? Well, you might point out that we could improve our performance by driving faster - over the speed limit 😱. This might work for a short time, but the police are waiting for you and end up pulling you over. This creates an unscheduled stop in your trip and the police take 30 minutes to write you your ticket. Costing you both money and time.

Some things like traffic, and road works are unavoidable, but we might be able to leave at a different time or plan better which means we can drive for longer at the speed limit.

Lastly, we could improve our quality. Here we could have saved distance by planning better again. We could make sure we know our route beforehand, and perform checks more often before we drive too far in the wrong direction.

Conclusion​

We have learnt how to use OEE using a practical example and now have an intuitive understanding of how we can use it to improve our production processes.

Go ahead and use these concepts to calculate OEE for your process, business, or even your cooking. The concept of OEE can be applied to many industries, and could help numerous businesses improve their efficiencies.

• How well do we utilise our kitchen (Restaurant)
• How many seat minutes are filled (Airlines, movie theaters)
• Utilization of your server (Software companies)

Feel free to leave any comments, and I look forward to discussions on how you applied OEE to make your business (or your cooking…) more efficient.

I'm in a technical role with some direct reports and like having a manager that knows at least something about the process that their team is working on. They do not need to know every detail, but they should know how to ask the right questions. Knowing which questions to ask, usually results from having some experience in the role. It is a painful experience watching someone try to manage a team without this quality.

Ideally, you want to be able to learn something from your manager. Their experience should be yours for the taking. You should be able to learn new ways of thinking, dealing with people and approaching problems with incomplete information.

A manager should build a working environment encouraging people to move if needed for personal growth, but do things for their team to make sure they don't want to move. This could involve supporting their employee's personal ambitions within the company, providing a budget for education and learning, or encouraging collaboration with other teams that the employee strives to be a part of.

Everyone typically wants the best for themselves. Ambitious people want to perform exceptionally on what they are working on. Managers should foster an environment that allows people to discuss things openly, without worrying if there could be negative consequences. Feelings of fear and personal consequences results in people not speaking up when they should. There should be a focus on the solution without fear of consequence for the person that raises it.

A manager should encourage people and show appreciation when they do a good job. They should avoid constant negative feedback when someone's giving their all but still struggling on something difficult. Bringing the person down might push them to solve the immediate problem, but it will result in resentment and hate.

Negative people should be removed from the team. A manager should identify these people that are a bad influence on those around them, and remove them if they manage to slip into the team.

This guide explains how I managed to set up a custom domain with my existing Gmail inbox, all without a monthly cost. The only cost is a domain if you don't already have one.

The goals:

• use your existing inbox with a custom domain,
• send and receive emails with the custom domain,
• be completely free (besides for the domain cost)

Things you need​

• An existing email account, I use Gmail.
• A Cloudflare account.
• A domain name using Cloudflare as your name server and DNS. Transfer your domain to them or buy one using the Cloudflare registrar.
• An email SMTP server such as Brevo, SendGrid or another of your choice. These allow you to send between 100 and 300 emails a day for free which should be fine for personal purposes.

Enable email forwarding​

The first step once you have your domain registered is to enable email forwarding. Do this within the Cloudflare admin page for your domain. After setting email forwarding up, email sent to a youremail@yourdomain.com is forwarded to another email address of your choice. You can also configure a 'catch all' address so that any emails sent to your domain will be forwarded to a specific inbox.

• Go to Email on the left menu bar.
• Enable email forwarding for your domain.
• Add the email DNS settings required by Cloudflare. This can be done automatically.
• Add a destination address, and verify it.
• You can either use a catch-all which forwards every email sent to your domain, or only forward specific email addresses.

In the end, you should have something that looks similar to this:

Enable email sending​

Now that you can receive emails on your domain, it's time to enable sending email.

This part will be done within your SMTP provider and Gmail (or another email provider of your choice). What we want to achieve is being able to send emails, and maximise the probability of them being delivered to the expected recipient.

You need to have decided on a service that allows you to send emails. As previously mentioned, I use Brevo.

SMTP service setup​

The first step is to set up the SMTP service to allow it to send emails on your behalf. On Brevo, the steps required look something like this:

• Authorize/validate your domain under Senders and IPs by adding all the records from the email sender to your Cloudflare DNS page.
• Configure SPF, DMARC and DKIM

Brevo will automatically setup DMARC and DKIM, but you may need to manually set up SPF and adjust the strictness of the email delivery.

Ultimately we want our DMARC management page in Cloudflare to look something like this:

SPF​

The Sender Policy Framework (or SPF) allows you to specify all IP addresses and hostnames that are permitted to send email on behalf of your domain. If you don't configure SPF on your domain your email may be blocked or marked as spam by email receivers.

For this to work, we need to add both Cloudflare and Brevo to the SPF record:

"v=spf1 include:_spf.mx.cloudflare.net include:spf.brevo.com mx -all"

This new SPF record now replaces the old record. It allows both Cloudflare and Brevo to send emails using your domain name, while all other senders are blocked using -all.

The options for controlling the strictness of the SPF record include:

• +all - will allow all emails to be received even if not sent by a trusted sender.
• ~all - will soft fail, so emails will be marked as spam if a trusted sender does not send them.
• -all - will reject all emails not sent by a trusted sender.

DMARC​

Domain-based Message Authentication Reporting and Conformance (or DMARC) tells an email receiver how to treat incoming emails from your domain that fail email security checks. You can also specify a reporting email address to receive regular reports about outgoing emails on your domain.

We want to specify that we want to reject all email that fails the DMARC authentication checks.

Create a TXT record in the DNS of your domain with the following value replacing the rua: mailto with the settings for your domain:

Record type: TXT Name: _dmarc Value: "v=DMARC1; p=reject; rua=mailto:<somehash>@dmarc-reports.cloudflare.net,mailto:rua@dmarc.brevo.com"

The p=reject means that emails will be dropped if they do not include the DKIM and SPF as configured for your domain.

DKIM​

DomainKeys Identified Mail (or DKIM) adds a public key to your DNS for email receivers to verify your outgoing signed email.

Brevo and Cloudflare will automatically create this.

Gmail setup​

Now, it's finally time to configure your Gmail inbox to be able to send and receive your emails from your domain account.

• Open Gmail settings and go to the Accounts and Import page.
• Click Add another email address.
• In the popup window, leave treat as Alias enabled and enter your custom email address that you wish to send from.
• In the SMTP page, fill in the SMTP settings from the SMTP provider you set up earlier.
• Validate the email address. This allows you to test that everything is working.
• Optional: Set this as your default email address in Gmail.

You can send and receive from as many email addresses as you like on your own domain.

Prevent your custom domain email from being detected as spam​

If you followed the instructions above, most of these steps should already be completed. In case you are having trouble, double-check that all these records exist:

• Have an MX record for your domain, so you're able to accept responses.
• Check the SPF, DMARC and DKIM record for your domain.
• Check to see if your sending IP address is on any blacklists. Get it off if you can.

Custom domains are often flagged by the spam filters on many email services so it might take a bit of time for your domain to become trusted if you recently registered it.

Conclusion​

Congrats, you should be able to send and receive your email on your custom domain. Completely for free!

Thanks for reading, and let me know if you encountered any issues.

I had successfully set up port forwarding, dynamic DNS and had a domain for the service without the VPN. I just couldn't figure out how to get it to work with the VPN. I had accepted that sometimes I wouldn't be able to access the service remotely.

I use Cloudflare as my DNS, and use their pages service to host this site. After browsing through their numerous services, they offer, I stumbled across Cloudflare Tunnel.

Cloudflare Tunnel

Cloudflare Tunnel exposes applications running on your local web server on any network with an internet connection without manually adding DNS records or configuring a firewall or router.

Bingo! Exactly what was needed.

I followed Cloudflare Tunnel · Cloudflare Zero Trust docs and within 30 minutes the service was accessible, with and without the VPN.

Now I have my service accessible at a fixed URL, reliably and without having to mess around with my router and Dynamic DNS.

It is based on the Coursera course: https://www.coursera.org/learn/leadership-collaboration

Leadership development​

Nature versus nurture​

Are leaders made or born?

There are two dominant theories of leadership: entity theory, also known as trait theory and incremental theory.

Entity theory argues that leadership is a trait, that one is born with. Incremental theory argues that one can learn to be a leader. I personally subscribe to the incremental theory of leadership. One can be born with certain traits that make one more naturally inclined to become a leader, but leadership skills can be grown and developed through hard work.

Mindset and growth​

Do you have a growth or fixed mindset? Do you believe that you can change who you are and improve your skills with work and practice or do you believe that you are who you are and anything you are good or bad at must stay that way? That is the difference between a fixed and growth mindset.

Growth mindsets are much more successful long term and should be nurtured in yourself and others. People with fixed mindsets don't like trying new things because the risk of failure implies that they aren't good enough. In contrast, people who have a growth mindset, see themselves and others as having potential. New things present an opportunity to learn and grow.

Have a growth mindset!

Leadership styles​

There are two main theories of leadership styles. Theory X leadership and Theory Y leadership.

Theory X leadership is mostly transactional. You do something and I'll give you a reward at the end, like a carrot dangling in front of a rabbit.

Theory Y leadership is based upon the idea that people want to contribute and do excellent work.

In my view, one needs to combine these leadership styles. People should be sufficiently rewarded monetarily, especially when they are contributing significantly to your own or the company's profits. However, the reward should not be the thing that motivates them to work for you. The reward should be a by-product of the intrinsic motivation to do something useful and meaningful.

Motivation, emotion, and cognition​

Intrinsic and extrinsic motivation​

Are you intrinsically or extrinsically motivated? What about your colleagues?

Often people believe that they themselves are intrinsically motivated and believe that others are extrinsically motivated. This obviously makes no sense, as why would most people believe that others are motivated by money, while themselves having some other internal motivation for what they do.

It is important to nurture and develop the intrinsic motivation in your colleagues and subordinates. You will get far better work from them.

Happiness​

You are either a happy person or not, right? Well, not exactly. According to some rough estimates about 50% of your happiness comes from your genetics, 40% from your intentional activities and 10% from material wealth.

You could take a few views from this:

1. If you are born an unhappy person that you are cursed for life.
2. If you consciously work on some practices, you can make yourself happy.
3. Without the other things, money can't buy you happiness, but it can certainly help.

Let's focus on the 40%. Few people are born with no intrinsic reason to be happy and if the 40% of intentional activity can be controlled, there is a good chance of being happy overall. Focus on behaviours and practices that we do voluntarily such as helping others, adopting a positive view, exercise and have goals you can work towards.

Decision making​

Most people believe that they are good decision makers. They are overconfident in their decision-making prowess. This leads to bias in the decision-making process. Let's discuss a few of these biases and how to overcome them.

1. The framing effect - The tendency for us to be risk-adverse regarding positive outcomes and risk taking regarding negative outcomes.
2. Confirmation bias - The tendency to actively seek out information that confirms your pre-existing viewpoint.
3. Decision fatigue - Making too many decisions, including insignificant decisions causes us to make poorer decisions when it really matters.
4. Team scaling fallacy - That by adding members to your team, you can solve the problem with linearly increasing efficiency.
5. Common information effect - The tendency for people in team to talk about the things that they all know, and not the things that they individually know which would add value.

Wherever you are working, it is important to be aware of these biases and work together to not fall prey to their effects.

People skills​

Emotional intelligence​

Think of your best boss. What characteristics stand out?

The things that stand out are likely not that they are intelligent, or that they had excellent grades at school. You probably thought about their people skills. Leaders are judged, not only on how clever they are, but also how they handle themselves and other people.

The skill related to understanding yourself, and others is known as emotional intelligence.

What are the implications for leadership? Emotional intelligence starts with self-awareness. People who are high in self-awareness are more empathic. You can improve your ability to self-regulate with rehearsal and practice and being more self-aware allows you to have more understanding for others.

Organisational intelligence​

People who span organisational divides and integrate knowledge and best practices around the organisation are much more useful than people that work in silos.

They connect people and ideas.

You want to have a network that is large and spans many areas of knowledge within your company to maximise value. This is known as organisational intelligence. Good leaders encourage this interconnected behaviour and themselves have good relationships across the organisation.

Cultural intelligence​

Cultural intelligence is a person's capability to adapt effectively to new cultural contexts.

The risk of talking about culture is that we may falsely stereotype people. A stereotype is a faulty belief that everyone from a given group or given culture is the same.

In general cultures across the globe can be defined into three main groups: face, honour, and dignity. It is important to be aware of other people's cultures and identify the overarching traits of theirs.

Face cultures are primarily East Asia. People from face cultures derive their self-worth from earning the respect of others.

Dignity cultures are primarily Europe North America, Australia, and New Zealand. People who embody the dignity culture believe strongly in speaking their mind and being self-reliant.

Honour cultures are primarily in the Middle East, South Asia, North Africa, and Latin America. Self-worth in honour cultures is socially claimed.

The multiple intelligences test​

Howard Gardner's theory of multiple intelligences challenges the traditional view that intelligence is a unitary capacity. Gardner proposes that there are at least eight intelligences that can be identified, and they influence how we learn, think, and interact.

Your leadership style is influenced by your own multiple intelligences. Gaining self-awareness of your own multiple intelligences can help you motivate and influence others more effectively.

You can do the Multiple Intelligence Survey yourself and read more about the The Components of Multiple Intelligences.

Here are my results.

The results show that my top three intelligences are logical-mathematical, bodily-kinaesthetic, and intra-personal. My weaknesses are linguistic and naturalistic.

It comes as no surprise that my top intelligence is logical-mathematical because of my role as a data scientist and engineer. In these roles it is critical to be able to formulate logical thoughts and understand complicated problems.

The next intelligence is the bodily-kinaesthetic. This is a strength for me as I have always participated in a variety of sports and been active. Learning new skills that require co-ordination and body-awareness come relatively easily.

The third intelligence is intra-personal. This involves self-reflection and an ability to learn from one's mistakes. I often review my actions and thoughts after events and figure out how different situations could be handled better in the future.

Getting to the weaknesses, by far my weakest area is linguistic intelligence. Writing, speaking, and learning languages doesn't come naturally to me. This requires work, and currently I am learning German. Based upon these results, I should invest more time writing, and working with language as effective communication is essential when working with others. One of the ways I intend to work on this intelligence is by writing for my blog. Over time, writing should become easier and more fun.

The next intelligence that I want to work on is my naturalistic intelligence. I enjoy being in nature, and spending time outdoors. I however don't learn much about various plants and animals and often just ignore any information that comes my way. To improve this, I should pay more attention to the details and try to learn and remember information to do with nature.

Working on these two intelligences will help my professional career, by improving communication and information sharing. They will help me become more aware of my surroundings and notice intricate details around me.

Conclusion​

If you made it this far, thanks for reading! I learnt a lot writing this post and found out where my strengths and weaknesses lie. Try out the exercise for yourself and come up with a plan to work on your own weaknesses and leverage your strengths in whatever environment you are currently working.

A good template allows you to:

• update all subprojects easily
• get to production quickly
• have a consistent development environment

Templates allow any improvements to be made in one place and then merged into all the projects that use the template as a base. They are vital when working in teams. Nobody likes repeating work many times over when you are working on multiple different projects at the same time.

Template repositories​

A template repository should include anything that fits into the standard project workflow that you use. You want to figure something out once, and document it's use by implementation in your template repository.

Things that you should include are:

• Readme
• standard env files for your setup
• development and production environment definitions
• deployment files
• CI/CD templates and scripts
• standard secret name
• formatting rules, pre-commit workflows

This is not an exhaustive list. Include everything you would use repetitively.

How to use templates​

You have two choices when deciding to use templates.

The first, is creating a new repository from a template, without including the git history. This is effectively a snapshot at the point in time of your template repository. Any changes you make to the template repository will need to be manually merged into all the projects that use it.

The second, and the method which I prefer, is to add a template repository as a remote repository on an empty git-repo. The advantage is that you can pull any changes from your template repository into your projects that derive from it.

Step one: create a repository​

I like to standardise my project names when they are based upon a template. For example, a project that is based on a template for a machine learning API could be named something like:

repository-name = project-name-ml-api

where project-name is the name of your project.

After deciding on a name, create the repository. Clone this new repository to your local machine.

Step two: add the template repository as a remote​

Follow the following steps to setup the new repository with your template repository. It will be configured as a remote where it is only possible to pull from.

Initially only the new repository origin will be a remote. You can check this by running the following command:

$ git remote -v
origin git@github.com:organisation/project-name-ml-api.git (fetch)
origin git@github.com:organisation/project-name-ml-api.git (push)

We will now add the template as a remote:

git remote add upstream git@github.com:organisation/template-ml-api.git

Then we want to ensure you cannot push to this remote:

git remote set-url --push upstream no_push

After executing the above commands, you should see the following output:

$ git remote -v
origin git@github.com:organisation>/project-name-ml-api.git (fetch)
origin git@github.com:organisation>/project-name-ml-api.git (push)
upstream        git@github.com:organisation/template-ml-api.git (fetch)
upstream        no_push (push)

Step three: start your project with the template​

Pull from the template repository and allow you to merge any changes that occurred in the base repository by running the following command:

git pull upstream master

Commit any changes and push back to your newly created repository. Congratulations, you have now created a new repository based on a template.

It should be possible to run git pull upstream master to merge any changes in the template into your project in the future.

You will need to update any files that need to be customised to your newly created project. This includes things like project naming, URLS, naming in Kubernetes files and environment variables etc.

Conclusion​

Template repositories are a good starting point for new projects. They help to speed up your development workflow and enable much faster development rates and efficiencies. You should also have a good understanding about what to put into a template repository. Happy templating!

I intend to update this as I learn more and use it to increase my accountability. As the saying goes, "We know that half of what we know about health is wrong, we just don't know which half."

There are a few different aspects to health that one needs to monitor and optimise:

• Exercise and physical health
• Mental emotional health
• Nutrition
• Sleep
• Medications or supplements
• Habits or productivity

We need to make a distinction between normal and optimal. Often guidance will be given around normal ranges, but we don't want to aim to be normal. We should aim for optimal and target the top 5% for age and gender.

While I am not an athlete or health professional, I have absorbed significant amounts of health-related books and podcasts. The below points have stood out.

Nutrition​

Diet and nutrition are frequently spoken about with a religious fervour. Ultimately, we don't know what the best diet is and never will. We can do things to maximise our probability of living a long life given the information we have access to today.

I have tried keto, paleo, and intermittent fasting. Of all three, I felt the best eating keto and had what felt like unlimited energy, but found it unsustainable.

In the end, I have tended towards a mixed diet, focussing on eating good quality grass fed meat, vegetables, and fruits. This combo ended up being the best mix for me, without being too strict about it. I probably eat too much chocolate and assorted delicious baked goods provided by my wife. Chocolate brownies are definitely a weakness. In general, aim to eat less, high quality food instead of copious amounts of low-quality junk.

Typically, I skip breakfast and have coffee with cream before eating a large meal around lunchtime.

I soak my vegetables in olive oil and generally try to avoid oils like sunflower, canola, or soybean. Recently, I have noticed many products bragging to be "palm oil free". These products are often significantly less healthy.

The fats I limit myself to are:

• olive oil
• butter
• coconut oil

Protein​

The primary focus is on getting enough protein, averaged out over the week. The RDA of around 0.8 grams per kilogram of body weight is too low, and the target should be around 2 grams per kilogram. This is especially important to maintain muscle mass as one ages.

It's difficult to get enough protein through whole foods, so I often supplement with whey and in Germany we have some delicious protein puddings which provide around 20 grams of protein per tub.

Glucose monitoring​

Glucose monitoring for a short time can be incredibly useful to learn how your body reacts to food.

It was good to see how my body reacted to different foods after using a glucose monitor for a couple of weeks.

Blood glucose readings should have a low average of less than 90 mg/dL as well as a low standard deviation. Ideally your blood glucose should never exceed 140 mg/dL.

Alcohol​

Alcohol is one of those things that one typically wants to drink less of. On the short term it affects sleep, reduces motivation, and interrupts your routines. Over the long term, the health effects are vast.

I use the app https://lessdrinks.com/ to track my alcohol consumption. It is surprisingly easy to forget how much alcohol one consumes, and this acts as a good reminder.

Exercise and physical health​

I like exercise. Without it, life would be miserable.

Sometimes it is not easy, but one feels better after doing some. It builds the mental toughness required to take on other challenges throughout life.

One should be strong, be able to do things for a long time and

I'm not an athlete. I exercise so that I can do things in life without having to worry about whether I am capable of it. Exercise is training for life. It allows one to do fun things when the opportunity arises or get out of a tricky situation. If you have done it in training, you can do it when it matters.

The aspects to work on are endurance, strength, and flexibility. All need to be worked on simultaneously to optimise one's physical health. My choices are running and cycling for endurance, and callisthenics and weight training for strength. Occasionally I fit in some yoga for flexibility, however this is definitely a pillar I need to work on.

Barefoot shoes​

Foot and leg health is taken for granted. Most people walk around never knowing what it's like to feel the ground.

I have used barefoot shoes since 2012. The transition took about a year before I could use them for all my activities. Honestly, there is no going back to regular shoes once you switch.

Once you go barefoot, there is no going back. - Me

Barefoot shoes have eliminated the knee pain (and other issues) I used to develop while running. After running a couple of marathons in them, I have only had pain in my muscles from being tired. Recovery is quick.

Other benefits that I have noticed include having naturally strong legs, improved balance and being able to w

One also develops a mental connection to the ground. You get so much more sensory input, that after a while you feel lost without this additional sense.

My favourite brand is Vivobarefoot, although I feel like their quality has started to deteriorate from their early days.

Here are three frequent questions I get about wearing barefoot shoes:

Q. Aren't you slower while running? Professional athletes don't use barefoot shoes.

A: I am slower, but I am not a professional athlete with naturally perfect running form. The benefits outweigh drawbacks. If I want to run a faster time, I should train harder. My running times can improve substantially before I need the minor speed increase of advanced running shoes. What is true for most people: It's not the shoes that stop you from running a 2:10 marathon. Maybe I'd look into racing in advanced running shoes while doing the majority of training in the barefoot shoes.

Q. Won't you injure yourself? You can't run on the road with no padding.

A. I have not injured myself once due to using barefoot shoes and running on the road. On the contrary, I used to injure myself all the time in normal shoes.

If you start slowly, and build up your muscle and tendon strength, your joints will be protected through better running form. Run for years without any injuries and recover faster from big events.

Q. Don't the shoes make you look ridiculous?

A. Another benefit is that you build the mental strength required to accept that you look ridiculous.

It is based on the work of my MSc thesis.

Introduction​

The introduction of renewable generation onto an electricity grid can have profound impact on its reliability, power quality and operation. I come from a background in South Africa where a significant amount of renewables will be connected to the medium voltage (MV) distribution network. A MV distribution network typically has voltages in the range of 11-33 kV. Large plants will be connected to the high voltage (HV) network at 66 kV to 400 kV.

Third world problems​

The power networks in countries such as South Africa, and other third world countries are significantly different to those in the first world.

The power networks can be characterised by:

• large distances between generation and load
• voltage drop rather than current constraints
• different load profiles

Generally, the amount of renewables that can be installed on the MV networks are limited due to the long, weak MV distribution networks that service most of the areas that renewables can be installed.

Over the next few years most of the potential renewable sites, with a strong connection to the grid, will have been used. Most of the open land and renewable resources lie where no existing infrastructure exists. In many cases renewables will have to be connected to weak feeders far away from substations and load centres. Costs rise dramatically if long dedicated power lines need to be built to access these resources.

Technical limitations​

Most distribution systems are passively controlled, designed for one way power flow and control problems are solved during the network planning stage using simple load flow tools. To connect the amount of renewables to the power network there are many technical challenges that must be overcome.

• Power quality: Voltage regulation, harmonic distortion, rapid voltage changes, flicker and voltage unbalance
• Protection: Relay co-ordination, anti-islanding, relay blinding, DG protection from fault currents
• Stability: Transient stability, long term dynamic stability and voltage collapse

There are basic guidelines to determine if detailed studies need to be performed when connecting a power plant to the grid. These guidelines can be summariezed by four main criteria:

1. Voltage regulation or voltage rise
2. Rapid voltage changes
3. Thermal limits
4. Protection limits

These criteria ensure that voltages and currents will never exceed the defined limits during worst case loading scenarios. Protection issues are more of a concern when integrating synchronous and induction generators into the distribution network. Fault current from generation, with power electronic converters, is limited to about double the rated power of the inverter.

The voltage rise, caused by a power plant, is dependent on the amount of power that the DG generates and the short circuit power level at it's point of connection. The EPRI planning guideline, specifies a 1% voltage change limit for renewable generation and a 5% voltage change limit for fixed generation.

On feeders that have a large amount of connected generation, the voltage can be supported along the length of the feeder, depending on where the generators are placed. There are various methods which will be discussed in subsequent posts, on how to regulate the voltage of the MV network when renewables have been installed.

Ultimately, I forsee the transformation of the distribution system from a passive network to an distributely controlled, intelligent network.

Objectives​

This series will address some of the issues relating to the installation of renewable generation onto the grid, such as:

1. Assess the impact of PV distributed generation on MV distribution feeders voltage and voltage control devices
2. Investigate various methods that can be used to increase penetration levels
3. Provide solutions to combine the various voltage control techniques to increase penetration levels further, without negatively impacting power quality

On a holiday in Turkey, I settled down to read the book Shantaram by Gregory David Roberts.

The book is a work of fiction but is based loosely on the life of Gregory after his escape from an Australian prison.

The story starts with the main character "Lin", who is on the run, having just arrived in Mumbai (Bombay) in the 80's. Lin ends up living a fascinating life and experiences the joys and hardships of Mumbai at the time.

In the book, it is hard to distinguish where the border lies between fiction and reality. The descriptions within the book are incredibly detailed. The story gets introduced in such a way and the character portrayals are so good, that one can mistake them as facts. These detailed descriptions are most prevalent and appreciated in the first half of the book. As the book progresses, they increasingly feel dragged out. Towards the end, one wants to get to the point.

Overall, the book is a particularly enjoyable read. I recommend it but keep in mind, that it takes work to get through.