Substrate framework introduction

Note: Slides used in this blog are copied from the other tutorials in YouTube etc.

The Substrate framework is an open-source blockchain development framework created by Parity Technologies. It empowers developers to build their own customized and scalable blockchain networks. Substrate provides a set of tools, libraries, and modules that simplify the process of creating and launching blockchains, enabling developers to focus on building unique features and applications.

The Substrate framework empowers developers to build highly customizable and interoperable blockchain networks while abstracting away much of the complexity associated with blockchain development. Its modular design, efficient runtime, and compatibility with Polkadot make it a powerful tool for creating innovative blockchain solutions.

Above digs is copied from Building blockchains the easy way | Substrate Seminar - YouTube

Key Features of the Substrate Framework:

1. Modularity and Customization: Substrate is designed with modularity in mind. It allows developers to create and combine different modules, known as "pallets," to build blockchains tailored to specific use cases. Pallets encapsulate specific functionalities, and developers can customize or create their own pallets to meet their project's requirements.
2. Efficiency and Performance: Substrate leverages WebAssembly (Wasm) as the runtime execution environment. This offers efficient and sandboxed execution of runtime logic. Additionally, Substrate's weight-based transaction execution system ensures fairness and prevents resource abuse.
3. Consensus Flexibility: Substrate supports various consensus mechanisms, allowing developers to choose the most suitable one for their blockchain network. Consensus mechanisms like Aura, Babe, and GRANDPA can be integrated to achieve different security and performance goals.
4. Interoperability with Polkadot: Substrate is designed to be compatible with Polkadot, which is a multi-chain network that enables interoperability between different blockchains. Substrate-based blockchains can be seamlessly connected to the Polkadot network, allowing for cross-chain communication and data sharing.
5. Security and Upgrades: Substrate includes mechanisms for handling runtime upgrades without disrupting the network. This allows for the introduction of new features, bug fixes, and optimizations while maintaining blockchain continuity. Runtime upgrades are managed with minimal disruptions to the network's operation.
6. Off-Chain Workers: Substrate provides support for off-chain workers, allowing developers to execute computations off the blockchain while still being connected to it. This is useful for tasks like data aggregation, interacting with external APIs, and more complex calculations.
7. Developer-Friendly Tools: Substrate offers developer-friendly tools such as the Substrate Node Template, which provides a starting point for building custom blockchains. The Substrate Developer Hub offers comprehensive documentation, tutorials, and examples to assist developers throughout the development process.
8. Smart Contracts with Ink!: Substrate integrates a smart contract language called Ink! that simplifies the creation and deployment of smart contracts. Developers can use Ink! to write and execute custom logic on the blockchain.
9. Web Interface: Substrate comes with a user interface called Polkadot.js that provides tools for interacting with the blockchain. It allows users to send transactions, manage accounts, and explore the blockchain's state.
10. Community and Ecosystem: Substrate has a growing community of developers, enthusiasts, and contributors. It also fosters an ecosystem of projects, tools, and libraries that enhance the capabilities of Substrate-based blockchains.

Architecture of Substrate: Architecture and Rust libraries | Substrate_ Docs

BlockDig :

At a high level, a Substrate node consists of two main parts:

• A core client with outer node services that handles network activity such as peer discovery, managing transaction requests, reaching consensus with peers, and responding to RPC calls.
• A runtime that contains all of the business logic for executing the state transition function of the blockchain.

Some of the most important activities that are handled by core client services involve the following components:

• Storage: The outer node persists the evolving state of a Substrate blockchain using a simple and highly efficient key-value storage layer.
• Peer-to-peer networking: The outer node uses the Rust implementation of the libp2p network stack to communicate with other network participants.
• Consensus: The outer node communicates with other network participants to ensure they agree on the state of the blockchain.
• Remote procedure call (RPC) API: The outer node accepts inbound HTTP and WebSocket requests to allow blockchain users to interact with the network.
• Telemetry: The outer node collects and provides access to node metrics through an embedded Prometheus server.
• Execution environment: The outer node is responsible for selecting the execution environment—WebAssembly or native Rust—for the runtime to use then dispatching calls to the runtime selected.

Runtime

The runtime determines whether transactions are valid or invalid and is responsible for handling changes to the blockchain state. Requests coming from the outside come through the client into the runtime, and the runtime is responsible for the state transition functions and storing the resulting state.

The Substrate runtime is designed to compile to WebAssembly (Wasm) byte code. This design decision enables:

• Support for forkless upgrades.
• Multi-platform compatibility.
• Runtime validity checking.
• Validation proofs for relay chain consensus mechanisms.

Core libraries

The Substrate libraries are divided into three main areas of responsibility:

• Core client libraries for outer node services.
• FRAME libraries for the runtime.
• Primitive libraries for underlying functions and interfaces for communication between the libraries.

Library Responsibility:

Core client libraries

The libraries that enable a Substrate node to handle its network responsibilities, including consensus and block execution are Rust crates that use the sc_ prefix in the crate name. For example, the sc_service library is responsible for building the networking layer for Substrate blockchains, managing the communication between the network participants and the transaction pool.

Primitive libraries

At the lowest level of the Substrate architecture, there are primitive libraries that give you control over underlying operations and enable communication between the core client services and the runtime. The primitive libraries are Rust crates that use the sp_ prefix in the crate name.

The primitive libraries provide the lowest level of abstraction to expose interfaces that the core client or the runtime can use to perform operations or interact with each other.

For example:

• The sp_arithmetic library defines fixed point arithmetic primitives and types for the runtime to use.
• The sp_core library provides a set of shareable Substrate types.
• The sp_std library exports primitives from the Rust standard library to make them usable with any code that depends on the runtime.

FRAME libraries for the runtime

The libraries that enable you to build the runtime logic and to encode and decode the information passed into and out of the runtime are Rust crates that use the frame_ prefix in the crate name.

The frame_* libraries provide the infrastructure for the runtime. For example, the frame_system library provides a basic set of functions for interacting with other Substrate components andframe_support enables you to declare runtime storage items, errors, and events.

In addition to the infrastructure provided by the frame_* libraries, the runtime can include one or more pallet_* libraries. Each Rust crate that uses the pallet_ prefix represents a single FRAME module. In most cases, you use the pallet_* libraries to assemble the functionality you want to incorporate in the blockchain to suit your project.

You can build a Substrate runtime without using the frame_* or pallet_* libraries using the primitives libraries. For Example Tuxedo runtime(UTXO-based Substrate Runtimes) is built without the Frame or pallet :

FYI

Tuxedo is

Tuxedo/README.md at main · Off-Narrative-Labs/Tuxedo (github.com)

Unlike account-based systems(Frame runtime ), where each user has a balance that can be updated by transactions, UTXO-based systems treat each transaction output as a separate entity that can be spent or unspent. This has several benefits, such as parallel processing and simpler state transitions. By implementing UTXO as one of the possible paradigms on substrate, the Tuxedo project aims to demonstrate the flexibility and interoperability of Polkadot.

UTXO MODEL(Tuxedo Runtime)

• All state is local to individual outputs, addresses are based on transaction hash.
• Transactions are fully deterministic
• On-chain code may be non-deterministic
• Transaction specifies exact input state
• Transaction specifies exact final state
• On-chain code just checks that constraints are satisfied

ACCOUNTS MODEL(Frame Runtime)

• Global storage items at known addresses
• On-chain logic must be deterministic
• Transactions execute on latest stateAnd they may put constraints on the state
• On-chain code calculates the final state

ACCOUNTS VS UTXOS

• Accounts - Great for reducing State bloat, not so great for privacy. Great for general computation, Not so great for parallelization
• UTXOS - Great for privacy, Not so great for State bloat, Not so great in terms of general computation(for now!), great for parallelization

CRYPTOCURRENCY EXAMPLE

Accounts:

AddressBalance0xA11ce50 coins0xB0bbb10 coins

UTXOs:

SerialAddressAmount0x11110xA11ce50 coins0x22220xB0bbb2 coins0x33330xB0bbb4 coins0x44440xB0bbb4 coins

TUXEDO

A framework for writing Substrate Runtimes that are based on the UTXO model.

• Github: https://github.com/Off-Narrative-Labs/Tuxedo/
• Rustdocs: https://off-narrative-labs.github.io/Tuxedo/tuxedo_core/index.html

PPT by Tuxedo team :Tuxedo: UTXOs for Substrate (off-narrative-labs.github.io)

Tuxedo Transaction looks like below :

Transaction in tuxedo_core::types - Rust (off-narrative-labs.github.io)

I will not go many details in to UTXO and Tuxedo, since Tuxedo is still under development, let's keep an eye on Tuxedo.

Pallet:

Let's try to study briefly about pallets which are building blocks of substrate based blockchains, without which it can't be possible to achieve the modularity and extensibility, seamless upgradability, reusability, interoperability of blockchain.

"pallet" is a modular component that encapsulates a specific set of functionalities within the blockchain runtime. Pallets are the building blocks of a Substrate-based blockchain and play a central role in achieving the framework's modularity and customization goals. Each pallet focuses on a specific feature or aspect of the blockchain's behavior, and developers can combine multiple pallets to create a blockchain network tailored to their requirements.

In essence, pallets are the core components of a Substrate-based blockchain's runtime. They enable the modular design of the blockchain, making it easier for developers to build, customize, and upgrade blockchain networks by composing and configuring these building blocks according to their specific use cases and requirements.

Here's how pallets contribute to the modularity of a Substrate-based blockchain:

1. Isolation and Separation of Concerns: Pallets enable the separation of different concerns within the blockchain's logic. Instead of having a monolithic codebase that contains all functionalities, the code is divided into distinct pallets. This isolation makes it easier to manage and maintain the blockchain's behavior and allows for clear separation of responsibilities with plug and play concept of pallets.
2. Customization and Composability: Pallets can be developed independently and composed to create a unique blockchain. Developers can choose the pallets that best suit their use case and combine them in a way that aligns with their project's requirements. This enables a high degree of customization without the need to reinvent the wheel for every project.
3. Reusable Components: Pallets are designed to be reusable across different blockchain projects. Once a pallet is developed and tested, it can be easily integrated into other Substrate-based blockchains, saving time and effort. This promotes the sharing of best practices and standardized functionality within the blockchain development community.
4. Focused Development and Testing: Each pallet focuses on a specific aspect of functionality. This enables developers to focus their efforts on implementing and testing that specific feature without being overwhelmed by the entire blockchain's complexity. It also allows for easier maintenance and debugging of specific functionalities.
5. Upgradability and Versioning: Pallets can be updated or replaced independently of the rest of the blockchain's logic. This makes it possible to introduce improvements, fix bugs, or add new features to specific parts of the blockchain without disrupting the entire network. Upgrading individual pallets ensures smoother network upgrades.
6. Consensus Agnostic: Pallets are designed to be consensus-agnostic. They can work with different consensus mechanisms seamlessly, which allows developers to choose the consensus mechanism that aligns with their network's goals and requirements.
7. Enhancing Protocol Upgrades: When upgrading the protocol of a Substrate-based blockchain, pallets can be upgraded separately, allowing for incremental changes. This can help prevent major disruptions during protocol upgrades by ensuring that specific features are modified or enhanced without affecting the rest of the blockchain.

How to build the substrate-FRAME based pallet : Collactable NFT. Setup (sacha-l.github.io)

Typical Pallet implementation using FRAME is like below (Copied from below Link: Runtime development | Substrate_ Docs

// Add required imports and dependencies
pub use pallet::*;

#[frame_support::pallet]
pub mod pallet {
 use frame_support::pallet_prelude::*;
 use frame_system::pallet_prelude::*;

 // Declare the pallet type
 // This is a placeholder to implement traits and methods.
 #[pallet::pallet]
 #[pallet::generate_store(pub(super) trait Store)]
 pub struct Pallet<T>(_);

 // Add the runtime configuration trait
 // All types and constants go here.
 #[pallet::config]
 pub trait Config: frame_system::Config { ... }

 // Add runtime storage to declare storage items.
 #[pallet::storage]
 #[pallet::getter(fn something)]
 pub type MyStorage<T: Config> = StorageValue<_, u32>;

 // Add runtime events
 #[pallet::event]
 #[pallet::generate_deposit(pub(super) fn deposit_event)]
 pub enum Event<T: Config> { ... }

 // Add hooks to define some logic that should be executed
 // in a specific context, for example on_initialize.
 #[pallet::hooks]
 impl<T: Config> Hooks<BlockNumberFor<T>> for Pallet<T> { ... }

 // Pallet internal functions
?impl<T: Config> Pallet<T> { ... }

 // Add functions that are callable from outside the runtime.
 #[pallet::call]
 impl<T:Config> Pallet<T> { ... }
}        

Pallets are broadly classified as System ,Functional and para chain pallets , please see link: FRAME pallets | Substrate_ Docs

Will try to write a separate article on how to develop the custom pallet, since the objective of this article is just general overview of substrate framework.

Runtime:

Next important topic is Runtime layer of substrate framework-based network:

The runtime layer is the heart of a Substrate-based blockchain, defining how the blockchain functions, processes transactions, maintains state, interacts with consensus mechanisms, and more. It encapsulates the core logic that distinguishes one blockchain network from another and allows developers to customize and create unique blockchain applications.

As described earlier the runtime layer is a crucial component of a Substrate-based blockchain. It defines the behavior and rules of the blockchain network by encapsulating the logic of the blockchain's operations, including transaction validation, consensus mechanisms, storage management, and more. In the context of Substrate, the runtime layer is written in Rust programming language and compiled into WebAssembly (Wasm) code for efficient execution on blockchain nodes.

Here's a more detailed explanation of the runtime layer in a Substrate blockchain:

1. Custom Logic and Functionality: The runtime layer houses the custom logic and functionality that make a blockchain network unique. It includes the implementation of various features, services, and actions that the blockchain can perform, such as token transfers, governance proposals, identity management, and any other application-specific functionalities.
2. Runtime Modules (Pallets): The runtime layer is composed of multiple runtime modules, often referred to as "pallets." Each pallet encapsulates a specific set of functionalities or features. These pallets can be created by the blockchain developers, and they define storage items, dispatchable functions (transactions), events, and other relevant components. Its already explained above.
3. Storage Management: The runtime manages the blockchain's on-chain storage using the storage items defined within the pallets. These storage items store the state of the blockchain and allow data to be read and modified in a structured manner. The runtime ensures that data integrity and consistency are maintained as transactions are processed.
4. Transaction Execution: When a user initiates a transaction, the runtime processes and executes the transaction's logic. This involves validating the transaction against predefined rules, updating the blockchain's state (if needed), emitting relevant events, and determining the outcome of the transaction.
5. Consensus Mechanisms: The runtime interacts with the consensus layer to participate in the consensus mechanisms of the blockchain. It includes the logic required to validate blocks, finalize transactions, and determine the order of transactions within a block.
6. Event Emission: The runtime emits events to provide information about changes or actions that have occurred within the blockchain. Events are used to notify other parts of the runtime or external applications about important occurrences, such as successful transactions, changes in storage, and more.
7. Smart Contracts and Off-Chain Logic: Some Substrate-based blockchains may incorporate smart contract functionality using languages like Ink! for creating and executing custom logic on the blockchain. Additionally, the runtime can support off-chain workers, which allow computations to be performed off the blockchain while still interacting with it.
8. WebAssembly (Wasm) Compilation: The runtime's Rust code is compiled into WebAssembly (Wasm) code. Wasm is a portable binary instruction format that offers secure and efficient execution of code within a sandboxed environment. Wasm execution is optimized for speed and can be executed on various platforms, making it suitable for blockchain environments.
9. Can achieve the forkless upgrade of runtime without actually stopping the running blockchain : Upgrade a running network | Substrate_ Docs

Consensus layer :

The consensus layer in the Substrate architecture is responsible for ensuring that all nodes in the blockchain network agree on the validity and order of transactions and blocks. It defines the rules by which nodes come to a consensus on the state of the blockchain. Substrate supports various consensus mechanisms, each with its own strengths and characteristics, allowing developers to choose the one that best suits their blockchain's requirements.

Here's a deeper look into the consensus layer in the Substrate architecture:

Consensus Mechanisms: Substrate allows developers to choose from different consensus mechanisms, each tailored for different use cases. Some of the supported consensus mechanisms include:

The Substrate node template uses a proof of authority consensus model also referred to as?authority round?or?Aura?consensus. The Aura consensus protocol limits block production to a rotating list of authorized accounts. The authorized accounts—authorities—create blocks in a round robin fashion and are generally considered to be trusted participants in the network.

AURA: Stands for Authority based round robin scheduling, it provides a slot-based block authoring mechanism. In Aura a known set of authorities take turns producing blocks.

Sourcecode: https://github.com/paritytech/substrate/blob/master/frame/aura

Aura is a simple and deterministic consensus mechanism suitable for small to medium-sized networks. It relies on a predefined list of validators to take turns producing blocks.

"predefined list of validators" refers to a set of validator nodes that have been selected to participate in the consensus process. These validators are responsible for producing and validating blocks in a decentralized blockchain network.

Here's how the process of validator selection and their predefined list works:

1. Validator Set: Before the network starts operating, a set of validator nodes is chosen to participate in the consensus algorithm. This set is often determined based on various factors, such as their stake in the network, their reputation, or other consensus-specific criteria.
2. Predefined List: The selected validator nodes form the predefined list of validators. This list is established before the network goes live and remains relatively stable over a certain period, which is often referred to as an "era" in the context of Substrate or Polkadot. The validators on this list are assigned specific roles, such as block production and block validation, based on the consensus algorithm being used.
3. Validator Rotation: In the Aura consensus algorithm, the predefined list of validators is used to determine the order in which validators take turns to produce blocks. Each validator is assigned a slot, and their order is deterministic. After each slot, the responsibility moves to the next validator in the list, ensuring that all validators get a chance to participate.
4. Decentralization: The predefined list of validators represents a decentralized network of nodes that collectively maintain the blockchain's security and integrity. The number of validators in this list can vary depending on the design and goals of the blockchain network. A larger number of validators often leads to increased security and decentralization.
5. Validator Attributes: Validator nodes on the predefined list typically have certain attributes associated with them, such as their public key, identity, and stake in the network. These attributes are used to determine the order of validators, their responsibilities, and their roles in the consensus process.
6. Updating the List: The predefined list of validators can change over time due to factors like validator performance, changes in stake, or network upgrades. Depending on the consensus algorithm and governance mechanisms in place, the list may be updated periodically or through governance proposals.

BABE(Blind Assignment for Blockchain Extension): is consensus mechanism designed for larger networks. It uses a verifiable random function to select validators for block production in a probabilistic manner. BABE is a consensus mechanism designed for the Polkadot blockchain. It is a block production mechanism that aims to improve the security and scalability of the network. BABE uses a verifiable random function (VRF) to determine which validator will produce the next block. It introduces randomness to the block production process, making it difficult for malicious actors to predict which validator will be chosen to create the next block. While BABE enhances the security of the blockchain, it is not directly related to Byzantine fault tolerance.

Source code: substrate/frame/babe at master · paritytech/substrate (github.com)

Byzantine Fault Tolerance: Byzantine fault tolerance is a concept in distributed computing and consensus algorithms. It refers to the ability of a distributed system to continue functioning correctly even when some of its nodes (participants) are faulty or malicious (Byzantine). Byzantine fault-tolerant consensus mechanisms ensure that the network can reach a consensus on the state of the system despite the presence of potentially malicious nodes.

In the context of blockchain, Byzantine fault tolerance is a critical property of consensus mechanisms to ensure the security and reliability of the network. Some well-known Byzantine fault-tolerant consensus mechanisms include Practical Byzantine Fault Tolerance (PBFT), HoneyBadgerBFT, and Tendermint.

Please note that while both BABE and Byzantine fault tolerance are important concepts in blockchain technology, they are not directly linked . BABE is a specific block production mechanism used in the Polkadot network, while Byzantine fault tolerance refers to a property of consensus mechanisms that ensures the system's resilience to malicious behavior.

GRANDPA: GRANDPA (GHOST-based Recursive ANcestor Deriving Prefix Agreement) is a finality gadget that provides finality guarantees for blocks. It complements the block production mechanism and helps achieve a secure and final consensus. So please remember "GRANDPA" refers to a finality gadget, not a consensus mechanism in the traditional sense. In blockchain terminology, a finality gadget is a component that helps determine when a block is considered irreversible or "finalized" in a blockchain network.

Source code : substrate/frame/grandpa at master · paritytech/substrate (github.com)

GRANDPA is a finality gadget used in the Polkadot blockchain ecosystem.It just listens to gossip about blocks that have been produced by block authoring nodes. GRANDPA validators vote on chains, not blocks,. GRANDPA validators vote on a block that they consider best and their votes are applied transitively to all previous blocks. After two-thirds of the GRANDPA authorities have voted for a particular block, it is considered final.

GRANDPA works in conjunction with Polkadot's nominated proof-of-stake (NPoS) consensus mechanism. It aims to achieve faster finality and consensus in a heterogeneous multi-chain environment, where different chains can have varying consensus mechanisms and block finalization speeds. GRANDPA helps to determine the canonical chain and ensure that blocks are finalized in a way that maximizes security and efficiency.

Block Production: Consensus mechanisms in Substrate are responsible for determining how blocks are produced and who is responsible for producing them. Different mechanisms use varying methods to select block producers, whether based on predefined schedules (like Aura) or probabilistic algorithms (like Babe).

Below slide syas about block production within the substrate :

Below slide says about block importing from other nodes:

Finality and Block Validation: Achieving consensus doesn't stop at block production. Finality is an essential aspect of ensuring that confirmed transactions cannot be reverted. Substrate's consensus mechanisms work together with finality gadgets like GRANDPA to establish the irreversible state of the blockchain.

Security and Fault Tolerance: Consensus mechanisms are designed to be secure and resilient against various attack vectors. BFT-based mechanisms like Babe and GRANDPA provide strong fault tolerance, ensuring that the network remains operational even in the presence of malicious nodes.

Interoperability with Polkadot: Substrate's consensus mechanisms are designed to interoperate with the Polkadot network. Substrate-based blockchains can connect to the Polkadot Relay Chain using bridges, enabling cross-chain communication and interoperability.

Consensus Upgrades: Substrate's modular design extends to the consensus layer as well. Developers can choose and switch between consensus mechanisms based on their blockchain's requirements. Upgrading the consensus mechanism can be done as part of a runtime upgrade without disrupting the entire network.

Consensus Agnostic Pallets: Pallets developed for Substrate-based blockchains are designed to be consensus-agnostic. This means that the same pallets can work with different consensus mechanisms seamlessly, offering flexibility when designing and deploying blockchain applications.

Cross-Consensus Messaging.

XCM stands for "Cross-Consensus Messaging." It's a protocol that allows communication and interaction between different blockchains or parachains within the Polkadot ecosystem, which is a multi-chain blockchain platform built using the Substrate framework.

XCM enables inter-chain communication by providing a standardized way for messages to be sent from one parachain to another, or even from one blockchain to another connected through Polkadot's relay chain. This communication includes transferring assets, invoking smart contracts, and triggering actions on remote chains. Essentially, XCM enables composability and interoperability between different chains in the Polkadot network.

Here are a few key points about XCM:

1. Message Format: XCM defines a message format that contains information about the origin, destination, and the action to be performed. It also includes information about the assets being transferred, smart contract calls, and more.
2. Composability: XCM allows for complex interactions between parachains and blockchains. Parachains can send messages to other parachains or to the relay chain, enabling decentralized applications (dApps) that span across multiple chains.
3. Interoperability: XCM is designed to support communication not only between parachains but also with external chains and systems. This facilitates interoperability between Polkadot and other blockchain networks.
4. Standardized API: XCM provides a standardized API for message passing, making it easier for developers to build applications that interact with multiple chains within the Polkadot ecosystem.
5. Security and Trust: XCM aims to provide secure cross-chain communication while maintaining the integrity and trust assumptions of the underlying chains. The relay chain ensures the validity of XCM messages and their execution.
6. Use Cases: XCM enables a wide range of use cases, including cross-chain asset transfers, decentralized finance (DeFi) applications that span multiple chains, data sharing, and more.

Forkless upgrade:

Reference: Upgrade a running network | Substrate_ Docs

The below slide says about the issues faced in bitcoin due to interger overflow bug :

Bitcoin decided to do hardfork and below slide says what problem happened due to hardfork:

Substrate-based blockchains, the runtime logic is indeed implemented as a WebAssembly (Wasm) module. Substrate supports forkless runtime upgrades, which means that the upgrade process does not require a hard fork or disruption to the network. Here's how the runtime upgrade process typically works in Substrate:

Preparing the New Runtime: Developers create an updated version of the runtime Wasm module, which includes the changes and improvements they want to introduce to the blockchain's logic.

Specifying the Runtime Upgrade: A runtime upgrade is coordinated through a governance process. Network participants (validators, stakeholders, or both) submit and discuss proposals for upgrading the runtime. These proposals include the new runtime Wasm code and related metadata.

Referendum and Voting: The network participants use the governance mechanisms of the blockchain to vote on the proposed upgrade. If the proposal receives enough support (as defined by the governance rules), it progresses to the next step.

Activation of the Upgrade:Once the referendum passes and the required threshold is met, the upgrade is scheduled to be activated at a specific block height. This is done in a way that validators and nodes are aware of the upcoming upgrade.

Pre-Upgrade Preparation: Before the upgrade, nodes download and verify the new runtime Wasm module. This can be done ahead of time to ensure a smooth transition.

Runtime Upgrade at Block Height: When the blockchain reaches the specified block height, the upgrade is triggered. The new runtime Wasm module is activated, replacing the old one. This is a seamless process because all nodes have already downloaded and verified the new module.

No Disruption to the Network: Since all nodes have prepared for the upgrade and are using the new runtime logic, there is no need for a hard fork. Transactions and consensus continue without interruption using the new logic.

Post-Upgrade Operations: Validators and nodes continue to validate and produce blocks using the updated runtime logic. Any new features or changes introduced by the upgrade become active.

Substrate's forkless upgrade mechanism is designed to ensure a smooth transition between runtime versions without requiring network-wide coordination for a hard fork. It promotes more frequent and secure upgrades, making it easier for blockchain projects to innovate and improve their systems over time.

Let's keep in mind that the specifics of the upgrade process might vary based on the Substrate version and any customizations made to the blockchain. We should always refer to the official Substrate documentation and resources for the most accurate and up-to-date information on runtime upgrades.

How the runtime WASM module is distributed to each node?

Runtime WebAssembly (Wasm) modules are distributed to each node in the network via the network itself. When a runtime upgrade is planned and approved through the governance process in a Substrate-based blockchain, the new Wasm module needs to be propagated to all the nodes in a secure and efficient manner. Here's how the distribution process generally works:

1. Distributing the Wasm Module: The new runtime Wasm module, along with any necessary metadata, is typically hosted on a central repository, such as a dedicated upgrade node, a content delivery network (CDN), or a version-controlled repository.
2. Announcing the Upgrade: Nodes on the network are informed about the upcoming runtime upgrade through the governance process and related mechanisms. Validators and nodes know when the upgrade will occur and can prepare for it.
3. Pre-Download Phase: Before the upgrade height is reached, nodes start downloading the new Wasm module. This phase allows nodes to retrieve and verify the Wasm code in advance.
4. P2P Networking: Substrate nodes communicate with each other using a peer-to-peer (P2P) networking protocol. During the pre-download phase, nodes share information about the new runtime version with each other.
5. Content Distribution: Nodes that have already downloaded the new runtime module can share it with other nodes that have not yet completed the download. This content distribution can occur via P2P networking.
6. Verification: Nodes that receive the new runtime Wasm module will verify its integrity and authenticity using cryptographic signatures and checksums. This ensures that the module has not been tampered with during transmission.
7. Activation at Upgrade Height: When the blockchain reaches the specified block height for the upgrade, nodes are already prepared with the new runtime module. The activation of the new runtime is coordinated, and the blockchain continues with the updated logic seamlessly.

It's important to note that the specifics of how the runtime module is distributed and the mechanisms used can vary based on the implementation, network setup, and consensus algorithm of the Substrate-based blockchain. In some cases, specialized tools or processes might be employed to ensure efficient and secure distribution.

To ensure the smoothest runtime upgrade process, it's recommended to follow best practices, such as having redundancy and fallback mechanisms in case of unexpected issues during the distribution process. Substrate's governance and upgrade mechanisms are designed to support these upgrade scenarios and make the process as reliable as possible.

Please read the below article for the Inherents,Transactions,extrinsics in below link:

Inherents,Transactions,extrinsics | LinkedIn

Important APIS or crates:

sp_api - Rust (docs.rs)

The Substrate runtime api is the interface between the node and the runtime. There isn’t a fixed set of runtime apis, instead it is up to the user to declare and implement these runtime apis. The declaration of a runtime api is normally done outside of a runtime, while the implementation of it has to be done in the runtime.

sp_core - Rust (docs.rs); Shareable Substrate types.

sp_metadata_ir - Rust (docs.rs): Intermediate representation of the runtime metadata.

Will be studying the node discovery (trusted and untrusted one ) and other advanced topics in later articles.

Info about nodes : Substrate-nodes | LinkedIn

Info about substrate architecture : Architecture and Rust libraries | Substrate_ Docs

Debugging guide : Debug | Substrate_ Docs

RUST_LOG=runtime=debug ./target/release/node-template --dev        

Thanks for reading till end, please comment if you have any.