Related to the architecture, new layer on top of Substrate

Question

In the context of Substrate, is it feasible to introduce a layer on top, such as Kafka, to improve request processing times? Could this architecture enable Substrate to be utilized solely for blockchain verification purposes, with the supplementary layer facilitating accelerated request handling by only requesting substrate when changes occur?

Answer 1

I suggest you do all the tutorials in substrate docs to understand more what you can do, it wont take long to finish all of them (around 3 days, which is worth it), after you finish the tutorials you will understand that substrate just a bunch of tools and libraries that you can combine however you want to build whatever you want, the only limitation is your imagination (and maybe your project requirements).

Let me give you an example to answer your question in your comment,currently we are working on a project(basically to do exactly what you are describing, scaling request processing) to validate a stream of events so most of our work was basically done offchain, we are runing a grpc server on each validator and gossiping some events using libp2p gossipsub and submitting transactions finally to substrate to be included in a block and listening for finalized blocks and returning responses to clients, all of this is done mostly in outer-node.

You can think of substrate as a state machine, and you can do whatever you want with state machines :)

Answer 2

Yes you can introduce a new layer or component into Substrate. You need to think at what level you want to do this i.e. on the runtime level, client level, off-chain level. I recommend you start by reading all of these.

Could this architecture enable Substrate to be utilized solely for blockchain verification purposes, with the supplementary layer facilitating accelerated request handling by only requesting substrate when changes occur?

If you can build a nice solution around this, the community would be grateful.