Crawler
A lightweight web crawler that can also split files.
for different kinds of informations.
This hackathon submission was made by:
When exploring a new smart contract language, after learning the syntax and data types, one of the best ways is through examples, boilerplates and exploring various strategies to implement something that you would have done in a previous language. Within the Web3 space there are a couple strategies that proven effective.
We can easily group these into "Bootstrappers" - "Boilerplate" - "Libraries" in a Magic Quadrant, these various strategies enable you quickly go from 0 - 1
in a new smart contract language.
It goes without saying, that project based learning is the most effective approach to becoming proficient in Soroban as it is with any other smart contract language or programming language in general. However, the daunting challenge that Soroban presents, especially for developers coming from more widely-used smart contract languages like Solidity, is its intimidating appearance at first glance. For those who are not familiar with Rust, Soroban's syntax and concepts can seem foreign and overwhelming, creating a significant barrier to entry. This unfamiliarity and perceived complexity cause blockchain developers to stick with the languages they are more comfortable with, despite the potential benefits Soroban may offer.
We built Soroban by Example is a comprehensive resource designed to help developers, especially those transitioning from other smart contract languages like Solidity, to learn and adopt Soroban through a project-based approach.
💡 However while keeping an ambitious goal in view, there were a couple challenges with trying to fit the world into Soroban. Essentially looking at the more popular blockchain ecosystems and their paradigms to fit into how smart contracts on Stellar can be written
EVMs have this concept of the ZERO_ADDRESS
for burning tokens and NFTs and performing a lot of modifer functions, however, that is irrelevant to the context of Soroban where ZERO_ADDRESS
is useless and instead of modifiers, modules are preferred
Solidity, the most popular smart contract language uses inheritance as a form of extensibility, but this does not apply to Soroban, as everything you will need from the host is available by passing &env
into functions to get access to the host environment. This means that in Soroban we favor functional composition over inheritance.
For some of us on the team, this was our first foray into the Soroban ecosystem, even Koha who has worked in DevRel roles for protocols and teams building within the EVM ecosystem, agrees that Soroban is more exciting 🤣. Having to fit into the standards/systems from more popular smart contract languages is boring.
Before getting to this point we made several attempts at achieving our objectives. Our first attempt — AI Code Generator was building out a Custom GPT to learn Soroban and get example contracts
💡 In order to achieve this initial milestone, we had to train a custom GPT with documents from the stellar developer docs. To achieve that, we wrote a simple scraper script fetch the developer docs and chunk them into text files, see our github repo for the scraper implementation.
A lightweight web crawler that can also split files.
However after a couple test runs, we realized that this GPT is not really great at understanding the JTBD (Job to be Done) and still hallucinates in rust
when asked to perform more advanced instructions like generate specific type contracts and tests.
For the AI Code Generator, we found some options like kapa.ai However upon further explorer, we realised that there would be limitations with that implementation as it would only cater to document search and suggestions.
This also meant we could not avoid manually curating the examples ourselves and reviewing each publication one by one.
After so many trials and errors, we decided that the best way to go about this was to manually curate each example contract ourselves and train / finetune an examples generator model called “Soroboy” — Soroboy will be different from a GPT, because this time it will live inside the context of an example smart contract page.
We first started by curating all contract examples in a Notion workspace and tracking their Okashi playground links as well. This meant starting out with core concepts and generating baseline contracts that are a great starting point.
When a user visits our example website, they can have a conversation with Soroboy to generate a variant of the contract example on that page.
In the example below, Soroboy is able to generate a variant of an only_owner
contract with multiple owners instead of one owner
To achieve this, we have to manually curate soroban contracts ourselves, review them exploration on this front focuses on training an AI model on examples, so that it can leverage that for contextual exploration, our idea around this is something that generates variations of an existing contract. The workflow is something like this.
💡 To implement this, we used a Vector storage library LanceDB which is an in-memory DB built with Rust and uses the [Apache-Arrow Data Format] for compact storage.
Lance was perfect for this exploration, because all of your vector embeddings are simply files that can be tracked on github. See our rag embeddings to get a sense of what this looks like.
For Generation capabilitie, API Calls are routed to OpenAI GPT-4o at the moment. To be able to inject the AI module into the documentation page, we have to implement a custom react library and API server to manage the interactions for this.
A couple things we finally executed that weren’t part of our original plan was to embed an Okashi playground into docs website.
If you take a look at our Root Notion Database You will notice that all of our docs page, when being prepared had their own Okashi playground. We had chosen Okashi because it was dead simple to use, and we didn’t have to maintain a large repo of example contracts with their own Cargo.toml
This was great for testing and exploration and during the development of this project, Our team wondered what the experience for our users would be like, if they could simply compile and run these contracts in a codesandbox
fashion.
Okashi was the first choice for us, however there are some limitations with Okashi we wish the maintainers would be open to support to greatly improve DX
If you take a look at the simple Soroban Loop Example on our website. You will notice that there is an Okashi Widget embedded within the page. As a matter of fact all the pages have an Okashi widget inside of them. We were able to do this by writing a custom_proxy_server
that acts as a middleware between our website and Okashi.
This server simply transpiles an Okashi request so we can get the full DOM object of Okashi and modify the views to suit our Widget capabilities.
We consider this to be a hack — a rather brute way of achieving our purpose, that enables us provide the intended experience for this project. We’ve already reached out to Morgan and the Okashi team on Discord, hopefully we can get their attention and have this as a feature in Okashi
In hindsight, I think we under-estimated how much work we would have to do in order to achieve the core objectives of this project, some of the artefacts we produced were
Soroban is an amazing smart contract language, and we intend to continue maintaining this project, as well as exploring Soroban and creating new projects with it.
Our goal with this project apart from increasing discoverability for the Soroban ecosystem and showing so-many simple use-cases and boilerplates that developers and individuals exploring the Soroban ecosystem can interact with. Will also be to offer
We took a lot of inspiration from projects within the ecosystem pushing the boundaries of what is possible with Soroban.
Featured ones: