Building an Ethical Text-to-Image AI-Art Site That Pays Royalties to Original Creators

A Comprehensive Guide on how to build an open source, ethical text to image Art generation site like with Bacalhau and the FVM. Waterlily.ai 🎮 Try it! Live on FVM at www.waterlily.ai https://youtu.be/oXNaCs0eyXI 📌 Quick Links Github: https://github.com/bacalhau-project/Waterlily Docs: https://docs.bacalhau.org 👩💻 What we'll do... In this tutorial I'll take you through how to go about building a similar application to on FEVM and Bacalhau, including: Waterlily.ai Building and deploying a solidity smart contract on the Filecoin Virtual Machine Using Lilypad for bridging FVM with AI functions like StableDiffusion Building the front-end UI interactions in Typescript Bonus: Creating an open-source Stable Diffusion model to run on Bacalhau Bonus: Finetuning models with Dreambooth Bonus: Creating a private Bacalhau cluster to run your scripts and models on Some knowledge of javascript and solidity is assumed in this tutorial. 🗺️ Overview There’s no doubt that AI has reached the mass adoption phase of the technology cycle and is reshaping industries and generating new opportunities across a variety of verticals. The use of Stable Diffusion models, for generative AI-Art, is no exception with , Midjourney, Dall-E and more making Stable Diffusion models a household name. Stability.ai These stable diffusion models raise some important social and legal questions though, not least of which is the problematic use of pre-existing images in models without proper attribution, consent, or revenue provided to the original creator. is a proof-of-concept application aiming to provide an alternative ethical solution by creating a new paradigm for AI image generation that pays royalties to the original creator and offers a new revenue stream for artists to explore. Waterlily.ai To achieve this, Waterlily leverages the transparency & trustlessness of blockchain, with smart contracts, combined with a decentralised edge compute platform in to deliver an intuitive user experience for generative AI-Art while compensating creators for their original work with every image generation call. FVM Bacalhau Each stable diffusion model on Waterlily is trained & finetuned to a selection of artist work that has been uploaded by the artist themselves or is available in the public domain. When a user navigates to the platform, they simply enter a text prompt for the images they want to create and choose an artist style that appeals to them. Waterlily.ai The small fee paid by the user is then distributed to the artist's wallet (except the fees required by the network to run the contract calls & return the generated images) - or to a vetted creator foundation in the case of public images. A user can then opt to download these generated images or mint them as NFTs. 🥞 The Tech Stack Waterlily relies on two main technology tools - (Filecoin Virtual Machine): the Filecoin EVM-compatible blockchain network to which the smart contracts are deployed and which acts as a payment layer, provides a verifiable record of transactions for art provenance, and hosts any NFTs. FVM : a verifiable peer-to-peer compute layer which both trains new artist models and also runs the stable diffusion scripts that generate images from a user's text prompt. Bacalhau The image below provides some great context for the Filecoin star map with FVM = Programmable Application and Bacalhau = Computation Services. Filecoin Virtual Machine (FVM) The Filecoin Virtual Machine went live on the 14th March this year enabling a new set of potential use cases on Filecoin. Fun fact: 14 March is 3.14 in American date-time or the first 3 PI digits which also happens to be the network id of the FVM mainnet chain! Bacalhau Bacalhau is aiming to help democratise the future of data processing by enabling off-chain computation over data without giving up the decentralisation values inherent to IPFS, Filecoin & Web3 more broadly. is a peer-to-peer open computation network that provides a platform for public, transparent and optionally verifiable computation processes where users can run Docker containers or Web Assembly images as tasks against data including data stored in IPFS (& soon Filecoin). It even has support for GPU jobs. Bacalhau any Making data processing and computation open and available to everyone & speeding up the processing times is possible in Bacalhau, firstly - by using batch processing across multiple nodes and secondly by putting the processing nodes where the data lives! FVM + Bacalhau = better together More on this later ;) Waterlily Tech Tooling Detail [Solidity, Open Zeppelin] Smart Contracts is an OO smart contract programming language for Ethereum (EVM) -compatible blockchains (used for the contracts) Solidity offers a security-audited implementation library of common smart contract components and contracts (we use this for the basis of the NFT contract) Open Zeppelin [Hardhat, Remix] Smart Contract IDE is a development environment for editing, compiling, debugging & deploying Ethereum software Hardhat , is a no-setup tool with an in-browser GUI for developing smart contracts. Remix Ethereum IDE [Filecoin Virtual Machine] Blockchain Network The FVM is an EVM-compatible mainnet built on the Filecoin blockchain [NFT.Storage] NFT Metadata Storage is a public good built on top of IPFS & Filecoin to store NFT Metadata immutably and persistently & offers free decentralised storage for NFTs and a javascript sdk. NFT.Storage [Web3.Storage] Artist Metadata Storage Similar to NFT.Storage, with web3.storage you get all the benefits of decentralized storage and other cutting-edge protocols with the frictionless experience you expect in a modern dev workflow. [NextJS / Typescript + NPM] Front-End We probably all know these... right? :P from client [Metamask, Ethers, Chainstack RPC Node] Smart Contract Interactions Using a - I can get read-only interactions with my blockchain contract. public RPC node With a provider (or similar wallet that specified by into the browser), we enable write calls to the blockchain contract. Metamask injects the Ethereum API EIP-1193 js is a library for interacting with EVM-compatible smart contracts Ethers AI [Python, Tensorflow, Docker] Text-To-Image Stable Diffusion Script is an open-source machine learning platform and library that provides pre-trained models and other data and ML tools. TensorFlow Decentralised for AI Text-To-Image Generation & Model Training [Bacalhau] Off-Chain Compute is a peer-to-peer open computation network that provides a platform for public, transparent and optionally verifiable computation processes. It's a decentralised off-chain data computation layer Bacalhau 🏗️ Building the Smart Contracts for Waterlily.ai Wooo, pre-amble complete - let's make some code! Creating the Smart Contracts on FEVM The FVM is a fully EVM-compatible network (hence the abbreviation - FEVM). This means that not only do we have access to all of the Filecoin decentralised storage network's we can also use solidity to write our contracts and take advantage of all the awesome tooling & audited contracts available to us on the EVM network as well (and enhance them!). API calls This project relies on 2 main contracts: An Artist Contract that can onboard and edit artist data An NFT Contract for NFT creation. The NFT Contract Let's start with the NFT Contract, something I've written about before in this guide: . https://developerally.com/build-your-own-ai-generated-art-nft-dapp When I built the demo application above, my main motivation was to help people learn how to use both FVM and Bacalhau. However, whilst I was making it, I was fairly unsettled by the fact I was using AI to generate art without any real knowledge of where those datasets (ie. the art) used to train the OSS ML model I used, were coming from. This was one of the motivations behind . Waterlily.ai The NFT contract in this app is actually no different to the one documented above - so feel free to go ahead and peruse the section "Building & Deploying the Solidity NFT Contract" if you'd like to replicate the NFT contract as well. 😎🫘 The Artist Contract // SPDX-License-Identifier: MIT pragma solidity >=0.8.4; import "hardhat/console.sol"; import "@openzeppelin/contracts/access/Ownable.sol"; /** @notice An experimental contract for POC work to call Bacalhau jobs from FVM smart contracts */ contract Waterlily is Ownable { //ok now what... } The Artist Contract requires the following information: Data Models An Artist Model (struct) to store data we need to know about an Artist such as their payment wallet address an identifier for their specific ML model to run when chosen (we've made special use of this id and made it an IPFS CID that contains metadata the artist input when signing up to Waterlily) details on their revenue and FIL payments from Waterlily some convenience helpers in the form of an id=>Artist mapping and an array for the UI to read from struct Artist { string id; // A CID containing all metadata about an artist address wallet; // Their FEVM wallet address for payments uint256 escrow; // amount of FIL owed to an Artist uint256 revenue; // total amt of revenue earned since joining uint256 numJobsRun; // total numbner of jobs run on the artist bool isTrained; // a flag for if the Artist Model is ready } // mapping of artist id onto artist struct mapping (string => Artist) artists; // we need this so we can itetate over the artists in the UI string[] artistIDs; An Image Model (struct) to store information on the generated images made including The image owner The original artist the image was generated from The prompt input used to generate the image A link to the image generated or the status of the image otherwise (ie. if there's an error) Convenience helpers: Mappings from customer to images owned by the customer and open Zeppelin Counters contract to generate the Image Id's /* ===Imports === */ import "@openzeppelin/contracts/utils/Counters.sol"; /* === Contract vars below === */ using Counters for Counters.Counter; Counters.Counter private _jobIds; // the image id - this is technically a Bacalhau job - hence the name _jobIds struct Image { uint id; // each image created has a unique id like an NFT // we're leveraging OpenZeppelin counter contract to make // these id's (see end of code here) uint jobId; // the id of the bacalhau job returned from lilypad bridge address customer; // the wallet owner for this image string artist; // the original artist model id of this image string prompt; // the text prompt used to generate the image string ipfsResult; // the image result: this is an IPFS CID returned by the script string errorMessage; // if error occurred running the job bool isComplete; // is the Stable Diffusion job complete? bool isCancelled; // if the job was cancelled. } mapping (uint => Image) images; // mapping of id to the Image mapping (address => uint[]) customerImages; // a map of customer address onto images they have submitted (setters, getters & events) - things we need to interact with the contract Methods Artist: * we need to know all the artists in the contract in order for the UI to display them getArtistIds() * fetch a specific artist getArtistById() *onlyArtist function - this function provides a method for the artist to withdraw their earnings at any time. The contract otherwise keeps funds in escrow until an artist has earned 5FIL in order to avoid the gas costs reducing their overall revenue earnings. artistWithdrawFunds(address payable _to) * Artist Signup is automated. This function is called when a new artist registers on the UI CreateArtist(string calldata _ArtistId) *ownerOnly function - this is to ensure bad actors uploading illegal or non-IP owned art are able to be removed from the platform DeleteArtist() *ownerOnly or artistOnly function - only the artist themselves or the owner of the contract (just in case of a change of public charity address or a verified Artist losing a private key) are able to change the artist wallet address! UpdateArtistWallet() Let's also create some events for useful-to-log occurrences on the contract like Events: When an artist is added When an artist is deleted When an artist wallet is updated When an artist is paid /** The events tracked by the contract *// event ArtistAdded(string indexed artistId); event ArtistDeleted(string indexed artistId); event ArtistWalletUpdated(string indexed artistId, address prevWallet, address curWallet); event ArtistPaid(string indexed artistId, address _paidTo, uint256 amountPaid); /** self explanatory XD **/ function getArtistIds() public view returns (string[] memory) { return artistIDs; // we defined this earlier! } /** self explanatory XD **/ function getArtistById(string calldata _artistId) public view returns (Artist memory) { return artists[_artistId]; // we defined this earlier! } /** @notice This function allows an Artist to withdraw their earnings at any time. Which will be possible to do through the UI in future. */ function artistWithdrawFunds(string calldata _artistId) public payable { require(bytes(artists[_artistId].id).length > 0, "artist does not exist"); // check that the artistID exists Artist storage artist = artists[_artistId]; require(artist.wallet == msg.sender, "only the artist's wallet can call this function"); // only the artist wallet associated with this artistId can withdraw the funds! require(artist.escrow > 0, "artist does not have any money to withdraw"); uint256 escrowToSend = artist.escrow; artist.escrow = 0; // No reentrancy issues for us please! address payable to = payable(msg.sender); to.transfer(escrowToSend); emit ArtistPaid(_artistId, to, escrowToSend); } /** @notice This function is used by the UI. The UI takes in all the artist details from an onboarding form and generates an IPFS CID from their public metadata. It then uses this as the unique artist Id. To disincentivise bad actors, we've also add a small upfront payment to sign on as an artist, which also covers our costs to generate images and update contract details of a new artist. While it's true that anyone could call this function with any string, there is no real value to doing so as no model will be trained nor will it display on the UI. We'll see more on that later! */ function createArtist(string calldata _artistId) external payable { require(bytes(_artistId).length > 0, "please provide an id"); require(bytes(artists[_artistId].id).length == 0, "artist already exists"); require(msg.value >= artistCost, "not enough FIL sent to pay for training artist"); // this is a variable defined in the contract by the admin of the contract & can be changed as needed for gas cost coverage. if(bytes(artists[_artistId].id).length == 0) { artistIDs.push(_artistId); } artists[id] = Artist({ id: _artistId, wallet: msg.sender, escrow: 0, revenue: 0, numJobsRun: 0, isTrained: false }); emit ArtistAdded(_artistId); } /** @notice This function takes advantage of Open Zeppelin's Ownable contact to ensure that only the contract deployer can remove an Artist. -> import "@openzeppelin/contracts/access/Ownable.sol"; -> contract Waterlily is Ownable {} */ function deleteArtist(string calldata id) public onlyOwner { require(bytes(artists[id].id).length > 0, "artist does not exist"); Artist storage artist = artists[id]; require(artist.escrow == 0, "please have the artist withdraw escrow first"); // they have money still to claim delete(artists[id]); // remove from artistIDs for (uint i = 0; i < artistIDs.length; i++) { if (keccak256(abi.encodePacked((artistIDs[i]))) == keccak256(abi.encodePacked((id)))) { artistIDs[i] = artistIDs[artistIDs.length - 1]; artistIDs.pop(); break; } } } /** @notice Example of UpdateArtist function to change an Artist's wallet address. Passing in the artistId helps us find the Artist details */ function updateArtistWallet(string calldata _artistId, address _newWalletAddress) public { require(bytes(artists[_artistId].id).length > 0, "artist does not exist"); Artist storage artist = artists[id]; require(artist.wallet == msg.sender || msg.sender == owner(), "only the artist's wallet can call this function"); //artist or owner artist.wallet = _newWalletAddress; } Hunky dory!!! Err... wait, what about the generated images?! Also, side point - hunky dory is a really weird expression... where did it come from even? Was there a particularly handsome Dory fish somewhere? But I digress... Images! getCustomerImages(address _ownerAddress) createImage(string calldata artistId, string calldata _textPrompt) ImageJobCall (wait... what's this name... read on friend!) Events: event ImageJobCalled(Image image) /** Takes the owner address & returns images associated with it **/ function getCustomerImages(address customerAddress) public view returns (uint[] memory) { return customerImages[customerAddress]; } /** Creates a Stable Diffusion image from a text prompt input with the specific Artist ML model... or does it...? **/ function createImage(string calldata _artistId, string calldata _prompt) external payable { require(bytes(artists[_artistId].id).length > 0, "Artist does not exist"); // make sure the Artist model exists require(artists[_artistId].isTrained, "Artist has not been trained"); // make sure the Artist ML model is trained require(msg.value >= imageCost, "Not enough FIL sent to pay for image generation"); // the _imageCost is another variable set by the contract. This is the money the artist will receive (outside of gas fees for returning the image.... more on that soon) _imageIds.increment(); // increment the image id aka Bacalhau jobId uint id = _imageIds.current(); // create the details of this new AI-generated image images[id] = Image({ id: id, jobId: 0, customer: msg.sender, artist: _artistId, prompt: _prompt, ipfsResult: "", errorMessage: "", isComplete: false, isCancelled: false }); customerImages[msg.sender].push(id); // add the image to the customer owned images emit ImageJobCalled(images[id]); // emit an event saying the image generation function has been called... // if they have paid too much then refund the difference assuming it's a mistake (a user can always donate to an Artist from the Waterlily UI :) ) uint excess = msg.value - imageCost; if (excess > 0) { address payable to = payable(msg.sender); to.transfer(excess); } } For the savvy readers out there, you might now be wondering.... but where and how exactly is the image generated? This createImage() function code just creates a data structure type of Image and stores it on the contract - it does nothing to generate an image... 🤔 actually Well, this is exactly where Project Lilypad comes in! Lilypad - Connecting the FEVM smart contracts to run Stable Diffusion on Bacalhau is a 'relayer' or 'bridge' which enables users to call Bacalhau jobs (like stable diffusion) from a smart contract and have the results of the deterministic job returned to their contract. Project Lilypad Quick refresher: Bacalhau is a peer-to-peer network of nodes (including GPU-enabled nodes) that can run any Docker job or WASM image. 🚩🫶 Very Important Note Here ie. you should probably stop scrolling & read me ! is currently in alpha stage - which means that while it's fully functional, there aren't guarantees on reliability or security of the network. It is deployed to the FVM Calibration Network, FVM Hyperspace Testnet and and connects to the public p2p . Lilypad FVM Mainnet Bacalhau network The team is currently also working on plans for to become much more developer-friendly and expansive, as well as include more comprehensive examples. Lilypad The good news for you early dev's - is it's all currently free to use outside of FVM network gas fees (I like that price!) you can have your say in Lilypad's direction by letting us know what use cases you're interested in our help to enable! and The way Lilypad works to connect FVM and Bacalhau in practice is (aka show me the code!): A developer implements the Lilypad Interface in their own smart contract /** === Lilypad Interface === **/ pragma solidity >=0.8.4; enum LilypadResultType { CID, StdOut, StdErr, ExitCode } interface LilypadCallerInterface { function lilypadFulfilled(address _from, uint _jobId, LilypadResultType _resultType, string calldata _result) external; function lilypadCancelled(address _from, uint _jobId, string calldata _errorMsg) external; } /** === User Contract Example === **/ contract MyContract is LilypadCallerInterface { function lilypadFulfilled(address _from, uint _jobId, LilypadResultType _resultType, string calldata _result) external override { // Do something when the LilypadEvents contract returns // results successfully } function lilypadCancelled(address _from, uint _jobId, string calldata _errorMsg) external override { // Do something if there's an error returned by the // LilypadEvents contract } } A developer also needs to call the LilypadEvents 'runLilypadJob()' function from their contract with a valid spec - which corresponds to a Docker Image. 💡 If you want to know more about what a valid image for Bacalhau would look like or see some examples, check out the Bacalhau or check out the stable diffusion code example provided in the . docs Lilypad github import "./LilypadEvents.sol"; //please check the github for up to date contract, instructions & needed addresses! import "./LilypadCallerInterface.sol"; /** === User Contract Example === **/ contract MyContract is LilypadCallerInterface { address payable public lilypadEventsAddress; constructor (address payable _eventsContractAddress) { lilypadEventsAddress = _eventsContractAddress; } function callLilypadEventsToRunBacalhauJob() external payable { // Require at least 0.03 FIL to be sent require(msg.value >= 30000000000000000, "Insufficient payment"); //this is the default amount 0.03FIL required to run a job. Check the github docs for more info string memory spec = "StableDiffusion"; // Call the function in the other contract and send fee (bool success, uint256 jobId) = lilypadBridgeAddress.call{value: lilypadFee}(abi.encodeWithSignature("runLilypadJob(address, bytes, uint256)", address(this), spec, uint256(LilypadResultType.CID))); require(success, "Failed to call the lilypad Events function to run the job."); //do something with the returned jobId } } 👉 See the LilypadEvents contract in remix 👉 See the LilypadCallerInterface in remix This is all that's needed on the contract side of things to invoke a Bacalhau job to run from your own FEVM smart contract! We just created a decentralised app that isn't just decentralised on-chain, but also uses a decentralised compute service to run an ML model! All that needs to happen now is to put the full contract together and deploy it! Side note on tutorial vs production codebase: The examples above show you how you could build a DApp with similar functionality to Waterlily.ai. In practice, however, Waterlily utilises a private LilypadEvents contract and bridge service (which ensures only compute jobs called from the Waterlily contract are run) and which connects to a private Bacalhau node cluster to run the unique stable diffusion jobs of each artist from the FEVM smart contract. It additionally simplifies the logic needed to handle the individual ML models of the artists. Waterlily.ai was designed this way to ensure we could protect the Artist ML models (which are not publicly owned IP) and ensure that images cannot be generated without payment fees for the original artists. ❤️ It also provides reliability guarantees on the GPU compute nodes in the Bacalhau cluster which are needed for users creating generative images on the DApp as well as for training new Artist models in their unique styles from supplied examples. You can skip to the bonus sections below to see more details on creating a private Bacalhau cluster as well as how you might train unique artist models & pass their spec in though! 😉 Full Contract Below is a full copy of a working solidity contract utilising the Lilypad StableDiffusion spec. 👉 Click here to open this contract in remix! // SPDX-License-Identifier: MIT pragma solidity >=0.8.4; import "hardhat/console.sol"; import "@openzeppelin/contracts/access/Ownable.sol"; import "@openzeppelin/contracts/utils/Strings.sol"; import "./LilypadEventsUpgradeable.sol"; import "./LilypadCallerInterface.sol"; error WaterlilyError(); contract Waterlily is LilypadCallerInterface, Ownable { /** General Variables */ address public lilypadBridgeAddress; uint256 lilypadFee = 0.03 * 10**18; //default fee // These are used in order to ensure the lilypad wallet can cover gas fees // for returning an image to the contract on FVM mainnet. (no charge on testnet, we auto topup tFIL) // The bacalhau network doesn't currently charge for compute use. uint256 public computeProviderEscrow; uint256 public computeProviderRevenue; uint256 public imageCost = 0.13 * 10**18; uint256 public artistCommission = 0.1 * 10**18; uint256 public artistCost = 0.1 * 10**18; uint256 public artistAutoPaymentMin = 3 * 10**18; /** Contract Events **/ event ArtistCreated(string indexed artistId); event ArtistDeleted(string indexed artistId); event ArtistWalletUpdated(string indexed artistId, address _prevWallet, address _curWallet); event ArtistPaid(string indexed artistId, address _paidTo, uint256 amountPaid); event ImageJobCalled(Image image); event ImageComplete(Image image); //we shouldn't broadcast the results publicly in reality. event ImageCancelled(Image image); /** Artist Data Structures / vars **/ struct Artist { string id; address wallet; uint256 escrow; uint256 revenue; uint256 numJobsRun; bool isTrained; } mapping (string => Artist) artists; string[] artistIDs; /** Image Events **/ event ImageCreated(Image image); /** Image Data Structures / vars **/ using Counters for Counters.Counter; Counters.Counter private _imageIds; struct Image { uint id; uint jobId; //the returned id for a job run by Lilypad. address customer; string artist; string prompt; string ipfsResult; string errorMessage; bool isComplete; bool isCancelled; } mapping (uint => Image) images; mapping (address => uint[]) customerImages; /** Initialise our contract vars **/ constructor(address _lilypadEventsContractAddress){ lilypadBridgeAddress = _lilypadEventsContractAddress; } /** Artist Functions **/ function getArtistIds() public view returns (string[] memory) { return artistIDs; } function getArtistById(string calldata _artistId) public view returns (Artist memory) { return artists[_artistId]; } function artistWithdrawFunds(string calldata _artistId) public payable { require(bytes(artists[_artistId].id).length > 0, "artist does not exist"); Artist storage artist = artists[_artistId]; require(artist.wallet == msg.sender, "only the artist's wallet can call this function"); require(artist.escrow > 0, "artist does not have any money to withdraw"); uint256 escrowToSend = artist.escrow; artist.escrow = 0; address payable to = payable(msg.sender); to.transfer(escrowToSend); } function createArtist(string calldata _artistId) external payable { require(bytes(_artistId).length > 0, "please provide an id"); require(bytes(artists[_artistId].id).length == 0, "artist already exists"); require(msg.value >= artistCost, "not enough FIL sent to pay for training artist"); if(bytes(artists[_artistId].id).length == 0) { artistIDs.push(_artistId); } artists[_artistId] = Artist({ id: _artistId, wallet: msg.sender, escrow: 0, revenue: 0, numJobsRun: 0, isTrained: false }); emit ArtistCreated(_artistId); // we don't directly trigger training of models via this function // though it is possible to do with lilypad } function deleteArtist(string calldata id) public onlyOwner { require(bytes(artists[id].id).length > 0, "artist does not exist"); Artist storage artist = artists[id]; require(artist.escrow == 0, "please have the artist withdraw escrow first"); // they have money still to claim delete(artists[id]); for (uint i = 0; i < artistIDs.length; i++) { if (keccak256(abi.encodePacked((artistIDs[i]))) == keccak256(abi.encodePacked((id)))) { artistIDs[i] = artistIDs[artistIDs.length - 1]; artistIDs.pop(); break; } } } function updateArtistWallet(string calldata _artistId, address _newWalletAddress) public { require(bytes(artists[_artistId].id).length > 0, "artist does not exist"); Artist storage artist = artists[_artistId]; require(artist.wallet == msg.sender || msg.sender == owner(), "only the artist's wallet can call this function"); //artist or owner artist.wallet = _newWalletAddress; } /** Image Functions **/ function getCustomerImages(address customerAddress) public view returns (uint[] memory) { return customerImages[customerAddress]; } function createImage(string calldata _artistId, string calldata _prompt) external payable { require(bytes(artists[_artistId].id).length > 0, "Artist does not exist"); require(artists[_artistId].isTrained, "Artist has not been trained"); require(msg.value >= imageCost, "Not enough FIL sent to pay for image generation"); _imageIds.increment(); uint id = _imageIds.current(); /** This is where we call the lilypad bridging contract */ /** NOTE: In reality each of your artists would need a different bacalhau job spec as each artist uses a unique model to generate images in their specific style. Note the "Image" provided - a generic stable diffusion job **/ string constant specStart = '{' '"Engine": "docker",' '"Verifier": "noop",' '"Publisher": "estuary",' '"Docker": {' '"Image": "ghcr.io/bacalhau-project/examples/stable-diffusion-gpu:0.0.1",' '"Entrypoint": ["python", "main.py", "--o", "./outputs", "--p", "'; string constant specEnd = '"]},' '"Resources": {"GPU": "1"},' '"Outputs": [{"Name": "outputs", "Path": "/outputs"}],' '"Deal": {"Concurrency": 1}' '}'; //How the spec is managed to add the prompt is a little awkward atm, but alpha's always improve :) string memory spec = string.concat(specStart, _prompt, specEnd); // Cast enum value to uint8 before passing as argument (yes this is DX we want to fix :P) uint8 resultType = uint8(LilypadResultType.CID); // Call the function in the other contract and send funds (bool success, bytes memory result) = lilypadBridgeAddress.call{value: lilypadFee}(abi.encodeWithSignature("runLilypadJob(address, string, uint8)", address(this), _spec, resultType)); require(success, "Failed to call the lilypad Events function to run the job."); uint jobId; assembly { jobId := mload(add(result, 32)) } images[id] = Image({ id: id, jobId: jobId, customer: msg.sender, artist: _artistId, prompt: _prompt, ipfsResult: "", errorMessage: "", isComplete: false, isCancelled: false }); customerImages[msg.sender].push(id); emit ImageJobCalled(images[id]); uint excess = msg.value - imageCost; if (excess > 0) { address payable to = payable(msg.sender); to.transfer(excess); } } /** Lilypad Functions (as I said - this is in alpha!) **/ function lilypadFulfilled(address _from, uint _jobId, LilypadResultType _resultType, string calldata _result) external override { //the image has come back successfully Image storage image = images[_jobId]; // sanity check that the image exists with that ID require(image.id > 0, "image does not exist"); require(image.isComplete == false, "image already complete"); require(image.isCancelled == false, "image was cancelled"); // get a reference to the artist for this image Artist storage artist = artists[image.artist]; // edge case where we deleted the artist in between the job // starting and completing require(bytes(artist.id).length > 0, "artist does not exist"); // update the result of the image image.ipfsResult = _result; image.isComplete = true; // update the artist details accordingly artist.escrow += artistCommission; artist.revenue += artistCommission; artist.numJobsRun += 1; computeProviderEscrow += imageCost - artistCommission; computeProviderRevenue += imageCost - artistCommission; //auto-pay the artist if their escrow is > AutopayMinValue if(artist.escrow > artistAutoPaymentMin){ uint256 escrowToSend = artist.escrow; address payable recipient = payable(artist.wallet); //should check contract balance here before proceeding artist.escrow = 0; recipient.transfer(escrowToSend); emit ArtistPaid(artist.id, recipient, escrowToSend); } emit ImageComplete(image); } function lilypadCancelled(address _from, uint _jobId, string calldata _errorMsg) external override { //something went wrong with the job - refund customer as much as possible Image storage image = images[_jobId]; // sanity check that the image exists with that ID require(image.id > 0, "image does not exist"); require(image.isComplete == false, "image already complete"); require(image.isCancelled == false, "image was cancelled"); // mark the image as cancelled and refund the customer image.isCancelled = true; image.errorMessage = _errorMsg; // in reality you might want to subtract the cost of the newtork gas costs for the bridge return here uint256 amountRefundable = imageCost; //-lilypad costs address payable to = payable(image.customer); to.transfer(amountRefundable); emit ImageCancelled(image); } } You might notice that there's a few extra pieces in this contract we haven't so far discussed including: This code pays out the artist when their escrow hits a specific amount. A technical decision was made against instant payments here so as to avoid the artist losing too much to gas fees. In future iterations though, micropayments could be enabled via an awesome project on the FVM mainnet called . Paying Artist in LilypadFulfilled: State Channels In practice, you'd also likely want to enable a way to update the global variables of lilypadBridgeAddress, lilypadFee, imageCost, artistCommission, artistCost and artistAutoPaymentMin. I will additionally note that it might be advantageous to also make this contract upgradeable with OpenZeppelin's great guide to upgradeable contracts and you could also take advantage of the for artist payments rather than handle this manually 😎 here OpenZeppelin PaymentSplitter contract Deploying the (Complete) Smart Contracts to FVM Right, we have a contract! Let's deploy this to the FVM! FVM Network details See See also https://docs.filecoin.io/networks chainlist.org Chain Name RPC ChainID BlockExplorer Faucet Filecoin Calibration Net ( ) testnet https://api.calibration.node.glif.io/rpc/v0 314159 , https://calibration.filscan.io/ https://faucet.calibration.fildev.network/ Filecoin Hyperspace ( ) testnet , , , https://api.hyperspace.node.glif.io/rpc/v1 https://hyperspace.filfox.info/rpc/v1 https://filecoin-hyperspace.chainstacklabs.com/rpc/v1 https://rpc.ankr.com/filecoin_testnet 3141 , , https://fvm.starboard.ventures/hyperspace/explorer/tx/ https://explorer.glif.io/ https://hyperspace.yoga/#faucet Filecoin Mainnet , , https://api.node.glif.io/rpc/v1 https://filecoin-mainnet.chainstacklabs.com/rpc/v1 https://rpc.ankr.com/filecoin 314 , , , https://fvm.starboard.ventures/ https://explorer.glif.io/ https://beryx.zondax.ch/ https://filfox.io/ There's quite a few ways to go about deploying this contract - I'll go through deployment on both remix (available in your browser) and utilising - an Ethereum developer environment tool. Hardhat Pre-requisites: To deploy to a Filecoin network, we'll need to wallet (you can use other compatible wallets, though I'll be using Metamask here) Set up & connect Metamask Obtain funds for deployment (if on testnet, you can use the faucets, otherwise see ) this guide To add the network you're deploying to in the wallet, you can either look it up on and add it via their interface, or you can add it manually. chainlist.org To manually add it, navigate to your Metamask wallet and click "Add network" button Fill out the following details and then click 'Save" If you're deploying to a testnet, you'll need to use the faucet to add funds to your wallet - see the above table for the faucet links. Note the Calibration Net faucet requires a filecoin type address - which you can find by connecting your metamask wallet at . https://www.glif.io/ If you're using mainnet, you can add funds via your ledger or from an exchange. See for more info. this guide Deploy on Remix It's fairly simple to deploy to FVM in remix using Metamask Wallet. To do so open the contract by clicking 👉 open in remix! You'll also need the Lilypad contracts, so go ahead and copy them into your open remix browser window too (you can find them - we're working on npm imports!). here Once you have the 2 contracts and the LilypadCallerInterface in remix, make sure your metamask wallet is connected to your desired network and you have funds in your wallet. Go to the compiler tab (solidity icon) and click compile Then go to the Ethereum tab (Ethereum icon) and choose "Injected Provider - Metamask" - this will link to your current Metamask network and wallet settings. Then you can deploy this using the big orange Deploy button on the left and passing in the LilypadEvents contract address (for the network you're deploying to) as a constructor argument. Metamask will ask you to approve the deployment transaction & fee. On confirming, the contract will be deployed to your connected network. You can look it up in a block explorer or play around with it in remix (bottom left): ⚠️ Don't forget to save the address for use in our front end!! ⚠️ Deploy on Hardhat is a development environment for editing, compiling, debugging & deploying Ethereum software. As the FVM is EVM-compatible we can take advantage of this tool Hardhat Hardhat also provides the benefit of enabling a local chain as well as a testing suite. To set up a hardhat project see their - they even have an extension for VS code editors. docs here Note: Hardhat for typescript requires some specific tsconfig options, that don't always play well with NextJs tsconfig needs. So if you're looking to have hardhat nested inside your front-end directory, you'll be giving yourself some setup challenges. The wallet private key, needed in the config for hardhat is available via Metamask -> account details -> export private key, and is here stored in a .env variable. hardhat.config.ts import { HardhatUserConfig, task } from 'hardhat/config'; import '@nomicfoundation/hardhat-toolbox'; import '@openzeppelin/hardhat-upgrades'; import 'hardhat-deploy'; import { config as dotenvConfig } from 'dotenv'; import { resolve } from 'path'; const dotenvConfigPath: string = process.env.DOTENV_CONFIG_PATH || './.env'; dotenvConfig({ path: resolve(__dirname, dotenvConfigPath) }); const walletPrivateKey: string | undefined = process.env.WALLET_PRIVATE_KEY; if (!walletPrivateKey) { throw new Error('Please set your Wallet private key in a .env file'); } const config: HardhatUserConfig = { solidity: '0.8.17', defaultNetwork: 'hardhat', namedAccounts: { admin: 0, }, networks: { hardhat: {}, localhost: {}, filecoinHyperspace: { url: 'https://api.hyperspace.node.glif.io/rpc/v1', //'https://rpc.ankr.com/filecoin_testnet', //https://filecoin-hyperspace.chainstacklabs.com/rpc/v1, "https://hyperspace.filfox.info/rpc/v1", wss://wss.hyperspace.node.glif.io/apigw/lotus/rpc/v1 chainId: 3141, accounts: [walletPrivateKey], }, filecoinCalibrationNet: { url: 'https://api.calibration.node.glif.io/rpc/v0', chainId: 314159, accounts: [walletPrivateKey], }, filecoinMainnet: { url: 'https://api.node.glif.io', //'https://rpc.ankr.com/filecoin_testnet', //https://filecoin-hyperspace.chainstacklabs.com/rpc/v1 chainId: 314, accounts: [walletPrivateKey], }, }, }; export default config; Now let's use a script to deploy Waterlily.sol! scripts/deploy.ts import hre, { ethers } from 'hardhat'; import type { Waterlily } from '../typechain-types/contracts/Waterlily'; import type { Waterlily__factory } from '../typechain-types/factories/contracts/Waterlily__factory'; async function main() { const walletPrivateKey: string | undefined = process.env.WALLET_PRIVATE_KEY; if (!walletPrivateKey) { throw new Error('Please set your Wallet private key in a .env file'); } const owner = new ethers.Wallet( walletPrivateKey, hre.ethers.provider ); console.log('Waterlily deploying from account: ', owner); console.log("Account balance:", (await owner.getBalance()).toString()); let LILYPAD_ADDRESS; if(hre.network.name == 'filecoinHyperspace') { LILYPAD_ADDRESS = "", //define me } if(hre.network.name == 'filecoinCalibrationNet') { LILYPAD_ADDRESS = "", //define me } if(hre.network.name == 'filecoinMainnet') { LILYPAD_ADDRESS = "", //define me } const WaterlilyFactory: Waterlily__factory = await ethers.getContractFactory('Waterlily') const waterlilyContract: Waterlily = await WaterlilyFactory.connect(owner).deploy( LILYPAD_ADDRESS ) await waterlilyContract.deployed() console.log('Waterlily deployed to ', waterlilyContract.address) } main().catch((error) => { console.error(error) process.exitCode = 1 }) You can run this script using the following in the command line of your project npx hardhat run scripts/deploy.js --network The contract address will show on successful deployment. ⚠️ Don't forget to save the address for use in our front end!! ⚠️ Celebrate! You just deployed your contracts to FVM! 📺 Building the User Interface The UI, or front-end is built on React (NextJS) and Typescript. To build the front end, I'm using NextJS and Typescript. Though, while I use NextJS's routing, I'm not taking advantage of any of NextJS's SSR (server side rendering) features, so you could really just go with a vanilla React set up (or any framework of your choice of course!). UI Overview The UI currently consists of two pages built on NextJS and Typescript. The main page is the home page. This is where a user is able to generate new images from a prompt input and browse through artist profiles to decide on a style for the generative art. It is also the page that shows a user's collection of previously generated images and NFTs The second page is the Artist Signup form for new artists to join the platform. UI Contract Connections There are 3 types of contract interactions here read-only calls to retrieve data from the chain without mutating it write calls that require a wallet to sign and pay gas ie. functions that change the state of the chain, like generating a new image, adding an artist or minting an NFT! event listeners - that listen for events emitted from the contract For all of these functions, we'll use the - a lightweight wrapper for the Ethereum API, to connect to our contract and perform calls to it. ethers.js library Connecting to the contract in read mode with a public RPC: //The compiled contract found in hardhat/artifacts/contracts import WaterlilyCompiledContract from '@Contracts/Waterlily.sol/Waterlily.json'; //On-chain address of the contract const contractAddress = ' '; //A public RPC Endpoint (see table from contract section) const rpc = '<chain rpc http'; const provider = new ethers.providers.JsonRpcProvider(rpc); const connectedReadWaterlilyContract = new ethers.Contract( contractAddress, WaterlilyCompiledContract.abi, provider ); Fetching data from the contract is a read event, so we can use the above to fetch the artist details or fetch the previously generated images belonging to a user's wallet const imageIDs = await connectedReadWaterlilydContract?.getCustomerImages(customerWallet); Listening for events on the contract is also a read-only (get) event, we can use the public RPC to listen for event emissions on-chain. //use the read-only connected Bacalhau Contract connectedReadWaterlilyContract.on( // Listen for the specific event we made in our contract 'ImageComplete', (Image: image) => { //DO STUFF WHEN AN IMAGEEVENT COMES IN // eg. filter if it is this user's and reload generated imagese-display or // } ); - this requires that the Ethereum object is being injected into the web browser by a wallet so that a user can sign for a transaction and pay for gas - which is why we're checking for a window.ethereum object. Connecting to the contract in write mode //Typescript needs to know window is an object with potentially and ethereum value. declare let window: any; //The compiled contract found in hardhat/artifacts/contracts import WaterlilyCompiledContract from '@Contracts/Waterlily.sol/Waterlily.json'; //On-chain address of the contract const contractAddress = ' '; //check for the ethereum object if (!window.ethereum) { //ask user to install a wallet or connect //abort this } // else there's a wallet provider else { // same function - different provider - this one has a signer - the user's connected wallet address const provider = new ethers.providers.Web3Provider(window.ethereum); const contract = new ethers.Contract( contractAddress, WaterlilyCompiledContract.abi, provider ); const signer = provider.getSigner(); const connectedWriteWaterlilyContract = contract.connect(signer); } Calling the CreateImage() function in the Waterlily contract with a value of $imageCost: const runStableDiffusionJob = async (prompt: string, artistid: string) => { try { console.log('Calling stable diffusion function in Waterlily contract...'); const tx = await connectedWriteWaterlilyContract.CreateImage(artistid, prompt, { value: imageCost }); //wait for netowrk to include tx in a block const receipt = await tx.wait(); //wait for transaction to be run and return a receipt } catch (error: any) { console.error(error); } } Calling the minting function for the NFT contract is also described in . this blog UI Wallet Connections Here are a few helpful wallet functions you might want, including how to check the chainId or wallet balance, and how to programmatically add a network to Metamask / wallet.You can interact with wallets using the Metamask Ethereum object directly -as I'm doing below or by using ethers.js or web3.js. declare let window: any; const fetchWalletAccounts = async () => { console.log('Fetching wallet accounts...'); await window.ethereum //use ethers? .request({ method: 'eth_requestAccounts' }) .then((accounts: string[]) => { return accounts; }) .catch((error: any) => { if (error.code === 4001) { // EIP-1193 userRejectedRequest error console.log('Please connect to MetaMask.'); } else { console.error(error); } }); }; const fetchChainId = async () => { console.log('Fetching chainId...'); await window.ethereum .request({ method: 'eth_chainId' }) .then((chainId: string[]) => { return chainId; }) .catch((error: any) => { if (error.code === 4001) { // EIP-1193 userRejectedRequest error console.log('Please connect to MetaMask.'); } else { console.error(error); } }); }; const fetchWalletBalance = async ( account: string ): Promise => { let balanceNumber: number; try { const balance = await window.ethereum.request({ method: 'eth_getBalance', params: [account], }); const formattedBalance = ethers.utils.formatEther(balance); balanceNumber = parseFloat(formattedBalance); } catch (error) { console.error('error getting balance', error); } } //!! This function checks for a wallet connection WITHOUT being intrusive to to the user or opening their wallet export const checkForWalletConnection = async () => { if (window.ethereum) { console.log('Checking for Wallet Connection...'); await window.ethereum .request({ method: 'eth_accounts' }) .then(async (accounts: String[]) => { console.log('Connected to wallet...'); // Found a user wallet return true; }) .catch((err: Error) => { console.log('Error fetching wallet', err); return false; }); } else { //Handle no wallet connection return false; } }; //Subscribe to changes on a user's wallet export const setWalletListeners = () => { console.log('Setting up wallet event listeners...'); if (window.ethereum) { // subscribe to provider events compatible with EIP-1193 standard. window.ethereum.on('accountsChanged', (accounts: any) => { //logic to check if disconnected accounts[] is empty if (accounts.length { // handle changed chain case }); // Subscribe to chainId change window.ethereum.on('balanceChanged', () => { // handle changed balance case }); } else { //handle the no wallet case } }; export const changeWalletChain = async (newChainId: string) => { console.log('Changing wallet chain...'); const provider = window.ethereum; try { await provider.request({ method: 'wallet_switchEthereumChain', params: [{ chainId: newChainId }], //newChainId }); } catch (error: any) { alert(error.message); } }; //AddChain Example export const addWalletNetwork = async () => { console.log('Adding the Hyperspace Network to Wallet...'); if (window.ethereum) { window.ethereum .request({ method: 'wallet_addEthereumChain', params: [ { chainId: '', rpcUrls: [ 'https://', 'https://', ], chainName: 'Filecoin ...', nativeCurrency: { name: 'tFIL', symbol: 'tFIL', decimals: 18, }, blockExplorerUrls: [ 'https://fvm.starboard.ventures/contracts/', 'https://hyperspace.filscan.io/', ], }, ], }) .then((res: XMLHttpRequestResponseType) => { console.log('added new network successfully', res); }) .catch((err: ErrorEvent) => { console.error('Error adding new network', err); }); } }; 💫 Bonus Code Section Bonus: Creating & Training the Artist ML Models I've previously written about how to create your own open source Stable Diffusion script and run it on Bacalhau . here https://www.youtube.com/watch?v=53uY48e1lis&t=1454s&embedable=true Underlying the finetuning is , which makes stable diffusion even more powerful with the ability to generate realistic looking pictures of humans, animals or any other object by just training them on 20-30 images. Dreambooth The specific models used for Waterlily, however, were created by from (he does some awesome work - check him and the team out!). Richard Blythman Algovera There's a great guide on how you could get started making scripts like this in the Bacalhau docs on this . here Check the following repo's out for more inspiration: Training and inference of images: https://github.com/JoePenna/Dreambooth-Stable-Diffusion Preparation of regularisation images: https://github.com/aitrepreneur/SD-Regularization-Images-Style-Dreambooth Bonus: Creating a Private Bacalhau Node Cluster A private cluster is a network of Bacalhau nodes completely isolated from any public node. That means you can safely process private jobs and data on your cloud or on-prem hosts.To create your own private Bacalhau node cluster - you can follow the guide . here 🌟 Possibilities for AI & Blockchain The main power of this POC application lies in its open-source tech stack and in combining the capability of decentralised and verifiable ML & AI with Blockchain payments and traceability. For me, the current AI trajectory just makes me more sure that blockchain is going to be an absolutely essential and powerful tool for the future - with its ability to provide both a settlement layer for payments as well as verifiable provenance of items on an immutable ledger. Proving for truth, authenticity and provenance of data and content are going to be essential in an AI-world. This applies not just to art content, but to any content being released, be it Tay Tay's new music album, or, more seriously, news content, scientific research papers and more. https://twitter.com/rpnickson/status/1639813074176679938?s=20&embedable=true Waterlily is aiming to provide a new revenue stream for original creators and perhaps tackle some of the current issues with AI-Art generation. It could also serve as a thought experiment for how we go about both tokenising and training datasets in fully decentralised ways that continues to give individuals authority over their own data. 🗺️ Roadmap is an open source repository. We welcome input via the github or submissions of ideas and feedback. Tell us what you think! Tell us what you think should happen next, or help us fix the 🐛🪲🐞 's Waterlily.ai If you're an artist, we'd also love to hear from you! Please get in touch :) ✍️ Keep in touch! Congrats if you read all the way through!!! A big thank you to the team that helped me develop and deploy Waterlily & Lilypad - including , , , , from and the rest of the crew - without whom this app would not have been possible. Luke Marsden Kai Davenport @Wes Floyd Simon Worthington Richard Blythman Algovera Bacalhau Keep in touch with us! #bacalhau Filecoin Project Slack With ♥️ DeveloperAlly Also published . here