This is the first post in this series where I build . In this post I focus on setting up the architecture to build an HTTP endpoint to do the following: a web chat powered by GCP free limits Anonymous users choose a username Users are ranked by recency of last access time After 10 minutes of inactivity, users are purged from the system A maximum of 100 users at a time Architecture and choice of components I want to have an effectively free infrastructure using GCP’s perpetually . The components for this post: free tier I will use to run the backend code. Cloud functions have a free limit of 2 million requests per month. I chose cloud functions because primarily because it requires little code to get started and I get observability and scalability out of the box. Cloud function: HTTP-based serverless cloud functions To store users’ data, I will use Redis because it is lightweight to run, fast and versatile for storing all kinds of data I will need for the chat app. But GCP’s implementation of Redis is not free. I work around this by installing Redis on a free compute instance (this somewhat limits scalability, but I have some room due to Redis’s memory efficiency). Redis: MemoryStore Connecting to rest of my VPC from cloud functions requires a . I work around this by connecting to Redis over the internet, but I need to ensure it is done over a secure connection. Connecting with Redis over the internet: (Serverless VPC access), which is not free VPC connector Installing Redis Google Cloud provides a free e2-micro instance as part of their monthly free tier. I will install Redis as a service on a compute instance. Providing an e2 micro instance in region -- let’s call the instance us-central1 pelican Reserve a static IP address to use and attach it to here pelican SSH into the instance and install the ( to get best TLS support) latest version Redis atleast version 7.0, If the compute instance is based on Debian, it is best to update the Redis config to be supervised by which is the default init system: change the file to update the supervision setting: systemd /etc/redis/redis.conf supervised systemd Remove the bind address to be nothing (all network interfaces on the instance should be able to access Redis) and the port to , add using the setting 12345 a password requirepass One last thing to do is to have access to the home directory -- this is only necessary because I store some files (generated later) in the home directory. allow the Redis service to be managed by systemd To do this, add or update the service file with this setting to read only: /etc/systemd/system/redis.service ProtectHome=read-only Reload systemd sudo systemctl daemon-reload Restart Redis sudo systemctl restart redis I can test Redis via the CLI locally on the instance mourjo@pelican:~$ redis-cli -p 12345 127.0.0.1:11219> AUTH somepassword OK 127.0.0.1:11219> set x y OK 127.0.0.1:11219> get x "y" I want to connect to Redis over the internet, so first allow the port on to be accessible. On the in GCP console, create a new allow list with the following settings: pelican firewall tab Priority: 70 (less than the default value of 100) Direction: ingress Target tags: A tag that I will apply to the compute instance Source IPv4 ranges: allows every host to connect to this port 0.0.0.0/0 Protocol: TCP on port 12345 Edit the instance to apply this firewall using the “network tags” setting and using the tag I created the firewall with above (note that this opens up my instance to be accessible to anyone, so it is highly recommended to set a password as above) It should now be possible to connect to Redis from outside of the machine, for example from a computer that has not SSH’ed into the instance 🎉 Enabling TLS support for Redis So far I have set a password to access Redis but that data is still being transmitted over an unencrypted TCP connection which has two problems: It is very easy to and read the password intercept traffic Anyone can connect and start even without authenticating, causing my tiny instance to be overwhelmed sending a flood of traffic Both problems can be solved by using a where both the server and the client need to establish their identity even before interacting with Redis. . Therefore only known clients with the right private keys can access my server over an encrypted connection. secure connection over TLS (or mTLS) ensures that both the client and the server verify each other Mutual TLS Digital Certificates and Trust Establishment Digital certificates are commonly used in TLS and HTTPS to verify the identity of participants. In short, . a certificate is cryptographically verifiable proof that someone is who they say they are as attested by another authority If I am connecting to google.com, I need to know that I am in fact connecting to google.com and not someone impersonating google.com. This happens via a trust chain established between me (the browser) and the server (google.com): Google sends a certificate that has been cryptographically signed by a certifying authority (CA) and that signature is verifiable by us. It is important to note that I need to implicitly . This usually happens on the web via known issuers of certificates baked into browsers and operating systems (eg stores the CAs a Linux system implicitly trusts). Once I see a certificate that is signed by someone I trust, I can verify that the party claiming to be who they are (here google.com) is actually Google and not a spoof pretending to steal information from us. In reality, there may be intermediate authorities that do the actual certificate signing, but the intermediate CAs themselves must also have certificates signed by the root CA. trust the CA without question /etc/ssl/certs Image source. of the party whose identity is being verified. Using the server’s public key from the certificate, the client can encrypt a payload that only the server can decrypt, thus making the communication secure: Certificates contain the public key no other observer of the network can decrypt the information being communicated. This is called an mTLS handshake. It is slightly more complicated in reality, but it is the same in principle: Generating the Certificates The first step to a TLS connection is to have a certificate for the server and the client, both of which are signed by a CA. I don’t want to purchase a certificate so I will generate my own CA certificate, which is not trusted in the wild, so I will need to tell the server and client to accept all certificates issued by this CA. Create the root CA key and certificate -- this will be implicitly trusted by both client and server Create a new private key for the root CA (optionally encrypt it with AES 256) openssl genrsa -aes256 -out ca.key 4096 Create the certificate valid for 10 years signed by this private key openssl req -new -x509 -days 3650 -key ca.key -out ca.crt Create the client certificate signed by the root CA Generate a new private key for the client (same command as 1a) openssl genrsa -aes256 -out client.key 2048 Create a certificate signing request, which contains the parameters the certificate will be created with openssl req -new -key client.key -out client.csr Create the certificate signed by the CA openssl x509 -req -days 3650 -in client.csr -CA ca.crt -CAkey ca.key -out client.crt -CAcreateserial Create the server certificate signed by the root CA Generate a new private key for the server (same command as 1a) openssl genrsa -aes256 -out server.key 2048 Create a certificate signing request (it is customary to use the server IP address when prompted for ) Common Name openssl req -new -key server.key -out server.csr Create a certificate signed by the CA openssl x509 -req -days 3650 -in server.csr -CA ca.crt -CAkey ca.key -out server.crt Setting up Redis with the certificates Using the certificates, let’s make these changes to Redis config: Enable TLS, disable the TCP port Add server certificate, server key Add CA certificate Enforce clients to be authenticated Update the file with these settings (here is the ): /etc/redis/redis.conf full config port 0 tls-port 12345 # server certificate and key: tls-cert-file /home/mourjo/certs/server.crt tls-key-file /home/mourjo/certs/server.key # the key file is encrypted, so add the password tls-key-file-pass thisisredacted # the CA certificate tls-ca-cert-file /home/mourjo/certs/ca.crt # make client authentication mandatory tls-auth-clients yes Starting the server should show the following logs sudo systemctl start redis tail /var/log/redis/redis-server.log 148924:M 27 Jan 2023 04:16:10.675 # Server initialized ... 148924:M 27 Jan 2023 04:16:10.676 * Ready to accept connections I can verify using OpenSSL’s that signed by my CA: s_client Redis does not accept a client that does not have a valid certificate # without any certificate openssl s_client -state -connect 35.209.163.139:11219 -servername 35.209.163.139 8610505984:error:1404C45C:SSL routines:ST_OK:reason(1116):/AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/ssl/tls13_lib.c:129:SSL alert number 116 # with a certificate not signed by the CA openssl s_client -state -cert random.crt -key random.key -connect <ip-address>:<port> -servername <ip-address> 8610505984:error:1404C418:SSL routines:ST_OK:tlsv1 alert unknown ca:/AppleInternal/Library/BuildRoots/810eba08-405a-11ed-86e9-6af958a02716/Library/Caches/com.apple.xbs/Sources/libressl/libressl-3.3/ssl/tls13_lib.c:129:SSL alert number 48 I can verify the SSL connection works when passing the right certificate: openssl s_client -state -cert client.crt -key client.key -connect <ip-address>:<port> -servername <ip-address> This should print out the necessary information about the server and security and also allow us to run Redis commands (because is human-readable): the Redis protocol Enter pass phrase for certprac/client.key: CONNECTED(00000003) depth=1 C = IN, ST = WB, L = Kolkata, O = mourjo.me, emailAddress = hello@mourjo.me verify error:num=19:self signed certificate in certificate chain verify return:0 write W BLOCK --- Certificate chain 0 s:/C=IN/ST=WB/L=Kolkata/CN=35.209.163.139/emailAddress=hello@mourjo.me i:/C=IN/ST=WB/L=Kolkata/O=mourjo.me/emailAddress=hello@mourjo.me 1 s:/C=IN/ST=WB/L=Kolkata/O=mourjo.me/emailAddress=hello@mourjo.me i:/C=IN/ST=WB/L=Kolkata/O=mourjo.me/emailAddress=hello@mourjo.me --- Server certificate -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- subject=/C=IN/ST=WB/L=Kolkata/CN=35.209.163.139/emailAddress=hello@mourjo.me issuer=/C=IN/ST=WB/L=Kolkata/O=mourjo.me/emailAddress=hello@mourjo.me --- No client certificate CA names sent Server Temp Key: ECDH, X25519, 253 bits --- SSL handshake has read 3049 bytes and written 1768 bytes --- New, TLSv1/SSLv3, Cipher is AEAD-CHACHA20-POLY1305-SHA256 Server public key is 2048 bit Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.3 Cipher : AEAD-CHACHA20-POLY1305-SHA256 Session-ID: Session-ID-ctx: Master-Key: Start Time: 1674968406 Timeout : 7200 (sec) Verify return code: 19 (self signed certificate in certificate chain) --- read R BLOCK read R BLOCK AUTH SMASH-workaday-sully +OK PING +PONG SET a b +OK GET a $1 b ^C I can also verify the SSL settings using an online tool like which reports that the SSL connection is set correctly, but issued by a CA that is not widely trusted in the world because I am using a CA who only I know. sslshopper Let us now connect using the Redis command-line client from outside the instance, passing the client certificate: redis-cli redis-cli -p 11219 --tls --cert client.crt --key client.key --cacert ca.crt Enter PEM pass phrase: 127.0.0.1:11219> auth thishasbeenredacted OK 127.0.0.1:11219> set xyz abc OK 127.0.0.1:11219> get abc Redis is now working on TLS over the internet! 🎉 Client Implementation I have set up the Redis server and it works with the default client but I want to build a simple backend using Redis: redis-cli Allow a user to use their username -- authentication is out of scope for this post Keep track of active users for 10 minutes and purge inactive users Allow a maximum of 100 users at a time I will deploy an HTTP cloud function that stores and communicates with the Redis server. The backend code is written in Java using the popular for communicating with Redis. Jedis library The business logic is fairly simple, every time I get a new user, I check the limit of 100 users, purge inactive users and store the current user in a with the current timestamp. sorted set double timeoutMillis = 10 * 60 * 1000D; int MAX_USERS = 100; // connect with TLS, pass connect timeout and socket timeout of 10 sec try (Jedis jedis = new Jedis(host, port, 10_000, 10_000, true)) { jedis.auth(redisPassword); // purge old users jedis.zremrangeByScore("recent_users", Double.NEGATIVE_INFINITY, System.currentTimeMillis() - timeoutMillis); // ensure that there only 100 users at max var total_users = jedis.zcount("recent_users", Double.NEGATIVE_INFINITY, Double.POSITIVE_INFINITY); if (total_users >= MAX_USERS) { throw new TooManyUsersException(total_users); } // add the current user to the sorted set with the timestamp jedis.zadd("recent_users", (double) System.currentTimeMillis(), user); // return a list of active users var activeUsers = jedis.zrangeWithScores("recent_users", 0, System.currentTimeMillis()); Collections.reverse(activeUsers); return activeUsers; } The rest of the client code uses the for cloud functions which wrap around the above business logic to respond to incoming requests. Java SDK The full repository is available . here Adding client certificates to Java Keystore The above code snippet although my Redis server only accepts TLS connections from clients that are trusted via my CA. This is as per the : , which ships directly with the JDK installation. So the client code while setting up the TLS connection transparently uses the certificates present in these protected files. makes no mention of certificates Java Cryptography Architecture (JCA) credentials/certificates are stored in password-protected files managed by the CLI utility keytool But before I start importing my certificates with , keytool I need to convert my certificates from the PEM format (which is the Linux default) to the PKCS12 format. Convert the CA root certificate to the format: PKCS12 openssl pkcs12 -export -in ca.crt -inkey ca.key -out ca.p12 Convert the client certificate (note that the key needs to be included in the p12 file): openssl pkcs12 -export -in client.crt -inkey client.key -out client.p12 As per common practice, will create two files with : one called the and the other . keytool keystore truststore The will store , that is the client’s private key and the client’s certificate that will allow us to prove to the Redis server that the client should be allowed to connect. keystore the keys/certificates the client needs to identify itself keytool -importkeystore -noprompt -srckeystore client.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype PKCS12 The will , that is the root CA’s certificate. truststore store the certificates the client will implicitly trust With this, when the server sends its certificate, the client will know that it can trust the server. The will have my CA certificate: truststore keytool -importkeystore -noprompt -srckeystore ca.p12 -srcstoretype PKCS12 -destkeystore truststore.jks -deststoretype PKCS12 Announce to the JVM where it should look for credentials/certificates and set the following options either as CLI options or load them in the code: System.setProperty("javax.net.ssl.keyStorePassword", "redacted"); // decryption password used while generating the keystore: System.setProperty("javax.net.ssl.keyStore", "keystore.jks"); System.setProperty("javax.net.ssl.keyStoreType", "PKCS12"); System.setProperty("javax.net.ssl.trustStorePassword", "redacted"); // decryption password used while generating the truststore System.setProperty("javax.net.ssl.trustStore", "truststore.p12"); System.setProperty("javax.net.ssl.trustStoreType", "PKCS12"); If all goes well, the following should return a from Redis server: PONG try (Jedis jedis = new Jedis(host, port, true)) { jedis.auth("theredispassword"); System.out.println(jedis.ping()); } Secret Storage The client code should now be able to access Redis. But before deploying this to a cloud function, I need to instead of having them hovering all over the codebase or via unencrypted environment variables. safely store secrets To do this, I will use GCP’s to store the following secrets: secret manager : The text password for authenticating with Redis JIBBER_REDIS_PASSWORD : The keystore file generated by above JIBBER_KEYSTORE keytool : The text password used to decrypt the keystore file JIBBER_KEYSTORE_PASSWORD : The truststore file generated by above JIBBER_TRUSTSTORE keytool : The text password used to decrypt the truststore file JIBBER_TRUSTSTORE_PASSWORD I need to also make sure that my cloud function has access to these secrets. I can do this by giving my default service account permission to read these secrets. I have to allow my service account to have the permission. Secret Manager Secret Accessor Deploying the Cloud Function In addition to the secrets, I also need to set a few environment variables (which are not secretive): Redis host and port Keystore and trust store locations The final command to deploy to cloud functions with secrets and environment variables looks like this: gcloud functions deploy jibber-function \ --entry-point me.mourjo.functions.Hello \ --runtime java17 \ --trigger-http \ --allow-unauthenticated \ --set-secrets '/etc/keystore:/keystore=JIBBER_KEYSTORE:1,/etc/truststore:/truststore=JIBBER_TRUSTSTORE:1,TRUSTSTORE_PASS=JIBBER_TRUSTSTORE_PASSWORD:1,KEYSTORE_PASS=JIBBER_KEYSTORE_PASSWORD:1,REDIS_PASSWORD=JIBBER_REDIS_PASSWORD:1' \ --set-env-vars 'REDIS_HOST=1.1.1.1,REDIS_PORT=12345,KEYSTORE_LOCATION=/etc/keystore/keystore,TRUSTSTORE_LOCATION=/etc/truststore/truststore' Deploying the code to my cloud function should now work! 🎉 Conclusion In this post, I covered the foundations of the chat web app by setting up a GCP Cloud Function that triggers via HTTP and communicates over the internet via a secure connection with Redis -- . all of which is part of GCP’s free tier This cost-efficient infrastructure comes with caveats: This is partly due to the Cloud function and partly due to the encrypted connection to Redis over the internet instead of the local network The endpoint is slow: Any data written to Redis is not backed up and there is no secondary Redis instance that might fill in if the compute instance goes down. Redis is not failsafe: pelican In the next post, I will write a browser-based client that can communicate with this cloud function to establish a user session and then hand it over to a WebSocket server based on Cloud Run. Stay tuned!
