How I Built a File Watcher Program After Forgetting Where I Saved My AWS S3 Files

AWS CloudFront Migration Issues - Hosting Your Website in S3 - a 4-Part Story

1 AWS Cloudfront DNS Migration Gotchas that no one tells you about

2 AWS Gotchas Part 2 - corrections from part 1 plus new gotchas, A MUST Read

3 Enabling AWS S3 to behave more like a Web Server

4 Forgot where I saved my AWS S3 files so I wrote this File Watcher Program

We’ll do something a little unusual and start with Part 4

Part 4 - Forgot where I saved my AWS S3 files so I wrote this File Watcher Program

The File Watcher in question is one that watches my local S3 repository for changes. When any file is changed my AWS S3 Bucket file is updated magically with the below code. With a simple NodeJS script, just save as myfile.js, open the CMD prompt, and enter node myfile.js to run.

Most may use GitHub as their website repository but for aws S3 any old local folder will do. An inconvenience of hosting on s3 is that you can't edit your HTML files right in the bucket. So after a few months with no edits, suddenly I asked myself: where did I save my files, and most importantly is this folder my most current version?

more below...

const AWS = require('aws-sdk'); //add these 2 dependencies
const fs = require('fs');

var file_name;
      //next line polls the origin folder while program is open
 const WATCH_TARGET = 'C:/Users/rickd/Downloads/backup 6-4-23/howtolearnjava.com/other/';
 fs.watch(WATCH_TARGET, (eventType, filename) => {
 console.log('File "' + filename + '" was changed: ' + eventType);
 file_name = filename;

if (eventType == 'change') {
       // a file change occurred
       //get aws user credentials
const s3 = new AWS.S3({
  accessKeyId: 'xxxxxxxxxxxxxxxxxxxxxxxxx',
  secretAccessKey: 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'
});

const bucketName = 'howtolearnjava.com';//need to define this here
const fileName2 = file_name;  //must be defined here
      // this is the origin file that we are changing
const origin_file = 'C:/Users/rickd/Downloads/backup 6-4-23/howtolearnjava.com/other/'+fileName2;
      //needs full path otherwise no such file or dir shows as error                                                                                                                                                                                                         //no fancy concat anymore...dont put syntax in the path
      //******very important..this program will not overwrite s3 file
      //if content type is absent (in cf and perhaps always)
      //here is where we read our file
const fileData = fs.readFileSync(origin_file);

s3.upload({
  Bucket: bucketName,
  Key: file_name, //this is only the name of the file
        // next line is ContentType: 'valid content type', //this is very major,
        //most tutorials leave this part out
        //when hosting on cloudfront default content type is
        //octetstream which causes file to download
        //to overcome we MUST specify content type on creation of file
  ContentType: 'text/html',
        //set content type here in top level params not in above const
  Body: fileData 

}, (err, data) => {
  if (err) {
    console.error(err);
  } else {
    console.log(`File uploaded successfully. ${data.Location}`);
  }
});

  }    //end if changed condition

});

Caveats

1. AWS Lambda cannot access the local Windows file system, making this task unfeasible.

2. Be cautious that you MUST specify the content type when creating an HTML file. Failing to do so will result in the default octet-stream format, causing a file download and potentially overwriting existing content (a known CloudFront issue).

3. It is imperative to utilize user credentials for S3 access. Instructions for this can be found here.

4. Upon initiating the program, a message will alert you that in 2023, it will be necessary to use AWS SDK (v3) instead of v2. As of now, this code may not function in a few months.

5. It's advisable to refrain from keeping S3 open while running this program, as it has been known to cause crashes. Although no damage has been reported so far, it's a potential concern. It's best to verify your files by accessing them through HTTP instead.

My Continuing Saga

Feeling concerned, I realized I could access my bucket and download the files manually. However, this process proved cumbersome, especially when attempting to select multiple files for download, as the download option disappeared.

So, I found a solution in the form of "fs.watch," a folder watcher. This handy tool automatically saves any changes to S3 whenever a file change is detected. By the way, it's important to note that you also need to manually copy updated files and paste them into your S3 bucket every time you make a change. I must have forgotten to mention this earlier. Despite these additional steps, I can say that my experience with managing S3 became much smoother once I implemented this program.

The program itself is quite simple, just a few lines of Node.js code. When executed, it sets up a listener that waits for updates. You only need to keep the program running while making changes to HTML files. When a file change is detected, the program sends the file name to an AWS upload function, which then uploads it to S3, overwriting the existing file. All you need to remember is the name of the Node.js file. Just open your command prompt, enter "node my_program.js," make your HTML updates, and the rest is like magic.

Part 1 - AWS Cloudfront DNS Migration Gotchas that no one tells you about

My overall observations: It takes a lot of digging to find these gotchas and many times AWS docs leave u hanging.

I migrated from an Apache Server using htaccess all the time, over to Cloudfront...started on June 7 and it is now June 18, and I still have issues but is working for the most part. This article assumes we already have some basic DNS and Networking knowledge but is still good for the beginner too.

I suspect that many developers go the Netlify route vs AWS Cloudfront due to an assumption of AWS complexity. I am not here to bash AWS, I think it is very worthwhile to do and HIGHLY recommend migrating to a better network neighborhood such as AWS S3 and Cloudfront.

First, the 4 major gotchas

1. All names (comma separated), not just Alternate names must be in the Alternate Domain Name Setting in the General tab on Cloudfront which is marked as 'optional'. The gotcha is that it says optional but won’t work till you fill it in. Doesn't 'optional' mean optional?

2. Cors issue is in 2 places, S3 and Cloudfront, don’t forget to enable cors also in Cloudfront. Gotcha is enabling only in s3 while forgetting to do it in Cloudfront. Otherwise, pre-migration links used on the outside will show Cors issue in the Dev Tools view. Go to the Behavior tab in Cloudfront and, under Origin Request policy, put in CORS-CustomOrigin and under Response headers policy add Cors with preflight and security.....

3. Using public DNS other than route53 does not migrate cleanly due to URL forwarding issues. Route53 is only 50 cents per month to host DNS on AWS Route53 vs GoDaddy or other DNS providers, and IT IS WELL WORTH IT. I was thrilled when I first found out that you don’t need route53 but ended up using it for a clean migration. The main issue was that my old links stopped working, they did not forward to my new page until I went back to using Route53. The thing about Route53 vs GoDaddy is that Route53 allows an A record to point to an alias and GoDaddy only allows an actual IP address. I think this is 301ing the right way and getting masking without really doing illegal masking.

4. This is not exactly a gotcha but could be. This is more of an FYI. Upon migration, pages now instantly live in the S3 bucket behind a Cloudfront distribution so be sure to change ur canonicals right away on each page with the Cloudfront path and submit a new sitemap to Google Search Console if u are on it. Also, any links on your homepage need the Cloudfront path changed right away. This applies to both buttons and href links. Even though many people say a sitemap is not necessary for a small site, I say submit a new sitemap for migration because if you are registered with Google search console this becomes a must-do.

Also, be sure robots.txt is used and points to ur sitemap so web crawlers can find you.
Also, another FYI is that Cloudfront seamlessly does a proper migration which includes your old name on the address bar. But if you do not use Route53 then your new Cloudfront address will appear and not your web name. In my case my old links onFacebook and Twitter stopped working because of this, they timed out. When I changed back to Route53 this fixed all my old link issues. To me, I could not have a situation where all my old links to my new home page now fail.

More gotchas below - but many are just FYIs

1. Need to re-seo everything because files now live in a new place, GSC still favors my now defunct files and sitemap. When I request indexing a duplicate issue arises where Google chooses old canonical over user-declared canonical....... still pending...will have an update on this soon. My new finding is that we do not have to re-seo, please see my updates in the gotchas part 2.

2. Beware that s3 requires the html extension until u tell it otherwise.....still pending
   My old pages had no file extension as directed by htaccess file in the Apache server. Now if user clicks on an old page link from Twitter or FB they get my 404 page which I finally revamped to accommodate this. Eventually, I need to try out the extension s3 option, but for now, it is not a priority.

   
   PS My index page old link works thank God\Update.. just rename the page in S3 and remove html extension and all old links will work now.

3. 304 forwarding not 301, index pg only....impact to SEO..but recently requesting indexing on the home page returned a 200 response and not the expected 304. This is an SEO issue because if Google sees the 304 response they assume there are no updates to the page and thus will not index the page.

Update - found out this is not true, see my gotchas part 2 about this.

:::

\

This is what it looks like:

function handler(event) {
    var request = event.request;
    var uri = request.uri;
    var requestURI = event.request.uri

    var newurl5 = `https://howtolearnjava.com/JDBC.html`;
    var newurl6 = `https://howtolearnjava.com/Java-Servlet.html`;


   //the uri is everything after the .com slash in our path including the slash and any subfolders


    if (requestURI === "/JDBC") {
            var response = {
                statusCode: 301,
                statusDescription: 'Found',
                headers:
                    { "location": { "value": newurl5 } }
                }
            return response;
        }

    if (requestURI === "/Java-Servlet") {
            var response = {
                statusCode: 301,
                statusDescription: 'Found',
                headers:
                    { "location": { "value": newurl6 } }
                }
            return response;
        }        



//Part 2 - below code will enable sub folders to host index.html as default
    // Check whether the URI is missing a file name. If so then it is a subfolder and we then concat and index.html file so when the user types in the subfolder, by default he gets the index.


    if (uri.endsWith('/')) {
        request.uri += 'index.html';
    } 
        // Check whether the URI is missing a file extension.
    else if (!uri.includes('.')) {
        request.uri += '/index.html';
    }
    return request;
}

Open AWS Cloudfront, click functions in the left menu, name and build the function, test it, publish it then associate it to a Cloudfront distribution.

Part 2 subfolders

Since S3 file storage is just simple storage, S3 has no knowledge of a subfolder or its index file. Luckily when we first encounter web enabling an S3 bucket, this part gives our index file default-like behavior. But generally, nothing is ever discussed about subfolders. In Apache, we make use of an htaccess file in each subfolder to deal with default indexes. But S3 and Cloudfront do not have an htaccess file so we hereby use a Cloudfront Function to mimic htaccess.

In our above example, the CloudFront Function is like a listener listening for HTTP requests (similar to Apache HTTP). Notice that when the user request comes in at the CDN edge our program asks if it is a subfolder and if it is then concat an index file to the URI. Voila, we have the behavior we want.

Part 3 Response Security Headers

In the 2 examples above, we are dealing with a request header but now we need to write a separate function to handle a response header. What we are doing here is providing our CloudFront config with some additional headers for security purposes. Normally, this is done once again in our Apache htaccess file but here we do it like this below. A separate function is needed because when we test it we want to check the box that says response and also want to do the same when we associate our published file. Likewise, with the request function, we check the box that says request.

Here we go:

function handler(event) {
     var response = event.response;
     var headers = response.headers;


   // config security headers

headers['strict-transport-security'] = { value: 'max-age=63072000; includeSubdomains; preload'}; 


headers['content-security-policy'] = { value: "script-src-elem 'self' 'unsafe-eval' https://javasqlweb.org/js_new.js 'unsafe-inline' https://ipinfo.io/ 'unsafe-inline' https://www.googletagmanager.com/ 'unsafe-inline'; style-src 'self' 'unsafe-inline"};


headers['x-content-type-options'] = { value: 'nosniff'}; 


headers['x-xss-protection'] = {value: '1; mode=block'};


headers['referrer-policy'] = {value: 'no-referrer-when-downgrade'};


headers['x-frame-options'] = {value: 'DENY'};


return response;
}

The content security policy header is the challenging one. In my case, I have a javascript file called js_new.js to enable and I also have 2 other js connections one to ipinfo and the other one to google tag manager. My last instruction for styles enables inline styles. It is best to google all these headers to gain a full understanding of their importance.

For thorough pre and post-testing of all the adjustments I mentioned, I rely on Screaming Frog. Without the appropriate security headers, it may raise concerns about their absence or incorrect implementation. Screaming Frog is undeniably a powerful tool to use regularly for website testing, helping you identify and rectify various issues.

It's also worthwhile to take some time to study these headers and understand what they signify. Another useful approach is to delve into the network tab of Chrome Dev Tools (accessed with ctrl-shift-j). From there, you can click on your page and inspect the response headers to gain insights into what's being returned.

Final Thoughts

With these minor refinements, we gradually coax S3 into behaving more like a genuine web server, even though it fundamentally serves as simple storage. Similar to how we enhance Apache using .htaccess, we enhance S3 with a Cloudfront distribution and a Cloudfront function to implement these tweaks. This has certainly improved my confidence in using S3 to host my website.