Static sites + microservices for science

Over the last few years, a debate has been raging in the software dev community about monolith architecture, microservices, serverless… while most software developers in the trenches are just trying to maintain or add features to a system that’s already working. However, if you do get a chance to build something new, I want to convince you that a static site + microservices is the way to go.

Unique things about biotech/pharma compute needs:

• Data is usually big (size), but slow (velocity). E.g. a scientist adds new data a few times per day, not a few times per second. (Robotics might be an exception here…)
• Heavy compute power is sometimes needed, but usually in burst-mode. E.g. find good primers for this design, then wait a couple days for the result from the sequencing team.
• If you do have a server running constantly doing some big compute task (e.g. processing sequencing data, molecular modeling), this is probably already a separate server (because I’m sure you don’t have that bogged-down machine serving critical apps for your users!). Congratulations: call this a microservice!
• Limited software developer resources.
• No seriously, really limited software dev resources. There is someone to build your site, but probably no resources devoted to QA, and possibly the app was built quickly without maintainability in mind. This ends up being a limitation for the scientist who just need “one little change” and nobody has the time to make and test it. This can be solved with more software dev team resources and better management, but more commonly people take the lower-friction path: bench scientists moonlight as developers, because they need to solve their own problems. Let’s embrace this!

Make your primary internal website STATIC

The term “static” does not mean that your site has no dynamic capabilities. It simply means that that your site consists of a set of “pre-built” files (html, js, and css files) that are directly served on request. There is no server! Data is pulled in to the html pages during a “build” step. Javascript interactive elements run code in the user’s browser.

• Hosting: So cheap and easy - just point your browser to the files on your internal drive if you like. Or host for free or nearly-free in the cloud, e.g. Netlify, Github, S3…
• Maintenance: One of the biggest reasons that servers get hacked is being behind on updates and patches. No server = no worries! (Well, unless your static site uses javascript libraries that have security vulnerabilities… Complex static sites are not “no maintenance”, but, still, lower maintenance.)
• Flexible: The build step can include almost anything: Want your build step to run an ML model based on the new data that was added? Your build step can be: “python my_ML_model.py && build”. (Just try not to make the build step too slow - see Con #1 below.)
• Easy for non-developers to update: Content edits are as easy as editing a text file (because that’s literally what feeds the build step for your site). If you want to add some complex dynamic features to your site, you’ll probably still need a developer for that. But even still, this is more easily done for static sites than dynamic because:
• Fast development workflow: after a change is made, just look at the new site in your browser… if you’re being careful, look in Chrome/IE/Safari… looks good? It is! Push the code! (For a server-backed site, you’d need a test server that matches your production server… it is easy for them to not match, by mistake, so the production site breaks… entire companies exist to help solve this problem.)
• An explosion of community resources: Hugo, Gatsby(https://www.gatsbyjs.org/), and Netlify(https://www.netlify.com/) are the most prominent players helping non-developers easily make static sites. We have open-sourced our science fork of the Netlify CMS. (CMS?? If the “editing text files” part earlier didn’t sound easy enough, edit pages in the browser with a CMS!)
• Speed: Basic static sites are incredibly fast! Of course you can bog it down by adding large javascript libraries or making the page wait for slow database calls, but, by default, your site will fly.

Offload heavy compute tasks

• A separate server (either on-premise or in the cloud) runs your compute-heavy tasks or hosts your database. What’s that you say? The server is down? No worries! The next “build” of your site will fail, but the previous build will still be available for all users, who will have no idea you are madly working to fix the server.
• E.g. If you have a high-throughput robotics setup, handle the robotics input/output as a completely separate application on the server. The processed robotics output could be served to the user-facing static site during the build step, or, if you want live data to show up dynamically for the user, various javascript libraries exist to make database queries interactive.
• The architecture described here would be called “microservices”, except in science, there’s probably nothing “micro” about it. You can process terabytes of data as a separate pipeline, store the results in a queryable database, and feed that to your static site.

CONS of this approach

• The site must be re-built to add new content. So a slow build step could severely limit the frequency that updates can happen. (The solution here is… make your build step fast! Even if it loads several libraries it should only take a few seconds. If data processing needs to happen, do that before you build.)
• Wait, non-developers can edit the site!? Heresy! (Solution: I’m sure you can come up with a management-based solution for this “problem”.)
• If you want to add features that require a “backend” (e.g. logins, forms, a database), the easy solutions are all cloud-based, e.g. Netlify, AWS, Google Cloud. These work great and are not expensive compared to paying a developer to build these features. I only mention this as a potential con because some biotech/pharma companies are nervous about using cloud services. (A solution, if you don’t use the cloud, and your static site is hosted entirely inside your firewall, maybe you don’t really need people to log in? Something to consider.)

I hope I’ve convinced you that static sites can, as it says on the Hugo site, “make building websites fun again”. Please reach out if you have any questions or have feedback/concerns about this!