September 29, 2014

November 25, 2019

Javascript, nginx and - help for search crawlers to render single page applications

Some time ago I've wrote a note about single page application vs server side rendering. I've mentioned about nginx and I've used for rendering pages for search crawlers because search crawlers doesn't render correctly pages which are written in javascript, like single page applications. I will just post my configuration here. I'm not using it anymore but maybe it will help someone.

Nginx server config.

server {
listen 80;
client_max_body_size 20M;
access_log /var/log/nginx/lastlog_access.log;
error_log /var/log/nginx/lastlog_error.log;
error_page 404 = @404;
location @404 {
rewrite .* / permanent;
root /projects/;
location / {
include prerender.conf;
try_files $uri $uri/ /index.html;
location ~ "^/([a-zA-Z0-9]+)$" {
if ($http_user_agent ~* "facebookexternalhit|LinkedInBot|(Google \(\+https\:\/\/developers\.google\.com\/\+\/web\snippet\/\))|Twitterbot|Pinterest") {
rewrite (.+) /api/content/item$1?social=1;
try_files $uri $uri/ /index.html;
location /api/ {
uwsgi_pass unix:///tmp/lastlog.sock;
include uwsgi_params;

As you can see all static files are in /projects/

I've created a special parameter for my API, social=1. is needed for web crawlers but we don't need this for pages like facebook, linkedin, etc. The fact is that when someone "likes" our page or tries to share on those portals some API from for example facebook tries to render our page and gets some data like title or read open graph tags. It doesn't need whole page. Above configuration checks if a request is from facebook, twitter, linkedin and google plus and if yes, it calls API with parameter social=1. API without social=1 parameter returns only JSON data but with social=1 it returns simple html:

<!DOCTYPE html>
<head prefix="og: fb:">
<title>{{ title }}</title>
<meta property="og:title" content="{{ title }}" />
content=" - Watch last logs from IT world!"
<meta property="og:url" content="{{ id }}" />
<meta property="og:image" content="{{ image }}" />
{{ title }}

That's all. Nothing more is needed. Let's back to and nginx.

/etc/nginx/prerender.conf (it is included in nginx server config)

set $needPrerender "";

if ($request_uri ~ '_escaped_fragment_') {
  set $needPrerender "Y";

if ($http_user_agent ~* (googlebot||bingbot||yandexbot||yahooseeker||slurp|feedfetcher|blekkobot|crawler) ) {
  set $needPrerender "Y";

#if ($http_accept ~* 'html') {
#  set $needPrerender "${needPrerender}ES";

if ($needPrerender = "Y") {
  rewrite .* /$scheme://$http_host$request_uri? break;
  proxy_pass http://localhost:3000;

As you can see prerender.conf contains proxy to http://localhost:3000. It is a proxy to I used supervisord for this. Here is a config - /etc/supervisor/conf.d/prerender.conf

command = /projects/ /projects/
directory = /projects/
autostart = true
autorestart = true
stopasgroup = true
process_name = %(program_name)s_%(process_num)02d
numprocs = 1
user = eshlox

What can i tell about this. It is a simple configuration file for supervisord.

  • environment - here is the path to environment (i'm using
  • command - here is a command to run prerender, i've created a config in lastlog.js file
  • directory - here is a

Below is my config for - lastlog.js (it is used in supervisord configuration)

#!/usr/bin/env node
var prerender = require("./lib")
var server = prerender({
workers: 2,
iterations: 4,
phantomBasePort: process.env.PHANTOM_CLUSTER_BASE_PORT,
messageTimeout: process.env.PHANTOM_CLUSTER_MESSAGE_TIMEOUT,

That's all. Just modify configuration files to your project. Reload nginx, run supervisord. Read documentation for because it has more options and.. very like RAM memory ;-)

© 2020 Przemysław Kołodziejczyk