Instantly share code, notes, and snippets.

Embed
What would you like to do?
How to run nightmare on Amazon Linux

Running nightmare on Amazon Linux

You may have thought of running nightmare on AWS Lambda. But before we can run it on Lambda, we need first to make it run on Amazon Linux.

Provision instance which replicates Lambda environment

According to AWS Documentation on Lambda Execution Environment and available Libraries we would need this AMI image with this alias amzn-ami-hvm-2016.03.3.x86_64-gp2. Keep in mind that AMI-image-id for this instance would be different in different regions (eg):

  • In eu-west-1 - ami-f9dd458a
  • In us-east-1 - ami-6869aa05

But you can find the right one in your "Community AMIs" section of "EC2 Launch Instance wizard".

If in your region you find more than one image with this name, you need to pick one with with description "Amazon Linux AMI 2016.03.3 x86_64 HVM GP2".

image

So now you can launch the Amazon Linux AMI in your preferred way, I would launch Amazon Linux Image via AWS CLI

Now let's connect to the instance via

ssh -i ~/.ssh/my-amazon-linux-keypair.pem ec2-user@instance.dns.name

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
22 package(s) needed for security, out of 80 available
Run "sudo yum update" to apply all updates.
Amazon Linux version 2016.09 is available.

$

Attempt to run nightmare

Now let's prepare instance and install some basic tools we need to try to run nightmare. (Note that we will use node 4.3.2 because that's what Lambda Environment docs say.

sudo yum update -y
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
nvm install v4.3.2
node -v
npm -v

Now let's install nightmare and try running sample

mkdir  nightmare-test && cd nightmare-test
npm install nightmare # This will automatically install  electron executable as dependency

So let's try running example.js:

[~/nightmare-test]$ node node_modules/nightmare/example.js
# Nightmare will return without any output on the console. Let's add debug flag:

[~/nightmare-test]$ DEBUG=nightmare node node_modules/nightmare/example.js
  nightmare queuing process start +0ms
  nightmare queueing action "goto" for http://yahoo.com +3ms
  nightmare queueing action "type" +2ms
  nightmare queueing action "click" +0ms
  nightmare queueing action "wait" +0ms
  nightmare queueing action "evaluate" +0ms
  nightmare running +1ms
  nightmare electron child process exited with code 127: command not found - you may not have electron installed correctly +19ms
  nightmare electron child process not started yet, skipping kill. +1ms

Now we get output like below, from which we can conclude that there's problem with running electron executable. But still we don't know what is the root cause of the problem. So let's try to run the electron executable manually:

[ec2-user@ip-172-31-5-1 nightmare-test]$ ./node_modules/nightmare/node_modules/electron/dist/electron
electron: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory

We found a root cause

Bingo! The dependency error was the first informative error and told us that electron can't run because of missing static library libgtk (which is installed by default on most of Desktop Ubuntu distros, but isn't always available on server distros, on CentOS or on Amazon Linux or Lambda).

It is logical to assume that in case one library is missing, there will be more. Let's check

$ cd node_modules/nigthmare/node_modules/electron/dist
[ec2-user@ip-172-31-5-1 dist]$ ldd electron  | grep 'not found'
        libgtk-x11-2.0.so.0 => not found
        libgdk-x11-2.0.so.0 => not found
        libatk-1.0.so.0 => not found
        libpangocairo-1.0.so.0 => not found
        libgdk_pixbuf-2.0.so.0 => not found
        libcairo.so.2 => not found
        libpango-1.0.so.0 => not found
        libXcursor.so.1 => not found
        libXdamage.so.1 => not found
        libXrandr.so.2 => not found
        libXfixes.so.3 => not found
        libXss.so.1 => not found
        libgconf-2.so.4 => not found
        libcups.so.2 => not found

We can see that there's total of 14 static libraries missing. About a year ago Yuanyi wrote great article on How to run electron on Amazon Linux. By now it is a bit outdated and only helps tackle 11 dependencies and misses few important in 2017 parts, but has definitely has saved me a day or two and showed the right approach (and right versions of the libraries to build from source).

However instead of simply rewriting a newer version of the article I have decided to put all of the commands and improvements into a script, so that the process can be automated. The tool is called eltool.sh and is available as gist

So as the next step we can download the tool into the home directory:

curl -o- https://gist.githubusercontent.com/dimkir/52054dfca586cadbd0ecd3ccf55f8b98/raw/2b5ebdf28f6a1aad760b5ab9cc581e8ad12a49f5/eltool.sh > ~/eltool.sh && chmod +x ~/eltool.sh 

Now we can proceed with installing missing electron dependencies and compiling from source certain libraries. The syntax of the tool is ./eltool.sh task1 task2 task3 and task names are made to be self explanatory:

$ ./eltool.sh dev-tools  # installs gcc compiler and some libs
$ ./eltool.sh dist-deps  # we install prebuilt dependencies from Amazon Linux repos by using yum
$ ./eltool.sh centos-deps # we install some  prebuil dependencies we can take from CentOS6 repo

# There's still a number of libraries which need to compile from source
$ ./eltool.sh gconf-compile gconf-install 
$ ./eltool.sh pixbuf-compile pixbuf-install
$ ./eltool.sh gtk-compile  # this will take 3 minutes on t2.small instance
$ ./eltool.sh gtk-install 

Now you have all dependencies available in the system directory /usr/local/lib, but some libraries need to be placed(hardlinked) into same directory as electron executable:

$ cd ~/nightmare-test/node_modules/nightmare/node_modules/electron/dist

# Let's create hardlinks to the required libraries
[dist]$  ln -PL /usr/local/lib/libgconf-2.so.4
[dist]$  ln -PL /usr/local/lib/libgtk-x11-2.0.so.0
[dist]$  ln -PL /usr/local/lib/libgdk-x11-2.0.so.0
[dist]$  ln -PL /usr/local/lib/libgdk_pixbuf-2.0.so.0  

# or alternatively you can use shorthand
[dist]$ ~/eltool.sh link

At this point in time you should not have any unresolved dependencies for electron, but let's double check:

[dist]$ ldd electron | grep 'not found' 
# Output should be empty

Let's run Xvfb and nightmare

Now all missing electron dependencies are present, let's try to run our example.js again:

[nightmare-test]$ node example.js
# Nothing happens, let's try to add DEBUG flag


[nightmare-test]$ DEBUG=nightmare node example.js 
  nightmare queuing process start +0ms
  nightmare queueing action "goto" for http://yahoo.com +3ms
  nightmare queueing action "type" +2ms
  nightmare queueing action "click" +0ms
  nightmare queueing action "wait" +0ms
  nightmare queueing action "evaluate" +1ms
  nightmare queueing action "screenshot" +0ms
  nightmare running +0ms
  nightmare electron child process exited with code 1: general error - you may need xvfb +42ms
  nightmare electron child process not started yet, skipping kill. +1ms

If you look carefully at this output, and compare it with the debug output we saw in the beginning of the article, you will notice two subtle differences :

  • exit code is 1 (and not 127 as was when we had problems with dependencies)
  • nightmare is suggesting actual error cause - missing Xvfb

So let's follow suggestion and install X-server and Xvfb to the instance:

 sudo yum -y install xorg-x11-server-Xorg xterm   # x-server
 sudo yum -y install xorg-x11-drv-vesa xorg-x11-drv-evdev xorg-x11-drv-evdev-devel  # x-drivers
 sudo yum -y install Xvfb   


# or alternatively you can use shortcut 
$ ~/eltool.sh xvfb-install 

And now let's finally run nightmare with Xvfb server running in the background:


# Upon successful execution the output should be an url related to nightmare. Any url you see would be sign of success.

[nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js
https://github.com/segmentio/nightmare

# Hurray! This seems to work!


# If you're still not convinced we can run it with DEBUG flag

[nigthmare-test]$ DEBUG=nightmare xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js
  nightmare queuing process start +0ms
  nightmare queueing action "goto" for http://yahoo.com +3ms
  nightmare queueing action "type" +2ms
  nightmare queueing action "click" +0ms
  nightmare queueing action "wait" +0ms
  nightmare queueing action "evaluate" +1ms
  nightmare running +0ms
  nightmare electron child process exited with code 0: success! +8s
https://github.com/segmentio/nightmare

CONGRATS! IT FINALLY WORKS! CONGRATS! IT FINALLY WORKS! CONGRATS! IT FINALLY WORKS!


Extras: Can we take a screenshot?

Modify example.js to take screenshot

To ensure everything really works, you probably do not want to limit yourself with single line of text returned by the example.js. Let's add screenshot functionality to example.js:

// ~/nightmare-test/example-screenshot.js
var Nightmare = require('nightmare');
var nightmare = Nightmare({ show: true })

var dt = (new Date()).getTime();
var filename = `/tmp/image-${dt}.png`;

nightmare
  .goto('http://yahoo.com')
  .type('form[action*="/search"] [name=p]', 'github nightmare')
  .click('form[action*="/search"] [type=submit]')
  .wait('#main')
  .evaluate(function () {
    return document.querySelector('#main .searchCenterMiddle li a').href
  })
  .screenshot(filename)
  .end()
  .then(function (result) {
    console.log(`Screenshot was saved to filename ${filename}`);
    console.log('Result: ', result)
  })
  .catch(function (error) {
    console.error('Search failed:', error);
  });

Run nightmare and make this screenshot


[nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node example-screenshot.js
Screenshot was saved to filename /tmp/image-1488912962584.png
Result:  undefined

# Let's check size of the screenshot
$ ls -lh /tmp/*.png
-rw-rw-r-- 1 ec2-user ec2-user 87K Mar  7 18:56 /tmp/image-1488912962584.png

Send image to email

If you still want to look at the actual image, you find many ways to get it to you: scp, upload to S3, send to email.

I will show the simplest way to email attachment from linux box using mutt command:

sudo yum install -y mutt  # let's install mutt mail client

echo "Sending screenshot" | mutt -s "Nightmare screenshot" email@domain.name -a  /tmp/image-1488912962584.png

# Be sure to check your Spam folder for this message =) 

And here's the reward: image

@sweinertjr

This comment has been minimized.

sweinertjr commented Mar 24, 2017

@dimkir Thanks for this writeup! I was able to get it running on an amazon linux AMI successfully. However, what are the next steps to actually get this running on a Lambda function? From what I know, there isn't a way to run the function with Xvfb. Any ideas? Thanks

@maksimlikharev

This comment has been minimized.

maksimlikharev commented Apr 11, 2017

you need to update as ORBit2-2.14.17-5.el6.x86_64.rpm no longer available and replaced by:
ORBit2-2.14.17-6.el6_8.x86_64.rpm

@rinogo

This comment has been minimized.

rinogo commented Apr 27, 2017

@sweinertjr Umm... So random to see you here! I just wrote up a fatty guide on getting NIghtmareJS running on CentOS 6. Obviously not very helpful for your predicament (hopefully you've figured it out), but it's crazy that in the whole big bad internet, I'd just happen to recognize your 32x32 avatar and associated username... Hope you're doing fantastic, my friend! :)

Oh, and thanks for the awesome guide, @dimkir ! :)

@jasonLiu001

This comment has been minimized.

jasonLiu001 commented Jun 20, 2017

Very helpful !! After i followed this guide, it run nightmare in centos 7 successfully. Thank you for this help.

@shmuelgutman

This comment has been minimized.

shmuelgutman commented Jul 4, 2017

Thanks for this gist, very helpful.
@maksimlikharev Also ORBit2-devel-2.14.17-5.el6.x86_64.rpm need to be replaced with ORBit2-devel-2.14.17-6.el6_8.x86_64.rpm

@sweinertjr

This comment has been minimized.

sweinertjr commented Sep 20, 2017

@rinogo Totally checking out your guide. Good to see a friendly face :) Hope all os well.

@uchoaaa

This comment has been minimized.

uchoaaa commented Oct 11, 2017

Hi , how to deploy a lambda function with this AMI? I followed the steps here and my AMI is ready, but I don't know how to deploy a lambda function associated with it. Any help?

@thanhbinh710

This comment has been minimized.

thanhbinh710 commented Dec 5, 2017

Hey I have the same question as @uchoaaa. Does anyone know it's possible?

@zwtan

This comment has been minimized.

zwtan commented Dec 6, 2017

If anyone tried using AWS CentOS 7 image to spin off to test run for Nightmare that having issue on Electron. I've used the following installation that resolve the issue. As the eltools.sh might be out of date or what so ever some packages could not be found.

Packages to be installed

sudo yum groupinstall "Development Tools"
sudo yum install gtk+-devel gtk2-devel

sudo yum install libasound.so.2 libXv.so.1 libXss.so.1 libSM.so.6 libXi.so.6 libXrender.so.1 libXrandr.so.2 libfreetype.so.6 libfontconfig.so.1 libz.so.1 libgthread-2.0.so.0 libstdc++.so.6 libQtDBus.so.4
libQtGui.so.4 pulseaudio-libs.i686

sudo yum install pango.x86_64 libXcomposite.x86_64 libXcursor.x86_64 libXdamage.x86_64 libXext.x86_64 libXi.x86_64 libXtst.x86_64 cups-libs.x86_64 libXScrnSaver.x86_64 libXrandr.x86_64 GConf2.x86_64 alsa-lib.x86_64 atk.x86_64 gtk3.x86_64 -y

@dimkir fyip

@iblessedi

This comment has been minimized.

iblessedi commented Jan 12, 2018

I got a problem while trying to install ORBIT. I had to edit eltool.sh and replace installing the ORBIT to

  sudo rpm -ivh http://mirror.centos.org/centos/6/os/x86_64/Packages/ORBit2-2.14.17-6.el6_8.x86_64.rpm
  sudo rpm -ivh http://mirror.centos.org/centos/6/os/x86_64/Packages/ORBit2-devel-2.14.17-6.el6_8.x86_64.rpm
@JonathanKolnik

This comment has been minimized.

JonathanKolnik commented Mar 16, 2018

Hi this is such a great walkthrough but I'm running into an issue

 $ ~/eltool.sh link
>>> Linking SO Static libraries to electron (creating symlinks in electron directory)...
ln: failed to access ‘/usr/local/lib/libgconf-2.so.4’: No such file or directory
‘./libgtk-x11-2.0.so.0’ => ‘/usr/local/lib/libgtk-x11-2.0.so.0’
‘./libgdk-x11-2.0.so.0’ => ‘/usr/local/lib/libgdk-x11-2.0.so.0’
‘./libgdk_pixbuf-2.0.so.0’ => ‘/usr/local/lib/libgdk_pixbuf-2.0.so.0’

This is the error

ln: failed to access ‘/usr/local/lib/libgconf-2.so.4’: No such file or directory

Any help would be appreciated!

edit: for posterity I was being dumb...I chose the latest ami that matched the naming convention but is from 2017, that could have been it? Also I didn't update the ORBIT files referenced in the comment above before running ./eltool.sh centos-deps

@djalmaaraujo

This comment has been minimized.

djalmaaraujo commented Jun 4, 2018

it worked, YOU ROCK!

@jmb64

This comment has been minimized.

jmb64 commented Jul 11, 2018

As of June 2018, the Orbit install paths in eltool.sh need to be edited as:

sudo rpm -ivh http://mirror.centos.org/centos/6/os/x86_64/Packages/ORBit2-2.14.17-7.el6.x86_64.rpm
sudo rpm -ivh http://mirror.centos.org/centos/6/os/x86_64/Packages/ORBit2-devel-2.14.17-7.el6.x86_64.rpm

@benonymus

This comment has been minimized.

benonymus commented Aug 29, 2018

hey when I am running this command curl -o- https://gist.githubusercontent.com/dimkir/52054dfca586cadbd0ecd3ccf55f8b98/raw/2b5ebdf28f6a1aad760b5ab9cc581e8ad12a49f5/eltool.sh > ~/eltool.sh && chmod +x ~/eltool.sh i get this response:
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6202 100 6202 0 0 189k 0 --:--:-- --:--:-- --:--:-- 189k

which seems ok I guess, but after that I can't run the following commands because it gives me this error: -bash: ./eltool.sh: No such file or directory I am on ubuntu.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment