dimkir/nightmare-on-amazon-linux.MD

## nightmare-on-amazon-linux.MD

      
    Raw
  

              nightmare-on-amazon-linux.MD
            
          
    Running nightmare on Amazon Linux

You may have thought of running nightmare on AWS Lambda. But before we can run it on Lambda, we need first to make it run on Amazon Linux.
Provision instance which replicates Lambda environment

According to  AWS Documentation on Lambda Execution Environment and available Libraries we would need this AMI image with this alias amzn-ami-hvm-2016.03.3.x86_64-gp2. Keep in mind that AMI-image-id for this instance would be different in different regions (eg):

In eu-west-1  - ami-f9dd458a
In us-east-1   - ami-6869aa05 

But you can find the right one in your "Community AMIs" section of "EC2 Launch Instance wizard".
If in your region you find more than one image with this name, you need to pick one with  with description "Amazon Linux AMI 2016.03.3 x86_64 HVM GP2".

So now you can launch the Amazon Linux AMI in your preferred way, I would launch Amazon Linux Image via AWS CLI
Now let's connect to the instance via
ssh -i ~/.ssh/my-amazon-linux-keypair.pem ec2-user@instance.dns.name

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
22 package(s) needed for security, out of 80 available
Run "sudo yum update" to apply all updates.
Amazon Linux version 2016.09 is available.

$

Attempt to run nightmare

Now let's prepare instance and install some basic tools we need to try to run nightmare. (Note that we will use node 4.3.2 because that's what  Lambda Environment docs say.
sudo yum update -y
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
nvm install v4.3.2
node -v
npm -v

Now let's install nightmare and try running sample
mkdir  nightmare-test && cd nightmare-test
npm install nightmare # This will automatically install  electron executable as dependency

So let's try running example.js:
[~/nightmare-test]$ node node_modules/nightmare/example.js
# Nightmare will return without any output on the console. Let's add debug flag:

[~/nightmare-test]$ DEBUG=nightmare node node_modules/nightmare/example.js
  nightmare queuing process start +0ms
  nightmare queueing action "goto" for http://yahoo.com +3ms
  nightmare queueing action "type" +2ms
  nightmare queueing action "click" +0ms
  nightmare queueing action "wait" +0ms
  nightmare queueing action "evaluate" +0ms
  nightmare running +1ms
  nightmare electron child process exited with code 127: command not found - you may not have electron installed correctly +19ms
  nightmare electron child process not started yet, skipping kill. +1ms


Now we get output like below, from which we can conclude that there's problem with running electron executable. But still we don't know what is the root cause of the problem. So let's try to run the electron executable manually:
[ec2-user@ip-172-31-5-1 nightmare-test]$ ./node_modules/nightmare/node_modules/electron/dist/electron
electron: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory

We found a root cause

Bingo! The dependency error was the first informative error  and told us that electron can't run because of missing static library libgtk (which is installed by default on most of Desktop Ubuntu distros, but isn't always available on server distros, on CentOS or on Amazon Linux or Lambda).
It is logical to assume that in case one library is missing, there will be more. Let's check
$ cd node_modules/nigthmare/node_modules/electron/dist
[ec2-user@ip-172-31-5-1 dist]$ ldd electron  | grep 'not found'
        libgtk-x11-2.0.so.0 => not found
        libgdk-x11-2.0.so.0 => not found
        libatk-1.0.so.0 => not found
        libpangocairo-1.0.so.0 => not found
        libgdk_pixbuf-2.0.so.0 => not found
        libcairo.so.2 => not found
        libpango-1.0.so.0 => not found
        libXcursor.so.1 => not found
        libXdamage.so.1 => not found
        libXrandr.so.2 => not found
        libXfixes.so.3 => not found
        libXss.so.1 => not found
        libgconf-2.so.4 => not found
        libcups.so.2 => not found


We can see that there's total of 14 static libraries missing.  About a year ago Yuanyi wrote great article on How to run electron on Amazon Linux. By now it is a bit outdated and only helps tackle 11 dependencies and misses few important in 2017 parts, but has definitely has saved me a day or two and showed the right approach (and right versions of the libraries to build from source).
However instead of simply rewriting a newer version of the article I have decided to put all of the commands and improvements into a script, so that the process can be automated.   The tool is called eltool.sh and is available as gist
So as the next step we can download the tool into the home directory:
curl -o- https://gist.githubusercontent.com/dimkir/52054dfca586cadbd0ecd3ccf55f8b98/raw/2b5ebdf28f6a1aad760b5ab9cc581e8ad12a49f5/eltool.sh > ~/eltool.sh && chmod +x ~/eltool.sh 


Now we can proceed with installing missing electron dependencies and compiling from source certain libraries. The syntax of the tool is ./eltool.sh task1 task2 task3 and task names are made to be self explanatory:
$ ./eltool.sh dev-tools  # installs gcc compiler and some libs
$ ./eltool.sh dist-deps  # we install prebuilt dependencies from Amazon Linux repos by using yum
$ ./eltool.sh centos-deps # we install some  prebuil dependencies we can take from CentOS6 repo

# There's still a number of libraries which need to compile from source
$ ./eltool.sh gconf-compile gconf-install 
$ ./eltool.sh pixbuf-compile pixbuf-install
$ ./eltool.sh gtk-compile  # this will take 3 minutes on t2.small instance
$ ./eltool.sh gtk-install 


Now you have all dependencies available in the system directory /usr/local/lib, but some libraries need to be placed(hardlinked) into same directory as electron executable:
$ cd ~/nightmare-test/node_modules/nightmare/node_modules/electron/dist

# Let's create hardlinks to the required libraries
[dist]$  ln -PL /usr/local/lib/libgconf-2.so.4
[dist]$  ln -PL /usr/local/lib/libgtk-x11-2.0.so.0
[dist]$  ln -PL /usr/local/lib/libgdk-x11-2.0.so.0
[dist]$  ln -PL /usr/local/lib/libgdk_pixbuf-2.0.so.0  

# or alternatively you can use shorthand
[dist]$ ~/eltool.sh link

At this point in time you should not have any unresolved dependencies for electron, but let's double check:
[dist]$ ldd electron | grep 'not found' 
# Output should be empty

Let's run Xvfb and nightmare

Now all missing electron dependencies are present, let's try to run our example.js again:
[nightmare-test]$ node example.js
# Nothing happens, let's try to add DEBUG flag


[nightmare-test]$ DEBUG=nightmare node example.js 
  nightmare queuing process start +0ms
  nightmare queueing action "goto" for http://yahoo.com +3ms
  nightmare queueing action "type" +2ms
  nightmare queueing action "click" +0ms
  nightmare queueing action "wait" +0ms
  nightmare queueing action "evaluate" +1ms
  nightmare queueing action "screenshot" +0ms
  nightmare running +0ms
  nightmare electron child process exited with code 1: general error - you may need xvfb +42ms
  nightmare electron child process not started yet, skipping kill. +1ms


If you look carefully at this output, and compare it with the debug output we saw in the beginning of the article, you will notice two subtle differences :

exit code is 1 (and not 127 as was when we had problems with dependencies)
nightmare is suggesting actual error cause - missing Xvfb

So let's follow suggestion and install X-server and Xvfb to the instance:
 sudo yum -y install xorg-x11-server-Xorg xterm   # x-server
 sudo yum -y install xorg-x11-drv-vesa xorg-x11-drv-evdev xorg-x11-drv-evdev-devel  # x-drivers
 sudo yum -y install Xvfb   


# or alternatively you can use shortcut 
$ ~/eltool.sh xvfb-install 

And now let's finally run nightmare with Xvfb server running in the background:

# Upon successful execution the output should be an url related to nightmare. Any url you see would be sign of success.

[nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js
https://github.com/segmentio/nightmare

# Hurray! This seems to work!


# If you're still not convinced we can run it with DEBUG flag

[nigthmare-test]$ DEBUG=nightmare xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js
  nightmare queuing process start +0ms
  nightmare queueing action "goto" for http://yahoo.com +3ms
  nightmare queueing action "type" +2ms
  nightmare queueing action "click" +0ms
  nightmare queueing action "wait" +0ms
  nightmare queueing action "evaluate" +1ms
  nightmare running +0ms
  nightmare electron child process exited with code 0: success! +8s
https://github.com/segmentio/nightmare


CONGRATS! IT  FINALLY WORKS!
CONGRATS! IT  FINALLY WORKS!
CONGRATS! IT  FINALLY WORKS!

Extras: Can we take a screenshot?

Modify example.js to take screenshot

To ensure everything really works, you probably do not want to limit yourself with single line of text returned by the example.js. Let's add screenshot functionality to  example.js:
// ~/nightmare-test/example-screenshot.js
var Nightmare = require('nightmare');
var nightmare = Nightmare({ show: true })

var dt = (new Date()).getTime();
var filename = `/tmp/image-${dt}.png`;

nightmare
  .goto('http://yahoo.com')
  .type('form[action*="/search"] [name=p]', 'github nightmare')
  .click('form[action*="/search"] [type=submit]')
  .wait('#main')
  .evaluate(function () {
    return document.querySelector('#main .searchCenterMiddle li a').href
  })
  .screenshot(filename)
  .end()
  .then(function (result) {
    console.log(`Screenshot was saved to filename ${filename}`);
    console.log('Result: ', result)
  })
  .catch(function (error) {
    console.error('Search failed:', error);
  });


Run nightmare and make this screenshot


[nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node example-screenshot.js
Screenshot was saved to filename /tmp/image-1488912962584.png
Result:  undefined

# Let's check size of the screenshot
$ ls -lh /tmp/*.png
-rw-rw-r-- 1 ec2-user ec2-user 87K Mar  7 18:56 /tmp/image-1488912962584.png


Send image to email

If you still want to look at the actual image, you find many ways to get it to you: scp, upload to S3, send to email.
I will show the simplest way to email attachment from linux box using mutt command:
sudo yum install -y mutt  # let's install mutt mail client

echo "Sending screenshot" | mutt -s "Nightmare screenshot" email@domain.name -a  /tmp/image-1488912962584.png

# Be sure to check your Spam folder for this message =) 


And here's the reward: