Building PNDA mirror behind proxy

proxy-env.sh

export http_proxy=http://proxy:80
export https_proxy=http://proxy:80
export JAVA_FLAGS="-Dhttp.proxyHost=proxy -Dhttp.proxyPort=80 -Dhttps.proxyHost=proxy -Dhttps.proxyPort=80 -Dhttp.nonProxyHosts=localhost,127.0.0.1,proxy"
export JAVA_OPTS="$JAVA_FLAGS"
export MAVEN_OPTS="$JAVA_FLAGS"

settings.xml

<settings>
  <proxies>
    <proxy>
      <id>default</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>proxy</host>
      <port>80</port>
    </proxy>
  </proxies>
</settings>

sbt version check in build-kafkamanager.sh is broken on CentOS, but it seems to proceed and "do the right thing" anyway.

Missing rpm dependencies on CentOS

• yum-utils
• python-six
• python-chardet

Version discrepancy

• Requires: systemd-libs = 219-30.el7_3.9
• Installed: systemd-libs-219-30.el7.x86_64

yum-config-manager is not available by default on CentOS. Need to install yum-utils to get it. But brings in other rpms that then get missed by create_mirror_rpm.

yum install policycoreutils-python selinux-policy-targeted. Fixed with new salt rule.

Need to disable firewalld on all nodes before running ./pnda-cli.py

Need to disable boto on GCE. Remove /etc/boto.cfg

Collecting boto==2.46.1 (from -r /opt/pnda/package-repository-develop/requirements.txt (line 4))
  Downloading http://10.142.1.2/pnda/mirror_python/packages/boto-2.46.1.tar.gz (1.5MB)
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-oaHfWR/boto/setup.py", line 37, in <module>
        from boto import __version__
      File "boto/__init__.py", line 1216, in <module>
        boto.plugin.load_plugins(config)
      File "boto/plugin.py", line 93, in load_plugins
        _import_module(file)
      File "boto/plugin.py", line 75, in _import_module
        return imp.load_module(name, file, filename, data)
      File "/usr/lib/python2.7/site-packages/google_compute_engine/boto/boto_config.py", line 29, in <module>
        from google_compute_engine import config_manager
    ImportError: No module named google_compute_engine

Ongoing problems with "audit: backlog limit exceeded" on lab CentOS devices. Now trying "auditctl -b 5000".

Need to remove lines from boostrap-scripts/base.sh for offline installs pypi.org will always fail when offline.

SYN-SENT    0      1      10.142.1.3:45386          151.101.128.223:https
 
% host pypi.org
pypi.org has address 151.101.0.223
pypi.org has address 151.101.192.223
pypi.org has address 151.101.64.223
pypi.org has address 151.101.128.223

python-carbon seems to get missed when building mirror_rpm

Yes, create_mirror_rpm.sh can completely miss rpms if the versions don't match. For example:

yumdownloader --resolve --archlist=x86_64 --destdir $RPM_REPO_DIR $RPM_PACKAGE_LIST
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.usinternet.com
 * epel: mirrors.mit.edu
 * extras: repo1.sea.innoscale.net
 * updates: repo1.sea.innoscale.net
No Match for argument redis-3.2.3-1.el7
...

Script continues past this.

As reported by the create_mirror script the create_mirror_rpm script is very problematic.