Skip to content

Instantly share code, notes, and snippets.

@rkmathi

rkmathi/LinuxConJapan2014-day1.md

Last active August 29, 2015 14:01
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save rkmathi/d5313d86b5baf5c32d76 to your computer and use it in GitHub Desktop.
Save rkmathi/d5313d86b5baf5c32d76 to your computer and use it in GitHub Desktop.
Download ZIP
LinuxConJapan2014
Raw
LinuxConJapan2014-day1.md

LINUXCON JAPAN 2014 day1

2014/05/20

Opening

$63.4B for R&D

  • MS, IBM, Google, ...

External R&D

  • Billions of dollars of free software harnessed to power core products and services.
  • How to manage my company

Open source collaboration requires new skills

  • Strategic Analysis

Choosing best open sourec projects to meet goals

  • Intellectual Property
  • Development Process
  • Business Process

Take away

  • Software is eating the technology industry
  • Open Source is eating the software industry
  • You need to be ready

Interoperability is Key to Accelerating SDN Adoption

OpenDaylight project

  • Why Software-Defined Networking (SDN) matters

Major Industry Debate

  • Overlay
  • White Box
  • OpenFlow
  • Application Policy
  • Border Gateway Protocol
  • Controller Federation

Industry Value

  • A common environment for users and application developers

Hydrogen Architecture

The Apache Way

Apache Group

  • Sharing code patches for the abandoned NCSA httpd
  • Equals, even from the beginning - no technical lead

Roots

  • Created from the beginning to be a flat management structure

The Numbers

  • Projects: 149
  • Incubating Projects: 34
  • ...

Projects

  • CloudStack
  • Cassandra
  • Ant
  • Hadoop
  • OpenOffice
  • Tomcat
  • ...

ASF: Leagal Structure

  • Member-based corporation - indivisuals, not companies can be members
  • Members nominate and elect new members
  • Members elect a board - 9 seats
  • Semi-annual meetings via IRC

Projects

  • Responsible for their own code, community and direction

Mission

  • Provide a means for indivisual volunteers to be sheltered from leagal suits directed at the Foundation's project
  • Protect the 'Apache' brand as applied to its software products, from being abused by other organizations

Vision

  • Support for the community of open-source software projects

Apache License

  • A liberal open source software license - BSD-like
  • Business friendly
  • Includes Patent Grant

The Apache way

  • Best ideas win
  • Peer-based
  • COnsensus decision making
  • Collaborative development
  • Responsible oversight
  • Indivisual Participation

Virtulization + Open Cloud Panel

Hypervisor is dead

Docker

Application layer

Security

How Juju makes cloud and Devops easy

What is DevOps?

  • Rate of agile developmnt and deployment requires deeper interaction between teams
  • A melding of development, deployment, and QA principles, methods, and practices
  • Fills the gap between developers and system administrators

What drives DevOps?

  • Speed of the deployment
  • Continuous Integration, Automated Testing, etc...
  • Fast change vs Stability

What does DevOps "deliver"?

  • Fast repeatable server setup, consistent environment
  • Abstract ops tasks to empower devs
  • Repeatable processes that let you scale out quickly

DevOpsolution

  • juju.ubuntu.com

So juju is ...

  • Like apt-get, but for sets of machines
  • Charms do all the work for you
  • Juju manages Services, not Machines

Using Juju deploy your application on Cloud

  • Joyent
  • Amazon EC2
  • HP Cloud
  • Azure
  • LXC containers
  • Vagrant

Juju provides service orchestration

  • Juju exposes re-usable service uints and well defined interfaces

Juju's internals

Juju treates indivisual services as atoms that are described as formulas and can be instantiated one or many times.

harms

  • Scalable application services defined

Relations

  • A high level interface described

Scaling services

Inside a Charm

  • Charms define how services integrate and how their service nits react to events in the distributed services.

Charm your application

DevOps with Vagrant and KVM/qemu

What is DevOps

"If you're' going to do operations reliably you need to make it reproducible and programmatic"

DevOps for developer

  • Gap between Development and Operations
  • Infrastructure engineering becomes like a development

Introduction to Vagrant

Launched at 2010

Development environemnt on VM, container or cloud

Same

  • among team members
  • among production and development *VirtualBox, AWS EC2

"Mature, stable, proven."

by author, Mr. Mitchell Hashimoto

Targeted Vagrant Users

  • Web application developers

Vagrant-KVM

Preconfigured VM images

  • Base for environment
  • Community shared

VagrantCloud.com

  • shares boxes

Shell, Puppet, Ansible, Docker, and Chef

Vagrant-KVM vs Vagrant-libvirt

Future plan

  • SUpport full features of Vagrant
  • Multiple architecture(ARM)
  • Linux kernel debug support

Infrastructure

New trend on DevOps

  • Infrastructure engineering

Blue Green Deployment

Packer

  • Create virtual machine image for several kind of cloud/virtulization infrastructures.

Vagrant and Docker

  • Prepare Docker environment on Vagrant

Introduction to a Virtulization Testsuite Based on Autotest Testing Framework

Parameter references

  • Problem: Massive Parameters
  • Solution: Cartesian Configuration

Pre&Post Processes

  • Initialize Resources
  • Setup and Cleanup Servies
  • Prepare Environment

OS Support

  • Just enough OS: JeOS
  • Based on Fedora
  • Less than 200Mib after compress

Unattended installation

Virt-test: Runner

  • Bootstrap
  • Configurations
  • List & Run tests

Add Tests(new provider)

Future work

  • A fully libvirt testsuite
  • Support more virtualization types
  • Bug fix & Enhancements

Google BigQuery for Open Data and the Open Source Community

BigQuery

GCP

  • Google Cloud Platform
  • PaaS

Google Big Data Scale at Glace

Dremel

True MPP(Massively Parallel Processing)

BigQuery Highlights

  • Big JOIN
  • BigQuery Streaming API
  • BigQuery Connector for Hadoop

BigQuery for Open Data

  • Massive open datasets available on BigQuery

HTTP Archive and BigQueri.es

BigQuery for Open Source Community

Fluentd: a popular OSS tool for log collection

Lambda Architecture on Google Cloud Platform

  • Fluentd
  • Norikra
  • BigQuery
  • Google Spreadsheet
  • Docker

Norikra: Real-time Complex Event Processing(CEP)

cloud.google.com

Raw
LinuxConJapan2014-day2.md

LINUXCON JAPAN 2014 day2

2014/05/21

Btrfs

http://marc.merlins.org/linux/talks/Btrfs-LC2014-JP/Btrfs.pdf

How The Linux Kernel is Developed

Getting involved

  • Documentation/HOWTO
  • Documentation/development-procedure

Linux Kernel Newbie

  • kernelnewbies.org

The Eudyptula Challenge

Presentation on how the Linux kernel is developed

Arch, Gentoo, SuSE, Fedora がVanillaカーネルの練習に良い

Ubuntuはダメ

Enhancement Activities on the Current Upstream Kernel for Mission-Critical Systems

Using Linux Kenerl for Mission-Critical Systems

  • Banking, Trains, ...

Memory dump deadlock

Memory dump deadlock

  • Kdump when serious problems which include panic

Kdump

  • Kernel crash dumping feature based on Kexec

Serial RX interrupt frequency

Serial devices are mainly used not only for embedded systems but also for mission-critical systems

  • maintenance
  • sensor feedback

We need configure for serial device driver

Added new I/F to the serial driver

  • Tunable RX interrupt trigger frequency
  • Usability problems of 1st/2nd patch
  • Change the ioctl(2) to systfs I/F after discussion in a Linux community

ftrace

ftrace is in-kernel tracer for debugging and analyzing problems

ftrace records PIDs and process names in order to specify process context of an

ftrace has saved_cmdlines file storing the PID-process name mapping table

  • It stores the list of 128 processes that hit a tracepoint

Develop tunable I/F

Solve the printk message fragmentation problem

printk message outputs error logging or debugging information in kernel

  • we handle automatically printk messages in user space in order to detect that the system has became unstable
  • We want the kernel to output printk as expected

mixed kernel messages

Mixed messages can occur when multiple printk() are executed at the same time

  • We named this problem "the owner problem"

Continuouts printk messages whith KERN_CONT can be also fragmented

  • We named this problem "the reconstruction problem"

Use the header information of /dev/kmsg

  • ,<sequence#>,,;

Users can roughly understand where the message outputed

Dive Into the Internals of Kernel Network Stack with Direct Code Execution

http://www.slideshare.net/hajimetazaki/conext13tazakidce

a testing framework for network stack is really needed

userspace version of network stack helps a lot

Development of network stack

Can we really test network stack?

  • to keep the pace of development
  • to keep the quality of software

Issues

  • OSPF is an ISP network
  • How to setup?
  • Limitation of network topology
  • How to reproduce a bug in a heavy load situation?
  • Reproducibility is important to ensure regression tests are meaningful

Regresssions we've seen in net-next tree

Alternative: VM

LXC, UML, OpenVZ...

  • Light-weight-virtualization
  • run many instances

High load with large numbers of vm

Alternative: OSv

Minimal Guest OS (for Cloud)

  • No system call
  • No user/kernel space

1 process/a VM

Very lightweight

With controllability

nfsim

a netfilter simulation environment

LD_PRELOAD=fakesockopt.so /sbin/iptables -L ...

Only a single (kernel) instance

Our solution is ...

Direct Code Execution (DCE)

DCE is

  • a userspace kernel network stack
  • with asm-generic based architecture
  • with multiple hosts by dlmopen
  • in a single userspace process

DCE makes

  • reproducible testing platform

Features

  • Functional Realism
  • Timing realism
  • Debuggability

Kernel Layer

Similar to Library OS

  • shared library
  • replaceable

Mapping via glue code

glue code in arch/sim

POSIX layer

Our POSIX implementation

  • Pass-through host library calls
  • system call => hijacking

Supported codes

  • iproute2
  • quagga
  • umip
  • ...

How to use it?

Recompile

  • Userspace as Position independent executable
  • kernelspace as shared library

Debubggability

gdb

Automated Testing

Jenkins CI

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment