# Kernel Dev Blog 1 -- Setup ## 4/4/2020 I want to grow, and there comes a point where you turn to yourself to do this. I started in calculator programming in grade school and then to web and graphics in high school. Web was a trajectory I was set on with my first job being a basic web developer contractor. It was forgiving and an excellent entry point into this career. From there my education as an undergraduate in Software Engineering opened me up and improved me considerably from the hacker-diy coder to a principled engineer with foresight. I learned architectural principles, methodologies and a variety of ancillary fields of mathematics and business that helped make my skillset more valuables to others. After college I did game development with a friend for years, working on the XBOX platform and PC platform mostly. This stretched me considerably as I had to develop for performance and scalability and put the Big^O principles into play. I learned about physics engines, data structures and algorithms such as pooling, quadtrees and pathfinding and even a considerable amount of packeted networking. Multi threaded development went from sandbox-theory to practice and proper object oriented development and memory management was essential for the code to survive our gaming dreams. Now, back in the web world I have been stretched more in the dev ops and infrastructure areas over the last few years. CI pipelines, EC2 autoscalers, apache bench testing, DDOS and threat mitigations and scalable architectures of applications have been very important. AWS has become my bread and butter, becoming certified and wielding it like there is no other option. But I find myself getting to that point of saturation, where the growth is beginning to diminish in my mid-thirties and I want to get better. Time to jump into a new pool of water. I truly respect those guys who now how to code for the machine, who speak its true language and provide incredible open source platforms for the world. Guys like Linus, Ken Thompson and John Carmack have always inspired me, so I figured I would take a stab at some lower level engineering: Linux, Device Drivers and Operating Systems in C. This is my walk into that. --- Using KernalNewbies to try and learn about Linux Kernel development: https://kernelnewbies.org/FirstKernelPatch ## 1. Start Installed VirtualBox with Ubuntu18 Server Installed my ashes setup Installed set of tools: `kerneldev_install.sh` ```shell sudo apt-get install libncurses5-dev gcc make git exuberant-ctags libssl-dev bison flex libelf-dev bc ``` ### About the Tools `gcc` is obviously the C compiler `make` is the tool to build projecs and link files properly `git` is obvious `libncurses5-dev` lets us build terminal gui editors for menus etc `exuberant-ctags` code index generator for editors `libssl-dev` TLS and SSL protocols for key generation and signing etc `bison` a "parser generator" for language grammars `flex` lexical analyser generator for text pattern matching (combined with parser generator?) `libelf-dev` Executable and Linkable Format standard file format for executable files, object code, libraries etc `bc` math processing language ## 2. Download the Code Cloned the repo on a specific branch ```shell git clone -b staging-testing git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git ``` Next is compiling, but reading first... --- ## Digress:: Device Drivers CH1 Book https://static.lwn.net/images/pdf/LDD3/ch01.pdf ### Driver Overview Drivers provide an API to hardware without exposing the internals of the hardware Separate "mechanism" from "policy", the drive should provide _what_ can be done with the hardware now _how_ it should be used > write kernel code to access the hardware, but don’t force particularpolicies on the user, since different users have different needs Drivers are a software layer between applications and hardware and should take into consideration: - concurrent usage - synchronous and asynchronous operation - being opened more than once - less software layers to reduce "policy" - library for easier usage with applications ### Kernel Overview Kernel manages four main things: - Process Management - Memory Management - Filesystems - Device Control (device drivers embedded into the kernel for ever physical hardware attached) - Networking Loadable Modules are a thing, meaning that object code can be loaded into the kernel at runtime to extend functionality Object code Modules can be linked and unlinked with `insmod` and `rmmod` Loading modules requires super user access and is a method that security vulnerabilities occur, root kits?a Kernel can also be compiled without supporting modules and only drivers compiled with it function ### Device and Module Classes __Char Devices__ (Character Devices) can be accessed as a stream of bytes __Block Devices__ host a filesystem for I/O larger than 512 bytes __Network Interfaces__ in charge of sending and receiving packets with other hosts ### Security Security code is mostly policy and should be addressed at higher levels of code You should however be conscious of dangerous operations that could damage systems or other users operations Also coding against vulnerabilities such as buffer overflows are important Other rules: - treat user input suspiciously - initialize all memory used to avoid random code existing in your runtime ### Version Numbering > As far as the kernel is concerned, the even-numbered kernel versions (i.e., 2.6.x) are the stable ones for general distribution > The odd versions (such as2.7.x), on the contrary, are development snapshots and are quite ephemeral --- ## 3. Copy Boot Configuration The config for drivers etc is not included in the source (since its machine dependent) so we are gonna copy from Ubuntu There is a config that lives in `/boot` that is thre kernel config file which is titled after the current release Get current release with ```shell $ uname -r # uname get system info, -r gets release version ``` Which pointed to a file `/boot/config-4.15.0-91-generic` Copied to repo codebase ```shell $ cp /boot/config-4.15.0-91-generic .config ``` Add a local version to the config file so you don't overwrite your OS release config E.g. `CONFIG_LOCALVERSION="-doomguyv1"` You can restore config values to default with `make olddefconfig` You can use a cool ncurses editor with `make menuconfig` ## 4. Make and Compile `make` or `make -jX` where "X" is the number of cores. (My vm was 2 core) Ran out of HD space with so upped to 40GB ```shell $ make -j2 ... LD [M] sound/usb/line6/snd-usb-pod.ko LD [M] sound/usb/line6/snd-usb-podhd.ko LD [M] sound/usb/line6/snd-usb-toneport.ko LD [M] sound/usb/line6/snd-usb-variax.ko LD [M] sound/usb/misc/snd-ua101.ko LD [M] sound/usb/snd-usb-audio.ko LD [M] sound/usb/snd-usbmidi-lib.ko LD [M] sound/usb/usx2y/snd-usb-us122l.ko LD [M] sound/usb/usx2y/snd-usb-usx2y.ko LD [M] sound/x86/snd-hdmi-lpe-audio.ko LD [M] virt/lib/irqbypass.ko ``` Done! ## 5. Install Kernel Seems you can do it yourself and update GRUB config to launch into Guide said Ubuntu has `installkernel` script already on it so we can use `make` to install > This script installs the kernel to /boot/, installs modules to /lib/modules/X.Y.Z/ (where X.Y.Z is something like 3.1.5), and updates file /boot/grub/grub.conf This should put a new install into `/boot` ```shell $ sudo make modules_install install ... INSTALL drivers/input/mouse/appletouch.ko INSTALL drivers/input/mouse/bcm5974.ko INSTALL drivers/input/mouse/cyapatp.ko ... DEPMOD 5.6.0-doomguyv1+ sh ./arch/x86/boot/install.sh 5.6.0-doomguyv1+ arch/x86/boot/bzImage \ System.map "/boot" run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 5.6.0-doomguyv1+ /boot/vmlinuz-5.6.0-doomguyv1+ run-parts: executing /etc/kernel/postinst.d/initramfs-tools 5.6.0-doomguyv1+ /boot/vmlinuz-5.6.0-doomguyv1+ update-initramfs: Generating /boot/initrd.img-5.6.0-doomguyv1+ run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 5.6.0-doomguyv1+ /boot/vmlinuz-5.6.0-doomguyv1+ run-parts: executing /etc/kernel/postinst.d/update-notifier 5.6.0-doomguyv1+ /boot/vmlinuz-5.6.0-doomguyv1+ run-parts: executing /etc/kernel/postinst.d/x-grub-legacy-ec2 5.6.0-doomguyv1+ /boot/vmlinuz-5.6.0-doomguyv1+ Searching for GRUB installation directory ... found: /boot/grub Searching for default file ... found: /boot/grub/default Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst Searching for splash image ... none found, skipping ... Found kernel: /boot/vmlinuz-4.15.0-91-generic Replacing config file /run/grub/menu.lst with new version Found kernel: /boot/vmlinuz-5.6.0-doomguyv1+ Found kernel: /boot/vmlinuz-4.15.0-91-generic Replacing config file /run/grub/menu.lst with new version Updating /boot/grub/menu.lst ... done run-parts: executing /etc/kernel/postinst.d/zz-update-grub 5.6.0-doomguyv1+ /boot/vmlinuz-5.6.0-doomguyv1+ Sourcing file `/etc/default/grub' Generating grub configuration file ... Found linux image: /boot/vmlinuz-5.6.0-doomguyv1+ Found initrd image: /boot/initrd.img-5.6.0-doomguyv1+ Found linux image: /boot/vmlinuz-4.15.0-91-generic Found initrd image: /boot/initrd.img-4.15.0-91-generic done ``` ## 6. Boot to Kernel We need to update the GRUB config file at `/etc/default/grub` I changed `GRUB_TIMEOUT_STYLE` and `GRUB_TIMEOUT` ```shell #GRUB_TIMEOUT_STYLE=hidden GRUB_TIMEOUT_STYLE=menu #GRUB_TIMEOUT=0 GRUB_TIMEOUT=10 ``` ```shell $ sudo update-grub2 Sourcing file `/etc/default/grub' Generating grub configuration file ... Found linux image: /boot/vmlinuz-5.6.0-doomguyv1+ Found initrd image: /boot/initrd.img-5.6.0-doomguyv1+ Found linux image: /boot/vmlinuz-4.15.0-91-generic Found initrd image: /boot/initrd.img-4.15.0-91-generic done $ sudo reboot ``` From there the GRUB menu appeared and I chose "Advanced Ubuntu Options" where it showed the kernel version ### !!!ERROR Attempting to boot into my build resulted in a long error string that ended in ```text [ 5.152144] ----[ end Kernel panic - not syncing: System deadlocked on memory ]--- ``` ### !!!SUCCESS A little bit of searching and I upped the memory size from 2GB to 4GB on the VM and it worked ```shell $ uname -a Linux doomkern 5.6.0-doomguyv1+ #2 SMP Sat Apr 4 22:23:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux ``` ## 7. GIT Email Setup Now that we have the Kernel compiling and launching we need text based email system We need to submit patches via GIT email which can be done with three functions: `sudo apt-get install esmtp` a mail transport client `sudo apt-get install mutt` a text based mail client that can send plan text patches `sudo apt-get install git-email` to send email patches with git ## 8. Build CTAGS for Kernel Codebase CTAGS is basically an indexer to find symbols in code (intellisense) We already installed it so we can build the ctags with ```shell $ make tags GEN tags ```