Archive for the ‘Linux’ Category

Bash cheatsheet - Bash shortcuts you have to master

Bash cheatsheet

In this article I will present what are in my opinion the most useful and worth learning BASH shortcuts. Learning this bash cheatsheet can substantially make you more productive and work faster with the terminal.
This is not intended to be a full list of all possible BASH shortcuts, and if you need it you better check the bash documentation.

You can also download a printer-friendly BASH cheat sheet PDF that you can print and keep on your desk. You will find a link to it at the end of the article.

Command Editing Shortcuts

ShortcutDescription
Ctrl + ago to the start of the command line
Ctrl + ego to the end of the command line
Ctrl + kdelete from cursor to the end of the command line
Ctrl + udelete from cursor to the start of the command line
Ctrl + wdelete from cursor to start of word (i.e. delete backwards one word)
Ctrl + ypaste word or text that was cut using one of the deletion shortcuts (such as the one above) after the cursor
Ctrl + xxmove between start of command line and current cursor position (and back again)
Alt + bmove backward one word (or go to start of word the cursor is currently on)
Alt + fmove forward one word (or go to end of word the cursor is currently on)
Alt + ddelete to end of word starting at cursor (whole word if cursor is at the beginning of word)
Alt + ccapitalize to end of word starting at cursor (whole word if cursor is at the beginning of word)
Alt + umake uppercase from cursor to end of word
Alt + lmake lowercase from cursor to end of word
Alt + tswap current word with previous
Ctrl + fmove forward one character
Ctrl + bmove backward one character
Ctrl + ddelete character under the cursor
Ctrl + hdelete character before the cursor
Ctrl + tswap character under cursor with the previous one

Command Control Shortcuts

ShortcutDescription
Ctrl + lclear the screen
Ctrl + sstops the output to the screen (for long running verbose command)
Ctrl + qallow output to the screen (if previously stopped using command above)
Ctrl + cterminate the command
Ctrl + zsuspend/stop the command

Command Recall Shortcuts

ShortcutDescription
Ctrl + rsearch the history backwards
Ctrl + gescape from history searching mode
Ctrl + pprevious command in history (i.e. walk back through the command history)
Ctrl + nnext command in history (i.e. walk forward through the command history)
Alt + .use the last word of the previous command

----

C++ - Simple command line argument manager

Command-line argument management is tricky to get right especially when the number of options that we support and the number of their combination is big. For this kind of applications, there are already a number of very effective libraries and tools supporting the programming in such a scenario. One such library is Boost::program_options which I highly encourage you to use as is it awesome.

But there are other cases where we need to quickly write a prototype or when the number of options and possible configuration is small where using Boost could be overkill. In such cases what we usually do is writing ad-hoc command-line options management code which will never be reuse and it is tailored for the specific application at hand. This approach costs time and it is boring once you did it once.

For this reason, I wrote a super simple and minimalistic C++17 command-line argument manager which allows you to:

  • Check if a set of options has been specified
  • Get the argument of each element of a set of options (as string).

The class is extremely easy to use. You only need to construct an instance of the class by passing the argc and argv variables and then you are ready to go. No more need for ugly and error prone command line argument parsing. For instance, as you can see from the code snippet below, I check for the existence of the -h or --help options and of all the other required options. If one of -X, -O, -P are not present or its required corresponding argument are not specified a help message is printed. Otherwise, the arguments are retrieved and returned as a vector of strings ready to be used.

The example code is listed below or can be downloaded from here (gist on GitHub). You can also try it live on wandbox.

#include <string>
#include <algorithm>
#include <vector>
#include <tuple>
#include <iostream>
#include <optional>


void usage(const std::string&amp; progName) {
  std::cout << progName << " -X FILE -P FILE -O DIRECTORY " << std::endl
            << "Options:"                        << std::endl
            << "-h  | --help    Print this help" << std::endl
            << "-X              Path to X file"  << std::endl
            << "-P              Path to P file"  << std::endl
            << "-O              Path to output directory." << std::endl;
}

class cmdline_args_parser{
    public:
        cmdline_args_parser (const int argc, char **argv){
            for (int i=0; i < argc; ++i)
                tokens.push_back(std::string(argv[i]));          
        }

        const std::optional<std::string> getCmdOption(const std::string &amp;option) const{
            auto  itr =  std::find(tokens.begin(), tokens.end(), option);
            if (itr != tokens.end() &amp;&amp; ++itr != tokens.end())
                return std::optional(*(itr));           
           return std::nullopt;
        }
    
        template<typename... Options>
        const auto get_all_options(const Options... ops) const{
            std::vector<std::optional<std::string>> v;
            (v.push_back(getCmdOption(ops)), ...);
            return v;
        }

        bool cmdOptionExists(const std::string &amp;option) const{
            return 
                std::find(tokens.begin(), tokens.end(), option) != tokens.end();
        }
 
        template<typename... Options>
        bool all_options_exists(const Options... opts) const{
            return (... &amp;&amp; cmdOptionExists(opts)); 
        }
        template<typename... Options>
        bool any_options_exists(const Options... opts) const{
            return (... || cmdOptionExists(opts)); 
        }
    
        const std::string&amp; get_program_name() const{ return tokens[0]; }
    
    private:
        std::vector<std::string> tokens;
};


auto process_args(const cmdline_args_parser&amp; p){
    auto opts = p.get_all_options("-X","-O","-P");
    if(p.any_options_exists("-h", "--help") || 
       !all_of(begin(opts), end(opts), [](const  auto&amp; x ){return x;}) )
    {
        usage(p.get_program_name());
        return;
    }
    for(const auto opt : opts){
        std::cout<<opt.value()<<std::endl;
    }
}

int main(int argc, char** argv){
cmdline_args_parser p (argc, argv);
process_args(p);    

  return 0;
}

Dynamic Message of the Day - motd - Fedora Linux

HOW-TO: Dynamic Message of the day

This article is  about setting up a dynamic message of the day (possibly informative and fun) as header of each newly opened shell.

The final result will be something like the following:

header_shell

It mixes a fun message using fortune and cowsay which you can install using


sudo dnf install fortune-mod cowsay

utility and some informative info about the status of the system as:

  • System load
  • Ram and Swap available and used
  • Disk space
  • Ip address

 

The script file can be easily configured and extended to suits the your needs. Colors can also be easily customized.

Read On…

Distributed Hadoop installation - Fedora Linux

Distributed  Hadoop and HBase installation - Fedora Linux

In this post I will describe how to get started with the latest versions of hadoop and hbase describing all the step to obtain a working hadoop installation. The steps described here can be easily used to perform a working installation on a large cluster (even tough it can requires additional steps as shared filesystem for instance).

Prerequisites

 sudo dnf install openssh openssh-askpass openssh-clients openssh-server 

Don't forget to start the ssh service using the following command:

 sudo service sshd start 

Add a dedicated Hadoop/HBase User (optional but reccomended)

Read On…

CUDA - Julia Set example code - Fractals

This article present a CUDA parallel code for the generation of the famous Julia Set. Informally a point of the complex plane belongs to the set if given a function f(z) the serie z_{n+1} = f(z_n) does not tend to infinity. Different function and different initial condition give raise eventually to fractals. One of the most famous serie is z_{n+1} = z_n^2 +c  (the one that generated the video below).
Some nice picture may be obtained with the following initial conditions:

  1. c = 1j # dentrite fractal
  2. c = -0.123 + 0.745j # douady's rabbit fractal
  3. c = -0.750 + 0j # san marco fractal
  4. c = -0.391 - 0.587j # siegel disk fractal
  5. c = -0.7 - 0.3j # NEAT cauliflower thingy
  6. c = -0.75 - 0.2j # galaxies
  7. c = -0.75 + 0.15j # groovy
  8. c = -0.7 + 0.35j # frost

Here a video, showing a sequence of picture generated using different initial conditions. A nice example of how math and computer science can produce art!

CUDA Julia set code

The code is such that it is very easy to change the function and the initial condition (just edit the device function functor).

#define TYPEFLOAT

#ifdef TYPEFLOAT
#define TYPE float
#define cTYPE cuFloatComplex
#define cMakecuComplex(re,i) make_cuFloatComplex(re,i)
#endif
#ifdef TYPEDOUBLE
#define TYPE double
#define cMakecuComplex(re,i) make_cuDoubleComplex(re,i)
#endif

__global__ void computeJulia(int* data,cTYPE c,float zoom){
	int i =  blockIdx.x * blockDim.x + threadIdx.x;
	int j =  blockIdx.y * blockDim.y + threadIdx.y;

	if(i<DIMX && j<DIMY){
		cTYPE p = convertToComplex(i,j,zoom);
		data[i*DIMY+j]=evolveComplexPoint(p,c);
	}
}


cTYPE c0;
__device__ cTYPE juliaFunctor(cTYPE p,cTYPE c){
	return cuCaddf(cuCmulf(p,p),c);
}

__device__ int evolveComplexPoint(cTYPE p,cTYPE c){
	int it =1;
	while(it <= MAXITERATIONS && cuCabsf(p) <=4){
		p=juliaFunctor(p,c);
		it++;
	}
	return it;
}


__device__ cTYPE convertToComplex(int x, int y,float zoom){

	TYPE jx = 1.5 * (x - DIMX / 2) / (0.5 * zoom * DIMX) + moveX;
	TYPE jy = (y - DIMY / 2) / (0.5 * zoom * DIMY) + moveY;
	return cMakecuComplex(jx,jy);

}


The code above are the GPU instructions that are devoted to the computation of a single set. The kernel computeJulia() is executed on DIMX*DIMY (the dimension of the image) threads that indipendently compute the evolveComplexPoint() function on the converted to complex corrensponding pixel indices, the viewport (each threads compute a single pixel that has integer coordinates). The evolveComplexPoint() take care of checking how fast che point diverges.

The code is available here on github. It generates a number of png tha can be then merged in a video (as the one I present here) using a command line tool as ffmpeg.

In order to compile and run it need the CUDA SDK,a CUDA capable device and libpng installed on your system.
To compile and/or generate the video simply hit:

#compilation
nvcc -I"/usr/local/cuda-6.5/samples/common/inc"  -O3 -gencode arch=compute_35,code=sm_35  -odir "src" -M -o "src/juliaSet.d" "../src/juliaSet.cu"

nvcc  -I"/usr/local/cuda-6.5/samples/common/inc" -O3 --compile --relocatable-device-code=false -gencode arch=compute_35,code=compute_35 -gencode arch=compute_35,code=sm_35  -x cu -o  "src/juliaSet.o" "../src/juliaSet.cu"

/usr/local/cuda-6.5/bin/nvcc --cudart static --relocatable-device-code=false -gencode arch=compute_35,code=compute_35 -gencode arch=compute_35,code=sm_35 -link -o  "JuliaSetDynParallelism"  ./src/juliaSet.o ./src/main.o   -lpng

#video generation
ffmpeg -f image2 -i "julia_%05d.png" -r 40 output.mov

Here another animation

 

Kernel Linux List - An Efficient Generic Linked List

Linux Kernel Linked List

Linux Kernel especially in the past years was plenty of different and replicated implementation of generic data structures such as linked lists, stacks and queues.  Kernel's developers decided to provide a unique set of API for using these kind of software machineries. We will briefly look in this article  at the list.h file (in the linux source tree under /usr/src/linux-headers-`uname -r`/include/) that provides macros and function for utilizing a doubly linked circular list. It is a very smart written piece of code that is worth to take a look at as it is  an opportunity  to improve  C programming skills and at the same time to see an "unconventional" implementation for a common data structure.

From canonical list to Linux kernel linked list

The canonical implementation of a C's linked list consists of a structure and two (for a doubly linked) recursive pointer fields to the structure itself. The links to the previous and subsequent element of the list.


struct point2DNode{
 int x,y;
 struct point2DNode* prev, next;
};

Read On…

Apple Xserve xeon Linux/Debian armed

Finally at my Departement we succesfully converted an  unsed  Apple xserve  cluster to a full functional Linux/Debian machine.

The cluster is make up of eleven node connected by Myrinet 10G low latency network and each node is a composed by a dual 4x cores Intel Xeon 2.8 Ghz Harpertown with 10Gb of DDR2 for a total of 88 cores. It is a quite old and small machine and hence was unused. We thought that  it could have been useful if well re-configured as an ad-hoc machine for small sized scientific computation session, as  developing and testing platform of parallel application or for training of undergraduate students that can experience directly with a real parallel machine. So we decided to cast a new light on it by configuring it as a Debian cluster (according to google just few people did it). 10949741_766948060061020_241789111_n

Read On…