[ previous ] [ Contents ] [ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ] [ 6 ] [ 7 ] [ 8 ] [ 9 ] [ 10 ] [ 11 ] [ 12 ] [ 13 ] [ 14 ] [ 15 ] [ A ] [ next ]

Debian Reference
Chapter 13 - Programming

Do not use "test" as the name of an executable test file. test is a shell built-in.

13.1 Where to start

References:

Many long info documents can be obtained as paperbacks from GNU.

The next four sections contain sample scripts in different languages for creating a text file of account information to be added to /etc/passwd using a batch processor such as the newusers program. Each script requires as input a file with lines of the form first_name last_name password. (Actual user home directories will not be created via these scripts.)

13.2 Shell

Reading shell scripts is the best way to understand how a Unix-like system works. Here, I give some pointers and reminders for shell programming. See Shell Mistakes to learn from mistakes.

13.2.1 Bash – GNU standard interactive shell

References for Bash:

Short program example (creates account entries for newusers from standard input): 

     #!/bin/bash
     # (C) Osamu Aoki Sun Aug 26 16:53:55 UTC 2001 Public Domain
     pid=1000;
     while read n1 n2 n3 ; do
     if [ ${n1:0:1} != "#" ]; then
     let pid=$pid+1
     echo ${n1}_${n2}:password:${pid}:${pid}:,,,/home/${n1}_${n2}:/bin/bash
     fi
     done

13.2.2 POSIX shells

Several packages provide a POSIX shell in Debian:

If you are writing a shell script for portability, it is best to write it as a POSIX shell script. Use /bin/sh linked to ash (or dash) to test its POSIX compliance. Avoid writing scripts with "bashisms" or the "zshisms". For example, avoid:

The description for the shell in this document applies only for the POSIX type shells and thus does not apply for the csh type shells including tcsh.

13.2.3 Shell parameters

Several special parameters to remember: 

     $0      = name of the shell or shell script
     $1      = first(1) shell argument
      ...
     $9      = ninth(9) shell argument
     $#      = number of positional parameters
     "$*"    = "$1 $2 $3 $4 ... $n"
     "$@"    = "$1" "$2" "$3" "$4" ... "$n"
     $?      = exit status of the most recent command
     $$      = PID of this shell script
     $!      = PID of most recently started background job

Basic parameter expansions to remember: 

         Form        If var is set       If var is not set
     ${var:-string}  $var                string
     ${var:+string}  string              null
     ${var:=string}  $var                string 
                                         (and run var=string)
     ${var:?string}  $var                (echo string and then exit)

Here, the colon `:' in all of these operators is actually optional.

Basic parameter substitutions to remember: 

         Form        Result
     ${var%suffix}   Remove smallest suffix pattern
     ${var%%suffix}  Remove largest suffix pattern
     ${var#prefix}   Remove smallest prefix pattern
     ${var##prefix}  Remove largest prefix pattern

13.2.4 Shell redirection

Basic redirection to remember (here the [n] is an optional number to specify the file descriptor): 

     [n]> file      Redirect stdout (or n) to file.
     [n]>> file     Append stdout (or n) to file.
     [n]< file      Redirect stdin (or n) from file.
     [n1]>&n2       Redirect stdout (or n1) to n2.
     2> file  >&2   Redirect stdout and stderr to file.
      > file 2>&1   Redirect stdout and stderr to file.
     | command      Pipe stdout to command.
     2>&1 | command Pipe stderr and stdout to command.

Here,

The shell allows you to open files using the exec built-in with an arbitrary file descriptor. 

     $ echo Hello >foo
     $ exec 3<foo 4>bar  # open files
     $ cat <&3 >&4       # redirect stdin to 3, stdout to 4
     $ exec 3<&- 4>&-    # close files
     $ cat bar
     Hello

Here n<&- and n>&- mean to close the file descriptor n.

13.2.5 Shell conditionals

Each command returns an exit status which can be used for conditional expressions:

Note that the use here of a 0 value to mean "true" differs from the usual convention in some other areas of computing. Also, `[' is the equivalent of the test command, which evaluates its arguments up to `]' as a conditional expression.

Basic conditional idioms to remember are: 

     command && if_success_run_this_command_too || true
     command || if_not_success_run_this_command_instead
     
     if [ conditional_expression ]; then  
      if_success_run_this_command
     else
      if_not_success_run_this_command
     fi

Here || true was needed to ensure this shell script will not exit at this line accidentally when shell is invoked with -e flag.

File comparison operators in the conditional expression are: 

     -e file         True if file exists.
     -d file         True if file exists and is a directory.
     -f file         True if file exists and is a regular file.
     -w file         True if file exists and is writable.
     -x file         True if file exists and is executable.
     file1 -nt file2 True if file1 is newer than file2. (modification)
     file1 -ot file2 True if file1 is older than file2. (modification)
     file1 -ef file2 True if they are the same device and inode numbers.

String comparison operators in the conditional expression are: 

          -z str    True if the length of str is zero.
          -n str    True if the length of str is non-zero.
     str1 == str2   True if the strings are equal.
     str1 =  str2   True if the strings are equal.
         ("=" should be used in place of "==" for strict POSIX compliance) 
     str1 != str2   True if the strings are not equal.
     str1 <  str2   True if str1 sorts before str2 (locale dependent).
     str1 >  str2   True if str1 sorts after str2 (locale dependent).

Arithmetic integer comparison operators in the conditional expression are -eq, -ne, -lt, -le, -gt, and -ge.

13.2.6 Command-line processing

The shell processes a script as follows:

Single quotes within double quotes have no effect.

Executing set -x in the shell or invoking the shell with -x option make the shell to print all of commands executed. This is quite handy for debugging.

13.3 Awk

References for Awk:

Short program example (creates newusers command entry): 

     #!/usr/bin/awk -f
     # Script to create a file suitable for use in the 'newusers' command,
     # from a file consisting of user IDs and passwords in the form:
     # first_name last_name password
     # Copyright (c) KMSelf Sat Aug 25 20:47:38 PDT 2001
     # Distributed under GNU GPL v 2, or at your option, any later version.
     # This program is distributed WITHOUT ANY WARRANTY.
     
     BEGIN {
         # Assign starting UID, GID
         if ( ARGC > 2 ) {
             startuid = ARGV[1]
             delete ARGV[1]
         }
         else {
             printf( "Usage:  newusers startUID file\n" \
               "  where:\n" \ 
               "    startUID is the starting userid to add, and\n" \
               "    file is an input file in form:\n" \
               "      first_name last_name password\n" \
             )
             exit
         }
     
         infile = ARGV[1]
         printf( "Starting UID: %s\n\n", startuid )
     }
     
     /^#/ { next }
     
     {
         ++record
         first = $1
         last = $2
         passwd = $3
         user= substr( tolower( first ), 1, 1 ) tolower( last )
         uid = startuid + record - 1
         gid = uid
         printf( "%s:%s:%d:%d:%s %s,,/home/%s:/bin/bash\n",  \
             user, passwd, uid, gid, first, last, user \
             )
     }

Two packages provide POSIX awk in Debian:

13.4 Perl

This is the interpreter on a Unix-like system.

References for Perl:

Short program example (creates newusers command entry): 

     #!/usr/bin/perl
     # (C) Osamu Aoki Sun Aug 26 16:53:55 UTC 2001 Public Domain
     $pid=1000;
     while (<STDIN>) {
             if (/^#/) { next;}
             chop;
             $pid++;
             ($n1, $n2, $n3) = split / /;
             print $n1,"_",$n2,":", $n3, ":",$pid,
                       ":",$pid,",,,/home/",$n1,"_",$n2,":/bin/bash\n"
     }

Install Perl module module_name: 

     # perl -MCPAN -e 'install module_name'

13.5 Python

It's a nice object-oriented interpreter.

References for Python:

Short program example (creates newusers command entry): 

     #! /usr/bin/env python
     import sys, string
     
     # (C) Osamu Aoki Sun Aug 26 16:53:55 UTC 2001 Public Domain
     # Ported from awk script by KMSelf Sat Aug 25 20:47:38 PDT 2001
     # This program is distributed WITHOUT ANY WARRANTY.
     
     def usages():
         print \
     "Usage:  ", sys.argv[0], " start_UID [filename]\n" \
     "\tstartUID is the starting userid to add.\n" \
     "\tfilename is input filename. If not specified, standard input.\n\n" \
     "Input file format:\n"\
     "\tfirst_name last_name password\n"
                     return 1
     
     def parsefile(startuid):
         #
         # main filtering
         #
         uid = startuid
         while 1:
             line = infile.readline()
             if not line:
                 break
             if line[0] == '#':
                 continue
             (first, last, passwd) = string.split(string.lower(line))
             # above crashes with wrong # of parameters :-)
             user = first[0] + last
             gid = uid
             lineout = "%s:%s:%d:%d:%s %s,,/home/%s:/bin/bash\n" %  \
                 (user, passwd, uid, gid, first, last, user)
             sys.stdout.write(lineout)
             +uid
     
     if __name__ == '__main__':
         if len(sys.argv) == 1:
             usages()
         else:
             uid = int(sys.argv[1])
             #print "# UID start from: %d\n" % uid
             if len(sys.argv) > 1:
                 infilename   = string.join(sys.argv[2:])
                 infile = open(infilename, 'r')
                 #print "# Read file from: %s\n\n" % infilename
             else:
                 infile = sys.stdin
             parsefile(uid)

13.6 Make

References for Make:

Simple automatic variables:

Rule syntax: 

     target: [ prerequisites ... ]
      [TAB]  command1
      [TAB]  -command2 # ignore errors
      [TAB]  @command3 # suppress echoing

Here [TAB] is a TAB code. Each line is interpreted by the shell after make variable substitution. Use \ at the end of a line to continue the script. Use $$ to enter $ for environment values for a shell script.

Implicit rules for the target and prerequisites can be written, for example, as: 

     %: %.c header.h

or, 

     %.o: %.c header.h

Here, the target contains the character % (exactly one of them). The % can match any nonempty substring in the actual target filenames. The prerequisites likewise use % to show how their names relate to the actual target name.

Suffix rules are the obsolete way of defining implicit rules for make. They are still supported in GNU make for compatibility, but use equivalent pattern rules whenever possible: 

     old suffix rule --> new pattern rule
     .c:             --> %  : %.c
     .c.o:           --> %.o: %.c

Automatic variables for the rule: 

     foo.o: new1.c new2.c old1.c new3.c
     $@ == foo.o                         (target)
     $< == new1.c                        (first one)
     $? == new1.c new2.c new3.c          (newer ones)
     $^ == new1.c new2.c old1.c new3.c   (all)
     $* == `%' matched stem in the target pattern.

Variable references: 

     foo1 := bar    # One-time expansion
     foo2  = bar    # Recursive expansion
     foo3 += bar    # Append
     SRCS := $(wildcard *.c)
     OBJS := $(foo:c=o)
     OBJS := $(foo:%.c=%.o) 
     OBJS := $(patsubst %.c,%.o,$(foo)) 
     DIRS  = $(dir directory/filename.ext) # Extracts "directory"
      $(notdir NAMES...), $(basename NAMES...), $(suffix NAMES...) ...

Run make -p -f/dev/null to see automatic internal rules.

13.7 C

Preparation: 

     # apt-get install glibc-doc manpages-dev libc6-dev gcc

References for C:

13.7.1 Simple C program (gcc)

A simple example to compile example.c with a library libm into an executable run_example: 

     $ cat > example.c << EOF
     #include <stdio.h>
     #include <math.h>
     #include <string.h>
     
     int main(int argc, char **argv, char **envp){
             double x;
             char y[11];
             x=sqrt(argc+7.5);
             strncpy(y, argv[0], 10); /* prevent buffer overflow */
             y[10] = '\0'; /* fill to make sure string ends with '\0' */
             printf("%5i, %5.3f, %10s, %10s\n", argc, x, y, argv[1]);
             return 0;
     }
     EOF
     $ gcc -Wall -g -o run_example example.c -lm
     $ ./run_example
             1, 2.915, ./run_exam,     (null)
     $ ./run_example 1234567890qwerty
             2, 3.082, ./run_exam, 1234567890qwerty

Here, -lm is needed to link library libm for sqrt(). The actual library is in /lib/ with filename libm.so.6, which is a symlink to libm-2.1.3.so.

Look at the last parameter in the output text. There are more than 10 characters even though %10s is specified.

The use of pointer memory operation functions without boundary checks, such as sprintf and strcpy, is deprecated to prevent buffer overflow exploits that leverage the above overrun effects. Instead, use snprintf and strncpy.

13.7.2 Debugging

13.7.2.1 Debugging with gdb

Preparation: 

     # apt-get install gdb

References for gdb:

Use gdb to debug a program compiled with the -g option. Many commands can be abbreviated. Tab expansion works as in the shell. 

     $ gdb program
     (gdb) b 1                # set breakpoint at line 1
     (gdb) run arg1 arg2 arg3 # run program
     (gdb) next               # next line
     ...
     (gdb) step               # step forward
     ...
     (gdb) p parm             # print parm 
     ...
     (gdb) p parm=12          # set value to 12

For debugging from within Emacs, refer to Editor command summary (Emacs, Vim), Section 11.3.4.

Since all installed binaries should be stripped on the Debian system by default, most debugging symbols are removed. In order to make gdb useful for debugging Debian packages, pertinent packages need to be rebuild with following care:

See Policy 10.1 for more info.

13.7.2.2 Check dependency on libraries

Use ldd to find out a program's dependency on libraries: 

     $ ldd /bin/ls
             librt.so.1 => /lib/librt.so.1 (0x4001e000)
             libc.so.6 => /lib/libc.so.6 (0x40030000)
             libpthread.so.0 => /lib/libpthread.so.0 (0x40153000)
             /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)

For ls to work in a chrooted environment, the above libraries must be available in your chrooted environment.

The following commands will also be useful:

13.7.2.3 Debugging with memory leak detection tools

There are several memory leak detection tools available in Debian.

Also check out Debugging Tools for Dynamic Storage Allocation and Memory Management.

13.7.3 Flex – a better Lex

flex is a fast lexical analyzer generator.

References for flex:

You need to provide your own main() and yywrap(), or your program.l should look like this to compile without a library (yywrap is a macro; %option main turns on %option noyywrap implicitly): 

     %option main
     %%
     .|\n    ECHO ;
     %%

Alternatively, you may compile with the -lfl linker option at the end of your cc command line (like AT&T-Lex with -ll). No %option is needed in this case.

13.7.4 Bison – a better Yacc

Several packages provide a Yacc-compatible LALR parser generator in Debian:

References for bison:

You need to provide your own main() and yyerror(). main() calls yyparse() which calls yylex(), usually created with FleX. 

     %%
     
     %%

13.7.5 Autoconf

autoconf is a tool for producing shell scripts that automatically configure software source code packages to adapt to many kinds of UNIX-like systems using the entire GNU build system.

autoconf produces the configuration script configure. configure automatically creates a customized Makefile using the Makefile.in template.

13.7.5.1 Compile and install a program

Debian does not touch files in /usr/local/ (see Supporting diversity, Section 2.5). So if you compile a program from source, install it into /usr/local/ so it will not interfere with Debian. 

     $ cd src
     $ ./configure --prefix=/usr/local
     $ make
     $ make install # this puts the files in the system

13.7.5.2 Uninstall program

If you still have the source and if it uses autoconf/automake and if you can remember how you configured it: 

     $ ./configure all-of-the-options-you-gave-it
     # make uninstall

Alternatively, if you are absolutely sure that the install process puts files only under /usr/local/ and there is nothing important there, you can erase all its contents by: 

     # find /usr/local -type f -print0 | xargs -0 rm -f

If you are not sure where files are installed, you should consider using checkinstall, which provides a clean path for the uninstall.

13.8 Web

Basic interactive dynamic web pages can be made as follows:

For security reasons it is better not to hand craft new hacks for parsing CGI parameters. There are established modules for them in Perl (see Perl, Section 13.4) and Python (see Python, Section 13.5). PHP comes with these functionality. When client data storage is needed, cookies are used. When client side data processing is needed, javascript is frequently used.

For more, see The Common Gateway Interface, The Apache Software Foundation, and JavaScript.

Searching "CGI tutorial" on Google by typing encoded URL http://www.google.com/search?hl=en&ie=UTF-8&q=CGI+tutorial directly to the browser address is a good way to see the CGI script in action on the Google server.

13.9 Document preparation

13.9.1 roff typesetting

Traditionally, roff is the main Unix text processing system.

See roff(7), groff(7), groff(1), grotty(1), troff(1), groff_mdoc(7), groff_man(7), groff_ms(7), groff_me(7), groff_mm(7), and info groff.

A good tutorial on -me macros exists. If you have groff (1.18 or newer), find /usr/share/doc/groff/meintro.me.gz and do the following: 

     $ zcat /usr/share/doc/groff/meintro.me.gz | \
          groff -Tascii -me - | less -R

The following will make a completely plain text file: 

     $ zcat /usr/share/doc/groff/meintro.me.gz | \
         GROFF_NO_SGR=1 groff -Tascii -me - | col -b -x > meintro.txt

For printing, use PostScript output. 

     $ groff -Tps meintro.txt | lpr
     $ groff -Tps meintro.txt | mpage -2 | lpr

13.9.2 SGML

Preparation: 

     # apt-get install debiandoc-sgml debiandoc-sgml-doc

References for debiandoc-sgml:

SGML enables management of multiple formats of a document. One easy SGML system is Debiandoc, which is used here. This requires minor conversion from original text files for the following characters:

To mark a section as a nonprintable comment, enter: 

     <!-- State issue here ... -->

To mark a section with a switchable comment, enter: 

     <![ %FIXME; [ State issue here ... ]]>

In SGML, the first definition of an entity wins. For example: 

     <!entity % qref "INCLUDE">
     <![ %qref; [ <!entity param "Data 1"> ]]>
     <!entity param "Data 2">
     &param;

This ends up as "Data 1". If the first line has "IGNORE" instead of "INCLUDE", this ends up as "Data 2" (the second line is a conditional statement). Also, repeating phrases can be defined in advance separately from the context. 

     <!entity whoisthis "my">
     Hello &whoisthis; friend.
     This is &whoisthis; book.

This results in the following: 

     Hello my friend.
     This is my book.

See the short SGML example sample.sgml in the examples.

When SGML documents become bigger, sometimes TeX which is used as the backend text processor may cause errors. See TeX/LaTeX, Section 13.9.3.

13.9.3 TeX/LaTeX

Preparation: 

     # tasksel # select Miscellaneous  --> TeX/LaTeX environment

References for LaTeX:

This is the most powerful typesetting environment. Many SGML processors use this as their back end text processor. Lyx provided by lyx, lyx-xforms, or lyx-qt and GNU TeXmacs provided by texmacs package offers nice WYSIWYG editing environment for LaTeX while many use Emacs and Vim as the choice for the source editor.

There are many online resources available:

When documents become bigger, sometimes TeX may cause errors. You must increase pool size in /etc/texmf/texmf.cnf (or more appropriately edit /etc/texmf/texmf.d/95NonPath and run update-texmf) to fix this.

13.9.4 Literate Programming

Instead of writing code containing documentation, the literate programmer writes documentation containing code. This approach ensures a good documentation for a program.

For more on literate-programming, see Literate Programming.

13.9.4.1 Noweb

Preparation: 

     # apt-get install nowebm

References for Noweb:

This is a WEB-like literate-programming tool which is simpler while providing extensibility and language-independence. [66] When noweb is invoked, it writes the program source code to the output files mentioned in the noweb file, and it writes a TeX file for typeset documentation.

The Debian ifupdown package is a fine example. 

     $ apt-get source ifupdown
     $ cd ifupdown*
     $ make ifupdown.pdf ifupdown.ps

13.9.4.2 Doxygen

Preparation: 

     # apt-get install doxygen doxygen-doc doxygen-gui

References for Doxygen (created by doxygen!):

It can generate HTML, RTF, Unix manual pages, PostScript, and PDF (using LaTeX) documentation for C++, C, Java, IDL, and to some extent PHP and C# programs. Doxygen is compatible to JavaDoc (1.1), Qt-Doc, KDOC and was specifically designed to be used for projects that make use of Troll Tech's Qt toolkit. It creates include dependency graphs, collaboration diagrams, and graphical class hierarchy graphs even for not documented programs. The output is similar to Qt's documentation.

13.10 Packaging

Preparation: 

     # apt-get install debian-policy developers-reference \
                       maint-guide dh-make debhelper
     # apt-get install packaging-manual # if Potato

References for packaging:

13.10.1 Packaging a single binary

Quick-and-dirty method to Package a single binary per Joey Hess. 

     # mkdir -p mypkg/usr/bin mypkg/DEBIAN
     # cp binary mypkg/usr/bin
     # cat > mypkg/DEBIAN/control
     Package: mypackage
     Version: 1
     Architecture: i386
     Maintainer: Joey Hess <joeyh@debian.org>
     Description: my little package
      Don't expect much.
     ^D
     # dpkg-deb -b mypkg

13.10.2 Packaging with tools

Use dh_make from the dh-make package to create a baseline package. Then, proceed according to instructions in dh-make(1). This uses debhelper in debian/rules.

An older approach is to use deb-make from the debmake package. This uses no debhelper scripts and depends only on the shell. Do not use this any more.

For examples of multiple-source packages, see "mc" (dpkg-source -x mc_4.5.54.dsc), which uses "sys-build.mk" by Adam Heath (doogie@debian.org), and "glibc" (dpkg-source -x glibc_2.2.4-1.dsc), which uses another system by the late Joel Klecker (espy@debian.org).

[ previous ] [ Contents ] [ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ] [ 6 ] [ 7 ] [ 8 ] [ 9 ] [ 10 ] [ 11 ] [ 12 ] [ 13 ] [ 14 ] [ 15 ] [ A ] [ next ]

Debian Reference

CVS, Thu Jan 18 11:52:15 UTC 2007

Osamu Aoki osamu#at#debian.org
Authors, Section A.1