Linux/390 Observations and Notes

Common Device Support

General Information

Overview of CDS interface concepts

get_dev_info_by_() - Retrieve Device Information

int get_dev_info_by_IRQ( int IRQ, dev_info_t *devinfo);
int get_dev_info_by_devno( unsigned int IRQ, dev_info_t *devinfo);

typedef struct {
unsigned int devno; /* device number */
unsigned int status; /* device status */
senseid_t sid_data; /* senseID data */
} dev_info_t;

//
// SenseID response buffer layout
//
typedef struct {
/* common part */
unsigned char reserved; /* always 0x'FF' */
unsigned short cu_type; /* control unit type */
unsigned char cu_model; /* control unit model */
unsigned short dev_type; /* device type */
unsigned char dev_model; /* device model */
unsigned char unused; /* padding byte */
/* extended part */
ciw_t ciw[62]; /* variable # of CIWs */
} senseid

typedef struct _ciw {
unsigned int et : 2; // entry type
unsigned int reserved : 2; // reserved
unsigned int ct : 4; // command type
unsigned int cmd : 8; // command
unsigned int count : 16; // count
} ciw_t;
Possible CIW entry types are:
#define CIW_TYPE_RDC 0x0; // read configuration data
#define CIW_TYPE_SII 0x1; // set interface identifier
#define CIW_TYPE_RNI 0x2; // read node identifier

Usage Notes

get_IRQ_by_devno() - Convert device identifiers

int get_IRQ_by_devno( unsigned int devno );
unsigned int get_devno_by_IRQ( int IRQ );

read_dev_chars() - Read Device Characteristics

int read_dev_chars( int IRQ, void **buffer, int length );

Usage Notes

request_IRQ() - Request Device Ownership

int request_IRQ( unsigned int IRQ, int (*handler)( int, void *, struct pt_regs *),
 unsigned long irqflags, const char *devname, void *dev_id);

typedef struct {
unsigned int devno; /* device number from irb */
unsigned int intparm; /* interrupt parameter */
unsigned char cstat; /* channel status - accumulated */
unsigned char dstat; /* device status - accumulated */
unsigned char lpum; /* last path used mask from irb */
unsigned char unused; /* not used - reserved */
unsigned int flag; /* flag : see below */
unsigned long cpa; /* CCW addr from irb at prim. status */
unsigned int rescnt; /* count from irb at primary status */
unsigned int scnt; /* sense count, if available */
union {
irb_t irb; /* interruption response block */
sense_t sense; /* sense information */
} ii; /* interrupt information */
} devstat_t;

Usage Notes

free_IRQ() - Release Device Ownership

void free_IRQ( unsigned int IRQ, void *dev_id);

Usage Notes

disable_IRQ() - Disable Interrupts for a given Device

int disable_IRQ( unsigned int IRQ );

The disable-IRQ() routine may return:

Usage Notes

void disable_IRQ( int IRQ);

enable_IRQ() - Enable Interrupts for a given Device

int enable_IRQ( unsigned int IRQ );

do_IO() - Initiate I/O Request

int do_IO( int IRQ, ccw1_t *cpa, unsigned long intparm, unsigned int lpm, unsigned long flag);

typedef struct {
char cmd_code; /* command code */
char flags; /* flags, like IDA addressing, etc. */
unsigned short count; /* byte count */
void *cda; /* data address */
} ccw1_t __attribute__ ((aligned(8)));

If device status DEVSTAT_FLAG_SENSE_AVAIL is indicated in field dev_id->flag, field dev_id->scnt describes the number of device specific sense bytes available in the sense area dev_id->ii.sense. No device sensing by the device driver itself is required.

typedef struct {
unsigned char res[32]; /* reserved */
unsigned char data[32]; /* sense data */
} sense_t;

Depending on the device status, multiple of those values may be set together. Please refer to the device specific documentation for details.

The devi_id->cstat field provides the (accumulated) subchannel status:

Usage Notes

halt_IO() - Halt I/O Request Processing

int halt_IO( int IRQ, /* subchannel number */
 int intparm, /* dummy intparm */
 unsigned int flag); /* operation mode */

Usage Notes

Miscellaneous Support Routines

s390irq_spin_lock( int IRQ);
s390irq_spin_unlock( int IRQ);

These two macro definitions are required to obtain the device specific IRQ lock. The lock needs to be obtained if the device driver intends to call do_IO() or halt_IO() from anywhere but the device interrupt handler (where the lock is already owned). Those routines should only be used if running enabled for interrupts. If running disabled already, the driver should use s390irq_spin_lock() and the corresponding unlock routine instead.

s390irq_spin_lock_irqsave( int IRQ, unsigned long flags);
s390irq_spin_unlock_irqrestore( int IRQ, unsigned long flags);

Special Console Interface Routines

set_cons_dev() - Set Console Device

int set_cons_dev( int IRQ);

reset_cons_dev() - Reset Console Device

int reset_cons_dev( int IRQ);

wait_cons_dev() - Synchronously Wait for Console Processing

int wait_cons_dev( int IRQ); 

The wait_cons_dev() function returns :

Usage Notes

Major and Minor Numbers

Major Block Device 95 - IBM S/390 DASD storage

0 = /dev/dasd0 	First DASD device, major
1 = /dev/dasd0a First DASD device, block 1
2 = /dev/dasd0b First DASD device, block 2
3 = /dev/dasd0c First DASD device, block 3
4 = /dev/dasd1 	Second DASD device, major
5 = /dev/dasd1a Second DASD device, block 1
6 = /dev/dasd1b Second DASD device, block 2
7 = /dev/dasd1c Second DASD device, block 3

Major Block Device 96 - IBM S/390 VM/ESA minidisk

0 = /dev/mnd0 	First VM/ESA minidisk
1 = /dev/mnd1 	Second VM/ESA minidisk

DASD Device Driver

Usage

Low-level format

typedef struct {
	int start_unit;
	int stop_unit;
	int blksize;
} format_data_t;

Make a filesystem

Bugs

• Performance sometimes is rather low because we do not fully exploit clustering

TODO-List

• Add IBM'S Disk layout to genhd
• Enhance driver to use more than one major number
• Enable usage as a module
• Support Cache fast write and DASD fast write (ECKD)

Files added to the Linux Distribution

Linux System Calls

Control Register Usage

Control Register 0

Control Register 1

Control Registers 2-5

Control Register 6

Control Register 7

Control Register 8

Control Registers 9-11

Control Register 12

Control Register 13

Control Register 14

Control Register 15

Access Register Usage

IPL under VM/ESA

Initial RAMDISK

Root Filesystem on VM Minidisk

Copying File Systems

static inline struct task_struct * get_current(void)
{
struct task_struct *current;
__asm__("lhi %0,-8192\n\t"
        "nr  %0,15"
        : "=r" (current) );
return current;
}

int FunctionA(int a) 
{ 
 int b; 
 FunctionC(int c) 
 { 
  b=c+1; 
 } 
 FunctionC(10); 
 return(b); 
}

0040037c int test(int b) 
{ # Function prologue below 
40037c: 90 de f0 34 	stm 	%r13,%r14,52(%r15) 	# Save registers r13 & r14 
400380: 0d d0 	basr 	%r13,%r0 		# Set up pointer to constant pool using 
400382: a7 da ff fa 	ahi 	%r13,-6 		# basr trick 
return(5+b); 
# Huge main program 
400386: a7 2a 00 05 	ahi 	%r2,5 			# add 5 to r2 
# Function epilogue below 
40038a: 98 de f0 34 	lm 	%r13,%r14,52(%r15) 	# restore registers r13 & 14 
40038e: 07 fe 	br 	%r14 			# return 
} 

Literal pool for main. 
400390: ff ff ff ec 	.long 0xffffffec 
main(int argc,char *argv[]) 
{ # Function prologue below 
400394: 90 bf f0 2c 	stm 	%r11,%r15,44(%r15) 	# Save necessary registers 
400398: 18 0f 		lr 	%r0,%r15 		# copy stack pointer to r0 
40039a: a7 fa ff a0 	ahi 	%r15,-96 		# Make area for callee saving 
40039e: 0d d0 		basr 	%r13,%r0 		# Set up r13 to point to 
4003a0: a7 da ff f0 	ahi 	%r13,-16 		# literal pool 
4003a4: 50 00 f0 00 	st 	%r0,0(%r15) 		# Save backchain 
return(test(5)); 						# Main Program Below 
4003a8: 58 e0 d0 00 	l 	%r14,0(%r13) 		# load relative address of test from 
# literal pool 
4003ac: a7 28 00 05 	lhi 	%r2,5 			# Set first parameter to 5 
4003b0: 4d ee d0 00 	bas 	%r14,0(%r14,%r13) 	# jump to test setting r14 as return 
 								# address using branch & save instruction. 
# Function Epilogue below 
4003b4: 98 bf f0 8c 	lm 	%r11,%r15,140(%r15)	# Restore necessary registers. 
4003b8: 07 fe 		br 	%r14 			# return to do program exit 
} 

make arch/s390/kernel/signal.i 

s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer 
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -E arch/s390/kernel/signal.c 
arch/s390/kernel/signal.i

# 1 "/home1/barrow/linux/include/asm/types.h" 1 

typedef unsigned short umode_t; 
typedef __signed__ char __s8; 
typedef unsigned char __u8; 
typedef __signed__ short __s16; 
typedef unsigned short __u16; 

make arch/s390/kernel/signal.s 
s390-gcc -D__KERNEL__ -I/home1/barrow/linux/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer 
-fno-strict-aliasing -D__SMP__ -pipe -fno-strength-reduce -S arch/s390/kernel/signal.c 
-o arch/s390/kernel/signal.s   

.LC54: 
.string "misaligned (__u16 *) in __xchg\n" 
.LC57: 
.string "misaligned (__u32 *) in __xchg\n" 
.L$PG1: # Pool sys_sigsuspend 
.LC192: 
.long -262401 
.LC193: 
.long -1 
.LC194: 
.long schedule-.L$PG1 
.LC195: 
.long do_signal-.L$PG1 
.align 4
.globl sys_sigsuspend 
.type sys_sigsuspend,@function 
sys_sigsuspend: 
# leaf function 0 
# automatics 16 
# outgoing args 0 
# need frame pointer 0 
# call alloca 0 
# has varargs 0 
# incoming args (stack) 0 
# function length 168 
        STM    8,15,32(15) 
        LR     0,15 
        AHI    15,-112 
        BASR   13,0 
.L$CO1: AHI    13,.L$PG1-.L$CO1 
        ST     0,0(15) 
        LR     8,2 
        N      5,.LC192-.L$PG1(13) 

objdump --source <victim program or object file> > <victims debug listing>

objdump --source vmlinux > vmlinux.lst 

objdump --disassemble-all --syms vmlinux > vmlinux.lst 

strace ping -c 1 127.0.0.1 

socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3 
getuid() = 0 
setuid(0) = 0 
stat("/usr/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory) 
stat("/usr/share/locale/libc/C", 0xbffff134) = -1 ENOENT (No such file or directory) 
stat("/usr/local/share/locale/C/libc.cat", 0xbffff134) = -1 ENOENT (No such file or directory) 
getpid() = 353 
setsockopt(3, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0 
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [49152], 4) = 0 
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(3, 1), ...}) = 0 
mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40008000 
ioctl(1, TCGETS, {B9600 opost isig icanon echo ...}) = 0 
write(1, "PING 127.0.0.1 (127.0.0.1): 56 d"..., 42PING 127.0.0.1 (127.0.0.1): 56 data bytes 
) = 42 
sigaction(SIGINT, {0x8049ba0, [], SA_RESTART}, {SIG_DFL}) = 0 
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {SIG_DFL}) = 0 
gettimeofday({948904719, 138951}, NULL) = 0 
sendto(3, "\10\0D\201a\1\0\0\17#\2178\307\36"..., 64, 0, {sin_family=AF_INET, 
sin_port=htons(0), sin_addr=inet_addr("127.0.0.1")}, 16) = 64 
sigaction(SIGALRM, {0x8049600, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
sigaction(SIGALRM, {0x8049ba0, [], SA_RESTART}, {0x8049600, [], SA_RESTART}) = 0
alarm(10) = 0 
recvfrom(3, "E\0\0T\0005\0\0@\1|r\177\0\0\1\177"..., 192, 0, 
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84 
gettimeofday({948904719, 160224}, NULL) = 0 
recvfrom(3, "E\0\0T\0006\0\0\377\1\275p\177\0"..., 192, 0, 
{sin_family=AF_INET, sin_port=htons(50882), sin_addr=inet_addr("127.0.0.1")}, [16]) = 84 
gettimeofday({948904719, 166952}, NULL) = 0 
write(1, "64 bytes from 127.0.0.1: icmp_se"..., 
5764 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=28.0 ms 

strace passwd 2>&1 | grep open 
produces the following output 
open("/etc/ld.so.cache", O_RDONLY) = 3 
open("/opt/kde/lib/libc.so.5", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libc.so.5", O_RDONLY) = 3 
open("/dev", O_RDONLY) = 3
open("/var/run/utmp", O_RDONLY) = 3
open("/etc/passwd", O_RDONLY) = 3
open("/etc/shadow", O_RDONLY) = 3
open("/etc/login.defs", O_RDONLY) = 4 
open("/dev/tty", O_RDONLY) = 4

#!/bin/bash 
/usr/bin/strace -o/t1 -f /usr/sbin/in.telnetd -h

Important options

• -o is used to tell strace to output to a file in our case t1 in the root directory
• -f is to follow children. For example, in our case above telnetd will start the login process & subsequently a shell like bash. You will be able to tell which is which from the process ID's listed on the left hand side of the strace output.
• -p<pid> will tell strace to attach to a running process, yup this can be done provided it isn't being traced or debugged already & you have enough privileges, the reason 2 processes cannot trace or debug the same program is that strace becomes the parent process of the one being debugged & processes (unlike people) can have only one parent.

However the file /t1 will get big quite quickly to test it telnet 127.0.0.1. Now look at what files in.telnetd execve'd

413 execve("/usr/sbin/in.telnetd", ["/usr/sbin/in.telnetd", "-h"], [/* 17 vars */]) = 0 
414 execve("/bin/login", ["/bin/login", "-h", "localhost", "-p"], [/* 2 vars */]) = 0

Whey it worked!

Other hints

If the program is not very interactive (i.e. not much keyboard input) and is crashing in one architecture but not in another you can do an strace of both programs under as identical a scenario as you can on both architectures outputting to a file then. do a diff of the two traces using the diff program. i.e.

and maybe you'll be able to see where the call paths differed, this is possibly near the cause of the crash.

More information

Look at man pages for strace and the various syscalls e.g. man strace, man alarm, man socket.

Debugging under VM

Addresses and values in the VM debugger are always hex never decimal. Address ranges are of the format <HexValue1>-<HexValue2> or <HexValue1>.<HexValue2.> e.g. The address range 0x2000 to 0x3000 can be described as 2000-3000 or 2000.1000.

The VM Debugger is case insensitive.

VM's strengths are usually other debuggers weaknesses you can get at any resource no matter how sensitive e.g. memory management resources, change address translation in the PSW. For kernel hacking you will reap dividends if you get good at it.

The VM Debugger displays operators but not operands, probably because some of it was written when memory was expensive and the programmer was probably proud that it fitted into 2k of memory and the programmers didn't want to shock hardcore VM'ers by changing the interface :-). Also, the debugger displays useful information on the same line and the author of the code probably felt that it was a good idea not to go over the 80 columns on the screen.

As some of you are probably in a panic now this isn't as unintuitive as it may seem as the 390 instructions are easy to decode mentally and you can make a good guess at a lot of them as all the operands are nibble (half byte aligned ). If you have an objdump listing also it is quite easy to follow. If you don't have an objdump listing keep a copy of the ESA Reference Summary and look at between pages 2 and 7 or alternatively the ESA principles of operation. e.g. even I can guess that

0001AFF8' LR 180F CC 0 

is a ( load register ) lr r0,r15

Also it is very easy to tell the length of a 390 instruction from the 2 most significant bits in the instruction (not that this info is really useful except if you are trying to make sense of a hexdump of code):

The debugger also displays other useful info on the same line such as the addresses being operated on destination addresses of branches and condition codes. For example:

00019736' AHI 	A7DAFF0E 			CC 1 
000198BA' BRC 	A7840004 	-> 000198C2'	CC 0 
000198CE' STM 	900EF068 	0FA95E78 	CC 2 

Useful VM debugger commands

I suppose I'd better mention this before I start to list the current active traces do

Q TR 

there can be a maximum of 255 of these per set (more about trace sets later). To stop traces issue:

TR END

To delete a particular breakpoint issue

TR DEL <breakpoint number> 

The PA1 key drops to CP mode so you can issue debugger commands, Doing alt-c (on my 3270 console at least ) clears the screen. Hitting b <enter> comes back to the running operating system from CP mode (in our case Linux ).

It is typically useful to add shortcuts to your PROFILE EXEC file if you have one (this is roughly equivalent to autoexec.bat in DOS or .profile in Linux). Here are a few from mine:

/* this gives me command history on issuing f12 */ 
set pf12 retrieve 
/* this continues */ 
set pf8 imm b 
/* goes to trace set a */ 
set pf1 imm tr goto a 
/* goes to trace set b */ 
set pf2 imm tr goto b 
/* goes to trace set c */ 
set pf3 imm tr goto c 

Instruction Tracing

Setting a simple breakpoint:

TR I PSWA <address> 

To debug a particular function try:

• TR I R <function address range>
• TR I
on its own will single step.
• TR I DATA <MNEMONIC> <OPTIONAL RANGE>
will trace for particular mnemonics. For example, TR I DATA 4D R 0197BC.4000 will trace for BAS'es (opcode 4D) in the range 0197BC.4000 if you were inclined you could add traces for all branch instructions and suffix them with the run prefix so you would have a backtrace on screen when a program crashes.
• TR BR <INTO OR FROM>
will trace branches into or out of an address. For example, TR BR INTO 0 is often quite useful if a program is getting awkward and deciding to branch to 0 and crashing as this will stop at the address before in jumps to 0.  

Displaying and Modifying Registers

• D G
will display all the gprs
• D X
will display all the control registers
• D AR
will display all the access registers
• D AR4-7
will display access registers 4 to 7
• CPU ALL D G
will display the GRPS of all CPUS in the configuration
• D PSW
will display the current PSW
• ST PSW 2000
will put the value 2000 into the PSW and probably crash your machine.
• D PREFIX
will display the prefix register

Displaying Memory

To display memory that was mapped using the current PSW's mapping try:

D <range>

To make VM display a message each time it hits a particular address and continue try:

• D I<range>
will disassemble and display a range of instructions.
• ST addr <32 bit word>
will store 32 bits of data data at a 32 bit aligned address
• ST Saddr <byte-sequence>
will store a byte sequence at a byte aligned address
• D T<range>
will display the EBCDIC in an address (if you are that way inclined)
• D R<range>
will display real addresses, without DAT, but with prefixing.

There are other complex options to display if you need to get at say home space but are in primary space the easiest thing to do is to temporarily
modify the PSW to the other addressing mode, display the stuff and then restore it.

Hints

If you want to issue a debugger command without halting your virtual machine with the PA1 key, then try prefixing the command with #CP:

#cp tr i pswa 2000 

Also suffixing most debugger commands with RUN will cause them not to stop just display the mnemonic at the current instruction on the console. If you have several breakpoints you want to put into your program and you get fed up of cross referencing with System.map you can do the following trick for several symbols.:

grep do_signal System.map 

which emits the following among other things:

0001f4e0 T do_signal 

Now you can do:

TR I PSWA 0001f4e0 cmd msg * do_signal 

This sends a message to your console each time do_signal is entered. (As an aside I wrote a perl script once which automatically generated a REXX script with breakpoints on every kernel procedure. This isn't a good idea because there are thousands of these routines and VM can only set 255 breakpoints at a time, you nearly had to spend as long pruning the file down as you would enter the messages by hand). However, the trick might be useful for a single object file.

Tracing Program Exceptions

If you get a crash which says something like illegal operation or specification exception followed by a register dump You can restart Linux and trace these using the tr prog <range | value> option.

The most common ones you will normally be tracing for is:

The full list of these is on page 22 of the current ESA Reference Summary. For example:

• tr prog 10
will trace segment translation exceptions.
• tr prog
on its own will trace all program interruption codes.

Trace Sets

On starting VM you are initially in the INITIAL trace set. You can do a Q TR to verify this. If you have a complex tracing situation where you wish to wait for instance till a driver is open before you start tracing IO, but know in your heart that you are going to have to make several runs through the code till you
have a clue whats going on.

What you can do is:

TR I PSWA <Driver open address>

Enter b to continue until the breakpoint is reached. Now do your:

TR GOTO B 

To got back to the initial trace set do:

TR GOTO INITIAL 

and the TR I PSWA <Driver open address> will be the only active breakpoint again.

Tracing Linux syscalls under VM

Syscalls are implemented on Linux for S390 by the Supervisor call instruction (SVC) there 256 possibilities of these as the instruction is made up of a 0x0a opcode and the second byte being the syscall number. They are traced using the simple command:

TR SVC <Optional value or range>

The syscalls are defined in linux/include/asm-s390/unistd.h. For example, to trace all file opens just do:

Producing Trace of System Flow

VM's ability to trace branch operations allows the production of system flow data. The output can (and is) quite voluminous but can be made human-friendly by the following process.

1. FTP the System.map to the VM user who will process the flow data or allow the EXEC to FTP it for you each time.
2. Create the trace set that will produce a printout of all branch operations. You can start the trace prior to booting Linux or prior to running your daemon or application:

#CP TR BR PRINT

3. Redirect the output of the command to a user who will process the data:

#CP SP P <user>

4. When you are ready to process the data, end the trace and close the print file:

#CP TR END
#CP CLOSE P

5. Run the following EXEC to process the trace:

/* */
parse upper arg Option .
signal on SYNTAX
MaxSym = 0
Cache. = ''
if Option <> '' then
   Stage  = ''
else
   Stage  = '| nlocate /memset/',
            '| nlocate /memcpy/',
            '| nlocate /memcmp/',
            '| nlocate /update_wall_time/',
            '| nlocate /printk/',
            '| nlocate /strcpy/',
            '| nlocate /strncpy/',
            '| nlocate /strcmp/',
            '| nlocate /strncmp/',
            '| nlocate /strchr/',
            '| nlocate /strlen/',
            '| nlocate /ExternalException/',
            '| nlocate /External+/',
            '| nlocate /do_timer/',
            '| nlocate /vsprintf/',
            '| nlocate /set_bit/',
            '| nlocate /free_pages/',
            '| nlocate /mem_init/'
'PIPE (name READ_MAP)',
   '| ftp ftp://<user>:<password>@<host.domain>/linux/System.map binary',
   '| xlate from 437 to 1047',
   '| deblock c',
   '| drop 1',
   '| strip',
   '| locate 1',
   '| nfind U' ||,
   '| spec w1 x2c 1 w1-* nw',
   '| stem Map.'
'PIPE (name READ_TRACE end ?)',
   '| reader',
   '| mctoasa',
   '| spec 2-* 1',
   '| a: locate /BASR/',
   '| b: faninany',
   '| spec w2-4 1 w6 nw',
   '| stem Trace.',
   '? a:',
   '| c: locate /LPSW/',
   '| b:',
   '? c:',
   '| locate / 07FE /',
   '| b:'
do I_Trace = 1 to Trace.0
   if ((I_Trace // 5000) = 0) then
      say '...'I_Trace
   parse var Trace.I_Trace From Branch BrType To
   if (Cache.From = '') then
   do
      FromSym = GET_ADDR(From,'F')
      Cache.From = FromSym
   end
   else
      FromSym = Cache.From
   if (Cache.To = '') then
   do
      ToSym   = GET_ADDR(To,'T')
      Cache.To = ToSym
   end
   else
      ToSym = Cache.To
   select
      when BrType = '0D1E' then
           Branch = '<---'
      when BrType = '0DEF' then
           Branch = '--->'
      when BrType = '07FE' then
           Branch = '<---'
      otherwise
           Branch = '--->'
   end
   Flow.I_Trace = FromSym LEFT(Branch,8) ToSym
end
Flow.0 = Trace.0
'PIPE (name WRITE_FLOW end ?)',
   '| stem Flow.',
   '| spec w1 1.'MaxSym 'w2-* nw',
   Stage,
   '| spec 1-* 5',
   '| > LINUX FLOW A'
exit
GET_ADDR:
   parse arg VAddr,Type
   parse var VAddr Addr"'"
   XAddr = X2C(Addr)
   Target    = Map.0
   NewTarget = 1
   Disp      = Map.0 % 2
   LastLow   = Target
   do forever
      parse var Map.Target 1 XSym 5 . . Symbol
      if XAddr = XSym then
      do
         LastLow = Target
         leave
      end
      if XAddr < XSym then
      do
         NewTarget = Target - Disp
         if (LastLow = NewTarget) then
            leave
         LastLow   = NewTarget
      end
      else
         NewTarget = Target + Disp
      Disp = Disp % 2
      if (Disp < 1) then
         Disp = 1
      Target = NewTarget
   end
   parse var Map.LastLow 1 XSym 5 . . Symbol
   Disp = X2D(Addr) - C2D(XSym)
   if Disp <> 0 then
      Symbol = Symbol'+'D2X(Disp)
   if ((LENGTH(Symbol) > MaxSym) & (Type = 'F')) then
      MaxSym = LENGTH(Symbol)
return Symbol
SYNTAX:
  say 'Error:' ERRORTEXT(Rc) 'at line' Sigl
  say SOURCELINE(Sigl)
  trace ?r; nop
exit -1

SMP Specific commands

• To find out how many CPUs you have:

• To find the cpu that the current cpu VM debugger commands are being directed at do:

Q CPU

• To change the current CPU VM debugger commands are being directed at do:

CPU <desired cpu no>

• On an SMP guest issue a command to all CPUs try prefixing the command with CPU ALL.

• To issue a command to a particular CPU, try CPU <cpu number>. For example:

CPU 01 TR I R 2000.3000 

• If you are running on a guest with several cpus and you have an I/O related problem and cannot follow the flow of code but you know it isn't SMP related. Then from the bash prompt issue:

Do a Q CPUS to find out how many CPUs you have; detach each one of them from your virtual machine except CPU 0 by issuing:

DETACH CPU 01-<number of CPUs in configuration> 

and re-boot Linux.

• TR SIGP
will trace inter processor signal processor instructions.  

Help for displaying ASCII text

Currently, text cannot be displayed in ASCII under the VM debugger (I love EBDIC too), I have written this little program which will convert a command line of hex digits to ASCII text which can be compiled under Linux and you can copy the hex digits from your x3270 terminal to your xterm if you are debugging from a Linux box.

This is quite useful when looking at a parameter passed in as a text string under VM (unless you are good at decoding ASCII in your head). For example, consider tracing an open syscall:

We have stopped at a breakpoint

000151B0' SVC 0A05 - 0001909A' CC 0 

Use D P SVC to check the SVC old PSW in the prefix area and see was it from user-space (for the layout of the prefix area consult page18 of the ESA 390 Reference Summary if you have it available).

SVC 0005  20 OLD 070C2000 800151B2   60  NEW 04080000 8001909A

The problem state bit wasn't set and it's also too early in the boot sequence for it to be a user-space SVC if it was we would have to temporarily switch the PSW to user space addressing so we could get at the first parameter of the open in gpr2. To display the parameter:

Now copy the text till the first 00 hex (which is the end of the string) to an xterm and do hex2ascii on it:

hex2ascii 2F646576 2F636F6E 736F6C65 00 

The resulting output is:

Decoded Hex:=/ d e v / c o n s o l e 0x00 

We were opening the console device.

You can compile the code below yourself for practice :-),

/* 
* hex2ascii.c 
* a useful little tool for converting a hexadecimal command line to ascii 
* 
* Author(s): Denis Joseph Barrow (djbarrow@de.ibm.com,barrow_dj@yahoo.com) 
* (C) 2000 IBM Deutschland Entwicklung GmbH, IBM Corporation. 
*/ 
#include <stdio.h 
int main(int argc,char *argv[]) 
{ 
 int cnt1,cnt2,len,toggle=0; 
 int startcnt=1; 
 unsigned char c,hex; 
 if(argc1&&(strcmp(argv[1],"-a")==0)) 
    startcnt=2; 
 printf("Decoded Hex:="); 
 for(cnt1=startcnt;cnt1<argc;cnt1++) 
 { 
    len=strlen(argv[cnt1]); 
    for(cnt2=0;cnt2<len;cnt2++) 
    { 
       c=argv[cnt1][cnt2]; 
       if(c='0'&&c<='9') 
          c=c-'0'; 
       if(c='A'&&c<='F') 
          c=c-'A'+10; 
       if(c='a'&&c<='F') 
          c=c-'a'+10; 
       switch(toggle) 
       { 
          case 0: 
             hex=c<<4;
             toggle=1; 
             break; 
          case 1: 
             hex+=c; 
             if(hex<32||hex127) 
             { 
                if(startcnt==1) 
                   printf("0x%02X ",(int)hex); 
                else 
                   printf("."); 
             } 
             else 
             { 
                printf("%c",hex);
                if(startcnt==1)
                   printf(" "); 
             }
             toggle=0;
             break; 
        }
    } 
 } 
 printf("\n");
} 

Alternatively, the following CMS PIPELINE will achieve the same thing:

/* */
parse arg XString
'PIPE (name E2A)',
   '| var Xstring',
   '| change / //',		/* Remove any blanks within string */
   '| spec 1-* x2c 1',    	/* Convert graphic hex to binary   */
   '| xlate from 437 to 1047',    /* Choose the code-page you prefer */
   '| cons' 

Stack tracing under VM

Here are the tricks I use 9 out of 10 times it works pretty well.

When your backchain reaches a dead end

This can happen when an exception happens in the kernel and the kernel is entered twice if you reach the NULL pointer at the end of the back chain you should be able to sniff further back if you follow the following tricks.

1. A kernel address should be easy to recognise since it is in primary space and the problem state bit isn't set and also The Hi bit of the address is set.
2. Another backchain should also be easy to recognise since it is an address pointing to another address approximately 100 bytes or 0x70 hex behind the current stackpointer.

Here is some practice.

• boot the kernel and hit PA1 at some random time

• d g to display the gprs, this should display something like

GPR 0  = 00000001 00156018 0014359C 00000000 
GPR 4  = 00000001 001B8888 000003E0 00000000
GPR 8  = 00100080 00100084 00000000 000FE000
GPR 12 = 00010400 8001B2DC 8001B36A 000FFED8 

• Note that GPR14 is a return address but as we are real men we are going to trace the stack. Display 0x40 bytes after the stack pointer:

D 0.40;BASEF
V000FFED8 000FFF38 8001B838 80014C8E 000FFF38
V000FFEE8 00000000 00000000 000003E0 00000000
V000FFEF8 00100080 00100084 00000000 000FE000
V000FFF08 00010400 8001B2DC 8001B36A 000FFED8   

• Ah now look at what is in sp+56 (sp+0x38) this is 8001B36A our saved r14 if you look above at our stackframe and also agrees with GPR14. Now backchain:

d 000FFF38.40 

• We now are taking the contents of SP to get our first backchain.

V000FFF38 000FFFA0 00000000 00014995 00147094
V000FFF48 00147090 001470A0 000003E0 00000000
V000FFF58 00100080 00100084 00000000 001BF1D0 
V000FFF68 00010400 800149BA 80014CA6 000FFF38 

This displays a 2nd return address of 80014CA6

• Now do d 000FFFA0.40 for our 3rd backchain

V000FFFA0 04B52002 0001107F 00000000 00000000 
V000FFFB0 00000000 00000000 FF000000 0001107F 
V000FFFC0 00000000 00000000 00000000 00000000
V000FFFD0 00010400 80010802 8001085A 000FFFA0 

Our 3rd return address is 8001085A. As the 04B52002 looks suspiciously like rubbish it is fair to assume that the kernel entry routines for the sake of optimization do not set up a backchain.

• Now look at System.map to see if the addresses make any sense:

Outputs among other things:

So 8001B36A is cpu_idle+0x66 (quiet the cpu is asleep, don't wake it!)

• Next:

grep -i 00014 System.map 

Produces among other things

00014a78 T start_kernel 

So 0014CA6 is start_kernel+0x22e.

• Then:

grep -i 00108 System.map 

This produces:

00010800 T _stext 

So 8001085A is _stext+0x5a

• Congratulations you've performed your first backchain!  

S390 IO Overview

I am not going to give a course in 390 IO architecture as this would take me quite a while and I'm no expert. Instead I'll give a 390 IO architecture summary for Dummies if you have the ESA principles of operation available read this instead. If nothing else you may find a few useful keywords in here and be able to use them on a web search engine like Altavista to find more useful information.

Unlike other bus architectures modern 390 systems do their IO using mostly fibre optics and devices such as tapes and disks can be shared between several mainframes, also S390 can support up to 65536 devices while a high end PC based system might be choking with around 64. Here is some of the common IO terminology.

General Concepts

Input Output Processors (IOP's) are responsible for communicating between the mainframe CPU's and the channel and relieve the mainframe CPU's from the burden of communicating with IO devices directly, this allows the CPU's to concentrate on data processing.

IOP's can use one or more links (known as channel paths) to talk to each IO device. It first checks for path availability and chooses an available one, then starts (and sometimes terminates IO). There are two types of channel path ESCON and the Parallel IO interface.

IO devices are attached to control units. Control units provide the logic to interface the channel paths and channel path IO protocols to the IO devices. They can be integrated with the devices or housed separately and often talk to several similar devices (typical examples would be RAID controllers or a control unit which connects to 1000 3270 terminals).

 
    +---------------------------------------------------------------+ 
    | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+     |
    | | CPU | | CPU | | CPU | | CPU | | Main     | | Expanded |     |
    | |     | |     | |     | |     | | Memory   | | Storage  |     |
    | +-----+ +-----+ +-----+ +-----+ +----------+ +----------+     |
    |---------------------------------------------------------------+
    |        IOP       |      IOP        |      IOP                 |
    |---------------------------------------------------------------| 
    | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | C | 
    +---------------------------------------------------------------+ 
         ||                                              || 
         ||  Bus & Tag Channel Path                      || ESCON 
         ||  ======================                      || Channel 
         ||  ||                  ||                      || Path 
    +----------+               +----------+         +----------+ 
    |          |               |          |         |          | 
    |    CU    |               |    CU    |         |    CU    | 
    |          |               |          |         |          | 
    +----------+               +----------+         +----------+ 
        |     |                     |                |       | 
+----------+ +----------+      +----------+   +----------+ +----------+ 
|I/O Device| |I/O Device|      |I/O Device|   |I/O Device| |I/O Device| 
+----------+ +----------+      +----------+   +----------+ +----------+ 
CPU = Central Processing Unit 
C = Channel 
IOP = IP Processor 
CU = Control Unit

The 390 IO systems come in 2 flavors the current 390 machines support both the older 360 and 370 interface, sometimes called the parallel I/O interface, sometimes called Bus-and Tag and sometimes Original Equipment Manufacturers Interface (OEMI).

This byte wide parallel channel path/bus has parity and data on the "Bus" cable and control lines on the "Tag" cable. These can operate in byte multiplex mode for sharing between several slow devices or burst mode and monopolize the channel for the whole burst. Upto 256 devices can be addressed on one of these ables. These cables are about one inch in diameter. The maximum unextended length supported by these cables is 125 Meters but this can be extended up to 2km with a fiber optic channel extended such as a 3044. The maximum burst speed supported is 4.5 megabytes per second however some really old processors support only transfer rates of 3.0, 2.0 and 1.0 MB/sec. One of these paths can be daisy chained to up to 8 control units.

IBM introduced ESCON, which is fiber optic based, in 1990. It uses 2 fiber optic cables and uses either LEDs or lasers for communication at a signaling rate of up to 200 megabits/sec. As 10 bits are transferred for every 8 bits of information this drops to 160 megabits/sec and to 18.6 Megabytes/sec once control information and CRC are added. ESCON only operates in burst mode.

ESCONs typical maximum cable length is 3km for the LED version and 20km for the laser version known as XDF (extended distance facility). This can be further extended by using an ESCON director which triples the above mentioned ranges. Unlike Bus and Tag as ESCON is serial. It uses a packet switching architecture. The standard Bus and Tag control protocol is however present within the packets. Up to 256 devices can be attached to each control unit that uses one of these interfaces.

A new fiber architecture has been released by IBM called FICON which improves on the performance of ESCON.

Common 390 Devices

• Network adapters typically OSA2, 3172's, 2116's and OSA-E gigabit Ethernet adapters
• Consoles 3270 and 3215 (a TTY emulated under Linux for a line mode console)
• DASD's direct access storage devices (otherwise known as hard disks)
• Tape Drives.
• CTC (Channel to Channel Adapters), ESCON or Parallel Cables used as a very high speed serial link between 2 machines. We use 2 cables under Linux to do a bi-directional serial link.

Debugging IO on S390 under VM

Now we are ready to go on with IO tracing commands under VM. First, a few self explanatory queries:

Q OSA 
Q CTC 
Q DISK 
Q DASD

Q OSA on my machine returns

OSA 7C08 ON OSA 7C08 SUBCHANNEL = 0000 
OSA 7C09 ON OSA 7C09 SUBCHANNEL = 0001
OSA 7C14 ON OSA 7C14 SUBCHANNEL = 0002 
OSA 7C15 ON OSA 7C15 SUBCHANNEL = 0003 

Now using the device numbers returned by this command we will trace the Io starting up on the first devices 7c08 and 7c09. In our simplest case we can trace the start subchannels:

TR SSCH 7C08-7C09 

Or the halt subchannels

 TR HSCH 7C08-7C09 

You can also trace MSCH's ,STSCH's, but I think you can guess the rest.

Ingo's favourite trick is tracing all the IO's and CCWS and spooling them into the reader of another VM guest so he can ftp the logfile back to his own machine. I'll do a small bit of this and give you a look at the output.

1. Spool stdout to VM guest Linux4's reader

SP PRT TO * 

2. Fill Linux4's reader with the trace

TR IO 7c08-7c09 INST INT CCW PRT RUN

3. Start up Linux
4. Finish the trace

TR END 

5. Close the reader

C PRT 

6. List reader contents

RDRLIST 

7. Copy it to Linux4's minidisk

RECEIVE / LOG TXT A1 ( replace 

8. FILELIST and press F11 to look at it. You should see something like:

00020942' SSCH B2334000 0048813C CC 0 SCH 0000 DEV 7C08 <
CPA 000FFDF0 PARM 00E2C9C4 KEY 0 FPI C0 LPM 80 
CCW 000FFDF0 E4200100 00487FE8 0000 E4240100 ........ 
IDAL 43D8AFE8 
IDAL 0FB76000
00020B0A' I/O DEV 7C08 - 000197BC' SCH 0000 PARM 00E2C9C4 
00021628' TSCH B2354000 00488164 CC 0 SCH 0000 DEV 7C08 
CCWA 000FFDF8 DEV STS 0C SCH STS 00 CNT 00EC 
KEY 0 FPI C0 CC 0 CTLS 4007
00022238' STSCH B2344000 00488108 CC 0 SCH 0000 DEV 7C08 

gdb on S390

Note, compiling for debugging with gdb works better without optimization (see Compiling programs for debugging).

Invocation

gdb <victim program <optional corefile>

Online help

help: gives help on commands. For example:

help 

help display 

Note gdb's online help is very good and we advise you to use it.

Assembly

• info registers
: displays registers other than floating point.
• info all-registers
: displays floating points as well.
• disassemble
: disassembles. For example,

Viewing and modifying variables

• print
or p: displays variable or register. For example:

• display
: prints variable or register each time program stops. For example:

• undisplay
: undo's display's

• info breakpoints
: shows all current breakpoints

• info stack
: shows stack back trace ( if this doesn't work too well, I'll show you the stacktrace by hand below ).

• info locals
: displays local variables.

• info args
: display current procedure arguments.

• set args
: will set argc and argv each time the victim program is invoked.

set <variable=value>

set argc=100 

set $pc=0 

Modifying execution

• step
: steps n lines of sourcecode step with no value steps 1 line.
• next
: like step except this will not step into subroutines
• stepi
: steps a single machine code instruction.
• nexti
: steps a single machine code instruction but will not step into subroutines.
• finish
: will run until exit of the current routine
• run
: (re)starts a program
• cont
: continues a program
• quit
: exits gdb.

breakpoints

• break
: sets a breakpoint. For example:

break main 
break *$pc 
break *0x400618 

• rbr
: Set a breakpoint for all functions matching REGEXP. This is really useful for large programs. For example:

rbr 390 

Will set a breakpoint with all functions with 390 in their name.

• info breakpoints
: lists all breakpoints

• delete
: delete breakpoint by number or delete them all. For example:

• watch
: This will set a watchpoint (usually hardware assisted). This will watch a variable till it changes. For example:

• info watchpoints
: Display currently active watchpoints

• condition
: Specify breakpoint number N to break only if COND is true. Usage is condition N COND, where N is an integer and COND is an expression to be evaluated whenever breakpoint N is reached. This is a particularly another useful command.

User defined functions/macros

• define: Define a macro/function. This is very very useful, simple and powerful. Usage: define <name> <list of commands> end. Examples which you should consider putting into .gdbinit in your home directory:

  define d 
  stepi 
  disassemble $pc $pc+10 
  end 
  define e 
  nexti 
  disassemble $pc $pc+10 
  end   

  Other hard to classify stuff

• signal n
: sends the victim program a signal. For example, signal 3 will send a SIGQUIT.

• info signals
: what gdb does when the victim receives certain signals.
• list
: Examples:

list 		lists current function source 

list 1,10 	list first 10 lines of current file. 

list test.c:1,10   

• directory: Adds directories to be searched for source if gdb cannot find the source. Note it is a bit sensitive about slashes. For example, to add the root of the filesystem to the searchpath do:

directory //   

• call <function>
: This calls a function in the victim program, this is pretty powerful. For example:

(gdb) call printf("hello world") 

Outputs:

$1 = 11 

You might now be thinking that the line above didn't work, something extra had to be done.

(gdb) call fflush(stdout)
hello world$2 = 0

As an aside the debugger also calls malloc and free under the hood to make space for the "hello world" string.  

Hints

1. Command completion works just like bash (if you are a bad typist like me this really helps). For example, type br <TAB>
2. If you have a debugging problem that takes a few steps to recreate put the steps into a file called .gdbinit in your current working directory if you have defined a few extra useful user defined commands put these in your home directory and they will be read each time gdb is launched.

A typical .gdbinit file might be:

break main 
run 
break runtime_exception 
cont   

Stack chaining in gdb by hand

• This is done using a the same trick described for VM:

This outputs:

$5 = 0x528f18 

(On my machine that is.)

• Now you can use:

info symbol (*($sp+56))&0x7fffffff 

You might see something like:

• Now do:

This outputs:

$6 = 0x528ed0

• Now do:

• Now do:

p/x (*(**$sp+56))&0x7fffffff 

and so on.

For more information

From your Linux box do:

man gdb or info gdb

Examining Core Dumps

A core dump is a file generated by the kernel ( if allowed ) which contains the registers, and all active pages of the program which has crashed. From this file gdb will allow you to look at the registers and stack trace and memory of the program as if it just crashed on your system. It is usually called core and created in the current working directory. This is very useful in that a customer can mail a core dump to a technical support department and the technical support department can reconstruct what happened. Provided the have an identical copy of this program with debugging symbols compiled in and the source base of this build is available.

In short it is far more useful than something like a crash log could ever hope to be.

In theory all that is missing to restart a core dumped program is a kernel patch which will do the following.

1. Make a new kernel task structure
2. Reload all the dumped pages back into the kernels memory management structures.
3. Do the required clock fixups
4. Get all files and network connections for the process back into an identical state (really difficult).
5. A few more difficult things I haven't thought of.

Why have I never seen one?

Probably because you haven't used the command:

to allow core dumps, now do:

ulimit -a 

to verify that the limit was accepted.

A sample core dump

• To create this I'm going to do:

ulimit -c unlimited 
gdb 

To launch gdb (my victim application).

• Now be bad and do the following from another telnet/xterm session to the same machine:

ps -aux | grep gdb 
kill -SIGSEGV <gdb's pid >

or alternatively use:

killall -SIGSEGV gdb

if you have the killall command.

• Now look at the core dump.

./gdb ./gdb core

The following will be displayed:

GNU gdb 4.18 
Copyright 1998 Free Software Foundation, Inc. 
GDB is free software, covered by the GNU General Public License, and you are 
welcome to change it and/or distribute copies of it under certain conditions. 
Type "show copying" to see the conditions. 
There is absolutely no warranty for GDB. Type "show warranty" for details. 
This GDB was configured as "s390-ibm-linux"... 
Core was generated by `./gdb'. 
Program terminated with signal 11, Segmentation fault. 
Reading symbols from /usr/lib/libncurses.so.4...done.
Reading symbols from /lib/libm.so.6...done. 
Reading symbols from /lib/libc.so.6...done.
Reading symbols from /lib/ld-linux.so.2...done.
#0 0x40126d1a in read () from /lib/libc.so.6
Setting up the environment for debugging gdb.
Breakpoint 1 at 0x4dc6f8: file utils.c, line 471.
Breakpoint 2 at 0x4d87a4: file top.c, line 2609.
(top-gdb) info stack 
#0 0x40126d1a in read () from /lib/libc.so.6 
#1 0x528f26 in rl_getc (stream=0x7ffffde8) at input.c:402 
#2 0x528ed0 in rl_read_key () at input.c:381 
#3 0x5167e6 in readline_internal_char () at readline.c:454 
#4 0x5168ee in readline_internal_charloop () at readline.c:507 
#5 0x51692c in readline_internal () at readline.c:521 
#6 0x5164fe in readline (prompt=0x7ffff810 "\177¢·¢xx\177¢·¢w¢X\177¢·¢xx¢@") 
 at readline.c:349 
#7 0x4d7a8a in command_line_input (prompt=0x564420 "(gdb) ", repeat=1,
 annotation_suffix=0x4d6b44 "prompt") at top.c:2091 
#8 0x4d6cf0 in command_loop () at top.c:1345
#9 0x4e25bc in main (argc=1, argv=0x7ffffdf4) at main.c:635   

LDD

This is a program which lists the shared libraries which a library needs. For example:

ldd ./gdb 

Outputs:

libncurses.so.4 = /usr/lib/libncurses.so.4 (0x40018000) 
libm.so.6 = /lib/libm.so.6 (0x4005e000) 
libc.so.6 = /lib/libc.so.6 (0x40084000)
/lib/ld-linux.so.2 = /lib/ld-linux.so.2 (0x40000000) 

Debugging modules

As modules are dynamically loaded into the kernel their address can be anywhere to get around this use the -m option with insmod to emit a load map which can be piped into a file if required.

The proc file system

This is a filesystem created by the kernel with files which are created on demand by the kernel if read, or can be used to modify kernel parameters. It is a powerful concept. For example:

cat /proc/sys/net/ipv4/ip_forward 

On my machine outputs:

0 

This tells me that ip_forwarding is not on. To switch it on I can do:

echo 1 /proc/sys/net/ipv4/ip_forward 

cat it again:

cat /proc/sys/net/ipv4/ip_forward 
1

That is, IP forwarding is now on.

There is a lot of useful info in here best found by going in and having a look around, so I'll take you through some entries I consider important.

• All the processes running on the machine have there own entry defined by /proc/<pid>.
• Let us have a look at the init process:

cd /proc/1 
cat cmdline 
init [2]

• Now look at the file descriptor entries:

cd /proc/1/fd 

This contains numerical entries of all the open files.

• The storage map for the process may be examined:

cat /proc/29/maps 
00400000-00478000 r-xp 00000000 5f:00 4103 /bin/bash 
00478000-0047e000 rw-p 00077000 5f:00 4103 /bin/bash
0047e000-00492000 rwxp 00000000 00:00 0 
40000000-40015000 r-xp 00000000 5f:00 14382 /lib/ld-2.1.2.so 
40015000-40016000 rw-p 00014000 5f:00 14382 /lib/ld-2.1.2.so
40016000-40017000 rwxp 00000000 00:00 0 
40017000-40018000 rw-p 00000000 00:00 0
40018000-4001b000 r-xp 00000000 5f:00 14435 /lib/libtermcap.so.2.0.8 
4001b000-4001c000 rw-p 00002000 5f:00 14435 /lib/libtermcap.so.2.0.8
4001c000-4010d000 r-xp 00000000 5f:00 14387 /lib/libc-2.1.2.so
4010d000-40111000 rw-p 000f0000 5f:00 14387 /lib/libc-2.1.2.so
40111000-40114000 rw-p 00000000 00:00 0
40114000-4011e000 r-xp 00000000 5f:00 14408 /lib/libnss_files-2.1.2.so 
4011e000-4011f000 rw-p 00009000 5f:00 14408 /lib/libnss_files-2.1.2.so 
7fffd000-80000000 rwxp ffffe000 00:00 0   

Showing us the shared libraries init uses where they are in memory and memory access permissions for each virtual memory area.

• /proc/1/cwd
is a softlink to the current working directory.
• /proc/1/root
is the root of the filesystem for this process.
• /proc/1/mem
is the current running process' memory, which you can read and write to like a file. strace uses this sometimes as it is a bit faster than the rather inefficient ptrace interface for peeking at DATA.

• Use cat /proc/1/status to display the status of the process:

Name: init 
State: S (sleeping) 
Pid: 1 
PPid: 0 
Uid: 0 0 0 0 
Gid: 0 0 0 0 
Groups: 
VmSize: 408 kB 
VmLck: 0 kB
VmRSS: 208 kB 
VmData: 24 kB 
VmStk: 8 kB 
VmExe: 368 kB 
VmLib: 0 kB 
SigPnd: 0000000000000000 
SigBlk: 0000000000000000
SigIgn: 7fffffffd7f0d8fc
SigCgt: 00000000280b2603
CapInh: 00000000fffffeff
CapPrm: 00000000ffffffff
CapEff: 00000000fffffeff 
User PSW: 070de000 80414146
task: 004b6000 tss: 004b62d8 ksp: 004b7ca8 pt_regs: 004b7f68
User GPRS: 
00000400 00000000 0000000b 7ffffa90 
00000000 00000000 00000000 0045d9f4
0045cafc 7ffffa90 7fffff18 0045cb08 
00010400 804039e8 80403af8 7ffff8b0
User ACRS: 
00000000 00000000 00000000 00000000 
00000001 00000000 00000000 00000000
00000000 00000000 00000000 00000000 
00000000 00000000 00000000 00000000
Kernel BackChain CallChain BackChain CallChain 
004b7ca8 8002bd0c 004b7d18 8002b92c 
004b7db8 8005cd50 004b7e38 8005d12a
004b7f08 80019114 

Showing among other things memory usage and status of some signals and the processes' registers from the kernel task_structure as well as a backchain which may be useful if a process crashes in the kernel for some unknown reason.

Starting points for debugging scripting languages etc.

bash/sh

Use the -x option to trace the script: bash -x <scriptname>. For example:

bash -x /usr/bin/bashbug 
+ MACHINE=i586 
+ OS=linux-gnu 
+ CC=gcc 
+ CFLAGS= -DPROGRAM='bash' -DHOSTTYPE='i586' -DOSTYPE='linux-gnu' -DMACHTYPE='i586-pc-linux-gnu' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./lib -O2 -pipe 
+ RELEASE=2.01 
+ PATCHLEVEL=1
+ RELSTATUS=release 
+ MACHTYPE=i586-pc-linux-gnu 

perl

Use the -d option of perl to invoke the interactive debugger: perl -d <scriptname>

Java

Use: jdb <filename> to invoke another fully interactive gdb style debugger. Type "?" for help when the debugger is invoked.

References

• Enterprise Systems Architecture Reference Summary
• Enterprise Systems Architecture Principles of Operation
• Hartmut Penners 390 stack frame sheet.
• IBM Mainframe Channel Attachment a technology brief from a CISCO webpage
• Various bits of man and info pages of Linux.
• Linux and GDB source.
• Various info and man pages.
• CMS Help on tracing commands.
   

Simplified Network Access

Configuring the LINUXTCP stack

LINUXTCP DTCPARMS

:nick.LINUXTCP :type.server
  :class.stack
  :owner.RVDHEIJ
  :diskwarn.

PROFILE TCPIP

device dev_700  CTC 700
link   link_linuxvm ctc 1 dev_700

home   192.168.16.254  link_linuxvm

port
  23 tcp intclien
internalclientparms
  port 23
endinternalclientparms

gateway
  192.168.16.2 =             link_linuxvm   defaultsize host
bsdroutingparms true
endbsdroutingparms
start dev_700

PROFILE EXEC

/* */
'VMLINK TCPIP 591 TCPIP 592 ( NONAMES'
'DEF CTC 700' ; 'COUPLE 700 TO LINUXVM 700'
'DEF CTC 701' ; 'COUPLE 701 TO LINUXVM 701'
'TCPRUN'

Verify the setup

Configure WARPHOLE

WARPHOLE EXEC

  \ tcplisten <i>from-port</i> user tcpip reuseaddr
  | spawner tcpclient linuxvm <i>target</i> user linuxtcp

/* WARPHOLE EXEC     TCP/IP Socket Relay                             */
/*                   Author: Rob van der Heij, 27 Oct 1999           */
ports = '23 25 80'
'PIPE(end \ name WARPHOLE.EXEC:3)',
   '\ var ports',
   '| split',
   '| spec ,tcplisten 100, 1 w1 n ,user tcpip reuseaddr, nw x4f nw',
          ',spawner tcpclient 192.168.16.2, nw w1 nw ,user linuxtcp, nw',
          'x4f nw ,cons, nw',
   '| join * ,\\,',
   '| var pipe1'
'PIPE (end \ name WARPHOLE.EXEC:7)' pipe1
return rc

SPAWNER REXX

/* SPAWNER  REXX     Spawn a subprocess like InetD does it           */
/*                   Author: Rob van der Heij, 27 Oct 1999           */
signal on error
arg process                           /* Process plus arguments      */
do forever
  'peekto req'                        /* Wait for connection request */
   'callpipe (end \ name SPAWNER.REXX:4)',
      '\ var req ',                   /* Take the request            */
      '| spec 65.16 1',               /* Requester address & port    */
      '| socka2ip',                   /* Make it readable            */
      '| spec w3 1 ,:, n w2 n',
      '| var whosthat'
  say whosthat 'calling' process
  'addpipe (end \ name SPAWNER.REXX:9)',
     '\ *.output: ',
     '| i: fanin ',
     '| tcpdata ',
     '|' process ,
     '| elastic ',
     '| i:'
  'callpipe( end \) *: | take',
     '| c: count lines',
     '| *:',                          /* Feed it one request         */
     '\ c: | var cnt'
  if cnt = 0 then 'readto'            /* Reject the connection       */
  'sever output'                      /* Cut it loose                */
end
error: return rc * ( rc ^= 12 )

Verify WarpHole

VM TCP/IP Netstat Level 320
Active Transmission Blocks
User Id  Conn  Local Socket            Foreign Socket            State
---- --  ----  ----- ------            ------- ------            -----
WARPHOLE 1860  *..10025                *..*                      Listen
WARPHOLE 1234  *..10023                *..*                      Listen
WARPHOLE 1363  *..10080                *..*                      Listen

Ported Packages

Common Issues

• Due to a bug with GCC, use
-O1 and not -O2 or -O3 when setting optimization on compiles.

This can be circumvented if you:

• Specify -fno-strict-aliasing to the CFLAGS, or,
• Upgrade to GCC 2.95.2

Also note that GCC will use all 16 floating point registers that are availableon IEEE-enabled machines. If your machine is not IEEE-capable then it will be emulated in software for you.

• For packages that use the configure script you will need to update the config.sub file to recognize s390-ibm-linux:
• Update

/etc/ld.so.conf with /usr/local/lib and run ldconfig -v.

Regina-0.08g

Results of Running REXXCPS.REXX

----- REXXCPS 2.1 -- Measuring REXX clauses/second -----
 REXX version is: REXX-Regina_0_08g 4.80 7 Aug 1999
       System is: UNIX
       Averaging: 10 measures of 30 iterations
     Performance: 113230 REXX clauses per second

----- REXXCPS 2.1 -- Measuring REXX clauses/second -----
 REXX version is: REXX370 4.01 20 Dec 1996
       System is: CMS
       Averaging: 10 measures of 30 iterations
     Performance: 74751 REXX clauses per second                                 

Results of Running Test Suite

========================== ../trip/arith.rexx ==========================
adding, multiplication, division, modulus, reminder, power, exponential,
numeric,
========================= ../trip/builtin.rexx =========================
REXX-Regina_0_08g 4.80
abbrev, abs, address, arg, b2x, bitand, bitor, bitxor, c2d, c2x, center,
changestr, compare, copies, countstr, datatype,
first= /1/
second=/0/
FuncTrip: error in  286:    call ch datatype('123.0000003','W'), '0'
first= /1/
second=/0/
FuncTrip: error in  287:    call ch datatype('123.0000004','W'), '0'
delstr, delword, digits,
d2c, d2x, errortext, form, format, fuzz, insert, lastpos, left, length, max,
min, overlay, pos, reverse, right, sign,
first= /-1/
second=/0/
FuncTrip: error in  645:    call ch sign('-0'),   0
sourceline,
first= /1158/
second=/1/
FuncTrip: error in  655:    call ch sourceline(),   lines(srcfile)   /* don't work for stdin */
space, strip, substr,
subword, symbol, translate, trunc, value, verify, word, wordindex,
wordlength, wordpos, words, xrange, x2c, x2d, justify,
========================== ../trip/files.rexx ==========================
simple, read, eof, zero_line, truncate, charin, faking, nofake, halfline,
crp/cwp, notready, holes, linecnt, flush, repos, persist/eof, stdio,
longlines, many, stick, close, empty_out,
========================== ../trip/funcs.rexx ==========================
arguments, comments, nullargs, labels, external, result, builtin, whole,
========================= ../trip/lexical.rexx =========================
========================= ../trip/limits.rexx ==========================
args, levels, comments, strings, hexstrings, symbols,
========================= ../trip/signal.rexx ==========================
halt, novalue, execution,  ...
Tripped in line 275: CALL ON HALT evalueate expression twice?.
   (triggered, return, rc/syntax,  ...
Tripped in line 353: SYNTAX condition was not triggered.
   (lineno1, lineno2,
in_delay,  ...
Tripped in line 436: Does not reset DELAY mode with CALL ON.
   (stacking,  ...
Tripped in line 458: did not trigger both ERROR and HALT.
   (result, sh: .//rc: cannot execute binary file
 ...
Tripped in line 476: ERROR not properly called.
   ( ...
Tripped in line 490: RESULT was set at return from CALL ON.
   (after_delay,  ...
Tripped in line 521: CALL ON HALT did not remove DELAYed mode.
   ( ...
Tripped in line 508: CALL ON HALT does not seem to work.
   (subrout, sh: .//rc: cannot execute binary file
sh: .//rc: cannot execute binary file
delay/sub, sh: .//rc: cannot execute binary file
sh: .//rc: cannot execute binary file
ctc/sub, sh: .//rc: cannot execute binary file
sh: .//rc: cannot execute binary file
on/off, sh: .//rc: cannot execute binary file
sigl/halt,  ...
Tripped in line 673: SIGL not set during CALL ON HALT.
   (symbols, drop,
========================== ../trip/stack.rexx ==========================
push, queue, queued, commands,
========================== ../trip/stats.rexx ==========================
general, address, arg, call, do-loops, drop, if, iterate, leave, parsing,
procedure, select, signal, trace,
========================== ../trip/time.rexx ===========================
date, time,
========================== ../trip/trip.rexx ===========================
comments, strings, hexstrings, binstrings, dataformats, continuations,
expr, random, arrays, lexical, labels, numbers, functions, recursing,
======================== ../trip/variable.rexx =========================
drop,  

THE-3.0

IMAP-4.7

Samba-2.0.6

OpenLDAP-1.2.8

Bind-8.2.2

Sendmail-8.9.3

• Create file BuildTools/OS/Linux.s390

#       @(#)Linux       8.4     (Berkeley)      3/22/1998
define(`confSTDIR', `/etc')
define(`confHFDIR', `/usr/lib')
define(`confDEPEND_TYPE', `CC-M')
define(`confMANROOT', `/usr/man/man')
define(`confCC', `gcc')
define(`confENVDEF', `-O1 -I/usr/local/include -fno-caller-saves')

• Create file cf/cf/linux.mc

divert(-1)
#
# Copyright (c) 1998 Sendmail, Inc.  All rights reserved.
# Copyright (c) 1983 Eric P. Allman.  All rights reserved.
# Copyright (c) 1988, 1993
#       The Regents of the University of California.  All rights reserved.
#
# By using this file, you agree to the terms and conditions set
# forth in the LICENSE file which can be found at the top level of
# the sendmail distribution.
#
#
divert(0)
VERSIONID(`@(#)linux.m4 8.7 (Berkeley) 5/19/1998')
define(`LOCAL_MAILER_PATH', /bin/mail.local)dnl
OSTYPE(linux)dnl
DOMAIN(generic)dnl
MAILER(local)dnl
MAILER(smtp)dnl

• Build
sendmail.cf using m4 m4/conf.m4 linux.mc >sendmail.cf
• Remove reference to
sendmail.cw in sendmail.cf (line 56)
• Issue the command
mkdir -p /var/spool/mqueue

Kaffe JVM

• Create conf/s390/linux subdirectory
• Create JIT/JNI source files:
• Locate .h and .c files containing "
BREAKPOINT" and rename to "JVM_BREAKPOINT" as it clashes with /usr/include/asm/s390-regs-common.h definition.