Η ιστορια του SMSQ απο τον Tony Tebby

[Wally]

Aπολαυστικο και ενδιαφερον (αν και ελαφρως παλιο) κειμενο στο οποιο ο δημιουργος του QDOS περιγραφει το ιστορικο δημιουργιας του "απογονου" SMSQ

Για τους κατοχους QL,απο ΕΔΩ μπορουν να το κατεβασουν ωστε να το δουνε στο Quill τους.

A BRIEF HISTORY of SMSQLe Grand Pressigny, FRANCE - Tony Tebby

"Incompatibilities & Improvements, Bugs & Features"

"Was It All a Terrible Mistake?" I knew right from the

start I should not do it, but so many people were asking for

SMS to come out from under its covers that, in early 1992, I

outlined a strategy (with Miracle Systems and Jochen Men) to

make a "QL compatible" version of SMS available.

The outline was quite simple. A QDOS compatible SMS kernel

existed (and had been in regular use since 1990). A complete

set of SuperBASIC procedures and functions existed. A

complete set of (extended QL style) device drivers for the

Atari ST series existed as well as "portable" disk, serial

and parallel port drives for other hardware. There was an

environment which supported QLiberator compiled programs.

All that was required was the core of a SuperBASIC

interpreter.

Following the success of the Gold Card, Miracle Systems were

looking for a legitimate operating system for their (as yet

undefined) forthcoming computer and Jochen Men needed an

operating system to legitimise the QL emulator for the Atari

ST series. It was clear that it would be best to provide a

version for the Gold Card as well but as Miracle Systems did

not want to get involved in selling software, the Gold Card

question was left.

I embarked on trying to find out what Sinclair's SuperBASIC

interpreter did. It was not difficult defining what it

should do, but Jan Jones had built it on the principles of a

GIGO (garbage in, garbage out) machine. With limited ROM

space, there was no room for real error checking so Jan just

tried to make sure that whatever rubbish SuperBASIC was

asked to deal with, it carried on and did something. The

"something" was not always obvious.

While WE (QJUMP, those wonderful people at QVIEW, Jochem

Merz, Albin Hessler etc.) would never deliberately exploit

"holes" in SuperBASIC, the same was not necessarily true of

other software suppliers or contributors to PD libraries. In

addition, even WE had been known to fall through a hole by

accident.

Compatibility, therefore, meant not only reproducing

SuperBASIC as it was intended, but reproducing as many

oddities as would be necessary to execute most QL software.

The two "compilers" for SuperBASIC programs provided a

starting point. The aim was to provide a BASIC interpreter

which would provide:

1. better compatibility with SuperBASIC than either

compiler,

2. execution at least half the speed of QLiberator,

3. an environment supporting both QLiberator and Turbo

compiled programs.

Clearly, as some software for the QL will not even work on

all QDOS ROMs, total compatibility with a particular QDOS

ROM can only be provided by copying that ROM code. Even

slight re-ordering of the QDOS ROM routines (as in the Thor

XVI) can cause considerable incompatibility. At this stage,

there was no intention of providing improvements. The

"Minerva Experience" had shown the extent to which the

slightest improvements could give rise to extensive

incompatibilities. So much for intentions!

The Birth of the QXL

"They should not have done it either!" In principle, the

implementation of SMSQ on a 680x0 processor embedded in a PC

should have been fairly straightforward. The Gold Card used

the IBM PC disk controller, the IDE hard disk interface is

not very different from the Hardcard used in the Miracle QL

Hard disk and the serial and parallel ports on the PC are

much the same as the serial and parallel ports that you find

anywhere except on the QL.

If the QXL had been designed as a card which plugged into a

standard AT motherboard (no processor or memory) and

provided with drivers to drive a standard keyboard

interface, a standard multi IO / floppy / IDE card and a

standard Super VGA card, it would have been simple.

To have done this, however, Miracle would have had to have

supplied the PC hardware (at a cost of about half the QXL

card for a single floppy / 110 Megabyte HD configuration

this would have seemed, to me, the obvious way to do it).

However, it would have meant that the machine would not have

been usable as a PC (sigh of relief) and would have ruled

out the use of portables and notebooks. In addition, second

hand PCs were widely available either free or for less than

$30.

As one of the most likely reasons for a second hand PC being

available at a very low price was that the hardware was not

quite a perfect clone, then there was no possibility of

having a version of SMSQ which would access the IO devices

directly. Also, if the QXL were to be put into a real

"working" PC, it would not only have to co-exist with its

host but would have to work through whatever low level

software (Stacker, Doublespace, Hypercache, Smartdrive etc.)

was used to improve the IO performance of the PC.

As a result, any direct access from the QXL to the PC was

ruled out and the QXL was to be hosted by a DOS program. A

logical decision, maybe, but, from the point of view of the

operating system software, it was a disaster.

Where is the problem? The PC comes complete with device

drivers for all of its peripherals all that needs to be done

is to pass data from the QXL to the PC device drivers (using

BIOS calls) and vice versa.

There are three problems with this.

1. The design of the PC BIOS does not take account of the

requirements of multitasking (it is, for example, impossible

to write something to disk while you are waiting for input

from a serial port).

2. While the accuracy of the reference manuals about the

QDOS operating system entry points left a lot to be desired,

the (in)accuracy of "reference" manuals for the PC gives a

whole new meaning to the word reference.

3. All the reference manuals (so far examined) for the PC

were written in the days of the PC and PC/XT. The PC BIOS

also dates from this period. The BIOS has been considerably,

and incoherently, changed while the manuals have been

superficially updated to take account of AT keyboards, hard

disk drives larger than 10 MByte and 3.5" and HD floppy

disks.

Take, for example, formatting a floppy disk on the PC. On

QDOS, it is a single operating system call. On the PC,

however, part Of the format operation is performed by the

application. The format routine for the Gold Card floppy

disk driver took a couple of hours to write. As the PC BIOS

does most of the work for you, it should be easy to write a

format routine on the PC.

There are a variety of DOS and BIOS calls to help you do

this: setting the device parameters, formatting and

verifying tracks etc. I have three reference manuals which

give example format programs.

I look at the first one, and think "this is very strange".

There seems to be no way in which you specify the density

and there seem to be no checks for whether the tracks have

been correctly formatted: it appears to be automatic. So I

try it. Fine, it makes all the right noises and tells me

that my DD disk has 1440 sectors. I try an HD disk. Fine, it

makes all the right noises but tells me that my HD disk has

only 1440 sectors. It try without a disk at all. Wonderful,

it formats much more quickly and tells me that I have 1440

sectors! - AU FREE!

I try the second program: this one checks the error return

from the "Format track and verify" call: it even allows me

to specify the density. I try it with a DD disk. Fine! it

tells me that there are 1440 sectors. I try it with an HD

disk. Fine! it tells me that there are 2880 sectors. I try

it without a disk: the format fails, excellent! I try it

with a bad DD disk telling that it is HD. Fine! it tells me

that it has 2880 sectors. Suspicious, I try to copy some

files to it using DOS. DOS refuses to recognise it. I try

the other two disks: neither is readable. Over to QDOS to

look at the disks: there are no sectors 1, 2 or 3 on any of

the tracks I look at. This would explain why the first

program did not bother to check the error return from

"Format track and verify": it does not verify!

On to the third program. This is similar to the other two,

but uses the old "INT 13h" BIOS calls rather than the more

powerful "DOS function 44h" calls (wonderful this DOS

terminology). This requires the use of a separate "verify

sectors" call. The verify sectors call seems to work: this

routine gets an error on every track: it is right, after

formatting, none of the tracks are readable on any type of

disk.

So, I try it myself. After a lot of experimenting with the

BIOS calls described in the various manuals, I am able to

write either DD or HD tracks and verify them. The only

problem is that I write too many sectors to a track: the

last sectors overwrite the first sectors on the track. After

a week of work, I can select the density and I can nearly

format a track.

Thinks! Microsoft can do it, I should be able to as well.

Now we start to see the problem: the DV3 floppy disk format

routine is less than 512 bytes. Microsoft's FORMAT program

is greater than 32 kbytes, almost the size of QDOS, all its

device drivers, SuperBASIC and all its procedures and

functions. No wonder all the manuals are wrong. It would

take about 200 pages just to list the MSDOS FORMAT program,

without trying to explain how it works!

Disassembling all of this program could take months. I

decide to trace the two paths of interest: 720k and 1440k

formats. It turns out that what you need to do is to poke

special values into various undocumented locations in low

memory. I note all the locations to be poked and set up a

format routine. Within this routine I poke all the required

locations, format the disk and restore all the locations to

their previous values.

Success, I can now format DD and HD floppy disks. The only

problem is that, despite my care in restoring all the poked

locations, after a format the PC refuses to recognise any

disk change until you hit the reset button. Two weeks have

passed and I still do not really know how to format a disk

using MSDOS.

Do I spend another 2 weeks finding out how to restore the

BIOS after a format operation?

Even if I succeed in making it work on my PC, will it work

on any other PC? How can anyone succeed in selling an

operating system where it takes two weeks to write a routine

using the operating system calls when it would only take two

hours to write the same routine accessing the hardware

directly?

Now we find the real cost of the QXL in development. Even

though the IO performance of the QXL is well below the

levels that it would be reasonable to expect, the

implementation of the QXL device drivers has cost between 10

and 20 times the cost of equivalent drivers for other 680x0

platforms. As a result, all the time that had been set aside

for the development of the SBASIC interpreter has been

swallowed up. For the first purchasers of the QXL, things

looked grim: poor IO performance, no SBASIC interpreter. Not

a very promising debut for SMSQ.

"You Take the High Road and I'11 Take the Low Road" The QXL

hardware strategy was not the only problem to be faced.

Miracle Systems, for reasons which should be obvious, wanted

the QXL to seem as much like the Trump Card and Gold Card as

possible, while Jochen Merz wanted an operating system which

was not just developed along the same line as the Atari QDOS

extended device drivers but one which went much further.

One man's improvement is another man's incompatibility. Now

we have the problem of developing (and maintaining) two

different variations of SMS: SMSQ, the basic QL-like version

and SMSQ/E, the extended version which is likely to diverge

evermore and more from SMSQ. Jochen Merz, therefore, decided

to supply SMSQ/E for the QXL as well as the Atari and Gold

Cards. Easy for him to decide: it was me that had to do the

work!)

More problems. It seems that computer users are not verysensitive about how much they have to pay for their

operating system. They are, however, very sensitive about

how much other users pay! Gold Card and Atari users do

not complain about having to pay for SMSQ/E (we told then it

would be necessary back in 1990), but they do object that

QXL users get a "free" version of SMSQ with their QXL. QXL

users do not seem to mind being asked to pay extra for

SMSQ/E (at the moment the differences are fairly small so it

is not usually worth "upgrading") but they do object that

Gold Card and Atari users are not being asked to pay more.

Even worse, there are some QXL users who seem to think that

they are being provided with a specially naff version of

SMSQ to oblige them to cough up a few extra pennies for an

upgrade!

Then to cap it all, Miracle Systems produce a Super Gold

Card which looks like a Gold Card, but turns out to be

rather different. We now have implementations of SMSQ on

three distinct hardware families, seven different hardware

variants, four different display types, with four different

68000 series processors, in three (and sometimes more)

languages. So far, there are more possible combinations than

there are users.

To avoid the necessity of producing a different version of

SMSQ/E for each user, SMSQ now uses a module structure which

has been borrowed from the Stella (Stella????) operating

system. This allows operating system modules to be selected

(or ignored) as the system is booted. In principle, a single

version of SMSQ could be delivered which would autoselect

the right modules for any hardware combination. In practice,

each hardware family (Gold Card, QXL and Atari ST/TT)

requires its own special loader, so that it is not worth

incorporating all the modules in each version.

Just as the number of users starts to take off, so does the

number of variations. Jochen Merz ships a copy of SMSQ/E

to a Gold Card user: the next day there is a message "SMSQ/E

does not work with the XXX keyboard". Not surprising, the

XXX keyboard uses a patched version of the JS ROM. The cure?

Another keyboard driver module for the Gold and Super Gold

Cards and another language module (the keyboard tables). The

net result is one new user and four new variations. Counting

variations is soon going to be like counting marbles in a

cookie jar.

"'Till Death us do Part" In the days of easy divorce for

reasons of mutual incompatibility, it is surprising to find

so many QL users wedded firmly to the old software packages

of the "use it at your own peril" style. Then I started the

evaluation of SBASIC, Miracle Systems sent me a bundle of

diskettes (about 10 Mbytes worth) of the type of software

that they thought might provide a test for the compatibility

of SBASIC.

I started looking at these disks on the Atari ST with JS and

the E level drivers. After resetting the Atari ST for the

tenth time without having found any software which even

started to work, I gave up and tried using a Gold Card.

After a day or so, I found two programs that could be

executed, played with and removed without crashing the

system. All the rest either crashed right at the start,

could not be made to do anything sensible, or could only be

removed by resetting (I began to understand why some users

have been asking for a quick reset). I have been told that a

lot more of the software would have worked if I had set the

memory size to 128 kbytes, but if you are going to reset you

machine to 128 kbytes use one program and then reset again,

there is no point at all in using SMSQ: you might as well

stick with QDOS on your old faithful QL. Seriously, does

anyone use this type of software anymore?

The first compatibility tests were very encouraging: all the

programs which crashed on a JS QL crashed with SMSQ. It

seemed that we had obtained better than 95% compatibility.

Moreover, one of the two programs that worked on the QL

worked with SMSQ: the figure was up to 98% compatibility.

"New Lamps for Old" One of the best ways of checking the

originality of software is to investigate the bugs. If two

items of software perform the same functions correctly, one

could be a copy of the other, or they could both be written

to the same specification. If, however, two items of

software exhibit the same bugs, it can be assumed that one

is copied from the other.

There are very few "first level" bugs (bugs which prevent

the system functioning correctly under "normal" conditions)

in QDOS. Because of the GIGO policy and the desire to limit

error checking to a minimum to maintain efficiency, there

are a much larger number of "second level" bugs (where the

system misbehaves when passed incorrect parameters or data

structures) and even more "holes" (where calling a system

function with deliberately incorrect parameters has a

reproducible if bizarre effect).

During testing of SMSQ and SBASIC, a large number of second

level bugs were uncovered in the JS ROMs. Many of these

showed up also in Minerva, none in SMSQ or SBASIC. From time

to time, users have uncovered a number of second level bugs

in SMSQ and SBASIC. All of these were entirely new and have

no connection with old QL ROM bugs: SMSQ and SBASIC are

entirely original!

Streamlining code has the effect of removing, altering or

introducing holes. It is not surprising, therefore, to find

that many of the holes that are exploited by some common

software, have either disappeared or been altered in Minerva

(giving rise to complaints of compatibility problems).

One such case is the xx.xxxxx SuperBASIC vector which is the

same in all QL ROMs. This vector is intended to be used with

data structures set up by the SuperBASIC interpreter. This

has three defined paths controlled by the value of one byte

(0, 2 or 3). Someone discovered that it could be made to

produce a bizarre effect if the passed a value of 1 in the

control byte. The resulting code fragment (which takes

longer than using a legitimate call) has been incorporated

into a utility, which has found its way into a large number

of programs for the QL. The streamlined Minerva code no

longer had this hole so a large amount of software stopped

working on Minerva. The Wizard did not manage to find the

real villain in the code, but succeeded in restoring

"compatibility" by setting a register to a value which it

would not normally have with SuperBASIC. This, in turn,

altered another hole and introduced different compatibility

problems.

In SBASIC, however, the hole never existed. Once the villain

code had been identified (a week's work) it was, therefore,

a simple matter of detecting the villain case and emulating

the hole directly. It was a waste of time and effort, and it

slows down SBASIC, but that's what it's about, isn't it?

The boundary between a bug and a hole is a very fine one and

if some software relies on a bug in the QL ROM do I need to

reproduce this bug? Unfortunately, the answer is sometimes

GRRRRRR YES.

Twice recently, I have received reports of "bugs" that have

appeared in the string handling in recent versions. These

"bugs" have been introduced into SBASIC to improve

compatibility with SuperBASIC (there are still three "bugs"

in SuperBASIC string handling which are not emulated in

SBASIC). Neither of these users was aware of that the bugs

existed in SuperBASIC: SBASIC is now being used where

SuperBASIC never went before.

"Whose Fault is it Anyway?" One rather tetchy letter

complained that SBASIC was very fragile by comparison with

SuperBASIC: using a well-known piece of commercial software:

"SBASIC crashed". This was misdirecting the blame. As the

software was invoked correctly by SBASIC and as it never

returned to SBASIC, SBASIC could hardly be to blame.

This well-known SuperBASIC extension started off by trying

to identify a fragment of the QL ROM, and, when it could not

find any QL ROM code (there is none in SMSQ), it jumped to a

completely arbitrary location. BANG. The cure: I re-wrote

the extension and incorporated it (with improvements) in

SBASIC.

In fact, SBASIC is more robust in this respect than

SuperBASIC: error trapping is much more thorough (and

forceful). If it had happened in a daughter SBASIC: it could

simply have been removed with no harmful side effects.

(Perhaps I should implement a keyboard "restart" for Job O.)

Another difference between QDOS and SMSQ which might give

the impression that SBASIC is more fragile than. SuperBASIC

is the default error handling: QDOS carries on but SMSQ

stops to allow a debugger to be started. If the job is

already being monitored by a debugger, there is no

difference. In the normal state, however, allowing a job

which has produced illegal instruction or address errors to

continue could easily result in widespread damage to the

system data structures, possibly resulting in the loss of

part or all the data on a hard disk. SMSQ is, therefore,

much safer, even if jobs appear to stop more often.

Many programs compiled with the current version of Turbo are

wonderful examples of this. At the start of these programs

we find some code which sets a location in memory to O.

Several times later on, the value in this location is moved

to register A2 and then there is the instruction to move the

contents of address O (A2) to 04, D2 is compared against the

new value in D4 and then there is a conditional branch.

MOVE.W (A2),D4

....... .......

SUB.W D2,D4

BLT.S ......

On the QL the MOVE sets D4 to 3 so the operation of the code

is dependent on whether D2 is greater or less than 3.

On Atari STs modified for the old QL emulator, the MOVE sets

D4 to 24,622. The behaviour of this Turbo code will,

therefore, be significantly different on these STs as the

operation now depends on whether D2 is greater or less than

24,622.

On unmodified STs with a patched version of the JS ROMs, the

MOVE will cause a "bus error" which QDOS ignores and so

execution will continue without changing the value in 04. 02

is, therefore, compared against an unknown value: this will

give yet different, and rather unpredictable, behaviour of

Turbo programs.

On unmodified STs with SMSQ, the MOVE is trapped and these

Turbo programs just stop. It is possible to set a special

"Turbo mode" (PROT_MEM O) which emulates the QL ROM access

by setting 04 to 3 and continuing. The behaviour of Turbo

programs on these STs is, therefore, the same as on a QL:

SMSQ is more compatible with the QL than QDOS!

"I See no Ships" While SMSQ with SBASIC marks a great

improvement in performance and capability over the old QDOS

ROMs, this has not been achieved without creating a few

problems. The original SMSQ and SBASIC had very few

intrinsic bugs, but many incompatibilities. Unfortunately,

it is one of the facts of life of computing that making any

changes to a existing software borders on vandalism. The

neat structure of the original conception begins to crumble

and soon each little "fix" risks introducing a host of new

problems. Fixing each one of these introduces more. It is a

tribute to Jan Jones original SuperBASIC conception that

during the early days of "active development" when new

features were being added every day, the ratio of changes to

bugs introduced was better than 10:1.

SBASIC is a much more complex piece of software which has

the disadvantage of being required to emulate all the

quirks of someone's first attempt at writing a BASIC

interpreter. It, therefore, starts off being not very neat

and it is prone to degenerate more quickly. SBASIC's ratio

of changes to bugs introduced is closer to 5:1 - small

enough to be convergent, but too large for comfort.

Fortunately, the bugs introduced from version 2.11

(experimental SBASIC) through version 2.25 (the first

"release" version) to version 2.42 (current at time of

writing) have usually been smaller than the ones they

replaced.

Although some compatibility problems with some hardware

variations remain to be resolved and there are one or two

program which still refuse to function with SMSQ, the

original aims for compatibility and performance have been

well exceeded and SMSQ is now establishing itself as a

living replacement for QDOS.

So, is SMSQ/E at last stable? The answer must be no. As more

and more people start using SMSQ/E there are more and morerequests for improvements (i.e. decreasing compatibility).

SMSQ/E has now passed the point of no return: there are more

requests for improved capability than for improved

compatibility. The current versions of SMSQ/E are at least

as reliable as any QL ROM version and are getting as close

to 100% compatibility as is possible while providing better

performance and more facilities.

SMSQ/E is a commercial product and as such needs to meet

users demands. If users require changes, and it is

commercially feasible to provide them, they will get them.

SMSQ/E cannot, however be developed in all directions at

once.

For software development, a 16 MHz 68000 based Mega STE (1

MIP with 2 Megabytes of memory running under SMSQ/E (my

"standard" configuration) is more than a match for a

"standard" 50 MIP 32 Mbyte workstation. A TT or a QXL on a

good 486 machine is more impressive still. The Gold and

Super Gold Cards have the same standards of raw performance,

but suffer from limited display capabilities, poor keyboard

and IO.

How much more is it reasonable to do with SMSQ which, by its

need for compatibility with a 10 year old computer, is

locked into a 10 year old design? Is there enough interest

in the type of operating system concepts pioneered by QDOS

to make it worthwhile producing a completely new system? The

future depends on your response.