1 (edited by Alcoholics Anonymous 2018-07-31 01:26:07)

Topic: Nail down esxdos api

Hi Miguel,

Is there any way we can finally formally document all of the esxdos api?  I have partial documentation you sent me ages ago and I have created a c api around that but there is much functionality not documented there.  In the meantime, nextos has adopted the esxdos api (because it's awesome smile ) and a lot of work went into reverse engineering dot commands and whatnot in order to document functions esxdos didn't formally document.  So what we have now is a documented nextos esxdos api that is probably correct for esxdos itself.  nextos has gone ahead and expanded the api by making available some services from the +3e api but that part can be ignored.

Writing documentation is annoying as hell (I avoid writing it myself) but in this case we have documentation written down and all that is needed is an edit to fix up what may be wrong.  Would it be possible to finally generate a similar doc as the nextos esxdos api for esxdos?

The nextos esxdos api is documented starting at page 20 in this but really the functions themselves are documented on page 27.
https://github.com/z88dk/techdocs/raw/m … os_api.pdf

2

Re: Nail down esxdos api

Hi Alvin,

With version 0.9.x the API will be considered "stable" and will be fully documented on release. The v0.8.x API was never considered stable, hence it was never publicly released. Everyone that asked for the API docs was alerted to this fact - unfortunately I was never contacted by the NextOS team (I think?) so I don't know if they are aware of this. This doesn't mean that all the work documenting/implementing it is wasted because most things will stay the same, but be prepared for some major changes.

Miguel

3

Re: Nail down esxdos api

Alcoholics Anonymous wrote:

a lot of work went into reverse engineering dot commands

Oh did it? Or did they just rip off UnoDOS 3 on the basis that they claimed it was just a copy of esxDOS? I don't suppose they've published the source code anywhere so I can check?

4

Re: Nail down esxdos api

aowen wrote:

Oh did it? Or did they just rip off UnoDOS 3 on the basis that they claimed it was just a copy of esxDOS? I don't suppose they've published the source code anywhere so I can check?

The +3e long predates either esxdos or unodos and that's what NextOS derives from.  The esxdos api is duplicated, not its implementation.  It's also expanded by introducing functionality from the +3e into the esxdos api, ie things like long filenames, memory resident drivers, memory allocation and so on.  The Next is a much more capable machine than a plain spectrum so proper support of the hardware necessitates adding a bunch of new things not relevant to the plain zx.

When esxdos dot commands were found not to work, they had to be disassembled to find out why.  Initially the implementation came from the documented portion of the api that Miguel supplied to some (with knowledge that these things may change).  I don't know if garry disassembled esxdos or looked at unodos to help define the existing api but it's not impossible.

5

Re: Nail down esxdos api

lordcoxis wrote:

With version 0.9.x the API will be considered "stable" and will be fully documented on release. The v0.8.x API was never considered stable, hence it was never publicly released. Everyone that asked for the API docs was alerted to this fact - unfortunately I was never contacted by the NextOS team (I think?) so I don't know if they are aware of this. This doesn't mean that all the work documenting/implementing it is wasted because most things will stay the same, but be prepared for some major changes.

Yes everyone's aware that the api was beta but at the same time, several years in the wild like that probably makes it one of the more stable spectrum apis out there big_smile

I hope you will take a look at the api as is in the next and consider if some of it fits into esxdos as it is evolving.  Speaking as a dev, I really would prefer to avoid having two slightly different apis to support.  However, at the same time, I've prepared for exactly that by keeping esxdos and nextos-esx separate inside z88dk so that if things divurge, asm or c programs can be made to work with either system with a simple recompile.

There are some things that I think should be brought into the esxdos api:

; ***************************************************************************
; * M_DOSVERSION ($88) *
; ***************************************************************************
; Get API version/mode information.
; Entry:
; -
; Exit:
; For esxDOS <= 0.8.6
; Fc=1, error
; A=14 ("no such device")
;
; For NextZXOS:
; Fc=0, success
; B='N',C='X' (NextZXOS signature)
; DE=NextZXOS version in BCD format: D=major, E=minor version
; eg for NextZXOS v1.94, DE=$0194
; HL=A=0 if running in NextZXOS mode (and zero flag is set)
; HL,A<>0 if running in 48K mode (and zero flag is reset)

This is how I am detecting whether a program is running under esxdos or nextos.  On the zx next both systems will be in use.  At the moment I check for carry flag set on return to determine if esxdos is running but you could add version information too like has been done for the nextos side with carry reset.

; ***************************************************************************
; * M_ERRH ($95) *
; ***************************************************************************
; Install error handler for dot command.
; Entry: HL=address of error handler within dot command
; (0 to change back to standard handler)
;
; NOTES:
; Can only be used from within a dot command.
; If any BASIC error occurs during a call to ROM3 (using RST $10 or RST $18)
; then your error handler will be entered with:
; DE=address that would have been returned to if the error had not
; occurred
; A=BASIC error code-1 (eg 8=9 STOP statement)

This one registers a function that is called if a basic error occurs before the dot is exited.  This is the only way to guarantee that file handles are properly cleaned up and on the next there is a larger problem of memory leaks if allocated pages are not returned to the system.  Without this ability, someone pressing space at the scroll prompt to break out of a dot command will cause file handles to leak.  Eventually the system will stop working because all the file handles are gone.

; ***************************************************************************
; * M_EXECCMD ($8f) *
; ***************************************************************************
; Execute a dot command.
; Entry:
; IX=address of commandline, excluding the leading "."
; terminated with $00 (or $0d, or ':')
; Exit (success):
; Fc=0
; Exit (failure):
; Fc=1
; A=error code (0 means user-defined error)
; HL=address of user-defined error message within dot command
;
; NOTES:
; The dot command name can be fully-pathed if desired. If just a name is
; provided, it is opened from the C:/BIN directory.
; eg: defm "hexdump afile.txt",0 ; runs c:/bin/hexdump
; defm "./mycommand.dot afile.txt",0 ; runs mycommand.dot in current
; ; directory
; If A=0, the dot command has provided its own error message but this is not
; normally accessible. It can be read using the M_GETERR hook.
; This hook cannot be used from within another dot command.

This one allows a dot command to be called by a program; it's not really essential to implement but in future I intend to generate libraries available as dot commands that can implement services outside of the spectrum's memory space.  Things like time functions, network functions, environment functions, etc, that can be made easily available to running programs at essentially no cost except the time to load a divmmc page.

6

Re: Nail down esxdos api

Alcoholics Anonymous wrote:

I hope you will take a look at the api as is in the next and consider if some of it fits into esxdos as it is evolving.  Speaking as a dev, I really would prefer to avoid having two slightly different apis to support.  However, at the same time, I've prepared for exactly that by keeping esxdos and nextos-esx separate inside z88dk so that if things divurge, asm or c programs can be made to work with either system with a simple recompile.

I will have a look at it when I have some free time, but there was a fundamental change in 0.9.x to make it more POSIX-y that means that it simply cannot be retro-compatible with 0.8.x API. And I do not want to handicap the final API just to make a few programs work without change (these can be recompiled).

Alcoholics Anonymous wrote:

There are some things that I think should be brought into the esxdos api:

; ***************************************************************************
; * M_DOSVERSION ($88) *
; ***************************************************************************

This is how I am detecting whether a program is running under esxdos or nextos.  On the zx next both systems will be in use.  At the moment I check for carry flag set on return to determine if esxdos is running but you could add version information too like has been done for the nextos side with carry reset.

This function exists since v0.8.0 but it seems I forgot to change the pointer to the routine (it's pointing to the 'ENOSYS' return code) and nobody noticed/warned me wink In 0.9.x there is another way to get the version, but I'll see if I can squeeze this in again.

Alcoholics Anonymous wrote:
; ***************************************************************************
; * M_ERRH ($95) *
; ***************************************************************************

This one registers a function that is called if a basic error occurs before the dot is exited.  This is the only way to guarantee that file handles are properly cleaned up and on the next there is a larger problem of memory leaks if allocated pages are not returned to the system.  Without this ability, someone pressing space at the scroll prompt to break out of a dot command will cause file handles to leak.  Eventually the system will stop working because all the file handles are gone.

In esxdos 0.8.x the file handles leak is automatically taken care of, but this seems like it could be used for other purposes as well.

Alcoholics Anonymous wrote:
; ***************************************************************************
; * M_EXECCMD ($8f) *
; ***************************************************************************

This one allows a dot command to be called by a program; it's not really essential to implement but in future I intend to generate libraries available as dot commands that can implement services outside of the spectrum's memory space.  Things like time functions, network functions, environment functions, etc, that can be made easily available to running programs at essentially no cost except the time to load a divmmc page.

Not sure what you mean about this one - it exists since esxdos v0.8.0 (it's how the NMI menu calls .snapload for example). Now the full-path thing - that was added more recently (0.8.6 maybe?), as requested by ub880d.

7

Re: Nail down esxdos api

Alcoholics Anonymous wrote:

The +3e long predates either esxDOS or UnoDOS and that's what NextOS derives from.

The +3e dates from 2000. divIDE came along fairly soon after that and esxDOS was in development soon after so it's not really true to say it "long predates" it. Also, given the Next team's "take other people's work without credit" approach to development, I find it unlikely that they would have bothered to reverse engineer the dot commands when I had already done it. They were certainly aware of the existence of UnoDOS 3 as they were using it as an excuse for ripping off ULAplus. If the source for the "NextOS" dot commands is ever published it will be easy to tell if that's the case. I don't expect them to keep that particular promise though.

The Next is a much more capable machine than a plain spectrum...

Depends on what you want to do with it. If you want to run all the existing software that's been written for the original Spectrum the Next falls short.

...so proper support of the hardware necessitates adding a bunch of new things not relevant to the plain ZX.

The job of an OS is to move data between disk and RAM. Adding lots of hardware proprietary stuff to the OS seems kind of pointless when people are just going to code for esxDOS. And frankly +3DOS is less capable than the old Sinclair microdrives when it comes to simple stuff like random file access from BASIC. I think that would be a much more useful edition to esxDOS than LFN support. I was going to add it to UnoDOS 3 but I've been kind of busy with Derby++ lately.

8 (edited by Alcoholics Anonymous 2018-08-18 07:24:05)

Re: Nail down esxdos api

aowen wrote:
Alcoholics Anonymous wrote:

The +3e long predates either esxDOS or UnoDOS and that's what NextOS derives from.

The +3e dates from 2000. divIDE came along fairly soon after that and esxDOS was in development soon after so it's not really true to say it "long predates" it.

The first private release of esxdos was in 2005.  The first public release was in 2009.  That's nine years.  It is very clear if you look at the api that the nextos version of the esxdos api is not using any esxdos code.  The api has been extended in several non-trivial respects by hooking into the already existing +3e codebase.

If the source for the "NextOS" dot commands is ever published it will be easy to tell if that's the case. I don't expect them to keep that particular promise though.

NextOS is written by Garry Lancaster and it belongs to him.  The dot commands were written by many individuals not part of the Next team. The Next team will not be releasing any of the source code nor do they have it.  Likewise they will not be releasing the source for esxdos because it's not theirs nor do they have it.

I find it unlikely that they would have bothered to reverse engineer the dot commands when I had already done it.

Honestly it's a five minute job to pass a dot command through a disassembler and identify the problematic esxdos calls.  Most of them are pretty simple.

If the source for the "NextOS" dot commands is ever published it will be easy to tell if that's the case. I don't expect them to keep that particular promise though.

Well I wrote five or six of them, all source at z88dk.org.  Garry probably a half dozen utility oriented things like defragging disk, installing drivers, parsing string variables for dot commands, creating swap space, etc where there is no existing equivalent afaik.  Another dozen or so for next-specific features.  I think out of 54 dot commands in my current bin about 30+ are new.  The rest of the ones I have are from the standard esxdos distribution.

The Next is a much more capable machine than a plain spectrum...

Depends on what you want to do with it. If you want to run all the existing software that's been written for the original Spectrum the Next falls short.

Kind of like every spectrum model big_smile  The only way to be 100% compatible is to have multiple hw implementations like you can do on an uno or a next.

The job of an OS is to move data between disk and RAM.

The OS has to manage shared resources.  On a 48k spectrum that's disk.  On the next that's disk, memory, network, resident drivers at minimum.

Adding lots of hardware proprietary stuff to the OS seems kind of pointless when people are just going to code for esxDOS. And frankly +3DOS is less capable than the old Sinclair microdrives when it comes to simple stuff like random file access from BASIC.

The current direction is to write for the next for maximum capability and to offer a reduced version for the standard spectrum via esxdos where it makes sense.  For programs that could be written for the standard spectrum, the standard esxdos api covers most things.  So, eg, dot commands written for the next (will) test if they are running on nextos or error out if not.  On the next, dot commands can be any size even a megabyte and not interfere with basic.  Basic programs can also be a megabyte.  This is made possible because the operating system is managing memory.

As for the rest, I think there's a lack of familiarity with NextOS smile  It is not a re-badge of the +3 or +3DOS.

9

Re: Nail down esxdos api

Alcoholics Anonymous wrote:

The first private release of esxdos was in 2005.  The first public release was in 2009.

I could have sworn it was older than that. I accept your points about the development of +3DOS to support the esxDOS 0.8.5 API. I know Garry's not a plagiarist. You'll have to forgive my cynicism about the Next in general, the main people driving it have an attitude like Microsoft in the bad old days.

The Next team will not be releasing any of the source code nor do they have it.

So much for being an open system. Just one of many broken promises.

If you want to run all the existing software that's been written for the original Spectrum the Next falls short.

Kind of like every spectrum model big_smile  The only way to be 100% compatible is to have multiple hw implementations like you can do on an uno or a next.

I didn't say 100% compatible, I said run all the existing software. The prototype ZX Spectrum SE could do that 18 years ago without needing clunky FPGA configuration. The Uno can do it now because its ZX core is basically a Spectrum SE.


The OS has to manage shared resources.  On a 48k spectrum that's disk.  On the next that's disk, memory, network, resident drivers at minimum.

Fair point. Wouldn't want the user having to interact directly with the next's awful memory paging system.

As for the rest, I think there's a lack of familiarity with NextOS smile  It is not a re-badge of the +3 or +3DOS.

This is the Next: https://en.wikipedia.org/wiki/NeXT_Computer
This is NextOS: https://www.nextiva.com/blog/its-here-n … today.html

The current TBBlue board is a clone of the ZX Uno with a bigger FPGA.
The ZX core that runs on the TBBlue board that offers access to things like the rip off of ULAplus is based on the +3.

As far as the OS is concerned, it's a development of the +3e. And the +3e is a development of +3DOS. Garry has said as much himself. He hasn't written a new OS from scratch. He's added an esxDOS API compatibility layer and FAT compatibility to +3DOS.

10

Re: Nail down esxdos api

aowen wrote:

The Next team will not be releasing any of the source code nor do they have it.

So much for being an open system. Just one of many broken promises.

I don't see how that's a broken promise.  They can't release source code to stuff they didn't make.  No one expected the esxdos source to be published along with the next way back when because they don't have it nor do they own it.  Likewise with nextzxos.  Both these systems will be used on the next.  The portion made by the next team itself will be open.

If you want to run all the existing software that's been written for the original Spectrum the Next falls short.

I didn't say 100% compatible, I said run all the existing software. The prototype ZX Spectrum SE could do that 18 years ago without needing clunky FPGA configuration. The Uno can do it now because its ZX core is basically a Spectrum SE.

There are fundamental differences in hw and sw that prevent all existing sw from running on any static machine.  You can get close but never 100%.

The next (and uno) are both capable of behaving identically with specific hw via change in core.  The next machine is imagined as a successor to the last spectrum model - the +3 - and is configured as such.  However the next core can also behave with different timings 48k,128k,pentagon by setting a parameter on boot and that's the origin of the different boot options.  The default is a next machine with +3 timing and nextzxos.  But there are other selectable options using the next core like 48k with standard rom and 48k timings, 128k with standard rom and 128k timings.  These latter types use esxdos.  In future I expect you'd see many cores like on the uno and some may be plain 48k/128k cores as well.  The uno people ported their spectrum core to the next as a test exercise last year.

Wouldn't want the user having to interact directly with the next's awful memory paging system.

The paging system is good.

Having the operating system manage allocation of 8k pages means you can write drivers that are resident in memory, you can make basic programs that are a megabyte long, you can write dot commands of any size.

The current TBBlue board is a clone of the ZX Uno with a bigger FPGA.
The ZX core that runs on the TBBlue board that offers access to things like the rip off of ULAplus is based on the +3.

There are many new things in the core that are not in the uno.
The +3 is one machine type the core implements and it happens to be the machine type chosen for the default next machine.

As far as the OS is concerned, it's a development of the +3e. And the +3e is a development of +3DOS. Garry has said as much himself. He hasn't written a new OS from scratch. He's added an esxDOS API compatibility layer and FAT compatibility to +3DOS.

It's an extension of the +3e but it is a significant one.  It's not just FAT and an esxdos api layer.

Anyway I'm not sure why you need to talk down a project.  The next is clearly breaking new ground for zx machines in several respects and that's what the backers wanted.

11

Re: Nail down esxdos api

Alcoholics Anonymous wrote:

I don't see how that's a broken promise.

The fact that they were never going to be able to keep the promise doesn't mean that it's not broken.

There are fundamental differences in hw and sw that prevent all existing sw from running on any static machine.

Aside from +3 specific software, there are no titles that do not have at least one release that runs on the Spectrum SE prototype from 18 years ago. And it can run most of the TS2068 software too.

In future I expect you'd see many cores like on the uno

I don't think the Next has a future.

There are many new things in the core that are not in the uno.

So you're not disagreeing with me that at the hardware level it's an Uno with a bigger FPGA then. Leaving aside the firmware that runs on the Z80, so far as I can tell the only things in the core that didn't first appear in the Uno, first appeared in the TSConf core for the ZX Evo.

I'm not sure why you need to talk down a project.

When people behave the way the people behind the Next have behaved, someone should hold them to account. But because they have the backing of the worst of the scene trolls, most people are afraid to speak out. Antonio Villena has alleged that the Next team has stolen the Spectrum core from the ZX Uno. If there was any Uno code in the core then the Next team would have been legally obliged to publish it as soon as the developer boards were delivered. So we must assume there is no Uno code in the core. But the decent thing to do would have been to publish the core. If the intention is to publish the source when the cased versions ship there is literally no good reason to keep the source a secret. Especially when applications are already being developed against it. A project like the Next ought to be able to withstand a little constructive criticism. The fact that anyone who speaks out critically in the official channels is banned is telling. People may think they are getting a new Spectrum. I submit that the Next has much more in common with the TK95.