IO

The IO Subsystem

The Nebula IO subsystem is a huge step forward from the Nebula1 and Nebula2 IO systems. The main design goals of the new IO subsystem are:

• use more standard mechanisms, like URIs to identify resource locations, and MIME types to identify data formats
• a flexible stream model, it shouldn't matter whether data comes from a file, from a memory buffer, an HTTP connection or somewhere else
• reading and writing data from and to a stream in different data formats should be more orthogonal, for instance it shouldn't matter if XML-formatted data is read from memory, from a file, from a network connection or from anywhere else
• extensibility, new stream and reader/writer classes can be registered with the IO subsystem at runtime
• portability without performance compromises, the entire IO subsystem must be able to use platform-specific IO functions under the hood instead of relying to CLib functions like fopen() for portability, which may come with an additional performance or memory overhead compared to the platform specific IO functions

The main concepts of the Nebula IO subsystem are:

• A central IO::Console object for all text input and output with attachable console handlers. It is guaranteed that all Nebula text output goes through the console as the one centralized in/out channel. Specialized console handlers can be used to handle text output in a special way (for instance writing the output to stdout, an ingame console, a log file or a network connection).
• Assigns as path aliases. The general functionality is the same as in Nebula1 and Nebula2, or the original AmigaOS assigns which inspired Nebula's assign mechanism. A new feature of Nebula assigns is that they can be aliases for complete URI's. For instance, the assign "textures:" could be defined as "http://www.radonlabs.de/textures", so that the shortcut resource path "textures:mytexture.dds" would be resolve to the absolute location "http://www.radonlabs.de/textures/mytexture.dds"
• Streams as basic in/out data channel. Streams are the replacement for Nebula2's nFile objects. Streams offer the same basic API with Open()/Close()/Read()/Write(), but may hide completely different transport or storage channels behind their common interface. Examples of stream classes are IO::FileStream, IO::MemoryStream, or Http::HttpStream
• Stream readers and writers are attached to streams and generally implement easy-to-use interfaces to read and write different data formats. For instance one could attach an IO::XmlReader to a IO::FileStream to read XML-formatted data from a filesystem file, or attach it to an IO::HttpStream to read XML-formatted data from a HTTP connection.

A good example to show the power of the Nebula in/out system is the following code fragment:

IO::FileServer::Instance()->CopyFile("https://en.wikipedia.org/wiki/Radon_Labs#/media/File:Radonlabslogo.svg", "temp:logo.svg");

This single line of code copies a file from a HTTP server into the current user's temp directory. With only a few lines more you could create a stream object pointing to a HTML file on a HTTP server, attach an XML reader to the stream, and parse the content of the HTML file without any intermediate storage file.

Standard Assigns

Nebula initializes the following standard assigns:

• home: Points to the application directory, in a US Windows installation, this is usually somewhere under "C:/Programs". Nebula applications should treat the home: location as a read only directory so that the user doesn't need administrator rights to run the application.
• user: This points to the currently logged in user directory for this Nebula application. In a US Windows installation, this is somewhere under "C:/Users/[username]/Documents". Nebula will automatically create a local directory in the user directory to prevent different applications to overwrite their data. It is generally safe to write data to the user directory. This is the place where configuration and save game data should be written, or any other data which should persist between application invokations.
• temp: This assign points to the current user's temp directory. This directory is generally writable. It should not be assumed that data in temp: survives until the next application start.
• bin: This points to the directory of the application's executable file. This may or may not be identical with the home: directory. The bin: assign should be treated as read-only.

Custom assigns may be defined at runtime by the application. Often this is used to define abstract path to resources like textures, sound data, and so on. That way the locations of those resources can be easily changes by setting a single assign instead of fixing all the resource paths. A nice side effect of assigns is that a path with assigns is often much shorter then an "absolute" path resulting in a smaller memory footprint.

URIs

Resource locations are generally defined through standard URIs in Nebula. URIs may consist of the following parts, some of them optional:

• a scheme, for instance "http:", "file:", etc... Nebula doesn't define any hardcoded schemes, instead, schemes are bound to stream classes by registering them with the IO::StreamServer singleton
• an optional user info field, often this is a login name and a password to authenticate with a remote FTP or HTTP host
• a hostname, like "www.radonlabs.de"
• an optional portname following the hostname
• a local path, pointing to a resource on the host
• an optional fragment, which often points to a location inside the resource
• an optional query part, which often contains arguments for a PHP script or some similar dynamic response mechamism

The class IO::URI is used to pass URIs around and to crack URI strings into its various components. It should be noted however, that an URI object has a bigger memory footprint compared to storing the URI in a simple string. So sometimes it may be better keep URIs around in strings and only use the IO::URI class to split the URI string into its parts.

Here are some examples for URI's:

file:///c:/temp/bla.txt
file://samba/temp/bla.txt
http://www.radonlabs.de/index.html
http://user:password@www.myserver.com:8080/index.html#main

By using Nebula assigns you can simplify those complex pathnames a lot. To reference a file in the application directory you can write for instance home:bla.txt which would resolve to something like file:///c:/programme/[myapp]/bla.txt.

Streams, Readers and Writers

Streams provide a common interface for storing or transporting raw data. They replace the nFile class of Nebula2 with a much more general approach for storing, retrieving and transporting data. A stream object provides the traditional Open()/Close()/Read()/Write()/Seek() interface. Some stream classes provide memory mapping, so that data can be read or written by direct memory access. Stream objects use an IO::URI object to define their resource location. Usually, one URI scheme maps to a specific stream class. For instance the URI scheme "http:" usually maps to the Net::HttpStream class, while the scheme "file:" maps to the IO::FileStream class. This mapping is implemented by the StreamServer which constructs a matching stream object given an URI. A Nebula application is responsible to provide the mapping of URI scheme to stream classes using the StreamServer::Register() method. This is also the way how new stream classes and schemes are registered with Nebula.

Important stream classes in Nebula are for instance:

• IO::FileStream: provides access to the host's filesystem
• IO::MemoryStream: a dynamic memory buffer with a stream interface
• IO::HttpStream: provides a stream interface to files on a HTTP server

To read and write formatted stream data in a more flexible way then Nebula2, stream reader and stream writer classes have been introduced in Nebula. Stream reader and writer classes provide a comfortable interface which is specialized on a specific data format. Here are some examples of stream readers and writers provided by Nebula:

• IO::BinaryReader/IO::BinaryWriter: read and write binary data
• IO::TextReader/IO::TextWriter: read and write text and character data
• IO::XmlReader/IO::XmlWriter: read and write XML formatted data
• Messaging::MessageReader/Messaging::MessageWriter: Message serialization

Here's a simple example how to access a file on a HTTP server with a XmlReader:

using namespace IO;
 
Ptr<Stream> stream = StreamServer::Instance()->CreateStream("http://www.radonlabs.de/index.html");
Ptr<XmlReader> xmlReader = XmlReader::Create();
xmlReader->SetStream(stream);
if (xmlReader->Open())
{
    // parse content here using the XmlReader interface
}

File Server

The Nebula IO::FileServer class provides a singleton which offers access to the hosts filesystem for global operations like defining assigns, copying, deleting and checking for existance of files and directories, listing directory contents, and so on.

Here's some sample code fragments for some of the more useful FileServer methods:

using namespace IO;
using namespace Util;
 
FileServer* fs = FileServer::Instance();
 
// check if a file or directory exists
bool fileExists = fs->FileExists("home:bla.txt");
bool dirExists = fs->DirectoryExists("temp:bla/blub");
 
// resolve a path with assigns into an absolute filesystem
// path, this is sometimes necessary to interface with 
// 3rd party libraries which don't understand Nebula paths directly
String absPath = fs->ResolveAssings("user:myapp/savegames");
 
// create a directory, note that all missing subdirectories will
// be created as well
fs->CreateDirectory("user:myapp/savegames");
 
// copy and delete files
fs->CopyFile("home:movie.mpg", "temp:movie.mpg");
fs->DeleteFile("temp:movie.mpg");
 
// list files in a directory matching a pattern
Array<String> files = fs->ListFiles("temp:", "*.txt");
 
// list all subdirectories in temp:
Array<String> dirs = fs->ListDirectories("temp:", "*");

The Nebula Console

[TODO]