[Ohrrpgce] File reading/writing script commands

Ralph Versteegen teeemcee at gmail.com
Sun Oct 21 17:44:05 PDT 2018

On SS the topic of reading/writing files with scripts has come up
again, and I realised I had this
email with my ideas on that, still sitting as an 18-month-old draft.

Thanks to the addition of find_file_portably for rungame, it's now a
bit easier to add these commands,
so here is a specific design for all the proposed commands.
I won't be adding them soon.

=== Restrictions ==

We ought to prevent people from doing certain mischief like creating gigabytes
of text files, or thousands of small ones, or creating binary files
like .exes, etc.

Here are some ideas for restrictions:
-a game can write up to 50MB of files each time it is run, counted as
the difference in
 file size between opening and closing (so you can repeatedly open,
erase, and rewrite a file)
-at most 256 files can be created each time the game is run
-at most 64 directories created each run.
-file extensions: either a whitelist (I was going to suggest: .txt
.log .csv .xml .html .hss .hsi)
 or a blacklist: .exe .bat .com .lnk ... plus some others on Windows
and Mac which I don't yet know about.
 Last time we discussed this, I think James suggested we didn't need
to blacklist any extensions.
-maybe only allow deleting or overwriting files you created in the
first place? But this could be a disaster if you
 accidentally ship your game with a script-created .txt and then when
the game is unzipped it can't
 write to it
-Location restrictions:
 Filenames must be relative to the .rpg location, and can't be in a
parent directory (only in the same
 directory or subdirectories)
 Note: I decided not to restrict rungame to only work for games in
subdirectories, since it's
 read-only, and useful for running "../maingame.rpg".
 But I wouldn't like to allow games to create files that freely
 We can allow a script to open the file browser to let the user pick
any directory or file, to get
 around this restriction. Either that command can return an open file,
or return a file path and whitelist
 that path.

=== find_file_portably ===

Further, I've realised I forgot many restrictions on permissible
portable file paths
that need to be added to find_file_portably (both when opening or
creating files):
-Can't begin with '.', can't contain any characters below 32 or above 127.
 Also, special characters on Windows: "*/:<>?\| and 127 need to be
disallowed everywhere.
 Strangely Microsoft's documentation never says so explicitly, but it
seems that to open a
 a file with a filename/path containing unicode is only possible if:
 -you use a short 8.3 version of the filename
 -the current codepage is set to something that can represent all the
characters in the filename.
 Windows doesn't tell you whether it performed
 I don't know yet how to handle Latin-1 characters above 128; probably
that needs special encoding.
-Microsoft recommends "Do not end a file or directory name with a
space or a period. Although the underlying
 file system may support such names, the Windows shell and user
interface does not."
-File paths should be limited to 200 characters. This is because
winapi calls by default enforce a limit of
 MAX_PATH (260) characters, whether the path is relative or absolute,
and we need to allow space for .rpgdir
 lumps, etc. To get around this limit is painful (using a "\\?\"
prefix plus using the unicode API).

On that note, we currently can't open files with filenames containing
unicode on windows.
I think fixing that requires replacing the DIR function on Windows
with a version that calls the Wide/Unicode winapi functions.

== Text file encoding ==

Because of newlines and unicode, text and binary files need to be
handled separately.

FB's file IO has builtin support for reading/writing utf8/utf16/utf32,
but sadly it appears it will reject
any file without a BOM (byte order mark) header, even UTF8, for which
using a BOM is not recommended! I was already doing some work
on FB's unicode I/O code, so can fix that.
We also have our own routines for encoding/decoding UTF8, so can work
around this FB bug in the UTF8 case.
We still won't be able to open UTF16 files without a BOM (which are rare).
Also, FB's OPEN does not do autodetection of the encoding, but we can
copy code from fbc (the compiler)
to autodetect based on the BOM.
Or port the very sophisticated code from HSpeak to do autodetection...
but it won't work if FB demands a BOM.

== Script commands ==

All commands that take a filename obey the restrictions listed under
"open text file".

If another process writes to the file, well, although that would be an
interesting way to implement
networked multiplayer, we don't need to deal with that problem.

open text file(filename, newfile = false, hide = false)
  Open a text file for reading or writing. Returns a file handle
(which is negative, so it can be distinguished
  from a string ID).
  If the file exists, the existing encoding (Latin1/UTf8/UTF16/UTF32)
is respected and followed.
  If the file doesn't exist it is created in a writable location:
  If hide == true: the player isn't expected to read it, put in the
.saves folder (it should be just a filename,
   no directories)
  Else: first try putting it next to the .rpg file, otherwise in a
subfolder of the user's Documents folder
     named after the RPG file/package name.
  If the file exists but is read-only then opening succeeds, but the
first attempt to write to it
  causes it to be copied to a writeable location.
  The directory must already exist
  If 'newfile' is true (it defaults to false), then a new file with a
different name
  is used if the given file already exists, appending a number to the file name.
  For example, if "a.txt" exists, "a 2.txt" is tried, then "a 3.txt", etc.
  Max 4 files open at once (oldest open file is closed automatically
  instead of failing).
is file(filename)
close file(handle)
clear file(handle, line number)
  Deletes contents after a certain line, defaults to whole file
delete file(handle or filename)
  Only works if the file was created by this game (list of created
files is stored
  in persistent storage). But kind of pointless to restrict that,
  since you can other write to or clear a file you didn't create.
  Closes handle.
file number of lines(handle)
browse for file(file type or extension, default location string id = -1)
  Let the user pick a file anywhere.
  The first argument is either:
   file type: a (negative) constant which gets mapped to a builtin
   extension: a string id for one of the allowed extensions
  Returns an open file handle, or false if cancelled.
  Also, the file path gets put on a list of file paths the game is allowed
  to open, which lasts until the game is quit. (Or should it go in
persistent storage? What if the .rpg is moved?)
browse for folder(default location string id = -1 [, into string id])
  The folder gets put on a list of file paths the game is allowed to open?
  Maybe this is too much.
  (In future "into string id" can be omitted and it returns a string)
pick filename(query string, extension string, default name string,
default location string)
  Let the user enter the name of a new file, and place it anywhere
(calls inputfilename()).
  extension should be one of the permitted extensions.
  Opens the file and returns a file handle, or false if cancelled.
  Also, the file path gets put on a list of file paths the game is
allowed to open.
file path(handle)
  Returns the full path to an open file, with certain folders like
Documents and Downloads shortened.
  This is useful for telling the player where a file is, and since the
game does not have complete
  control over where a file is placed.
  E.g. opening "character log.txt" might result in the path
"Documents\Dungeoneer\character log 3.txt"
file read line(handle, line number [, into string id])
  Read one line of the file, with newline stripped.
  Default to reading the first file, then the next, etc
  (In future "into string id" can be omitted and it returns a string)
file write line(handle, from string id)
  Append one or more lines (if the string contains newlines) to a file

Future commands:

open binary file(filename, newfile = false, hide = false)
file read data(handle, [offset, length])
  Read some bytes into an array, or the whole file.

More information about the Ohrrpgce mailing list