Beware of the closures

 The other day I was debugging ChessKISS looking for strange calls to system in order to increase performance and surprisingly I found too many calls to GetMem() , very strange I thought, I check the call stack and I end up in a very innocent function called ReceiveDataFromWinboard

function TChessEngine.ReceiveDataFromWinboard(out aMove: TMove): TMoveStatus;
var
  cmd: TString;
  notification: TNotification<AnsiString>;

begin
  aMove := NO_MOVE;
  result := msNone;

  if FWinboardTask = nil then
  begin
    notification := FWinboard.GetMessage;
    FWinboardTask := TFuture<AnsiString>.Create(notification, tpNormal);
  end;

  if FWinboardTask.Available then
  begin
    try
      cmd := TString(FWinboardTask.GetValue);
      if FWinboardTask.Error <> '' then
      begin
        FLog.Add(FWinboardTask.Error);
        raise FWinboardTask.GetException;
      end;

      if (not IsCommand(cmd)) and (FGameStatus = gsPlay) then
        result := TCommandHandler.Instance.ParseCommand(cmd, aMove);
    finally
      FreeAndNil(FWinboardTask);
    end;
  end;
end;

Again nothing to be worried, a new thread is created every time the remote        application sends a message, so I had to disassembler in order to see what was    going on and voila!

004A0F38 55               push ebp
004A0F39 8BEC             mov ebp,esp
004A0F3B 6A00             push $00
004A0F3D 53               push ebx
004A0F3E 56               push esi
004A0F3F 8BD8             mov ebx,eax
004A0F41 33C0             xor eax,eax
004A0F43 55               push ebp
004A0F44 68AF0F4A00       push $004a0faf
004A0F49 64FF30           push dword ptr fs:[eax]
004A0F4C 648920           mov fs:[eax],esp
004A0F4F B201             mov dl,$01
004A0F51 A1880E4A00       mov eax,[$004a0e88]
004A0F56 E8393EF6FF       call TObject.Create
004A0F5B 8BF0             mov esi,eax
004A0F5D 8D45FC           lea eax,[ebp-$04]
004A0F60 8BD6             mov edx,esi
004A0F62 85D2             test edx,edx
004A0F64 7403             jz $004a0f69
004A0F66 83EAF8           sub edx,-$08
004A0F69 E84E84F6FF       call @IntfCopy

WTF!?, Delphi is automatically creating an object whose name is xxx$ActRec and    then copying some information. Well, we have to isolated the problem and see if   this still happening or it is a combination of closures with threads.
procedure TForm39.OnTimer(Sender: TObject);
begin
  if FCount = 0 then
  begin
    FCount := 100;

    Call(
      function: integer
      begin
        result := Random(100);
      end
    );
  end else
    Dec(FCount);
end;

procedure TForm39.Call(aTest: TTest);
begin
  aTest();
end;

procedure TForm39.FormCreate(Sender: TObject);
begin
  FCount := 0;
end

So here we go, in this example a Timer is calling a function which ONLY calls a   closure once every 100 times, so I checked again the assembler code generated and I got very disappointed, Delphi automatically does his stuff every time there is a closure function, usually this is more than ok on a regular application, but on  an application like ChesskISS which is heavily based on threads and the main loop is called millions of times per second this is not acceptable.
The solution was to add a new constructor in the TFuture<T> class adding a simple notificator as a parameter, so the code was clean again, sadly CheckKISS still    slow, shame on me...

Big files

 In a big project you might end up with tons of files, maybe you group them by directory, but it can be quite handy to group them all into a single file, for that purpose we have the TBigFile class, which contains the following methods:

  TBigFile = class
    constructor Create;
    destructor Destroy; override;
    procedure AddStream(const aName: string; aStream: TStream);
    procedure AddFile(const aFileName: string);
    procedure BuildFrom(const aFileName: string);
  	procedure Clear;
    function GetFiles: TStrings; overload;
    function GetFiles(const aPath, aMask: string): TStrings; overload;
    function GetDirectories: TStrings;
    function Load(const aPath, aFileName: string): TStream; overload;
    function Load(const aFileName: string): TStream; overload;
    procedure Save(const aFileName: string);

    property Signature: AnsiString read FSignature write FSignature;

  end;

In the demos file, there is a project named BigFile.dpr which shows how to create and load a big file, let's make a summary:

Create the class

FBig := TBigFile.Create;
Add some files

FBig.AddStream('test.txt', stream);
Since the parameter is a stream we can add any kind of resource, aditionally there is a method for directly add physical files called AddFile(). 

You can also specify different folders just add the folder name prior to the file name like 'one/test.txt'

Save the big file

FBig.Save('TheOne.big');
Load the big file

FBig.BuildFrom('TheOne.big');
Loading files

stream := FBig.Load('test.txt')
In the BB API, there are many load functions that support streams, so one can do 
things like 
ent.LoadStream(FBig.Load('donut.asc'));
Getting files

strings := FBig.GetFiles;
Now strings will contain all files located inside the big file, there is an overloaded method to retrieve files per directory and mask


strings := FBig.GetFiles('first', '*.bmp');
These are the classes that support streams:
TEnt
TAnimations
TSimpleSprite
TLayer
TIni
TImageEx
TSurface
Of course those streams can be compresses/decompressed, an easy way is to use the helper class TStreamHelper which holds several methods to help you:

  TStreamHelper = class helper for TStream
public 
  function ToString: string; 
  procedure WriteString(const aString: string);
  function Bof: boolean;
    function Eof: boolean; 
  class function Compress(aStream: TStream): TStream; 
  class function Decompress(aStream: TStream): TStream; 
  class function StreamToString(aStream: TStream): string; 
  class function StringToStream(const aString: string): TStream; 
  class function MemoryToStream(aBuffer: pointer; aSize: int64): TStream; 
  class function StringToCompressed(const aString: string): TStream; 
  class function CompressToString(aStream: TStream): string; 
  class function ComponentToStream(aComponent: TComponent): TStream; 
  class function ComponentToString(aComponent: TComponent): string; 
  class procedure Save(const aFileName: string; aStream: TStream); 
end;