Memory Safety and Delphi in the news

Microsoft 2019 Microsoft: 70 percent of all security bugs are memory safety issues | ZDNET

"Users who often read vulnerability reports come across terms over and over again. Terms like buffer overflow, race condition, page fault, null pointer, stack exhaustion, heap exhaustion/corruption, use after free, or double free --all describe memory safety vulnerabilities.

Speaking at the BlueHat security conference in Israel last week, Microsoft security engineer Matt Miller said that over the last 12 years, around 70 percent of all Microsoft patches were fixes for memory safety bugs."

The Microsoft report above says :
image

MiraclePtr< T> to address UseAfterFree bugs in Chrome …

ā€œThe BackupRefPtr algorithm is based on reference counting. It uses support of Chrome’s own heap allocator, PartitionAlloc, which carves out a little extra space for a hidden reference count for each allocation.
Raw_ptr< T> increments or decrements the reference count when it’s constructed, destroyed or modified.
When the application calls free/delete and the reference count is greater than 0, PartitionAlloc quarantines that memory region instead of immediately releasing it. The memory region is then only made available for reuse once the reference count reaches 0.ā€

Surely the only point you’ve made from this thread is to show Delphi isn’t memory safe and shouldn’t be on the list. Where Delphi doesn’t typically suffer from buffer overflow bugs (the only reason I can think of for it being on the list) it certainly doesn’t protect from many of the other issues like use after free etc

Some basic changes to the language like automatically initialising local variables would go a long way to helping make things much safer.

Agree, in so far as I don’t understand the reasons it ended up in the list. My de facto position before any of this came up was that OP is a nicer, more elegant, less spiky C++. So it was a surprise to me, as I said, to see it there.

That said, people put effort into these reports. Someone had reasons for making that evaluation, so we should probably figure out explicitly the pros and cons for our language in this context. AND create and promote guidelines for how to best utilise it for a certain level of ā€œsafeā€ outcome.

Also … hopefully, between this in the news,
and the efforts updating the C++Builder product line, Embarcadero will be energised to put new development attention into our favourite language.

The advertising has been done, essentially.
We have a chance to try to live up to it.

The link above about MiraclePtr suggests eg that maybe FastMM could be modified to do something similar.

@GrahameDGrieve already gave us a presentation that involved him doing his own memory manager. Maybe we need another talk on that subject.

The link above about MiraclePtr suggests eg that maybe FastMM could be modified to do something similar.

There is a great feature in C++Builder called CodeGuard which has been there for 25-odd years (but seems to be a bit flakey in recent times and now requires not using runtime libraries, I haven’t used it for a long time) that can catch a lot of memory safety issues. Using it does affect runtime performance but it can be enabled/disabled.

List of access errors detected:

  • Access in freed memory
  • Method called on freed object
  • Reference to freed resource
  • Method called on illegally casted object
  • Access overrun
  • Access underrun
  • Access in uninitialized stack
  • Access in invalid stack

Access Errors - RAD Studio

Would be great to see these added to FastMM.

1 Like

FastMM does some of those things, but others are very hard indeed for a memory manager to do. Or impossible.

I have written my own memory manager, but not anymore - I use fastMM. I do have a base object that does tracking / manual reference counting / debugging leak support, and I have talked a little about this. I’m happy to talk again about it, but it’ll have to virtual - I’m on the road for the next 9 months

Grahame

1 Like

Yes. Google Gemini has been surveiling your location. :grin:

3 Likes

Or is this you?

1 Like

FastMM does some of those things, but others are very hard indeed for a memory manager to do. Or impossible.

I believe that some of them work as follows (I’d have to check my old article on CodeGuard to see if I knew how they worked back then - my memory manager deallocated that knowledge years ago :smile:):

  • Access in freed memory: when a block is freed the mm overwrites the block with a pattern and delays actually releasing the memory for a while. Later, when it is time to release the memory it checks that the pattern is still intact and actually releases the memory. If the pattern was modified then an invalid write occurred. This doesn’t catch read access though. This technique detects write access to freed memory if it would have otherwise been reallocated for another purpose
  • Method called on freed object: Hook all class member functions. When the hook function is called check the implicit Self parameter against valid memory blocks and call the original function
  • Access overrun (also available in FastMM) and underrun: Add some guard memory to the start and end of each allocated block and fill with a pattern. When the object is freed check that the patterns are still intact. Again, won’t catch an invalid read
  • Access uninitialized/invalid stack: Function hooks, check stack pointer and frame have just been initialized, maybe push a pattern on the stack, call the original function, maybe check pattern on return

To help catch access in freed memory immediately, at least where the memory is not reallocated, and to catch reads, you could use something to trigger AV’s like the following instead of calling .Free:

procedure FreeAndInvalidate(var Obj);
var
  Temp: TObject;
begin
  Temp := TObject(Obj);
  Pointer(Obj) := $C0FFEE;
  Temp.Free;
end;

  var o := TThing.Create;
  ...
  FreeAndInvalidate(o);
  ...
  // Access member field or call a method that does = AV: read of address 00C0FFEE
  if o.Something = 42 then ...

Calling FreeAndNil() also works but that would get mixed in with ā€˜access before allocated’ or other uninitialised ref code.

1 Like

I had more thoughts on trying to deal with memory access issues, only experimental, so I’ve moved them to a github pages website : Memory-Safety-Experimental | Experimenting with ways to increase memory safety in Delphi

For an extremely good presentation around this area, see @GrahameDGrieve 's ADUG talk on preventing Double Frees from November 2023:

1 Like



Delphi is usually pretty good at warning about uninitialised variables.

I will just add an appropriate var := value; statement, even when (rarely) I think it has it wrong.
Sometimes annoying when a newer version of the compiler warns that the initialisation value is not used… What to do then? Probably just leave it in, with a comment.

For shame. I was forgetting …

Delphi Memory Management. Dalija Prasnikar.

https://dalija.prasnikar.info/delphimm/index.html

Table of Contents:

Introduction

Clean Code

Part 1. Memory management basics and terminology

  • Short overview of memory management models

  • How memory works

    • Overview
    • Initialization and instantiation
    • The stack, the heap, and the differences between them
  • Variables and types

    • Value types, pointers and references, unmanaged and managed types
    • Automatic initialization of variables
  • Invalid pointers or references - wild, dangling, stale

  • Scope and lifetime

  • Classes

  • Resource management - RAII

Part 2. Object instances

  • To be, or not to be

    • Nil - The Billion Dollar Mistake
    • Passing nil, returning nil
    • Nil (Null) Object Pattern
    • Nullable types
  • Ownership

  • Object Instance Lifecycle

    • Lifecycle diagram
    • Difference in object instance lifecycle between classic and ARC compiler
    • Exceptions and exception handling in lifecycle methods
    • Creating and destroying an object instance
    • Overview of object lifecycle methods
  • Lifecycle Coding Patterns

    • Constructors and destructors
    • AfterConstruction and BeforeDestruction
    • Adding the override directive
    • Calling the inherited
    • Class constructors and destructors
    • Object construction through metaclasses
    • Object factories
    • Object factory with metaclass
    • OnCreate, OnDestroy and similar patterns
    • Deep copy - Assign method

Part 3. Manual memory management

  • Releasing object instances

    • Methods for releasing object instances
    • Destroy
    • Free
    • FreeAndNil
    • Releasing through ownership models
    • Collections with ownership
    • Handling exceptions during ownership transfer
    • TComponent ownership
    • TComponent notification system

Part 4. ARC memory management

  • Automatic Reference Counting - Overview

    • Golden Rule of ARC
    • Strong & weak references - [weak], [unsafe], pointer
    • Ownership
    • Strong reference cycles
  • Automatic Reference Counting in Delphi

    • ARC in classic compiler - Interfaces
    • Delphi NextGen ARC compiler
  • Automatic Reference Counting - Concepts

    • How the reference counting mechanism works
    • Reference counting triggers
    • Forming of a reference cycle
    • When is it safe to use [unsafe]?
  • DisposeOf - ARC with a twist

    • DisposeOf in the context of ARC memory management
    • DisposeOf and object instance lifecycle
    • Disposed property
    • Why does DisposeOf exist in Delphi ARC compilers?
    • TComponent notification system and ARC
    • Pitfalls of DisposeOf
  • Interfaces in the classic compiler

    • Mixing object and interface references
    • Disabling reference counting
    • Chameleon class
    • Memory leak on const parameter
    • Creating a fake interface
    • Interface list
  • Storing weak and unsafe references in collections

  • Anonymous methods

Part 5. Coding patterns

  • Smart Pointers

  • Lazy

  • Weak

Appendix

  • References

  • Quality Portal Reports

This SO answer by Barry Kelly, former Delphi compiler engineer, summarises which variables are initialised:

1 Like

How about those C++ people to actually enable their warnings (I am sure clang and GCC have something similar): Compiler Warning (level 1 and level 4) C4700 | Microsoft Learn

Zeroing stack space does not come at zero cost. :roll_eyes:

This is how I would deal with the issue
a update to Delphi
1/ TObject.free should always set the pointer to Nil so when writing code you know if the memory has been freed by looking at the pointer!!! everyone is thinking that way - can the complier or assembly code find the object’s pointer from inside the free statement to set the pointer to Nil in TObject.Free?

I have been working on an object of my own and I have a very irritating complier issue
when I declare a value with a type in my object’s procedure
1/ The type is always from another place than in the file
2/ It sometimes cannot identify the type!!!
It reminds me of a bad wiring joint that works when it wants to work!!!.
Very common types to Delphi
var
W: Word;
I64: Int64;
it is rarely I: Integer;
These a very common types I’m using but they are sometimes rejected by the complier
so i make Word64 = NativeUInt; in my file and no problems
Var
w64: Word64
why?
I have not had the problem with
Strings;
ShortStrings;

I need to admit I put my computer in sleep mode many times
And the editor sometimes does not fully work recovering from a sleep for some reason

Based on an extremely lively debate we had a couple of years ago there would be many people who would disagree with you :joy:

There is the FreeAndNil(x) construct for exactly the purpose you describe.

So its about the pro’s and con’s if TObject.Free NILed the pointer of TObject as the object was freed as Delphi user.
1/ I can test the pointer to see if its nil before so I do not create another TObject and loose the old pointer value. And that lowers the risk of memory leeks!!!
2/ another point is the creation of TObject and the pointer value already used (not Nil). That the object does not get created but instead old the pointer value is maintained until it is set to Nil. And that blocks the risk of memory leeks!!!

Example TObject,Create(Var NewObjectPtr: TObject) ; virtual;

Just an idea, you may have another or use both idea’s

The project I’m working on right now is only to have one object from the file to ever be executed at any one time. It has a single pointer in the file of this main object that other stuff uses in the file. When I create my Object I cannot stop the creation so I have to run a TList to remember created objects and then produce a delete of one of the objects.
I’m saying its messy code?

One Example TObject,CheckCreate(Var NewObjectPtr: TObject; NoAllowedObjects: integer default 0) ; virtual; Override; // 0 means unlimited objects
something like that as What I’m saying is a pointer is allocated to an object for a single task and you use memory for the object or you do not use memory controlled buy one pointer. You can still put the pointer value in TList and then Nil the pointer in code for reuse as you have made it harder to forget the pointer is in use before reusing it.

This is just in my head and you may hate them.

The other option is a TList containing all created objects that gets cleared when the application closes. the con how much mothering does the programmer need?