Memory Safety and Delphi in the news

Microsoft 2019 Microsoft: 70 percent of all security bugs are memory safety issues | ZDNET

"Users who often read vulnerability reports come across terms over and over again. Terms like buffer overflow, race condition, page fault, null pointer, stack exhaustion, heap exhaustion/corruption, use after free, or double free --all describe memory safety vulnerabilities.

Speaking at the BlueHat security conference in Israel last week, Microsoft security engineer Matt Miller said that over the last 12 years, around 70 percent of all Microsoft patches were fixes for memory safety bugs."

The Microsoft report above says :
image

MiraclePtr< T> to address UseAfterFree bugs in Chrome ā€¦

ā€œThe BackupRefPtr algorithm is based on reference counting. It uses support of Chromeā€™s own heap allocator, PartitionAlloc, which carves out a little extra space for a hidden reference count for each allocation.
Raw_ptr< T> increments or decrements the reference count when itā€™s constructed, destroyed or modified.
When the application calls free/delete and the reference count is greater than 0, PartitionAlloc quarantines that memory region instead of immediately releasing it. The memory region is then only made available for reuse once the reference count reaches 0.ā€

Surely the only point youā€™ve made from this thread is to show Delphi isnā€™t memory safe and shouldnā€™t be on the list. Where Delphi doesnā€™t typically suffer from buffer overflow bugs (the only reason I can think of for it being on the list) it certainly doesnā€™t protect from many of the other issues like use after free etc

Some basic changes to the language like automatically initialising local variables would go a long way to helping make things much safer.

Agree, in so far as I donā€™t understand the reasons it ended up in the list. My de facto position before any of this came up was that OP is a nicer, more elegant, less spiky C++. So it was a surprise to me, as I said, to see it there.

That said, people put effort into these reports. Someone had reasons for making that evaluation, so we should probably figure out explicitly the pros and cons for our language in this context. AND create and promote guidelines for how to best utilise it for a certain level of ā€œsafeā€ outcome.

Also ā€¦ hopefully, between this in the news,
and the efforts updating the C++Builder product line, Embarcadero will be energised to put new development attention into our favourite language.

The advertising has been done, essentially.
We have a chance to try to live up to it.

The link above about MiraclePtr suggests eg that maybe FastMM could be modified to do something similar.

@GrahameDGrieve already gave us a presentation that involved him doing his own memory manager. Maybe we need another talk on that subject.

The link above about MiraclePtr suggests eg that maybe FastMM could be modified to do something similar.

There is a great feature in C++Builder called CodeGuard which has been there for 25-odd years (but seems to be a bit flakey in recent times and now requires not using runtime libraries, I havenā€™t used it for a long time) that can catch a lot of memory safety issues. Using it does affect runtime performance but it can be enabled/disabled.

List of access errors detected:

  • Access in freed memory
  • Method called on freed object
  • Reference to freed resource
  • Method called on illegally casted object
  • Access overrun
  • Access underrun
  • Access in uninitialized stack
  • Access in invalid stack

Access Errors - RAD Studio

Would be great to see these added to FastMM.

1 Like

FastMM does some of those things, but others are very hard indeed for a memory manager to do. Or impossible.

I have written my own memory manager, but not anymore - I use fastMM. I do have a base object that does tracking / manual reference counting / debugging leak support, and I have talked a little about this. Iā€™m happy to talk again about it, but itā€™ll have to virtual - Iā€™m on the road for the next 9 months

Grahame

1 Like

Yes. Google Gemini has been surveiling your location. :grin:

3 Likes

Or is this you?

1 Like

FastMM does some of those things, but others are very hard indeed for a memory manager to do. Or impossible.

I believe that some of them work as follows (Iā€™d have to check my old article on CodeGuard to see if I knew how they worked back then - my memory manager deallocated that knowledge years ago :smile:):

  • Access in freed memory: when a block is freed the mm overwrites the block with a pattern and delays actually releasing the memory for a while. Later, when it is time to release the memory it checks that the pattern is still intact and actually releases the memory. If the pattern was modified then an invalid write occurred. This doesnā€™t catch read access though. This technique detects write access to freed memory if it would have otherwise been reallocated for another purpose
  • Method called on freed object: Hook all class member functions. When the hook function is called check the implicit Self parameter against valid memory blocks and call the original function
  • Access overrun (also available in FastMM) and underrun: Add some guard memory to the start and end of each allocated block and fill with a pattern. When the object is freed check that the patterns are still intact. Again, wonā€™t catch an invalid read
  • Access uninitialized/invalid stack: Function hooks, check stack pointer and frame have just been initialized, maybe push a pattern on the stack, call the original function, maybe check pattern on return

To help catch access in freed memory immediately, at least where the memory is not reallocated, and to catch reads, you could use something to trigger AVā€™s like the following instead of calling .Free:

procedure FreeAndInvalidate(var Obj);
var
  Temp: TObject;
begin
  Temp := TObject(Obj);
  Pointer(Obj) := $C0FFEE;
  Temp.Free;
end;

  var o := TThing.Create;
  ...
  FreeAndInvalidate(o);
  ...
  // Access member field or call a method that does = AV: read of address 00C0FFEE
  if o.Something = 42 then ...

Calling FreeAndNil() also works but that would get mixed in with ā€˜access before allocatedā€™ or other uninitialised ref code.

1 Like

I had more thoughts on trying to deal with memory access issues, only experimental, so Iā€™ve moved them to a github pages website : Memory-Safety-Experimental | Experimenting with ways to increase memory safety in Delphi

For an extremely good presentation around this area, see @GrahameDGrieve 's ADUG talk on preventing Double Frees from November 2023:

1 Like



Delphi is usually pretty good at warning about uninitialised variables.

I will just add an appropriate var := value; statement, even when (rarely) I think it has it wrong.
Sometimes annoying when a newer version of the compiler warns that the initialisation value is not usedā€¦ What to do then? Probably just leave it in, with a comment.

For shame. I was forgetting ā€¦

Delphi Memory Management. Dalija Prasnikar.

https://dalija.prasnikar.info/delphimm/index.html

Table of Contents:

Introduction

Clean Code

Part 1. Memory management basics and terminology

  • Short overview of memory management models

  • How memory works

    • Overview
    • Initialization and instantiation
    • The stack, the heap, and the differences between them
  • Variables and types

    • Value types, pointers and references, unmanaged and managed types
    • Automatic initialization of variables
  • Invalid pointers or references - wild, dangling, stale

  • Scope and lifetime

  • Classes

  • Resource management - RAII

Part 2. Object instances

  • To be, or not to be

    • Nil - The Billion Dollar Mistake
    • Passing nil, returning nil
    • Nil (Null) Object Pattern
    • Nullable types
  • Ownership

  • Object Instance Lifecycle

    • Lifecycle diagram
    • Difference in object instance lifecycle between classic and ARC compiler
    • Exceptions and exception handling in lifecycle methods
    • Creating and destroying an object instance
    • Overview of object lifecycle methods
  • Lifecycle Coding Patterns

    • Constructors and destructors
    • AfterConstruction and BeforeDestruction
    • Adding the override directive
    • Calling the inherited
    • Class constructors and destructors
    • Object construction through metaclasses
    • Object factories
    • Object factory with metaclass
    • OnCreate, OnDestroy and similar patterns
    • Deep copy - Assign method

Part 3. Manual memory management

  • Releasing object instances

    • Methods for releasing object instances
    • Destroy
    • Free
    • FreeAndNil
    • Releasing through ownership models
    • Collections with ownership
    • Handling exceptions during ownership transfer
    • TComponent ownership
    • TComponent notification system

Part 4. ARC memory management

  • Automatic Reference Counting - Overview

    • Golden Rule of ARC
    • Strong & weak references - [weak], [unsafe], pointer
    • Ownership
    • Strong reference cycles
  • Automatic Reference Counting in Delphi

    • ARC in classic compiler - Interfaces
    • Delphi NextGen ARC compiler
  • Automatic Reference Counting - Concepts

    • How the reference counting mechanism works
    • Reference counting triggers
    • Forming of a reference cycle
    • When is it safe to use [unsafe]?
  • DisposeOf - ARC with a twist

    • DisposeOf in the context of ARC memory management
    • DisposeOf and object instance lifecycle
    • Disposed property
    • Why does DisposeOf exist in Delphi ARC compilers?
    • TComponent notification system and ARC
    • Pitfalls of DisposeOf
  • Interfaces in the classic compiler

    • Mixing object and interface references
    • Disabling reference counting
    • Chameleon class
    • Memory leak on const parameter
    • Creating a fake interface
    • Interface list
  • Storing weak and unsafe references in collections

  • Anonymous methods

Part 5. Coding patterns

  • Smart Pointers

  • Lazy

  • Weak

Appendix

  • References

  • Quality Portal Reports

This SO answer by Barry Kelly, former Delphi compiler engineer, summarises which variables are initialised:

1 Like

How about those C++ people to actually enable their warnings (I am sure clang and GCC have something similar): Compiler Warning (level 1 and level 4) C4700 | Microsoft Learn

Zeroing stack space does not come at zero cost. :roll_eyes: