Manual & Automatic Deletion

This describes the differences between soft-deletion and permanent deletion. Entities can either be manually deleted by a user using the UI or automatically based on configurable deletion rules. There are two types of manual deletion. Which one is used depends on the entity that is deleted by the user.

Manual Soft-deletion

When an entity is soft-deleted it is not removed from the database. Instead it's just flagged as deleted, so the entity and all child entity are no longer visible to users.

It applies to all core entities and also samples and pathogen tests.

To be done: When an entity is soft-deleted the user has to give a reason why the entity is deleted. In addition users that have the right to delete entities should also be able to display those entities using a dedicated filter in the directory views. This allows them to review deletions done by other users and to revert the deletion if needed.

How soft deletion works

1. It flags the entity as deleted

2. For some entities related entities are soft-deleted as-well:

• Case: Contacts are deleted and samples that are not linked to other entities (e.g. contact)

• Contact: Samples that are not linked to other entities

• Event participant: Samples that are not linked to other entities

• Event: Event participants are deleted

3. For cases and events that are shared with external surveillance tools a deletion request is sent.

4. Links to other entities are removed:

• Case: Removed from related contacts, samples, event participants, travel entries and immunizations

• Contact: Removed from related samples and case exposures

• Event: Removed from all subordinate events

5. When an entity is flagged as deleted it is no longer displayed in the user interface and no longer provided via ReST.

6. This also applies for children of those entities (e.g. tasks of soft-deleted cases).

7. Persons of soft-deleted entities are only shown to the user when they are linked to another entity that is not deleted.

Manual Permanent Deletion

Children of those core entities are considered "content data" (similar to the text in a field) and don't support soft-deletion. When users delete this data it is permanently deleted from the database.

This applies to the following entities, all of which are managed in lists and are direct or indirect children of a core entity:

• Action, ActivityAsCase, AdditionalTest, ClinicalVisit, EventGroup, Exposure, Location (addresses), PersonContactDetail, Prescription, PreviousHospitalization, SurveillanceReport, Task, Treatment, Visit

• Document: Will be marked as deleted first. The actual deletion is done by a nightly cronjob

Important: Since the deletion of history data is done by a database trigger it means that deleting the history of those entries will also be deleted.

Automatic Deletion

Automatic deletion takes care of permanently deleting entities based on rules that are defined in the deletion configuration.

Currently those rules can only be defined by accessing the database. By default there are already two entries for each entity type - one for automatic deletion of manually soft-deleted entities and an additional one. Both don't define a deletion period, so by default no automatic deletion is done.

The deletion is configured using the following parameters.

• Entity type: Automatic deletion is available for all core entities (except campaigns)

• Deletion reference (see below)

• Deletion period in days with a minimum of 7.

• Subset of fields to be deleted - TBD Extend deletion configuration for field-wise deletion #7716

Deletion References

• CREATION: Takes the creationDate of the entity as reference

• ORIGIN: Only implemented for travel entries. Takes the dateOfArrival

• END: Take the endOfProcessDate of the entity as reference
  TBD: Automatic deletion based on end of process date instead of change date #8996

• MANUAL_DELETION: Takes the changeDate of entities that were soft-deleted.
  This allows the automatic permanent deletion of those entities after a short period of time that allows a dedicated user to review the deletion.

How automatic deletion works

A nightly cronjob executes the following for each deletion rule of each entity type that supports automatic deletion:

1. Identify the date for automatic deletion (e.g. today minus 90 days)

2. Identify all entities whose deletion reference date is older

3. All identified entities are deleted permanently as described in the section for manual permanent deletion

Afterwards all persons that are no longer referenced by a core entity are permanently deleted as-well.

How permanent deletion works

The following is done when permanently deleting a core entity:

1. Children are deleted form the database, e.g. Tasks, Samples, Documents, Treatments, Prescriptions, ClinicalVisits, Visits. For Samples and Visits this is only done when they are no longer used by other entities

2. SORMAS2SORMAS sharing meta data is deleted

3. For cases and events that are shared with external surveillance tools a deletion request is sent

4. Cases and contacts are unlinked from merged cases/contacts

5. Links to other entities are removed (see how soft-deletion works above)

6. The entity itself is deleted from the database

7. When deleting an entity from the database the related history table entries are automatically deleted as-well.

Example Deletion Configuration

Automatic permanent deletion of manually soft-deleted cases after 90 days:
entityType: CASE, deletionReference: MANUAL_DELETION, deletionPeriod: 90 SQL for all entities: UPDATE public.deletionconfiguration SET deletionperiod=90 WHERE deletionreference='MANUAL_DELETION';

Automatic permanent deletion of cases 10 years after end of processing:
entityType: CASE, deletionReference: END, deletionPeriod: 3650

Automatic permanent deletion of travel entries, 14 days after date of arrival:
entityType: TRAVEL_ENTRY, deletionReference: ORIGIN, deletionPeriod: 14 SQL: UPDATE public.deletionconfiguration SET deletionreference='ORIGIN', deletionperiod=14 WHERE entitytype='TRAVEL_ENTRY' and deletionreference IS NULL;

Automatic deletion in the UI

When a user open's a core entity in the UI the system will display the calculated deletion date of the entity.

Additional Information

Implementation epic: Automatic deletion of personal data #7736